Guidelines for Performing Safe Measurement on the Internet
Tor Project
irl@torproject.org
safety
measurement
monitoring
research
internet
Researchers from industry and academia often use Internet measurements
as part of their work. While these measurements can give insight into the
functioning and usage of the Internet, they can come at the cost of user privacy.
This document describes guidelines for ensuring that such measurements can
be carried out safely.
Comments are solicited and should be addressed to the research group's mailing list at pearg@irtf.org and/or the author(s).
The sources for this draft are at:
https://github.com/irl/draft-safe-internet-measurement
Performing research using the Internet, as opposed to an isolated testbed
or simulation platform, means that experiments co-exist in a space with other
users. This document outlines guidelines for academic and industry researchers
that might use the Internet as part of scientific experimentation to mitigate
risks to the safety of other users.
Following the guidelines contained within this document is not a substitute for
any institutional ethics review process, although these guidelines
could help to inform that process. Similarly, these guidelines are not legal
advice and local laws must also be considered before starting any experiment
that could have adverse impacts on user safety.
The scope of this document is restricted to guidelines that mitigate exposure
to risks to Internet user safety when measuring properties of the Internet:
the network, its constiuent hosts and links, or its users traffic.
For the purpose of this document, an Internet user is an individual or
organisation that uses the Internet to communicate, or maintains Internet
infrastructure.
A threat is a potential for a security violation, which exists when
there is a circumstance, capability, action, or event that could
breach security and cause harm .
Every Internet measurement study has the potential to subject Internet users
to threat actions, or attacks.
Many of the threats to user safety occur from an instantiation (or
combination) of the following:
Surveillance: An attack whereby an Internet user's
information is collected. This type of attack covers not only data but
also metadata.
Inadequate protection of collected data: An attack where data, either in
flight or at rest, was not adequately protected from disclosure.
Failure to adequately protect data to the expectations of the user is an attack
even if it does not lead to another party gaining access to the data.
Traffic generation: An attack whereby traffic is generated to traverse the
Internet.
Traffic modification: An attack whereby the Internet traffic of users is
modified.
Any conceivable Internet measurement study might be considered an attack
on an Internet user's safety. It is always necessary to consider the best
approach to mitigate the impact of measurements, and to balance the risks of
measurements against the benefits to impacted users.
Internet measurement studies can be broadly categorized into two groups:
active measurements and passive measurements. Active measurements generate
or modify traffic while passive measurements use surveillance of existing
traffic.
The type of measurement is not truly binary and many studies will include
both active and passive components.
The measurement of generated traffic may also lead to insights into other users'
traffic indirectly.
XXX On-path/off-path
XXX One ended/two ended
Consequences of attacks
Breach of Privacy: data collection. This impact also covers the case of an Internet user's data
being shared beyond that which a user had given consent for.
Impersonation: An attack where a user is impersonated during a measurement.
XXX Legal
XXX Other Retribution
System corruption: An attack where generated or modified traffic causes the
corruption of a system. This attack covers cases where a user's data may be
lost or corrupted, and cases where a user's access to a system may be affected.
XXX Data loss, corruption
XXX Denial of Service (by which self-censorship is covered)
XXX Emotional Trauma
XXX a user is best placed to balanced risks vs benefits themselves
In an ideal world, informed consent would be collected from all users that
may be placed at risk, no matter how small a risk, by an experiment. In cases
where it is practical to do so, this should be done.
For consent to be informed, all possible risks must be presented to the users.
The considerations in this document can be used to provide a starting point
although other risks may be present depending on the nature of the measurements
to be performed.
A researcher would like to use volunteer owned mobile devices to collect
information about local Internet censorship. Connections will be made from the
volunteer's device towards known or suspected blocked webpages.
This experiment can carry substantial risk for the user depending on the
circumstances, from disciplinary action from their employer to arrest or
imprisonment. Fully informed consent ensures that any risk that is being taken
has been carefully considered by the volunteer before proceeding.
In cases where it is not practical to collect informed consent from all
users of a shared network, it may be possible to obtain proxy consent. Proxy
consent may be given by a network operator or employer that would be more
familiar with the expectations of users of a network than the researcher.
In some cases, a network operator or employer may have terms of service that
specifically allow for giving consent to 3rd parties to perform certain
experiments.
A researcher would like to perform a packet capture to determine the TCP
options and their values used by all client devices on an corporate wireless
network.
The employer may already have terms of service laid out that allow them to
provide proxy consent for this experiment on behalf of the employees (the users
of the network). The purpose of the experiment may affect whether or not they
are able to provide this consent. For example, to perform engineering work on
the network then it may be allowed, whereas academic research may not be
covered.
In larger scale measurements, even proxy consent collection may not be
practical. In this case, implied consent may be presumed from users for some
measurements. Consider that users of a network will
have certain expectations of privacy and those expectations may not align with
the privacy guarantees offered by the technologies they are using. As a thought
experiment, consider how users might respond if asked for their informed
consent for the measurements you'd like to perform.
Implied consent should not be considered sufficient for any experiment that
may collect sensitive or personally identifying information. If practical,
attempt to obtain informed consent or proxy consent from a sample of users to
better understand the expectations of other users.
A researcher would like to run a measurement campaign to determine the
maximum supported TLS version on popular web servers.
The operator of a web server that is exposed to the Internet
hosting a popular website would have the expectation that it may be included in
surveys that look at supported protocols or extensions but would not expect that
attempts be made to degrade the service with large numbers of simultaneous
connections.
A researcher would like to perform A/B testing for protocol feature and how
it affects web performance. They have created two versions of their software
and have instrumented both to report telemetry back. These updates will be
pushed to users at random by the software's auto-update framework. The telemetry
consists only of performance metrics and does not contain any personally
identifying or sensitive information.
As users expect to receive automatic updates, the effect of changing the
behaviour of the software is already expected by the user. If users have
already been informed that data will be reported back to the developers of the
software, then again the addition of new metrics would be expected. There are
risks in pushing any new software update, and the A/B testing technique can
reduce the number of users that may be adversely affected by a bad update.
The reduced impact should not be used as an excuse for pushing higher risk
updates, only updates that could be considered appropriate to push to all users
should be A/B tested. Likewise, not pushing the new behaviour to any user
should be considered appropriate if some users are to remain with the old
behavior.
In the event that something does go wrong with the update, it should be easy
for a user to discover that they have been part of an experiment and roll back
the change, allowing for explicit refusal of consent to override the presumed
implied consent.
Wherever possible, use a testbed. An isolated network means that there are no
other users sharing the infrastructure you are using for your experiments.
When measuring performance, competing traffic can have negative effects on
the performance of your test traffic and so the testbed approach can also
produce more accurate and repeatable results than experiments using the public
Internet.
WAN link conditions can be emulated through artificial delays and/or packet
loss using a tool like . Competing traffic can
also be emulated using traffic generators.
If your experiment is designed to trigger a response from infrastructure that
is not your own, consider what the negative consequences of that may be. At the
very least your experiment will consume bandwidth that may have to be paid for.
In more extreme circumstances, you could cause traffic to be generated that
causes legal trouble for the owner of that infrastructure. The Internet is a
global network crossing many legal jurisdictions and so what may be legal for
you is not necessarily legal for everyone.
If you are sending a lot of traffic quickly, or otherwise generally deviate
from typical client behaviour, a network may identify this as an attack which
means that you will not be collecting results that are representative of what
a typical client would see.
When performing active measurements on a shared network, maintain a list of
hosts that you will never scan regardless of whether they appear in your target
lists. When developing tools for performing active measurement, or traffic
generation for use in a larger measurement system, ensure that the tool will
support the use of a "Do Not Scan" list.
If complaints are made that request you do not generate traffic towards a
host or network, you must add that host or network to your "Do Not Scan" list,
even if no explanation is given or the request is automated.
You may ask the requester for their reasoning if it would be useful to your
experiment. This can also be an opportunity to explain your research and offer
to share any results that may be of interest. If you plan to share the reasoning
when publishing your measurement results, e.g. in an academic paper, you must
seek consent for this from the requester.
Be aware that in publishing your measurement results, it may be possible to
infer your "Do Not Scan" list from those results. For example, if you measured a
well-known list of popular websites then it would be possible to correlate the
results with that list to determine which are missing.
When collecting, using, disclosing, and storing data from a measurement, use
only the minimal data necessary to perform a task. Reducing the amount of data
reduces the amount of data that can be misused or leaked.
When deciding on the data to collect, assume that any data collected might
be disclosed. There are many ways that this could happen, through operation
security mistakes or compulsion by a judicial system.
When directly instrumenting a protocol to provide metrics to a passive
observer, see section 6.1 of RFC6973 for data
minimalization considerations specific to this use case.
XXX: Discard data that is not required to perform the task.
When performing active measurements be sure to only capture traffic that you
have generated. Traffic may be identified by IP ranges or by some token that is
unlikely to be used by other users.
Again, this can help to improve the accuracy and repeatability of your
experiment. , for performance benchmarking,
requires that any frames received that were not part of the test traffic are
discarded and not counted in the results.
XXX: Mask data that is not required to perform the task. Particularly useful
for content of traffic to indicate that either a particular class of content
existed or did not exist, or the length of the content, but not recording the
content itself. Can also replace content with tokens, or encrypt.
XXX: Binning, categorizing, geoip, noise.
When collecting data, consider if the granularity can be limited by using
bins or adding noise. XXX: Differential privacy.
XXX: Do this at the source, definitely do it before you write to disk.
presents a case-study on the in-memory
statistics in the software used by the Tor network, as an example.
The benefits should outweigh the risks.
Consider auxiliary data (e.g. third-party data sets) when assessing the risks.
Take reasonable security precautions, e.g. about who has access to your data sets or experimental systems.
This document has no actions for IANA.
Many of these considerations are based on those from the
adapted and generalised to be applied to
Internet research.
Other considerations are taken from the Menlo Report
and its companion document
.
Network emulation with NetEm
Open Source Development Lab
Benchmarking Methodology for Network Interconnect Devices
This document is a republication of RFC 1944 correcting the values for the IP addresses which were assigned to be used as the default addresses for networking test equipment. This memo provides information for the Internet community.
Tor Research Safety Board
Tor Project
Internet Security Glossary, Version 2
Privacy analysis of Tor's in-memory statistics
Tor Project
The Menlo Report: Ethical Principles Guiding Information and Communication Technology Research
Applying Ethical Principles to Information and Communication Technology Research
Privacy Considerations for Internet Protocols
CDT
Nokia Siemens Networks
Skype
NeuStar, Inc.
ULD
Janet