Active and Passive Metrics and Methods
(and everything in-between, or Hybrid)AT&T Labs200 Laurel Avenue SouthMiddletown, NJUSAacmorton@att.comThis memo provides clear definitions for Active and Passive
performance assessment. The construction of Metrics and Methods can be
described as Active or Passive. Some methods may use a subset of both
active and passive attributes, and we refer to these as Hybrid Methods.
This memo also describes multiple dimensions to help evaluate new
methods as they emerge.The adjectives "active" and "passive" have been used for many years
to distinguish two different classes of Internet performance assessment.
The first Passive and Active Measurement (PAM) Conference was held in
2000, but the earliest proceedings available on-line are from the second
PAM conference in 2001
[https://www.ripe.net/ripe/meetings/pam-2001].The notions of "active" and "passive" are well-established. In
general:An Active metric or method depends on a dedicated measurement
packet stream and observations of the stream.A Passive metric or method depends *solely* on observation of one
or more existing packet streams. The streams only serve measurement
when they are observed for that purpose, and are present whether
measurements take place or not.As new techniques for assessment emerge it is helpful to have clear
definitions of these notions. This memo provides more detailed
definitions, defines a new category for combinations of traditional
active and passive techniques, and discusses dimensions to evaluate new
techniques as they emerge.This memo provides definitions for Active and Passive Metrics and
Methods based on long usage in the Internet measurement community, and
especially the Internet Engineering Task Force. This memo also describes
the combination of fundamental Active and Passive categories, which are
called Hybrid Methods and Metrics.The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119.The scope of this memo is to define and describe Active and Passive
versions of metrics and methods which are consistent with the long-time
usage of these adjectives in the Internet measurement community and
especially the Internet Engineering Task Force. Since the science of
measurement is expanding, we provide a category for combinations of the
traditional extremes, treating Active and Passive as a continuum and
designating combinations of their attributes as Hybrid methods.Further, this memo's purpose includes describing multiple dimensions
to evaluate new methods as they emerge.This section defines the key terms of the memo. Some definitions use
the notion of "stream of interest" which is synonymous with "population
of interest" defined in clause 6.1.1 of ITU-T Recommendation Y.1540
. The definitions are consistent with .The standard definition of a quantity, produced in an assessment of
performance and/or reliability of the network, which has an intended
utility and is carefully specified to convey the exact meaning of a
measured value. (This definition is consistent with that of
Performance Metric in and ).The procedure or set of operations having the object of determining
a Measured Value or Measurement Result.See section 2 of for this definition (a
location in the network where packets can be observed), and related
definitions. The comparable term defined in IETF literature on Active
measurement is Measurement Point, see section 4.1 of . Both of these terms have come into use describing
similar actions at the identified point in the network path.Active measurement methods have the following attributes:Active methods generate packet streams. Commonly, the packet
stream of interest is generated as the basis of measurement.
Sometimes, the adjective "synthetic" is used to categorize Active
measurement streams . Accompanying packet
stream(s) may be generated to increase overall traffic load,
though the loading stream(s) may not be measured.The packets in the stream of interest have fields or field
values (or are augmented or modified to include fields or field
values) which are dedicated to measurement. Since measurement
usually requires determining the corresponding packets at multiple
measurement points, a sequence number is the most common
information dedicated to measurement, and often combined with a
timestamp.The Source and Destination of the packet stream of interest are
usually known a priori.The characteristics of the packet stream of interest are known
at the Source at least, and may be communicated to Destination as
part of the method. Note that some packet characteristics will
normally change during packet forwarding. Other changes along the
path are possible, see .When adding traffic to the network for measurement, Active
Methods influence the quantities measured to some degree, and those
performing tests should take steps to quantify the effect(s) and/or
minimize such effects.An Active Metric incorporates one or more of the aspects of Active
Methods in the metric definition.For example, IETF metrics for IP performance (developed according
to the framework) include the Source packet
stream characteristics as metric input parameters, and also specify
the packet characteristics (Type-P) and Source and Destination IP
addresses (with their implications on both stream treatment and
interfaces associated with measurement points).Passive measurement methods arebased solely on observations of undisturbed and unmodified
packet stream of interest (in other words, the method of
measurement MUST NOT add, change, or remove packets or fields, or
change field values anywhere along the path).dependent on the existence of one or more packet streams to
supply the stream of interest.dependent on the presence of the packet stream of interest at
one or more designated observation points.Some passive methods simply observe and collect information
on all packets that pass Observation Point(s), while others filter the
packets as a first step and only collect information on packets that
match the filter criteria, and thereby narrow the stream of
interest.It is common that passive methods are conducted at one or more
Observation Points. Passive methods to assess Performance Metrics
often require multiple observation points, e.g., to assess latency of
packet transfer across a network path between two Observation Points.
In this case, the observed packets must include enough information to
determine the corresponding packets at different Observation
Points.Communication of the observations (in some form) to a collector is
an essential aspect of Passive Methods. In some configurations, the
traffic load associated with results export to a collector may
influence the network performance. However, the collection of results
is not unique to Passive Methods, and the load from management and
operations of measurement systems must always be considered for
potential effects on the measured values.Passive Metrics apply to observations of packet traffic (traffic
flows in ).Passive performance metrics are assessed independent of the packets
or traffic flows, and solely through observation. Some refer to such
assessments as "out-of-band".One example of passive performance metrics for IP packet transfer
can be found in ITU-T Recommendation Y.1540 ,
where the metrics are defined on the basis of reference events
generated as packet pass reference points. The metrics are agnostic to
the distinction between active and passive when the necessary packet
correspondence can be derived from the observed stream of interest as
required.Hybrid Methods are Methods of Measurement which use a combination
of Active Methods and Passive Methods, to assess Active Metrics,
Passive Metrics, or new metrics derived from the a priori knowledge
and observations of the stream of interest. ITU-T Recommendation
Y.1540 defines metrics that are also
applicable to the hybrid categories, since packet correspondence at
different observation/reference points could be derived from "fields
or field values which are dedicated to measurement", but otherwise the
methods are passive.There are several types of Hybrid methods, as categorized
below.With respect to a *single* stream of interest, Hybrid Type I
methods fit in the continuum as follows, in terms of what happens at
the Source (or Observation Point nearby):If you generate the stream of interest => ActiveIf you augment or modify the stream of interest, or employ
methods that modify the treatment of the stream => Hybrid Type
IIf you solely observe a stream of interest => PassiveAs an example, consider the case where the method generates traffic
load stream(s), and observes an existing stream of interest according
to the criteria for Passive Methods. Since loading streams are an
aspect of Active Methods, the stream of interest is not "solely
observed", and the measurements involve a single stream of interest
whose treatment has been modified both the presence of the load.
Therefore, this is a Hybrid Type I method.We define Hybrid Type II as follows: Methods that employ two or
more different streams of interest with some degree of mutual
coordination (e.g., one or more Active streams and one or more
undisturbed and unmodified packet streams) to collect both Active and
Passive Metrics and enable enhanced characterization from additional
joint analysis. presents
a problem statement for Hybrid Type II methods and metrics. Note that
one or more Hybrid Type I streams could be substituted for the Active
streams or undisturbed streams in the mutually coordinated set. It is
the Type II Methods where unique Hybrid Metrics are anticipated to
emerge.Methods based on a combination of a single (generated) Active
stream and Passive observations applied to the stream of interest at
intermediate observation points are also a type of Hybrid Methods.
However, already defines these as Spatial
Metrics and Methods. It is possible to replace the Active stream of
with a Hybrid Type I stream and measure
Spatial Metrics (but this was un-anticipated when was developed).The Table below illustrates the categorization of methods (where
"Synthesis" refers to a combination of Active and Passive Method
attributes).There may be circumstances where results measured with Hybrid
Methods can be considered equivalent to Passive Methods. Referencing
the notion of a "class C" where packets of different Type-P are
treated equally in Section 13 of and the
terminology for paths from Section 5 of :Hybrid Methods of Measurement that augment or modify packets of a
"class C" in a host should produce equivalent results to Passive
Methods of Measurement, when hosts accessing and links transporting
these packets along the path (other than those performing
augmentation/modification) treat packets from both categories of
methods (with and without the augmentation/modification) as the same
"class C". The Passive Methods of Measurement represent the Ground
Truth for comparisons of results between Passive and Hybrid methods,
and this comparison should be conducted to confirm the class C
treatment.This section illustrates the definitions and presents some
examples.If we compare the Active and Passive Methods, there are at least
two dimensions on which methods can be evaluated. This evaluation
space may be useful when a method is a combination of the two
alternative methods.The two dimensions (initially chosen) are:"Effect of the measured stream on network
conditions." The degree to which the stream of interest biases
overall network conditions experienced by that stream and other
streams. This is a key dimension for Active measurement error
analysis. (Comment: There is also the notion of time averages - a
measurement stream may have significant effect while it is
present, but the stream is only generated 0.1% of the time. On the
other hand, observations alone have no effect on network
performance. To keep these dimensions simple, we consider the
stream effect only when it is present, but note that reactive
networks defined in may exhibit bias for
some time beyond the life of a stream.)"a priori Stream Knowledge." The degree to
which stream characteristics are known a priori. There are
methodological advantages of knowing the source stream
characteristics, and having complete control of the stream
characteristics. For example, knowing the number of packets in a
stream allows more efficient operation of the measurement
receiver, and so is an asset for active measurement methods.
Passive methods (with no sample filter) have few clues available
to anticipate what the protocol first packet observed will use or
how many packets will comprise the flow, but once the standard
protocol of a flow is known the possibilities narrow (for some
compliant flows). Therefore this is a key dimension for Passive
measurement error analysis.There are a few examples we can plot on a two-dimensional space. We
can anchor the dimensions with reference point descriptions.We recognize that method categorization could be based on
additional dimensions, but this would require a different graphical
approach.For example, "effect of stream of interest on network conditions"
could easily be further qualified into:effect on the performance of the stream of interest itself: for
example, choosing a packet marking or Differentiated Services Code
Point (DSCP) resulting in domain treatment as a real-time stream
(as opposed to default/best-effort marking).effect on unmeasured streams that share the path and/or
bottlenecks: for example, an extremely sparse measured stream of
minimal size packets typically has little effect on other flows
(and itself), while a stream designed to characterize path
capacity may affect all other flows passing through the capacity
bottleneck (including itself).effect on network conditions resulting in network adaptation:
for example, a network monitoring load and congestion conditions
might change routing, placing some flows to alternate paths to
mitigate the congestion.We have combined 1 and 2 on the Y-axis, as examination of
examples indicates strong correlation of effects in this pair, and
network adaptation is not addressed.It is apparent that different methods of IP network measurement can
produce different results, even when measuring the same path at the
same time. The two dimensions of the graph help to understand how the
results might change with the method chosen. For example, an Active
Method to assess throughput adds some amount of traffic to the network
which might result in lower throughput for all streams. However, a
Passive Method to assess throughput can also err on the low side due
to unknown limitations of the hosts providing traffic, competition for
host resources, limitations of the network interface, or private
sub-networks that are not an intentional part of the path, etc. And
Hybrid Methods could easily suffer from both forms of error. Another
example of potential errors stems from the pitfalls of using an Active
stream with known bias, such as a periodic stream defined in . The strength of modelling periodic streams (like
VoIP) is a potential weakness when extending the measured results to
other application whose streams are non-periodic. The solutions are to
model the application streams more exactly with an Active Method, or
accept the risks and potential errors with the Passive Method
discussed above.In , an IPv6 Option
Header for Performance and Diagnostic Measurements (PDM) is described
which (when added to the stream of interest at strategic interfaces)
supports performance measurements. This method processes a user
traffic stream and adds "fields which are dedicated to measurement"
(the measurement intent is made clear in the title of this option).
Thus:The method intends to have a small effect on the measured
stream and other streams in the network. There are conditions
where this intent may not be realized.The measured stream has unknown characteristics until it is
processed to add the PDM Option header. Note that if the packet
MTU is exceeded after adding the header, the intent to have small
effect will not be realized.We conclude that this is a Hybrid Type I method, having at
least one characteristic of both active and passive methods for a
single stream of interest.Draft , proposed to color
packets by re-writing a field of the stream at strategic interfaces to
support performance measurements. This method processes a user traffic
stream and inserts "fields or values which are dedicated to
measurement". Thus:The method intends to have a small effect on the measured
stream and other streams in the network (smaller than PDM above).
There are conditions where this intent may not be realized.The measured stream has unknown characteristics until it is
processed to add the coloring in the header, and the stream could
be measured and time-stamped during that process.We note that proposes a
method similar to , as ippm-list
discussion revealed.We conclude that this is a Hybrid Type I method, having at least
one characteristic of both active and passive methods for a single
stream of interest.Many Operations, Administration, and Management (OAM) methods exist
beyond the IP-layer. For example, defines
several different measurement methods which we would classify as
follows:Loss Measurement (LM) occasionally injects frames with a count
of previous frames since the last LM message. We conclude LM is
Hybrid Type I becauseThis method processes a user traffic stream,and augments the stream of interest with frames having
"fields which are dedicated to measurement".Synthetic Loss Measurement (SLM) and Delay Measurement (DM)
methods both inject dedicated measurement frames, so the "stream
of interest is generated as the basis of measurement". We conclude
that SLM and DM methods are Active Methods.We also recognize the existence of alternate terminology used in
OAM at layers other than IP. Readers are encouraged to consult for MPLS Loss and Delay measurement terminology,
for example.When considering security and privacy of those involved in
measurement or those whose traffic is measured, there is sensitive
information communicated and observed at observation and measurement
points described above, and protocol issues to consider. We refer the
reader to the security and privacy considerations described in the Large
Scale Measurement of Broadband Performance (LMAP) Framework , which covers active and passive measurement
techniques and supporting material on measurement context.This memo makes no requests for IANA consideration.Thanks to Mike Ackermann for asking the right question, and for
several suggestions on terminology. Brian Trammell provided key terms
and references for the passive category, and suggested ways to expand
the Hybrid description and types. Phil Eardley suggested some hybrid
scenarios for categorization as part of his review. Tiziano Ionta
reviewed the draft and suggested the classification for the "coloring"
method of measurement. Nalini Elkins identified several areas for
clarification following her review. Bill Jouris, Stenio Fernandes, and
Spencer Dawkins suggested several editorial improvements. Tal Mizrahi,
Joachim Fabini, Greg Mirsky and Mike Ackermann raised many key
considerations in their WGLC reviews, based on their broad measurement
experience.Internet protocol data communication service - IP packet
transfer and availability performance parametersOperation, administration and management (OAM) functions and
mechanisms for Ethernet-based networks