Network Working Group G. Almes, Advanced Network & Services Internet Draft W. Cerveny, Advanced Network & Services P. Krishnaswamy, BellCore J. Mahdavi, Pittsburgh Supercomputer Center M. Mathis, Pittsburgh Supercomputer Center V. Paxson, Lawrence Berkeley Labs Expiration Date: January 1997 July 1996 Framework for IP Provider Metrics 1. Status of this Memo This document is an Internet Draft. Internet Drafts are working doc- uments of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute work- ing documents as Internet Drafts. Internet Drafts are draft documents valid for a maximum of six months, and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet Drafts as reference material or to cite them other than as ``work in progress''. To learn the current status of any Internet Draft, please check the ``1id-abstracts.txt'' listing contained in the Internet Drafts shadow directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast). This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited. 2. Introduction The purpose of this memo is to define a general framework for partic- ular metrics to be developed by the IP Provider Metrics (IPPM) effort within the Benchmarking Methodology Working Group (BMWG) of the Oper- ational Requirements Area (OR) of the IETF. We begin by laying out several criteria for the metrics that we adopt. These criteria are designed to promote an IPPM effort that will maximise an accurate common understanding by Internet users and Internet providers of the performance and reliability both of end-to- end paths through the Internet and of specific 'IP clouds' that Almes et al. [Page 1] ID Framework for IP Provider Metrics July 1996 comprise portions of those paths. We next define some Internet vocabulary that will allow us to speak clearly about Internet components such as routers, paths, and clouds. We next define three fundamental concepts, metrics, measurement methodology, and uncertainties/errors, that will allow us to speak clearly about specific metrics. Given these concepts, we proceed to discuss how they relate to the analytical framework shared by many aspects of the Internet engineering discipline. We then introduce the notion of empirically defined metrics, and continue to discuss two forms of composition. In some sections of the memo, we will surround some commentary text with the brackets {Comment: ... }. We stress that this commentary is only commentary, and is not itself part of the framework document or a proposal of particular metrics. In some cases this commentary will discuss some of the properties of metrics that might be envisioned, but the reader should assume that any such discussion is intended only to shed light on points made in the framework document, and not to suggest any specific metrics. 3. Criteria for IP Provider Metrics The overarching goal of the IP Provider Metrics effort is to achieve a situation in which users and providers of Internet transport ser- vice have an accurate common understanding of the performance and reliability of the Internet component 'clouds' that they user/provide. To achieve this, performance and reliability metrics for paths through the Internet must be developed. In several meetings of the BMWG criteria for these metrics have been specified: + The metrics must be concrete and well-defined, + A methodology for a metric should have the property that it is repeatable: if the methodology is used multiple times under iden- tical conditions, the same measurements should result in the same measurements. + The metrics must exhibit no bias for IP clouds implemented with identical technology, + The metrics must exhibit understood and fair bias for IP clouds implemented with non-identical technology, Almes et al. [Page 2] ID Framework for IP Provider Metrics July 1996 + The metrics must be useful to users and providers in understanding the performance they experience or provide, + The metrics must avoid inducing artificial performance goals. 4. Terminology for Paths and Clouds The following list defines terms that need to be precise in the development of path metrics. We proceed from low-level notions of host, router, and link, then proceed to define the notions of path and notions of IP cloud and exchange that allow us to segment a path into relevant pieces. host A computer capable of communicating using the Internet protocols; includes "routers". link A single link-level connection between two (or more) hosts; includes leased lines, ethernets, frame relay clouds, etc. router A host which facilitates network-level communication between hosts by forwarding IP packets. path A sequence of the form < h0, l1, h1, ..., ln, hn >, where n >= 0, each hi is a host, each li is a link between hi-1 and hi, each h1...hn-1 is a router. In an appropriate operational configura- tion, the links and routers in the path facilitate network-layer communication of packets from h0 to hn. Note that path is a unidi- rectional concept. subpath Given a path, a subpath is any subsequence of the given path which is itself a path. (Thus, the first and last element of a subpath is a host.) cloud An undirected (possibly cyclic) graph whose vertices are routers and whose edges are links that connect pairs of routers. Formally, ethernets, frame relay clouds, and other links that connect more than two routers are modelled as fully-connected meshes of graph edges. Note that to connect to a cloud means to connect to a router of the cloud over a link; this link is not itself part of the cloud. exchange A special case of a link, an exchange directly connects either a host to a cloud and/or one cloud to another cloud. Almes et al. [Page 3] ID Framework for IP Provider Metrics July 1996 cloud subpath A subpath of a given path, all of whose hosts are routers of a given cloud. path digest A sequence of the form < h0, e1, C1, ..., en, hn >, where n >= 0, h0 and hn are hosts, each e1 ... en is an exchange, and each C1 ... Cn-1 is a cloud subpath. 5. Three Fundamental Concepts 5.1. Metrics In the operational Internet, there are several quantities related to the performance and reliability of the Internet that we'd like to know the value of. When such a quantity is carefully specified, we term the quantity a metric. We anticipate that there will be sepa- rate RFCs for each metric (or for each closely related group of met- rics). In some cases, there might be no obvious means to effectively measure the metric; this is allowed, and even understood to be very useful in some cases. It is required, however, that the specification of the metric be as clear as possible about what quantity is being speci- fied. Thus, difficulty in practical measurement is sometimes allowed, but ambiguity in meaning is not. Each metric will be defined in terms of standard units of measure- ment. The international metric system will be used, with the follow- ing points specifically noted: + When a unit is expressed in simple meters (for distance/length) or seconds (for duration), appropriate related units based on thou- sands or thousandths of acceptable units are acceptable. Thus, distances expressed in kilometers (Km), durations expressed in milliseconds (msec), or microseconds (usec) are allowed, but not centimeters (because the prefix is not in terms of thousands or thousandths). + When a unit is expressed in a combination of units, appropriate related units based on thousands or thousandths of acceptable units are acceptable, but all such thousands/thousandths must be grouped at the beginning. Thus, kilo-meters per second (Km/sec) is allowed, but meters per millisecond is not. Almes et al. [Page 4] ID Framework for IP Provider Metrics July 1996 + The unit of information is the bit. + When metric prefixes are used with bits or with combinations including bits, those prefixes will have their metric meaning (related to decimal 1000), and not the meaning conventional with computer storage (related to decimal 1024). In any RFC that defines a metric whose units include bits, this convention will be followed and will be repeated to ensure clarity for the reader. + When a time is given, it will be taken in UTC. Note that these points apply to the specifications for metrics and not, for example, to packet formats where octets will likely be used in preference/addition to bits. Finally, we note that some metrics may be defined purely in terms of other metrics; such metrics are call 'derived metrics'. 5.2. Measurement Methodology For a given set of well-defined metrics, a number of distinct mea- surement methodologies may exist. A partial list includes: + Direct measurement of a performance metric using injected test traffic. Example: measurement of the round-trip delay of an IP packet of a given size over a given route at a given time. + Projection of a metric from lower-level measurements. Example: given accurate measurements of propagation delay and bandwidth for each step along a path, projection of the complete delay for the path for an IP packet of a given size. + Estimation of a consituent metric from a set of more aggregated measurements. Example: given accurate measurements of delay for a given one-hop path for IP packets of different sizes, estimation of propagation delay for the link of that one-hop path. + Estimation of a given metric at one time from a set of related metrics at other times. Example: given an accurate measurement of flow capacity at a past time, together with a set of accurate delay measurements for that past time and the current time, and given a model of flow dynamics, estimate the flow capacity that would be observed at the current time. This list is by no means exhaustive. The purpose is to point out the variety of measurement techniques. When a given metric is specified, a given measurement approach might be noted and discussed. That approach, however, is not formally part of the specification. A methodology for a metric should have the property that it is repeatable: if the methodology is used multiple times under identical conditions, it should result in consistent measurements. Almes et al. [Page 5] ID Framework for IP Provider Metrics July 1996 Backing off a little from the word 'identical' in the previous para- graph, we could more accurately use the word 'continuity' to describe a property of a given methodology: a methodology for a given metric exhibits continuity if, for small variations in conditions, it results in small variations in the resulting measurements. Slightly more precisely, for every positive epsilon, there exists a positive delta, such that if two sets of conditions are within delta of each other, then the resulting measurements will be within epsilon of each other. At this point, this should be taken as a heuristic driving our intuition about one kind of robustness property rather than as a precise notion. A metric that has at least one methodology that exhibits continuity is said itself to exhibit continuity. Note that some metrics, such as hop-count along a path, are integer- valued and therefore cannot exhibit continuity in quite the sense given above. Note further that, in practice, it may not be practical to know (or be able to quantify) the conditions relevant to a measurement at a given time. For example, since the instantaneous load (in packets to be served) at a given router in a high-speed wide-area network can vary widely over relatively brief periods and will be very hard for an external observer to quantify, various statistics of a given met- ric may be more repeatable, or may better exhibit continuity. In that case those particular statistics should be specified when the metric is specified. Finally, some measurement methodologies may be 'conservative' in the sense that a measurement that may themselves modify the value of the performance metric they attempt to measure. {Comment: for example, in a wide-are high-speed network under modest load, a test using sev- eral small 'ping' packets to measure delay would likely not interfere (much) with the delay properties of that network as observed by oth- ers. The corresponding statement about tests using a large flow to measure flow capacity would likely fail.} 5.3. Measurements, Uncertainties, and Errors Even the very best measurement methodologies for the very most well behaved metrics will exhibit errors. Those who develop such measure- ment methodologies, however, should strive to: Almes et al. [Page 6] ID Framework for IP Provider Metrics July 1996 + minimise their uncertainties/errors, + understand and document the sources of uncertainty/error, and + quantify the amounts of uncertainty/error. by doing so, the measurement community will work together to improve our ability to understand the performance and reliability of the Internet. For example, when developing a method for measuring delay, understand how any errors in your clocks introduce errors into your delay mea- surement, and quantify this effect as well as you can. In some cases, this will result in a requirement that a clock be at least up to a certain quality if it is to be used to make a certain measure- ment. As a second example, consider the timing error due to measurement overheads within computer making the measurement, as opposed to delays due to the Internet component being measured. The former is a measurement error, while the latter reflects the metric of interest. Note that one technique that can help avoid this overhead is the use of a packet filter/sniffer, running on a separate computer that records network packets and timestamps them accurately. The result- ing trace can then be analysed to assess the test traffic, minimising the effect of measurement host delays, or at least allowing those delays to be accounted for. Finally, we note that derived metrics (defined above) or metrics that exhibit spatial or temporal composition (defined below) offer occa- sion for the analysis of measurement uncertainty of related measure- ments to be themselves related. 6. Metrics and the Analytical Framework As the Internet has evolved from the early packet-switching studies of the 1960s, the Internet engineering community has evolved a common analytical framework of concepts. This analytical framework, or A- frame, used by designers and implementers of protocols, by those involved in measurement, and by those who study computer network per- formance using the tools of simulation and analysis, has great advan- tage to our work. A major objective here is to generate network characterisations that are consistent in both analytical and practi- cal settings, since this will maximise the chances that non-empirical network study can be better correlated with, and used to further our understanding of, real network behavior. Whenever possible, therefore, we would like to develop and leverage the A-frame. Thus, whenever a metric to be specified is understood to be closely related to concepts (such as the Internet components Almes et al. [Page 7] ID Framework for IP Provider Metrics July 1996 defined above) within the A-frame, we will attempt to specify the metric in the A-frame's terms. In such a specification we will develop the A-frame by precisely defining the concepts needed for the metric, then leverage the A-frame by defining the metric in terms of those concepts. Such a metric will be called an 'analytically specified metric' or, more simply an analytical metric. {Comment: Examples of such analytical metrics might include: propagation time of a link The time, in seconds, required by a single bit to travel from the output port on one Internet host across a single link to another Internet host. bandwidth of a link for packets of size k The capacity, in bits/second, where only those bits of the IP packet are counted, for a packet of size k bytes. route The path, as defined in Section 4, from A to B at a given time. hop count of a route The value 'n' of the route path. } Note that we make no a priori list of just what A-frame concepts will emerge in these specifications, but we do encourage their use and urge that they be carefully specified so that, as our set of metrics develops, so will a specified set of A-frame concepts tech- nically consistent with each other and consonent with the common understanding of those concepts within the general Internet commu- nity. These A-frame concepts will be intended to abstract from actual Internet components in such a way that: + the essential function of the component is retained, + properties of the component relevant to the metrics we aim to cre- ate are retained, + a subset of these component properties are defined as analytical metrics, and + those properties of actual Internet components not relevant to defining the metrics we aim to create are dropped. {Comment: for example, when considering a router in the context of packet forwarding, we might model the router as a component that receives packets on an input link, queues them on a FIFO packet queue Almes et al. [Page 8] ID Framework for IP Provider Metrics July 1996 of finite size, employs tail-drop when the packet queue is full, and forwards them on an output link. The transmission speed (in bits/second) of the input and output links, the latency in the router (in seconds), and the maximum size of the packet queue (in bits) are relevant analytical metrics.} In some cases, such analytical metrics used in relation to a router will be very closely related to specific metrics of the performance of Internet paths. For example, an obvious formula (L + P/B) involv- ing the latency in the router (L), the packet size (in bits) (P), and the transmission speed of the output link (B) might closely approxi- mate the increase in packet delay due to the insertion of a given router along a path. We stress, however, that well-chosen and well-specified A-frame con- cepts and their analytical metrics will support more general metric creation effort in less obvious ways. {Comment: for example, when considering the flow capacity of a path, it may be of real value to be able to model each of the routers along the path as packet forwarders as above. Techniques for estimating the flow capacity of a path might use the maximum packet queue size as a parameter in decidedly non-obvious ways. For example, as the maximum queue size increases, so will the ability of the router to continuously move traffic along an output link despite fluctuations in traffic from an input link. Estimating this increase, however, remains a research topic.} The key role of these concepts is to abstract the properties of the Internet components relevant to given metrics. Judgement is required to avoid making assumptions that bias the modeling and metric effort toward one kind of design. {Comment: for example, routers might not use tail-drop, even though tail-drop might be easier to model analytically.} Note that, when we specify A-frame concepts and analytical metrics, we will inevitably make simplifying assumptions. Further, as noted above, judgement is required in making these assumptions in order to make them best suit our purposes. Finally, note that different elements of the A-frame might well make different simplifying assumptions. For example, the abstraction of a router used to further the definition of delay might treat the router's packet queue as a single FIFO queue, but the abstraction of a router used to further the definition of the handling of an RSVP- enabled packet might treat the router's packet queue to support bounded delay -- a contradictory assumption. This is not to say that Almes et al. [Page 9] ID Framework for IP Provider Metrics July 1996 we make contradictory assumptions at the same time, but that two dif- ferent parts of our work might refine the simpler base concept in two divergent ways for different purposes. 7. Empirically Specified Metrics There are useful performance and reliability metrics that do not fit so neatly into the A-frame, usually because the A-frame lacks the complexity or power for dealing with them. For example, "the best flow capacity achievable along a path using an RFC-1122-compliant TCP" would be good to be able to measure, but we have no analytical framework of sufficient complexity to allow us to cast that flow capacity as an analytical metric. These notions can still be well specified by instead describing a reference methodology for measuring them. Such a metric will be called an 'empirically specified metric', or more simply, an empirical metric. Such empirical metrics should have three properties: + we should have a clear definition for each in terms of real-world Internet components, + we should have at least one effective means to measure them, and + to the extent possible, we should have an (necessarily incomplete) understanding of the metric in terms of the A-frame so that we can use our measurements to reason about the performance and reliabil- ity of A-frame components and of aggregations of A-frame compo- nents. 8. Two Forms of Composition 8.1. Spatial Composition of Metrics In some cases, it may be realistic and useful to define metrics in such a fashion that they exhibit spatial composition. By spatial composition, we mean a characteristic of some path met- rics, in which the metric as applied to a (complete) path can also be defined for various subpaths (cf. definition above), and in which the appropriate A-frame concepts for the metric suggest useful relation- ships between the metric applied to these various subpaths (including the complete path, the various cloud subpaths of a given path digest, and even single routers along the path). The effectiveness of spa- tial composition depends: Almes et al. [Page 10] ID Framework for IP Provider Metrics July 1996 + on the usefulness in analysis of these relationships as applied to the relevant A-frame components, and + on the practical use of the corresponding relationships as applied to metrics and to measurement methodologies. {Comment: for example, consider some metric for delay of a 100-byte packet across a path P, and consider further a path digest of P. The definition of such a metric might include a conjecture that the delay across P is very nearly the sum of the corresponding metric across the exhanges (ei) and clouds (Ci) of the given path digest. The definition would further include a note on how a corresponding relation applies to relevant A-frame components, both for the path P and for the exchanges and clouds of the path digest.} When the definition of a metric includes a conjecture that the metric across the path is related to the metric across the subpaths of the path, that conjecture constitutes a claim that the metric exhibits spatial composition. The definition should then include: + the specific conjecture applied to the metric, + a justification of the practical utility of the composition in terms of making accurate measurements of the metric on the path, and + a justification of the usefulness of the composition in terms of making analysis of the path using A-frame concepts more effective. 8.2. Temporal Composition of Formal Models and Empirical Metrics In some cases, it may be realistic and useful to define metrics in such a fashion that they exhibit temporal composition. By temporal composition, we mean a characteristic of some path met- rics, in which the metric as applied to a path at a given time T is also defined for various times t0 < t1 < ... < tn < T, and in which the appropriate A-frame concepts for the metric suggests useful rela- tionships between the metric applied at times t0, ..., tn and the metric applied at time T. The effectiveness of temporal composition depends: + on the usefulness in analysis of these relationships as applied to the relevant A-frame components, and + on the practical use of the corresponding relationships as applied to metrics and to measurement methodologies. {Comment: for example, consider some metric for the expected flow capacity across a path P during the five-minute period surrounding the time T, and suppose further that we have the corresponding values for each of the four previous five-minute periods t0, t1, t2, and t3. Almes et al. [Page 11] ID Framework for IP Provider Metrics July 1996 The definition of such a metric might include a conjecture that the flow capacity at time T can be estimated from a certain kind of extrapolation from the values of t0, ..., t3. The definition would further include a note on how a corresponding relation applies to relevant A-frame components. Note: any (spatial or temporal) compositions involving flow capacity are likely to be subtle, and temporal compositions are generally more subtle than spatial compositions, so the reader should understand that the foregoing example is intentionally naive.} When the definition of a metric includes a conjecture that the metric across the path at a given time T is related to the metric across the path for a set of other times, that conjecture constitutes a claim that the metric exhibits temporal composition. The definition should then include: + the specific conjecture applied to the metric, + a justification of the practical utility of the composition in terms of making accurate measurements of the metric on the path, and + a justification of the usefulness of the composition in terms of making analysis of the path using A-frame concepts more effective. 9. Security Considerations This memo raises no security issues. 10. References [1] Vern Paxson, ftp://ftp.ee.lbl.gov/papers/metrics-framework- INET96.ps.Z 11. Authors' Addresses Guy Almes Advanced Network & Services, Inc. 200 Business Park Drive Armonk, NY 10504 USA Phone: +1 914/273-7863 Bill Cerveny Advanced Network & Services, Inc. 200 Business Park Drive Almes et al. [Page 12] ID Framework for IP Provider Metrics July 1996 Armonk, NY 10504 USA Padma Krishnaswamy Bell Communications Research 445 South Street Morristown, NJ 07960 USA Jamshid Mahdavi Pittsburgh Supercomputing Center 4400 5th Avenue Pittsburgh, PA 15213 USA Matt Mathis Pittsburgh Supercomputing Center 4400 5th Avenue Pittsburgh, PA 15213 USA Vern Paxson MS 50B/2239 Lawrence Berkeley National Laboratory University of California Berkeley, CA 94720 USA Phone: +1 510/486-7504 Almes et al. [Page 13]