Network Working Group Jerry Perser INTERNET-DRAFT Spirent Expires in: August 2001 David Newman Network Test Sumit Khurana Telcordia Shobha Erramilli Telcordia Scott Poretsky Avici February 2001 Terminology for Benchmarking Network-layer Traffic Control Mechanisms Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Table of Contents 1. Introduction ............................................... 2 2. Existing definitions ....................................... 3 3. Term definitions .............................................3 3.1 Channel Capacity ..........................................3 3.2 Classification ............................................4 3.3 Code Point Forwarding Rate ................................4 3.4 Conforming ................................................5 3.5 Congestion ................................................5 3.6 Congestion Management .....................................6 3.7 Delay .....................................................6 3.8 Expected Output Vector ....................................7 Perser, Newman, & Poretsky [Page 1] INTERNET-DRAFT Differentiated Services Termology Feb. 2001 3.9 Flow ......................................................8 3.10 Jitter ...................................................8 3.11 Nonconforming ............................................9 3.12 Offered Load Vector .....................................10 3.13 Policy ..................................................10 3.14 Provision ...............................................11 3.15 Sequence number .........................................11 3.16 Shaping .................................................12 3.17 Stream ..................................................12 3.18 Tail dropping ...........................................13 3.19 Unburdened Response .....................................14 4. Security Considerations .....................................14 5. References ..................................................14 6. Author's Address ............................................15 7. Full Copyright Statement ....................................16 1. Introduction Driven by Internet economics, service providers and enterprises alike have shown strong interest in adding traffic-control capabilities to network devices. These capabilities would enable network operators to define and deliver minimum or maximum levels of bandwidth, delay, and jitter for multiple classes of traffic. Perhaps more importantly, network operators would be able to set pricing according to the level of service delivered. Networking device manufacturers have responded with a bewildering array of mechanisms for controlling network traffic. While there are numerous _policy management_ and _quality of service_ frameworks, many of them rely on one of two network-layer mechanisms for the actual control of forwarding rate, delay, and jitter. These two mechanisms are the IP precedence setting in the IP header and the diff-serve code point (DSCP) defined in [3]. This document describes the various terms to be used in benchmarking devices that implement traffic control based on IP precedence or DSCP criteria. This document is narrowly focused, in that it describes only terms for measuring behavior of a device or a group of devices using one of these two mechanisms. End-to-end and service-level measurements are beyond the scope of this document. This document introduces several new terms that will allow measurements to be taken during periods of congestion. New terminology is needed because most existing benchmarking terms assume the absence of congestion. For example, throughput is one of the most widely used measurements _- yet RFC 1242 defines throughput as a rate in the absence of loss. As a result, throughput is not a meaningful measurement where congestion exists. Another key difference from existing benchmarking terminology is the definition of measurements as observed on egress as well as ingress of a device/system under test. Again, the existence of congestion Perser, Newman, Khurana, Erramilli, Poretsky [Page 2] INTERNET-DRAFT Differentiated Services Termology Feb. 2001 requires the addition of egress measurements as well as those taken on ingress; without observing traffic leaving a device it is not possible to say whether traffic-control mechanisms effectively dealt with congestion. The principal measurements introduced in this document are forwarding rate, latency, and jitter, all of which can be observed with or without congestion of the DUT/SUT. 2. Existing definitions RFC 1242 "Benchmarking Terminology for Network Interconnect Devices" and RFC 2285 " Benchmarking Terminology for LAN Switching Devices" should be consulted before attempting to make use of this document. RFC 2474 " Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers" section 2, contains discussions of a number of terms relevant to network-layer traffic control mechanisms and should also be consulted. For the sake of clarity and continuity this RFC adopts the template for definitions set out in Section 2 of RFC 1242. Definitions are indexed and grouped together in sections for ease of reference. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 3. Term definitions 3.1 Channel Capacity Definition: The maximum forwarding rate of a link or set of aggregated links at which none of the offered packets are dropped by the DUT/SUT. Discussion: Channel capacity measures the data rate at the egress interface(s) of the DUT/SUT. In contrast, throughput as defined in RFC 1242 measures the data rate is based on the ingress interface(s) of the DUT/SUT. Ingress-based measurements do not account for congestion of the DUT/SUT. Channel capacity, as an egress measurement, does take congestion into account. Understanding channel capacity is a necessary precursor to any measurement involving congestion. Throughput numbers can be higher than channel capacity because of queueing. Perser, Newman, Khurana, Erramilli, Poretsky [Page 3] INTERNET-DRAFT Differentiated Services Termology Feb. 2001 Measurement units: N-octet packets per second Issues: See Also: Throughput [1] 3.2 Classification Definition: Local mapping of Policies to the IP Precedence or DSCP value in each frame. Discussion: Could be based on the DS field or IP Precedence in the packet header. Could be based on MF criteria such as IP Source and destination addresses, protocol and port number. Covers reclassification. A Hop does not know if a previous Hop classified the packets. Since the scope of the document is a per-hop-behavior, we avoid end-to-end discussions. Classification should cover all cases defined in policy. This may include rules mandating packet drops, not just forwarded packets. Measurement units: n/a Issues: See Also: 3.3 Code Point Forwarding Rate Definition: The number of packets per second in for all packets containing a single DSCP (or IP precedence) that a device can be observed to successfully transmit to the correct destination interface in response to a specified offered load. Discussion: - CP based, not port nor IP stream based. - Per-hop measurement. A DUT may change the DSCP for a multiple-hop measurement. Perser, Newman, Khurana, Erramilli, Poretsky [Page 4] INTERNET-DRAFT Differentiated Services Termology Feb. 2001 Measurement units: N-octet packets per second Issues: See Also: 3.4 Conforming Definition: A packet meeting a certain policy. Discussion: Consider a policy that allows a given application to consume only X bit/s of channel capacity and no more. In a congestion scenario, some individual packets will be conforming and others will not. It cannot be said that an entire stream or flow is conforming, since an offered load that exceeds policy boundaries will necessarily carry some nonconforming traffic. Measurement units: n/a Issues: See Also: 3.5 Congestion Definition: A condition in which one or more egress interfaces are offered more packets than can be forwarded at any given instant. Discussion: This condition is a superset of the overload definition [2]. That definition assumes the overload is introduced by strictly by the tester on ingress of a DUT/SUT. That may or may not be the case here. Another difference is that with multiple-DUT measurements, congestion may occur at multiple points. For example, multiple edge devices collectively may congest a core device. In contrast, overload [1] deals only with overload on ingress. Perser, Newman, Khurana, Erramilli, Poretsky [Page 5] INTERNET-DRAFT Differentiated Services Termology Feb. 2001 Ingress observations alone are not sufficient to cover all cases in which congestion may occur. A device with an infinite amount of memory could buffer an infinite amount of packets, and eventually forward all of them. However, these packets may or may not be forwarded during the test duration. Even though ingress interfaces accept all packets without loss, this hypothetical device may still be congested. The definition presented here explicitly defines congestion as an event observable on egress interfaces. Regardless of internal architecture, any device that cannot forward packets on one or more egress interfaces is congested. Measurement units: n/a Issues: See Also: 3.6 Congestion Management Definition: An implementation of one or more per-hop-behaviors to avoid or minimize the condition of congestion. Discussion: Congestion management may seek either to control congestion or avoid it altogether. Such mechanisms classify packets based upon IP Precedence or DSCP settings in a packet's IP header. Congestion avoidance mechanisms seek to prevent congestion before it actually occurs. Congestion control mechanisms gives one or more service classes preferential treatment over other classes during periods of congestion. Measurement units: n/a Issues: See Also: 3.7 Delay Definition: Perser, Newman, Khurana, Erramilli, Poretsky [Page 6] INTERNET-DRAFT Differentiated Services Termology Feb. 2001 The time interval starting when the last bit of the input frame reaches the input port of the DUT/SUT and ending when the last bit of the output frame is seen on the output port of the DUT/SUT. Discussion: Delay measurement is measured the same regardless of the type of DUT/SUT. Latency [1] require some knowledge of whether the DUT/SUT is a "store and forward" or a "bit forwarding" device. The fact that a DUT/SUT's technology has a lower delay than another technology should be visible in the measurement. The measurement point of the last bit is more like the way an IP datagram is processed. A datagram is not passed up or down the stack unless it is complete. Competition happens at the last bit of the datagram. Delay can be run at any offered load. Recommend at or below the channel capacity for non-congested delay. For congested delay, run the offered load above the channel capacity. Measurement units: Seconds. Issues: See Also: 3.8 Expected Output Vector Definition: A vector describing the expected output rate of packets having a specific code-point for each code-point in the code-point set depending on the offered load vector and device PHB configurations of the DUT/SUT. Discussion: The DUT is configured in a certain way in order that service discrimination happens for behavior aggregates when a specific traffic mix consisting of multiple behavior aggregates is applied. This term attempts to capture the expected behavior for which the device is configured given a certain load. The actual algorithms or mechanism, that the DUT uses to achieve service discrimination, is not important in describing the expected output vector. Measurement units: bits per second N-octets per second Perser, Newman, Khurana, Erramilli, Poretsky [Page 7] INTERNET-DRAFT Differentiated Services Termology Feb. 2001 See Also: Offered Load Vector 3.9 Flow Definition: A flow is a one or more of packets sharing a common intended pair of source and destination interfaces. Discussion: Packets are grouped by which interface they ingress and egress the DUT/SUT. A flow can contain multiple source IP addresses and/or destination IP addresses. All packets in a flow must enter on the same ingress interface and exit on the same egress interface, and have some common network layer content. Microflows [3] are a subset of flows. As defined in [3], microflows require application-to-application measurement. In contrast, flows use lower-layer classification criteria. Since this document focuses on network-layer classification criteria, we concentrate here on the use of network-layer identifiers in describing a flow. Flow identifiers also may reside at the data-link, transport, or application layers of the ISO model. However, identifiers other than those at the network layer are out of scope for this document. A flow may contain a single code point/IP precedence or may contain multiple values destined for a single egress interface. This is determined by the test methodology. Measurement units: n/a Issues: See Also: Microflow [3] Streams 3.10 Jitter Definition: Variation in delay. Discussion: Perser, Newman, Khurana, Erramilli, Poretsky [Page 8] INTERNET-DRAFT Differentiated Services Termology Feb. 2001 Jitter as the absolute value of the difference between the delay measurement of two adjacent packets |D(i) - D(i+1)|. The jitter between two packets is reported as the "instantaneous jitter". Packets lost are not counted in the jitter measurement. Average Jitter is the average of the instantaneous jitter measured during the test duration. Peak-to-peak jitter is the maximum delay minus the minimum delay of the packets forwarded by the DUT/SUT. Measurement units: Seconds (instantaneous) Seconds P-P (peak to peak) Seconds avg (average) Issues: See Also: 3.11 Nonconforming Definition: A packet in violation of a given policy. Discussion: A packet can be said to be nonconforming when it violates the terms of a given policy. For example, a policy may set an upper bound on bandwidth for a given class of traffic. When a DUT/SUT is offered packets of that class in excess of the terms allowed by the policy, the DUT/SUT must discard some packets of that class. The discarded packets are nonconforming to policy. Note that the decision to drop or forward packets is essentially time-based. Even though two packets may share a common DSCP or IP precedence value, one may be forwarded and one may be dropped depending on when the packets are offered to the DUT. It is the combination of classification and policy that determines whether packets are conforming or nonconforming. Measurement units: n/a Issues: Perser, Newman, Khurana, Erramilli, Poretsky [Page 9] INTERNET-DRAFT Differentiated Services Termology Feb. 2001 See Also: 3.12 Offered Load Vector Definition: A vector describing the rate at which packets having a specific code-point are offered to the DUT/SUT, for each code-point in a code-point set. Discussion: Offered loads across the different code-point classes, constituting a code-point set, determine the metrics associated with a specific code-point traffic class. Measurement Units: bits per second N-octets per second Issues: Packet size. Traffic descriptors such as peak rate, average rate etc. See Also: Expected output vector 3.13 Policy Definition: A rule or set of rules used to classify packets. Discussion: In the context of data networking, policies consist of one or (usually) more rules to administer, manage, and control access to network resources. These resources include applications, devices, bandwidth, and any other elements attached to a common network. In the context of network devices, PHBs are the mechanisms by which a policy may be enforced. Packets offered to a DUT/SUT may be conforming or nonconforming to a given policy. A PHB may cause some packets to be dropped or forwarded based on the rules laid down by a given policy. Measurement units: n/a Issues: See Also: Perser, Newman, Khurana, Erramilli, Poretsky [Page 10] INTERNET-DRAFT Differentiated Services Termology Feb. 2001 Conforming Nonconforming PHB 3.14 Provision Definition: Configuration of the DUT/SUT to match a specified rule or rules in the policy. Discussion: A DUT/SUT must be configured in accordance with the rules of a policy for that policy's rules to be enforced. Provision may be done on a dynamic or a static basis. Measurement units: n/a Issues: See Also: Conforming Nonconforming PHB Policy 3.15 Sequence number Definition: A field in the IP payload portion of the packet that is used to verify the order of the packets on the egress of the DUT/SUT. Discussion: The traffic generator sets the sequence number value and the traffic receiver checks the value upon receipt of the packet. The traffic generator changes the value on each packet transmitted based on an algorithm agreed to by the traffic receiver. The traffic receiver keeps track of the sequence numbers on a per stream basis. In addition to number of received packets, the traffic receiver may also report number of in-sequence packets, number of out-sequence packets and number of duplicate packets. The recommended algorithm to use to change the sequence number on sequential packets is an incrementing value. Measurement units: Perser, Newman, Khurana, Erramilli, Poretsky [Page 11] INTERNET-DRAFT Differentiated Services Termology Feb. 2001 n/a Issues: See Also: Stream 3.16 Shaping Definition: DUT/SUT control of code point forwarding rate so that a flow or stream does not exceed or drop below a specified rate set in the policy. Discussion: [3] defines a behavior aggregate in which a DUT/SUT performs a specific forwarding treatment on a group of packets. Shaping is one such treatment: A device may drop or forward packets within a stream or flow depending on whether the packets are in or out of the terms of the policy. Measurement units: n/a Issues: See Also: Conforming Nonconforming Policy 3.17 Stream Definition: A group of packets tracked as a single entity by the traffic receiver. A stream shares a common content such as type (IP, UDP), frame size, or payload. Discussion: Streams are tracked by "sequence number" or "unique signature field" (RFC 2889). Streams define how individual packet's statistics are grouped together to form an intelligible summary. Common stream groupings would be by egress interface, destination address, source address, DCSP, or IP precedence. A stream using sequence numbers can track the ordering of packets as they transverse the DUT/SUT. Streams are not restricted to a pair of source and destination interfaces as long as all packets are tracked as a single Perser, Newman, Khurana, Erramilli, Poretsky [Page 12] INTERNET-DRAFT Differentiated Services Termology Feb. 2001 entity. A mulitcast stream can be forward to multiple destination interfaces. Measurement units: n/a Issues: See Also: Flow MicroFlow [3] Sequence number 3.18 Tail dropping Definition: The condition in which a DUT/SUT discards newly arriving packets. Discussion: Every DUT/SUT has a finite amount of traffic it can forward, beyond which congestion occurs. Once the offered load crosses the congestion threshold, the device may discard any additional traffic that arrives until congestion clears. Tail dropping is typically a function of offered load exceeding a DUT/SUT's buffer capacity, but other factors internal to the DUT/SUT may also come into play. In terms of what is externally observable, tail dropping can be said to occur only when offered load exceeds channel capacity. Since a DUT/SUT may buffer traffic on ingress, the actual threshold for tail dropping may be higher than channel capacity. Measurement units: n/a Issues: Some congestion management mechanisms seek to avoid tail dropping by discarding packets before offered load exceeds channel capacity. In the presence of such mechanisms, neither congestion nor tail dropping should occur. See Also: Channel capacity Congestion Perser, Newman, Khurana, Erramilli, Poretsky [Page 13] INTERNET-DRAFT Differentiated Services Termology Feb. 2001 3.19 Unburdened Response Definition: A performance measure obtained when mechanisms used to support DiffServ are disabled. Discussion: Enabling Diffserv mechanisms such as scheduling algorithms may impose an additional processing overhead for packets, which may cause the aggregate response to suffer even when traffic belonging to only one class, the best effort class, is offered to the device. Comparisons with "unburdened performance" may thus be in order when obtaining metrics to ensure that enabling Diffserv mechanisms doesn't impose an excessive performance penalty. Measurement units: TBD. 4. Security Considerations Documents of this type do not directly effect the security of the Internet or of corporate networks as long as benchmarking is not performed on devices or systems connected to operating networks. 5. References [1] Bradner, S., Editor, "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, July 1991. [2] Mandeville, R., "Benchmarking Terminology for LAN Switching Devices", RFC 2285, February 1998. [3] K. Nichols, S. Blake, F. Baker, D. Black,"Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, December 1998. [4] S. Blake, D. Black, M. Carlson, E. Davies, Z. Wang, W. Weiss, "An Architecture for Differentiated Services", RFC 2475, December 1998. Perser, Newman, Khurana, Erramilli, Poretsky [Page 14] INTERNET-DRAFT Differentiated Services Termology Feb. 2001 6. Author's Address Jerry Perser Spirent Communications 26750 Agoura Road Calabasas, CA 91302 USA Phone: + 1 818 676 2300 EMail: jerry.perser@spirentcom.com David Newman Network Test 321 Newark St., Suite 301 Hoboken, NJ 07030 USA Phone: + 1 201 798 9999 EMail: dnewman@networktest.com Sumit Khurana Telcordia Technologies 445 South Street Morristown, NJ 07960 USA Phone: + 1 973 829 3170 EMail: sumit@research.telcordia.com Shobha Erramilli Telcordia Technologies 331 Newman Springs Road, Navesink, NJ 07701 USA Phone: + 1 732 758 5508 EMail: shobha@research.telcordia.com Scott Poretsky Avici Systems 101 Billerica Ave_Building #6 N. Billerica, MA 01862 USA Phone: + 1 978 964 2287 EMail: sporetsky@avici.com Perser, Newman, Khurana, Erramilli, Poretsky [Page 15] INTERNET-DRAFT Differentiated Services Termology Feb. 2001 7. Full Copyright Statement Copyright (C) The Internet Society (1998). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Perser, Newman, Khurana, Erramilli, Poretsky [Page 16]