Current Meeting Report

2.3.3 Benchmarking Methodology (bmwg)

NOTE: This charter is a snapshot of the 52nd IETF Meeting in Salt Lake City, Utah USA. It may now be out-of-date. Last Modified: 10-Oct-01
Kevin Dubray <>
Operations and Management Area Director(s):
Randy Bush <>
Bert Wijnen <>
Operations and Management Area Advisor:
Randy Bush <>
Mailing Lists:
To Subscribe:
In Body: subscribe your_email_address
Description of Working Group:
The major goal of the Benchmarking Methodology Working Group is to make a series of recommendations concerning the measurement of the performance characteristics of various internetworking technologies; further, these recommendations may focus on the systems or services that are built from these technologies.

Each recommendation will describe the class of equipment, system, or service being addressed; discuss the performance characteristics that are pertinent to that class; clearly identify a set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of benchmarking results.

Because the demands of a class may vary from deployment to deployment, a specific non-goal of the Working Group is to define acceptance criteria or performance requirements.

An ongoing task is to provide a forum for discussion regarding the advancement of measurements designed to provide insight on the operation internetworking technologies.

Goals and Milestones:
Done   Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches.
Done   Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC.
Done   Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately.
Done   Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft.
Done   Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices.
Done   Submit initial draft of Benchmarking Methodology for LAN Switches.
Done   Submit Terminology for IP Multicast Benchmarking draft for AD Review.
Done   Submit Benchmarking Terminology for Firewall Performance for AD review
Done   Progress ATM benchmarking terminology draft to AD review.
Done   Submit Benchmarking Methodology for LAN Switching Devices draft for AD review.
Done   Submit first draft of Firewall Benchmarking Methodology.
Done   First Draft of Terminology for FIB related Router Performance Benchmarking.
Done   First Draft of Router Benchmarking Framework
Done   Methodology for ATM Benchmarking for AD review.
Done   Progress Frame Relay benchmarking terminology draft to AD review.
Done   Terminology for ATM ABR Benchmarking for AD review.
Mar 01   Router Benchmarking Framework to AD review.
Jul 01   Terminology for FIB related Router Performance Benchmarking to AD review.
Nov 01   Methodology for IP Multicast Benchmarking to AD Review.
Nov 01   Firewall Benchmarking Methodology to AD Review
Nov 01   Net Traffic Control Benchmarking Terminology to AD Review
Nov 01   Resource Reservation Benchmarking Terminology to AD Review
Nov 01   EGP Convergence Benchmarking Terminology to AD Review
Dec 01   First Draft of Methodology for FIB related Router Performance Benchmarking.
Feb 02   First draft Net Traffic Control Benchmarking Methodology.
Feb 02   Resource Reservation Benchmarking Methodology to AD Review
Feb 02   Basic BGP Convergence Benchmarking Methodology to AD Review.
Jun 02   Methodology for FIB related Router Performance Benchmarking to AD review.
Nov 02   Net Traffic Control Benchmarking Methodology to AD Review.
Request For Comments:
RFC1242 Benchmarking Terminology for Network Interconnection Devices
RFC2285 Benchmarking Terminology for LAN Switching Devices
RFC2432 Terminology for IP Multicast Benchmarking
RFC2544 Benchmarking Methodology for Network Interconnect Devices
RFC2647 Benchmarking Terminology for Firewall Performance
RFC2761 Terminology for ATM Benchmarking
RFC2889 Benchmarking Methodology for LAN Switching Devices
RFC3116 Methodology for ATM Benchmarking
RFC3133 Terminology for Frame Relay Benchmarking
RFC3134 Terminology for ATM ABR Benchmarking

Current Meeting Report

Benchmarking Methodology WG Minutes

WG Chair: Kevin Dubray

Minutes reported by Kevin Dubray.

The BMWG met at the 52nd IETF in Salt Lake City, Utah, on Tuesday, December 11, 2001.

The agenda was offered and subsequently approved as:

1. Administration.

2. Router Resource Reservation Benchmarking - status update.

3. Benchmarking Network-layer Traffic Control Mechanisms - update on changes to the metrics.

4. Issues regarding the specification of a unified benchmarking methodology for Basic BGP convergence.

5. Goals for next period.

1. Administration.
The I-D on "Terminology for Forwarding Information Base (FIB) based Router Performance, <draft-ietf-bmwg-fib-term-04.txt>, was moved to the RFC editor queue for RFC publication.

The "Methodology for IP Multicast Benchmarking" I-D, <draft-ietf-bmwg-mcastm-07.txt> was revised, issued, and was the basis of a WG Last Call. It was thought the I-D was in need of further edits before advancing.

The I-Ds:


represent revised Internet-Drafts that were posted during this period. Of these, the Firewall draft appears to be nearing last call, as is Resource Reservation Benchmarking Terminology draft. Also note that the editors indicate that <draft-ietf-bmwg-benchres-method-01.txt> was revised and submitted, but it never got posted.

2. Router Resource Reservation Benchmarking

Krisztian Nemeth gave a quick update as to what's new in the Router Resource Reservation Benchmarking Terminology draft. Essentially, one new metric, Session Refreshing Capacity, was added. Details of the new metric are noted in the slides.

A slide showing processor utilization versus the number of sessions being maintained by an RSVP implementation was offered.

Krisztian asked whether anyone thought that the draft was not ready for last call. There was no dissenting views. The chair asked folks to read the draft and post comments to the list, as there was little input or discussion evidenced.

3. Benchmarking Network-layer Traffic Control Mechanisms
Jerry Perser outlined changes from the 01 to the 02 versions of the related draft. He clarified the notion of vectors and gave examples. He indicated that the next step would be to begin moving the terminology draft to WG last call and start applying the terminology to a corresponding methodology draft. Jerry's slides can be found in the proceedings.

Follow-up commentary regarding the use of the term "congestion" in the draft ensued. Jerry indicated by congestion he meant "the condition by which the device is unable to forward packets." It was offered that another look at what the draft states and what Jerry means might be advantageous. It was suggested to take follow-up discussions to the list.

Another question sought to elucidate the difference between the concepts of delay and latency. Jerry indicated a major difference was which portion of the data unit that was formed the basis of the measurement: with Latency, it's the frame. With Delay, it's the IP packet only, exclusive of the packet's framing considerations.

A question regarding what input did the I-D's editors get regarding the definition of "policy" was posed. Jerry indicated that the editors didn't seek to define the term policy. They hope to gather input over time and adapt the notion as needed.

There were questions regarding packet sequencing and the draft. Was goodness just a non-reversed sequence of 2 packets? Or was it anticipated that some effort would be made to determine how far out of order a particular packet train was? Jerry thought the degree the packets were out of order was not as important as the notion that packets were out of order. Others disagreed. It was mentioned that packet sequencing benchmarks were discussed at the last IPPM meeting. One of the attending IPPM co-chairs (Matt Zekauskas) was asked as to the current status of that effort. Matt indicated that the WG was still waiting on an I-D from the proponents.

4. Benchmarking BGP convergence issues.
Marianne Lepp, Alvaro Retana, Padma Krishnaswamy, and Sue Hares conspired to present and discuss some of the outstanding issues that have been experienced in developing the BGP convergence benchmarking I-D. The slides used are available in the proceedings.

The group was reminded that issues related to the topic are wider and deeper than the scoping of this initial methodology will avail.

In the context of sending "a packet stream from TR," slide 6, it was asked when was the packet actually sent. It was indicated that the packet was intended to be sent after the BGP session is set up.

Slide 9, "Benchmarking Convergence Approach," provided lots of discussion as to the goals of the benchmark. Was the goal to provide a realistic view of device operation or provide characterization at some upper bound? The intent, as it was explained, was to provide a "level playing field" by which good characterization data could be collected on a device in typical scenarios. An attendee stated that "real" scenarios are better than contrived, upper bounds scenarios. This was countered by another attendee who stated many ISP's RFP's require the characterization of "boundary" conditions...

On route mixtures, slide 13, the notion that prefix distribution and and node distribution are very related was reinforced. Moreover, most of those factors distill into two primary issues: how the device's tables are built, and how the information is subsequently conveyed. With regards to packet trains, the group was asked if the train has jitter, was it important to replicate that jitter?

In the discussion of slide 14, Prefix Distribution, it was asked whether it was desirable to replicate the distribution of the table or the updates. One replied, you could do either one or even both; the were dissenting voices on the wisdom of employing the "both" part of that reply.

A question was posed as to why the choice for DUT stimulus appeared to be biased towards synthetic traces over recorded traces from actual network elements. It was replied that repeatability was one consideration, but the primary reasons were cited in slide 10 (e.g., dependency on vantage point, consider future operating conditions, etc.). It was replied that the importance of synthetic traces should not be discounted.

On the subject of BGP attribute distribution, it was emphasized that a BGP table contains many _different_ "attribute combinations." Prefixes that share an attribute aren't necessarily grouped together - repeatability of routing information does impact the results. It was stated by an attendee that commonality of nomenclature in these related I-Ds and the research literature could prove to be an issue.

Packet packing, slide 28, was thought to be another area that could benefit from input; what is a good set of parameters to use for a test?

Slide 33 addressed multiple peers in a test environment and that a most realistic operation was to stagger the peers' start. The group was asked to comment on what is a accurate staggering.

One attendee asked, based on slide 35, Peer Specifics, whether the focus was on characterizing the behavior related to the "loading" of a DUT or response to "changes in state" experienced by the DUT. It was thought the former would be best for intial focus; we can worry about changes in state later.

Another general theme thought to require addressing: one can test with real data, but how does one "push on" a variety of parameters to form _useful_ test(s) and which parameters are they?

It was thought that input on the topic of route mixtures was crucial to moving forward, as was resolving whether synthetic traces were preferred to actual recorded traces.

5. Goals for next period.

- Get Multicast methodology draft to AD review.
- Move Resource Reservation Router benchmarking terminology draft to last call.
- Provide input to open BGP convergence methodology issues.
- Tie up last revision(s) of the Firewall I-D.
- Progress other drafts as needed.


BGP Convergence Measurement Issues
DiffServ Measurement
Router Resource Reservation Benchmarking - status update