2.3.3 Benchmarking Methodology (bmwg)

NOTE: This charter is a snapshot of the 48th IETF Meeting in Pittsburgh, Pennsylvania. It may now be out-of-date. Last Modified: 17-Jul-00


Kevin Dubray <kdubray@ironbridgenetworks.com>

Operations and Management Area Director(s):

Randy Bush <randy@psg.com>
Bert Wijnen <bwijnen@lucent.com>

Operations and Management Area Advisor:

Randy Bush <randy@psg.com>

Mailing Lists:

General Discussion:bmwg@ironbridgenetworks.com
To Subscribe: bmwg-request@ironbridgenetworks.com
Archive: http://www.alvestrand.no/archives/bmwg/

Description of Working Group:

The major goal of the Benchmarking Methodology Working Group is to make a series of recommendations concerning the measurement of the performance characteristics of various internetworking technologies; further, these recommendations may focus on the systems or services that are built from these technologies.

Each recommendation will describe the class of equipment, system, or service being addressed; discuss the performance characteristics that are pertinent to that class; clearly identify a set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of benchmarking results.

Because the demands of a class may vary from deployment to deployment, a specific non-goal of the Working Group is to define acceptance criteria or performance requirements.

An ongoing task is to provide a forum for discussion regarding the advancement of measurements designed to provide insight on the operation internetworking technologies.

Goals and Milestones:



Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches.



Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC.



Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately.



Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft.



Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices.



Submit initial draft of Benchmarking Methodology for LAN Switches.



Submit Terminology for IP Multicast Benchmarking draft for AD Review.



Submit Benchmarking Terminology for Firewall Performance for AD review



Progress ATM benchmarking terminology draft to AD review.



Submit Benchmarking Methodology for LAN Switching Devices draft for AD review.

Jul 00


Submit first draft of Firewall Benchmarking Methodology.

Jul 00


First Draft of Terminology for FIB related Router Performance Benchmarking.

Jul 00


First Draft of Router Benchmarking Framework

Jul 00


Methodology for FIB related Router Performance Benchmarking to AD review.

Sep 00


Methodology for IP Multicast Benchmarking to AD Review.

Sep 00


Methodology for ATM Benchmarking for AD review.

Sep 00


Progress Frame Relay benchmarking terminology draft to AD review.

Nov 00


Terminology for ATM ABR Benchmarking for AD review.

Jan 01


Firewall Benchmarking Methodology to AD Review

Jan 01


Terminology for FIB related Router Performance Benchmarking to AD review.

Mar 01


First Draft of Methodology for FIB related Router Performance Benchmarking.

Mar 01


Router Benchmarking Framework to AD review.


Request For Comments:







Benchmarking Terminology for Network Interconnection Devices



Benchmarking Terminology for LAN Switching Devices



Terminology for IP Multicast Benchmarking



Benchmarking Methodology for Network Interconnect Devices



Benchmarking Terminology for Firewall Performance



Terminology for ATM Benchmarking

Current Meeting Report

The bmwg met Monday morning 7/31/00. Scott Bradner acted as chair due to the normal chair's being unable to attend.

The first item on the agenda was a discussion on a proposal to have the bmwg develop documents about testing of performance in a wireless environment. The decision of the AD and the ad-hoc chair was that at this time it did not look like the proposed work fits within the bmwg area so the offer of a presentation was declined.

The group then discussed draft-khurana-bmwg-diffservm-00.txt. There was a lot of discussion on trying to figure out if testing of a device that implements diffserv also needs to take into account the actions of end devices running TCP for example TCP backing off in the face of packet discard. The consensus seemed to be that any exploration of this should be done after the methodology of testing isolated diffserv performance had been completed.

The final document that the working group discussed was draft-feher-benchresres-00.txt. Feher gave a short presentation explaining what was involved in testing devices implementing RSVP. There was support for continued work on this proposal but the suggestion was made to split the ID into separate terminology and methodology documents.


None received.