2.4.3 Benchmarking Methodology (bmwg)

NOTE: This charter is a snapshot of the 50th IETF Meeting in Minneapolis, Minnesota. It may now be out-of-date. Last Modified: 14-Mar-01


Kevin Dubray <kdubray@juniper.net>

Operations and Management Area Director(s):

Randy Bush <randy@psg.com>
Bert Wijnen <bwijnen@lucent.com>

Operations and Management Area Advisor:

Randy Bush <randy@psg.com>

Mailing Lists:

General Discussion:bmwg@ietf.org
To Subscribe: bmwg-request@ietf.org
In Body: subscribe your_email_address
Archive: ftp://ftp.ietf.org/ietf-mail-archive/bmwg/

Description of Working Group:

The major goal of the Benchmarking Methodology Working Group is to make a series of recommendations concerning the measurement of the performance characteristics of various internetworking technologies; further, these recommendations may focus on the systems or services that are built from these technologies.

Each recommendation will describe the class of equipment, system, or service being addressed; discuss the performance characteristics that are pertinent to that class; clearly identify a set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of benchmarking results.
Because the demands of a class may vary from deployment to deployment, a specific non-goal of the Working Group is to define acceptance criteria or performance requirements.
An ongoing task is to provide a forum for discussion regarding the advancement of measurements designed to provide insight on the operation internetworking technologies.

Goals and Milestones:



Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches.



Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC.



Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately.



Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft.



Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices.



Submit initial draft of Benchmarking Methodology for LAN Switches.



Submit Terminology for IP Multicast Benchmarking draft for AD Review.



Submit Benchmarking Terminology for Firewall Performance for AD review



Progress ATM benchmarking terminology draft to AD review.



Submit Benchmarking Methodology for LAN Switching Devices draft for AD review.

Jul 00


Submit first draft of Firewall Benchmarking Methodology.



First Draft of Terminology for FIB related Router Performance Benchmarking.



First Draft of Router Benchmarking Framework

Jul 00


Methodology for FIB related Router Performance Benchmarking to AD review.

Sep 00


Methodology for IP Multicast Benchmarking to AD Review.

Sep 00


Methodology for ATM Benchmarking for AD review.

Sep 00


Progress Frame Relay benchmarking terminology draft to AD review.

Nov 00


Terminology for ATM ABR Benchmarking for AD review.

Jan 01


Firewall Benchmarking Methodology to AD Review

Mar 01


First Draft of Methodology for FIB related Router Performance Benchmarking.

Mar 01


Router Benchmarking Framework to AD review.

Jul 01


Terminology for FIB related Router Performance Benchmarking to AD review.

Request For Comments:






Benchmarking Terminology for Network Interconnection Devices



Benchmarking Terminology for LAN Switching Devices



Terminology for IP Multicast Benchmarking



Benchmarking Methodology for Network Interconnect Devices



Benchmarking Terminology for Firewall Performance



Terminology for ATM Benchmarking



Benchmarking Methodology for LAN Switching Devices

Current Meeting Report

Benchmarking Methodology WG Minutes

WG Chair: Kevin Dubray
Minutes reported by Kevin Dubray.

The BMWG met at the 50th IETF in Minneapolis, MN, on Monday, March 19, 2001.

The proposed agenda:

1. Administration
2. <draft-berkowitz-bgpcon-00.txt>
3. <draft-ietf-bmwg-dsmterm-00.txt>

was approved.

1. Administration.

Activity over the last period was summarized as:

The I-Ds: <draft-ietf-bmwg-atm-term-abr-02.txt>, <draft-ietf-bmwg-fr-term-05.txt>
now reflect IESG inputs and should be on their way to RFC editor.
[Chair's note: Currently, there is some confusion over edit versions of these drafts. This should be resolved soon.]

The I-D <draft-ietf-bmwg-atm-method-03.txt> passed a WG last call and should be in the ADs' hands for Info RFC consideration.

The I-Ds: <draft-ietf-bmwg-mcastm-07.txt>, <draft-ietf-bmwg-fib-term-01.txt>
were revised but missed the I-D cutoff. They will be resubmitted after the IETF meeting.

2. <draft-berkowitz-bgpcon-00.txt>

Sue Hares and Howard Berkowitz were present to give insight on their draft about measuring BGP convergence. It was stated that a second draft was produced, but it was unable to be submitted before the Internet Draft cut off time. The presentation and discussion, however, surrounded that second draft. Slides for the presentation can be found at:

The presenters reinforced the notion that the scope of this work was very constrained. Moreover, they further explained that there were many topics of convergence that would be left to future work. Examples of some of these future convergence topics include BGP and IGP interaction, flapping, and policy. While there were some concerns of worm can opening, there appeared to be attendee interest in this work. The presenters were requested to submit the revised draft to the Internet-Draft secretariat, after which a proposal to undertake this new initiative would be made to the working group at large.

2. <draft-ietf-bmwg-dsmterm-00.txt>

The authors of this first draft on terminology for benchmarking network layer traffic control mechanisms were on hand to summarize the status of the draft and to field questions. Essentially, it was stated that this first draft was a work in progress and, as such, it was not near completion. The bulk of the comments offered counsel to align this new work with previous BMWG work, wherever possible.

It was noted that some drafts had expired, specifically the Router benchmarking framework document. The chair ask the group to consider the continuation of this dormant work items.

Lastly, the chair invited the group to consider and offer comments on the individually submitted draft regarding the standardization of metrics in the IETF. The draft ID is:

The goals for next period are:

* Work to bring the firewall, multicast, and FIB I-Ds to a Working Group Last Call.

* Progress the traffic control mechanisms draft and Resource Reservation drafts.

* Assess the proposed new work item on BGP convergence benchmarking.


None received.