2.4.2 Benchmarking Methodology (bmwg)

NOTE: This charter is a snapshot of the 43rd IETF Meeting in Orlando, Florida. It may now be out-of-date. Last Modified: 25-Nov-98


Kevin Dubray <kdubray@ironbridgenetworks.com>

Operations and Management Area Director(s):

Harald Alvestrand <Harald.Alvestrand@maxware.no>
Bert Wijnen <wijnen@vnet.ibm.com>

Operations and Management Area Advisor:

Harald Alvestrand <Harald.Alvestrand@maxware.no>

Mailing Lists:

General Discussion:bmwg@ironbridgenetworks.com
To Subscribe: bmwg-request@ironbridgenetworks.com
Archive: http://www.alvestrand.no/archives/bmwg/

Description of Working Group:

The major goal of the Benchmarking Methodology Working Group is to make a series of recommendations concerning the measurement of the performance characteristics of various internetworking technologies; further, these recommendations may focus on the systems or services that are built from these technologies.

Each recommendation will describe the class of equipment, system, or service being addressed; discuss the performance characteristics that are pertinent to that class; clearly identify a set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of benchmarking results.

Because the demands of a class may vary from deployment to deployment, a specific non-goal of the Working Group is to define acceptance criteria or performance requirements.

An ongoing task is to provide a forum for discussion regarding the advancement of measurements designed to provide insight on the operation internetworking technologies.

Goals and Milestones:



Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches.



Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC.



Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately.



Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft.



Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices.



Submit initial draft of Benchmarking Methodology for LAN Switches.



Submit Terminology for IP Multicast Benchmarking draft for AD Review.

Sep 98


Incorporate BMWG input and continue to progress the Cell/Call Terminology Draft. Reissue draft as appropriate.

Sep 98


Submit first draft of Latency Benchmarking Terminology

Dec 98


Submit Benchmarking Terminology for Firewall Performance for AD review

Mar 99


Submit Terminology for Cell/Call Benchmarking draft for AD review.

Mar 99


Submit Benchmarking Methodology for LAN Switching Devices draft for AD review.

Jul 99


Submit Latency Benchmarking Terminology draft for AD review


Request For Comments:







Benchmarking Terminology for Network Interconnection Devices



Benchmarking Methodology for Network Interconnect Devices



Benchmarking Terminology for LAN Switching Devices



Terminology for IP Multicast Benchmarking

Current Meeting Report

Benchmarking Methodology WG Minutes

WG Chair: Kevin Dubray

Minutes reported by Kevin Dubray.

The BMWG met at the 43rd IETF on Monday, December 7th, 1998. The group consisted of over 40 people.

The chair presented the agenda:

1. Agenda/Administration

The agenda was approved as presented. Dubray announced that the multicast benchmarking terminology draft was announced as RFC 2432. Unfortunately, the LAN switch benchmarking methodology draft did not progress enough to merit discussion at this time. Additionally, the editor of the latency terminology draft believed that the scope of the work merited investigation before proceeding further.

With regards to the multicast terminology draft, Dubray announced that a volunteer to help edit the draft was still needed. Interested parties should apply.

2. Benchmarking Terminology for Firewall Performance.

David Newman was then introduced to lead a discussion on the Firewall Draft. David gave a quick overview of the modifications to the draft. Notable changes include:

Security Association
Session Establishment
Session Maintenance
Session Teardown
Unit of Transfer
Enhancement of connection-related terms.

David also communicated that some other minor cleanup had also ocurred. In discussing some of the changes, David announced that he added the notion of security association to address an issue brought up in earlier discussion: one does not want to undertake a forwarding rate type of measurement until "all hands have been shaken."

On the term "session," David was keen to point out that session is not a measurement itself; rather it was a list of things that needs to happen before measurement can happen. Likewise, the need to finish forwarding rate measurements should occur before session teardown.

In exploring the term "Unit of Transfer," Newman stated that he felt the alternative term "transaction" was too generic. He also concluded that other definitions were too narrow. The suggested term "Unit of Work" was attractive, but David thought it didn't reflect the movement of data. In the UOT discussion, David thought it was important to communicate that while UOT is characteristically broad, users must keep the comparisons narrow to be useful.

David closed the summary, saying that he believe the draft was ready for WG Last Call.

A question was raised as to the communicated duality of term "session," - is it data or a metric? David articulated that he would attempt to clear up the ambiguity.

Presuming the inclusion of discussed issues in a subsequent draft, the chair surveyed the group as to whether a WG Last Call should be issued on the draft. There was consensus to issue the Last Call.

4. Presentation: Latency Characterization in LAN switching devices.

The chair announced that a reordering of the agenda was going to take place. John Dawson was introduced to give his presentation on Latency. (see slides in the proceeding or the Powerpoint presentation on the BMWG archive).

Some questions were posed after the presentation. One person asked whether it was possible for latency measurements to yield a negative number. Dawson indicated that it was, indeed, possible - especially in devices with cut-through architectures. He added that it was his experience that these architecture are not predominant. He cited that handling mixed media speeds made cut-through more problematic than store-and-forward architectures.

Another person asked whether the latency proposals operated at Layer 2 or Layer 3. John indicated the focus was on a packet from one interface to the next.

There was discussion about the notion that 100% line rate is susceptible in regards to clock variances. Someone queried as to whether 100% is a practical case, anyways. John didn't believe so as many protocols require acknowledgement in practice.

Another question was posed as to whether clock synchronization was an issue. John thought is was primarily an issue when synchronizing the DUT's clock with the test gear's clock.

A question asked whether the drift of multiple streams cancel the net effect of the drift. Dawson said no.

Dawson made the recommendation that a review of latency related items in existing RFCs may make make the metrics more meaningful. More pointedly, Dawson stated that he believes that current working of the metrics or their related methodology lends to novel workarounds that demonstrate something other than true latency.

Dawson said he would forward the presentation and related paper to the chair for posting on the BMWG list.

3. Terminology for Cell/Call Benchmarking

With the editors of the draft not present, Dubray proceeded to attempt a review of the current draft.

Many of the comments reflected that the metrics currently presented seemed too link-centric and did not necessarily offer insight into characterizing devices that combined network and link based forwarding technology such as edge devices.

Another comment suggested that the measuring buffering ability and impact may be useful.

With that Dubray closed the working group session and stated the WG goals for the next period:

** Minutes after the WG adjournment, the Cell/Call terminology draft's editors arrived having experienced flight delays. An auxillary meeting was announced and was held on Tuesday, Dec 8.

The minutes of that meeting follow:

Terminology for Cell/Call Benchmarking

Jeff Dunn and Cynthia Martin gave a quick overview of the presentation to meeting attendees. The presentation is to be posted to the list and in the BMWG minutes.

Following the presentation, some issues were discussed.

It was asked, "What is the goal of the draft given that other organizations (ITU-T, ATMF, etc.) are operating in this space. The editors responded that they were leveraging existing work, because the this work helps differentiate the parameters that may affect the IP context. For example, CDV is a factor that will affect TCP on the user layer.

The authors communicated taking a "layered" approach, e.g., how does the service provider affect the service user(s). For example, if you have an error at the ATM layer, it has an implication at the AAL level, which has implications on the IP layer and TCP layer. The document needs to point out that there is correlation to higher layer(s).

Authors also stated their belief that restricting the document's scope to ATM may allow for quicker prototypical work.

The authors put forward the notion that they would like to advance the methodological work - if nothing else, they would like to start discussion on how to architect the methodolgy. The goal would be to produce a usable framework for testing, independent of implementation.

When queried as to the changes for the next version of the terminology draft, the editors stated that they would like to develop the discussion section and enhance the measurements offered.


Call/Cell Benchmarking Terminology ID
Latency Under Load