2.4.1 Benchmarking Methodology (bmwg)

NOTE: This charter is a snapshot of the 41st IETF Meeting in Los Angeles, California. It may now be out-of-date. Last Modified: 19-Feb-98


Kevin Dubray <kdubray@baynetworks.com>

Operations and Management Area Director(s):

John Curran <jcurran@bbn.com>
Michael O'Dell <mo@uu.net>

Operations and Management Area Advisor:

John Curran <jcurran@bbn.com>

Mailing Lists:

General Discussion:bmwg@baynetworks.com
To Subscribe: bmwg-request@baynetworks.com
Archive: ftp://ndtl.harvard.edu/pub/bmwg/mailing.list

Description of Working Group:

The major goal of the Benchmarking Methodology Working Group is to make a series of recommendations concerning the measurement of the performance characteristics of various internetworking technologies; further, these recommendations may focus on the systems or services that are built from these technologies.

Each recommendation will describe the class of equipment, system, or service being addressed; discuss the performance characteristics that are pertinent to that class; clearly identify a set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of benchmarking results.

Because the demands of a class may vary from deployment to deployment, a specific non-goal of the Working Group is to define acceptance criteria or performance requirements.

An ongoing task is to provide a forum for discussion regarding the advancement of measurements designed to provide insight on the operation internetworking technologies.

Goals and Milestones:



Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches.

Aug 97


Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC.

Aug 97


Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately.

Aug 97


Incorporate BMWG input and continue to progress the Cell/Call Terminology Draft. Reissue draft as appropriate.

Jan 98


Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices.

Jan 98


Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft.

Mar 98


Submit initial draft of Benchmarking Methodology for LAN Switches.

Aug 98


Submit Terminology for IP Multicast Benchmarking draft for AD Review.

Dec 98


Submit Terminology for Cell/Call Benchmarking draft for AD review.

Mar 99


Submit Benchmarking Methodology for LAN Switching Devices draft for AD review.


Request For Comments:







Benchmarking Terminology for Network Interconnection Devices



Benchmarking Methodology for Network Interconnect Devices



Benchmarking Terminology for LAN Switching Devices

Current Meeting Report

Minutes of the Benchmarking Methodology (bmwg) Working Group

Reported by Kevin Dubray

Kevin Dubray opened the Benchmarking Methodology Working Group session. Upon regarding the light turnout (about 30) he commented that holding a session in the last meeting slot on the last day before the Plenary might not be such a good thing...

Light turnout notwithstanding, Dubray presented the agenda for the 41st IETF BMWG meeting.


I. Administration
1.1 Agenda Bashing
1.2 RFC 2285
II. Benchmarking Terminology of Firewall Performance
III. Terminology for IP Multicast Benchmarking
IV. Goals for Next Period.

I. Administration

Dubray announced that this would be an abbreviated session since unexpected scheduling changes forced some speakers to amend their plans.

David Newman asked for an item on the agenda to be added to ascertain interest in a new work item. The chair amended the agenda to reflect this change.

The chair announced that the work formerly known as the LAN Switching Benchmarking Terminology draft was approved by the IESG as an Informational Request for Comments. The RFC editor designated this work as RFC 2285. Dubray thanked the editor, Bob Mandeville, and the other people who had contributed to the draft.

II. Benchmarking Terminology of Firewall Performance

David Newman was introduced to lead a discussion on the Firewall Performance Benchmarking draft. David began with a quick overview of the changes to the draft. He describe the two major changes revolving around the notion of:

A. Connection
B. Forwarding Rate

Newman then engaged in a detailed discussion of the changes to this latest version of the draft. He noted that the term "connection" was changed pursuant to the comments at the Washington DC IETF. He noted that the term "session" was modified as well. David noted that given the relationship between the two terms, it might be a good idea to combine the two at some point.

He went on to say that the concept of "external" was modified to reflect the idea of an "unprotected" network segment. David said the draft also offered a definition for the term, "Firewall." David conveyed that the definition might not be the most rigorous or encompassing, but it should characterize the behavior of most common firewall devices. David said he based the definition on historical references on firewalls.

In presenting the term "forwarding rate," Newman articulated that he believed it was significantly different than the term in RFC 2285. A comment was offered that, at first glance, the only seemingly different notions were that of the measurement units "bits" versus "frames" per second and that of the more specific term "firewall" in lieu of "DUT/SUT". The follow-up question then asked how different were the notions of "Forwarding Rate" in this draft over RFC 2285? Newman thought that they were substantially different. A trailing question remarked that the two terms were ambiguous outside the context of the separate discussion.

Newman acknowledged that the overlap was troubling. He asked for suggestions. One suggestion offered changing the name to something more firewall specific - perhaps a "Firewall Forwarding Rate" metric. Another suggestion was made to align the metric's title closer to the primary difference, "bit forwarding." This was met with general approval excepting a caveat that the term not necessarily be restricted to just firewalls. David asked for clarification of what was the consensus, firewall forwarding rate or bit forwarding rate. The generic bit-forwarding term was reaffirmed.

Upon visiting the term "Session," a discussion ensued regarding the differentiation between the terms session and connection. Dubray commented that earlier in the session, Newman referred to Session as a State - "data flowing" - though the term "state" was not expressly used in the draft. Dubray thought it might be useful to explicitly define session as a state. For example, "A state in which higher level transactions can occur or is occurring between associated peers."

It was further noted that Newman verbally referred to a connection as a state, too. The draft, however, refers to connection not as a state, but as a "path." A question was posed as to whether the draft, in defining connection and session, was attempting to define a hierarchy between a data path and data transfer state. The example was offered that many sessions could be offered over a single connection. David reiterated his view that a "connection," as defined, was as state - communicating a hierarchy was not a goal. David expressed that he viewed connection as a table entry in a device.

[At the meeting's conclusion, Kim Martin offered that a single, unifying concept - circuit, could be perhaps used. It's historical use is consistent to the tone of the discussion. This was noted by Mr. Newman as a good idea.]

David then moved the discussion to the modifications surrounding stateful inspection versus stateful packet filtering. He noted the wording was changed, per the DC meeting, to be less vendor related.

As the discussion moved on to security considerations, a question was offered regarding the merits of assessing firewall functionality. David strongly stated that is was not the draft's intention to provide device validation or other verification-style tests. A voice was heard to say that an understanding could not be gathered on whether performance may compromise functionality.

A general question was poised to Mr. Newman. The query went along the lines of what level did Mr. Newman perceived the end to end testing? Layer 3? Layer 4? Firewall functionality vs. Forwarding path? David communicated, "all the above."

In closing the discussion, final questions were offered Mr. Newman. In his opening remarks, Mr. Newman referred to two major metrics of the draft: a) forwarding rate, and b) connection. Forwarding rate was straightforward, but how does one make a metric of a data path or state? David said that you could measure the number of concurrent connections. The person posing the question articulated that number of connections seems more like a measurement unit than a metric. Using connection as a metric may be ambiguous. David acknowledged the comment. David thanked the group for its input.

III. Terminology for IP Multicast Benchmarking

Kevin Dubray began the discussion of <draft-ietf-bmwg-mcast-03.txt> with a quick overview of the draft's initiatives and areas of concentration. He noted that the review at this meeting marked the coverage of all the metrics presented by the Multicast Terminology draft.

He then gave a high level summary of changes to the current draft. Two areas of change Dubray focused on was a) the notion of "interaction," and b) the fact that "Fairness" was removed from the draft. In regards to the latter, Dubray articulated that Fairness was removed from the draft in a direct response to queries regarding QoS as being a possible area of future BMWG work.

The section on Interaction was added as a direct response to input from the Washington, D.C. BMWG meeting and input from the BMWG mailing list.

Dubray presented the metric Encapsulation Throughput. Dubray indicated that Decapsulation Throughput and Re-encapsulation throughput were defined similarly. When a question was posed as to why three separate metrics were needed, Kevin pointed out that these individual metrics were presented as one metric, Translational Throughput in the last draft. Dubray said it was intended that the methodology be the defining factor. Dubray indicated that the working group thought it would be better to define three discrete metrics. The members of this session reaffirmed the previous wisdom.

With old business behind, Dubray ensued in a discussion of the last sections of the draft: Overhead, Capacity, and Interaction.

Dubray indicated that feedback from the mailing list indicated three thoughts, not necessarily all predominant:

Dubray presented the wording to relieve the condition cited in A. The group offered no dissenting views. Dubray communicated his thoughts that items B and D were better suited for the corresponding methodology document. The group agreed. He thought that the associated "burdened response" metric to be presented later would raise the interest index cited in condition C.

On Group Join Delay (3.4.1), there was a question as to what constitutes the successful transaction. Kevin thought it was a good question - but, again, thought it was a question better answered in the methodology document. The group agreed.

The group further concurred that the additions of Interaction (3.6) and Burdened Response (3.6.1) were welcome to the draft.

On the topic of Forwarding-Burdened Multicast Latency, there was a question regarding the nature of the metric. Was the goal to parameterized the latency metric? Dubray said yes, especially with respect to load and associated forwarding rate. An additional question queried as to what should be gleaned from the metric. Dubray indicated the intent was not to limit the statistical information obtained from the measurement. So, from the set of n latencies, the appropriate statistics could be collected.

There was little discussion on the remaining part of the presentation.

With that, Dubray announced that the major parts of the draft had been presented and discussed. He asked if there were any issues. A question was raised on the Multicast Group Capacity metric. While the metric tried to identify the maximum number of multicast groups a device can support, it did so with the qualification that traffic be correctly forwarded. Specifically, what level/rate of traffic needed to be processed to indicate goodness.

Dubray indicated that he didn't know. Moreover, he didn't think the actual rate was important. He thought it was more important to ascertain the correct delivery. Regardless, he thought it was an issue best addressed by the corresponding methodology document.

Dubray asked that if the changes suggested were made, would the document be ready for final review. There were no dissenting reviews.

Dubray then invited David Newman to present the new work item.

David Newman said that he wished to make a proposal for Bob Mandeville, who was unable to make the meeting. David informed the group that Bob was hoping to do some work in the QoS area using latency as vehicle. David asked if there was interest. The group indicated that there was. Someone suggested that the work would be more useful if it were expanded beyond latency. Others agreed. The chair asked that Mandeville submit a workplan for WG consideration.

Interest was again indicated in the area of the Cell/Call benchmarking draft. Dubray indicated that Raj Jain had hoped to attend, but he was called away at last minute. Dubray indicated that he anticipated a rise in activity in this direction. In the interim, he asked that folks start to open the discussion on the BMWG mailing list.

With no new business, Dubray reviewed the next period's goals:


None Received

Attendees List

go to list