NOTE: This charter is a snapshot of the 47th IETF Meeting in Adelaide, Australia. It may now be out-of-date. Last Modified: 03-Feb-00
Chair(s):
Kevin Dubray <kdubray@ironbridgenetworks.com>
Operations and Management Area Director(s):
Randy Bush <randy@psg.com>
Bert Wijnen <bwijnen@lucent.com>
Operations and Management Area Advisor:
Randy Bush <randy@psg.com>
Mailing Lists:
General Discussion:bmwg@ironbridgenetworks.com
To Subscribe: bmwg-request@ironbridgenetworks.com
Archive: http://www.alvestrand.no/archives/bmwg/
Description of Working Group:
The major goal of the Benchmarking Methodology Working Group is to make a series of recommendations concerning the measurement of the performance characteristics of various internetworking technologies; further, these recommendations may focus on the systems or services that are built from these technologies.
Each recommendation will describe the class of equipment, system, or service being addressed; discuss the performance characteristics that are pertinent to that class; clearly identify a set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of benchmarking results.
Because the demands of a class may vary from deployment to deployment, a specific non-goal of the Working Group is to define acceptance criteria or performance requirements.
An ongoing task is to provide a forum for discussion regarding the advancement of measurements designed to provide insight on the operation internetworking technologies.
Goals and Milestones:
Done |
|
Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches. |
Done |
|
Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC. |
Done |
|
Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately. |
Done |
|
Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft. |
Done |
|
Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices. |
Done |
|
Submit initial draft of Benchmarking Methodology for LAN Switches. |
Done |
|
Submit Terminology for IP Multicast Benchmarking draft for AD Review. |
Done |
|
Submit Benchmarking Terminology for Firewall Performance for AD review |
Dec 99 |
|
Progress ATM benchmarking terminology draft to AD review. |
Dec 99 |
|
Submit Benchmarking Methodology for LAN Switching Devices draft for AD review. |
Jan 00 |
|
Submit first draft of Firewall Benchmarking Methodology. |
Mar 00 |
|
Progress Frame Relay benchmarking terminology draft to AD review. |
Internet-Drafts:
· Benchmarking Methodology for LAN Switching Devices
· Terminology for Frame Relay Benchmarking
· Methodology for IP Multicast Benchmarking
Request For Comments:
RFC |
Status |
Title |
RFC1242 |
Benchmarking Terminology for Network Interconnection Devices | |
RFC2285 |
Benchmarking Terminology for LAN Switching Devices | |
RFC2432 |
Terminology for IP Multicast Benchmarking | |
RFC2544 |
Benchmarking Methodology for Network Interconnect Devices | |
RFC2647 |
Benchmarking Terminology for Firewall Performance | |
RFC2761 |
Terminology for ATM Benchmarking | |
Benchmarking Methodology WG Minutes
WG Chair: Kevin Dubray
Minutes reported by Cynthia Martin.
1. Administrative Overview.
Kevin Dubray provided an overview of the meeting agenda. No questions or comments were given from the audience.
2. Overview of ID "ATM ABR Benchmarking Terminology".
Jeff Dunn provided an overview of the "ATM ABR Benchmarking Terminology" ID including updates and changes to the ID from previous versions. There is interest in the group to create an "ATM ABR Benchmarking Methodology" ID, but no proposed dates or timelines were discussed. Jeff requested that anyone interested in creating the methodology ID should contact Jeff at Jeffrey.Dunn@worldnet.att.net or via the BMWG mailing list. We need people with operational experience to provide input. The "ATM ABR Benchmarking Terminology" ID will go into last call after this BMWG meeting if no comments or questions are raised.
3. Overview of ID "Frame Relay Benchmarking Terminology".
Jeff Dunn provided an overview of the "Frame Relay Benchmarking Terminology" ID including updates and changes to the ID from previous versions. The comment was made to add additional PDH (DS-3) and ADSL information in this ID or to create a new ID. This issue will be put on the list so everyone may comment on what they feel is the appropriate solution. The authors believe that a separate ID should be created for ADSL. The "Frame Relay Benchmarking Terminology" ID will go into last call after this BMWG meeting if no comments or questions are raised.
4. Overview of ID "ATM Benchmarking Methodology".
Jeff Dunn provided an overview of the "ATM Benchmarking Methodology " ID including updates and changes to the ID from previous versions. Many comments and questions were raised including:
a) Should the required histograms be normalized or absolute?
After a discussion, it was decided that both should be available to the user and that a uniform format is more important. The ID will reflect this decision.
b) How do we test cell rate margin?
The Cell Rate Margin is defined the terminology ID, but we have not come up with a good test scenario for this. The authors would like to remove this test from the ID.
c) The filter definitions are still not defined. Should we include them in the ID? Are the filters for ATM or IP?
Does it effect throughput? These questions will be put on the list so everyone may comment. We need an answer to these for the final version of the ID. The authors would like to remove this from the ID.
The "ATM Benchmarking Methodology" ID will go under one more revision before the next IETF and then into last call.
5. Overview of IP QOS ID proposal
Jerry Perser provided an overview of the "IP QOS" ID. Many comments and questions were raised including:
a) Many people are interested in the ID, but who is willing to work and comment on it?
b) IP QOS is not defined or understood, how will this ID be developed?
c) Should we address DIFFSERV? How does this interact with IP QOS? DIFF SERV does not define specific metrics or yardsticks for service agreements. How will this effect the development of the ID?
d) Should we address RSVP? How does this interact with IP QOS?
e) Should we address TOS? How does this interact with IP QOS?
f) Should this be a configuration effort, referring to other work and re-interpreting existing metrics?
g) Should we also address IP over ATM QOS and IP over Frame Relay QOS to narrow the scope?
h) Is this tied to certain mechanisms and algorithms?
i) Is this ingress traffic policing or something else on the egress? Is this traffic policing conformance?
The attendees agreed that the goals of the proposal need to be reworked and articulated to the BMWG mailing list.
6. Overview of Routing Metrics proposals
a) Guy Trotter provided an overview of the proposed "Forwarding Performance based on Forwarding Table Size" ID. A few questions were raised including:
i. Will the ID address forwarding table or route table?
The author clarified that the ID will address forwarding table only, the nomenclature needs to be consistent.
ii. Will the ID address issues like how soon after the routing distribution is complete will traffic forwarding occur?
The ID needs to address this.
iii. Will the ID address the effects of how the routing entries are placed into the table and if it effects traffic forwarding? What are the boundaries on the ID? The ID needs to address this.
b) Jeff Dunn provided an overview of the proposed "Framework for IPv4 Routing Benchmarking" ID and "IPv4 Routing Metrics" ID. A few questions were raised including:
i. Will the ID address MPLS or QOS?
The author clarified that the ID will not address QOS that is for the QOS ID discussed earlier. The ID will not address MPLS at this time. This will be put on the list so everyone may comment. The authors believe that MPLS should be discussed in another ID, similar to the ATM and Frame Relay ID's.
ii. What is the scope of the ID?
The ID needs to be very clear. With more people working in the ID, the authors believe that a clearly defined, well-scoped ID can be created.
iii. Does this proposed ID overlap with the proposed "Forwarding Performance based on Forwarding Table Size" ID?
Yes. The proposed "Forwarding Performance based on Forwarding Table Size" is dependent on the proposed "Framework for IPv4 Routing Benchmarking", but they can be developed in parallel. The framework is necessary to develop all other router based benchmarking ID's.
c) Although a consensus was not taken, the group agreed that both proposals could be developed at the same time. Since the BMWG meeting, the authors of both ID's decided to work together on the "Framework for IPv4 Routing Benchmarking" ID. A first draft of this ID should be available in 1-2 months from the BMWG meeting date. A first draft of the proposed "Forwarding Performance based on Forwarding Table Size" ID was not discussed, but after the meeting the author did indicate that a first draft would be available within a few weeks from the BMWG meeting date.
None received.