NOTE: This charter is a snapshot of the 46th IETF Meeting in Washington, DC. It may now be out-of-date. Last Modified: 18-Oct-99
Chair(s):
Kevin Dubray <kdubray@ironbridgenetworks.com>
Operations and Management Area Director(s):
Randy Bush <randy@psg.com>
Bert Wijnen <wijnen@vnet.ibm.com>
Operations and Management Area Advisor:
Randy Bush <randy@psg.com>
Mailing Lists:
General Discussion:bmwg@ironbridgenetworks.com
To Subscribe: bmwg-request@ironbridgenetworks.com
Archive: http://www.alvestrand.no/archives/bmwg/
Description of Working Group:
The major goal of the Benchmarking Methodology Working Group is to make a series of recommendations concerning the measurement of the performance characteristics of various internetworking technologies; further, these recommendations may focus on the systems or services that are built from these technologies.
Each recommendation will describe the class of equipment, system, or service being addressed; discuss the performance characteristics that are pertinent to that class; clearly identify a set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of benchmarking results.
Because the demands of a class may vary from deployment to deployment, a specific non-goal of the Working Group is to define acceptance criteria or performance requirements.
An ongoing task is to provide a forum for discussion regarding the advancement of measurements designed to provide insight on the operation internetworking technologies.
Goals and Milestones:
Done |
|
Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches. |
Done |
|
Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC. |
Done |
|
Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately. |
Done |
|
Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft. |
Done |
|
Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices. |
Done |
|
Submit initial draft of Benchmarking Methodology for LAN Switches. |
Done |
|
Submit Terminology for IP Multicast Benchmarking draft for AD Review. |
Done |
|
Submit Benchmarking Terminology for Firewall Performance for AD review |
Dec 99 |
|
Progress ATM benchmarking terminology draft to AD review. |
Dec 99 |
|
Submit Benchmarking Methodology for LAN Switching Devices draft for AD review. |
Jan 00 |
|
Submit first draft of Firewall Benchmarking Methodology. |
Mar 00 |
|
Progress Frame Relay benchmarking terminology draft to AD review. |
Internet-Drafts:
· Terminology for Cell/Call Benchmarking
· Benchmarking Methodology for LAN Switching Devices
· Terminology for Frame Relay Benchmarking
· Methodology for IP Multicast Benchmarking
Request For Comments:
RFC |
Status |
Title |
RFC1242 |
Benchmarking Terminology for Network Interconnection Devices | |
RFC2285 |
Benchmarking Terminology for LAN Switching Devices | |
RFC2432 |
Terminology for IP Multicast Benchmarking | |
RFC2544 |
Benchmarking Methodology for Network Interconnect Devices | |
RFC2647 |
Benchmarking Terminology for Firewall Performance | |
Benchmarking Methodology WG Minutes
WG Chair: Kevin Dubray
Minutes reported by Kevin Dubray.
The BMWG met at the 46th IETF in Washington, D.C., on Wednesday, November 10, 1999. There were over 20 attendees.
The proposed agenda follows:
1. Administrivia
1.1 Status of <draft-ietf-bwmg-atm-term-00.txt>.
2. Methodology for ATM Benchmarking. (Jeff Dunn/Cynthia Martin)
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-atm-method-00.txt
3. Terminology for Frame Relay Benchmarking. (Jeff Dunn/Cynthia Martin)
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-fr-01.txt
4. Benchmarking Methodology for LAN Switching Devices (Jerry Perser)
http://www.netcomsystems.com/perser/draft-ietf-bmwg-mswitch-01a.txt
The agenda was left unmodified.
1. Administrivia.
Draft-ietf-bwmg-atm-term-00.txt was announced to have gone through a WG Last Call and was in the hands of the IESG. [It has since been forwarded from the IESG to the RFC Editor.]
Jeff Dunn was then introduced to lead a discussion regarding the first two agenda items.
2. Terminology for Frame Relay Benchmarking.
Jeff began by stating that an ATM ABR Terminology draft was is in progress and a first draft would be submitted shortly after the meeting.
Jeff stated that he would start the presentations slightly out of order from the agenda by tackling the Frame Relay (FR) terminology draft first.
Jeff's presentation followed his slides (submitted with these minutes). The presentation communicated the motivation, focus, and anticipated milestones of this effort.
Jeff reiterated that the goal of the work was to identify metrics that would characterize IP or higher layer performance over a Frame Relay infrastructure. Said another way, produce metrics that would help characterize the effects of impairments in the lower layers on the upper layers' performance.
In general, the authors feel that the draft needs focus and clarification - this is where they turn to the WG for help.
A question came up on congestion. Was the focus on DUT congestion or congestion external to the DUT? Jeff indicated this was exactly the type of insight that the authors needed from the WG.
Cynthia Martin indicated another area of focus is whether to characterize single Virtual Circuit (VC) performance versus multiple VC performance.
There was some discussion about the notion of "spectral" coverage - the implication was the testing was "real world" versus clinical. It was communicated that the IPPM handled more "real world" network characterizations, albeit from a more generic perspective. Jeff indicated that he believe the notion of spectral was associated more to a DLCI than time.
Discard Eligibility (DE) was a Frame Relay term that might be expanded or more closely bound to the "upper layers". Additionally, it may be necessary to identify new terms to help draw out the appropriate associations. David Newman offered to make some proposals and send to the WG mailing list.
Newman also thinks that the draft will need to present the necessary context, where appropriate. The notion of VCs and their associated performance may be differ if taken from a vendor's perspective than from a provider's perspective.
It was thought that DS-3 channels should be given the same level of consideration in the draft as T1/E1. In general, Jeff indicated that he ignored things plesiochronous. He stated that they might have to reconsider this - especially with regards to alarm conditions.
Jeff commented that it was the authors intent to get the drafts solidified by its third revision. He reiterated that the authors could only do that with WG participation.
With that Jeff opened the floor for general comments.
A suggestion was offered that the draft would make more sense if the terminology were more logically group as opposed to being presented alphabetically. It was also thought that a Table of Contents would be very useful.
It was noted that the BECN definition needed some work. It was also stated that the FCS section might give consideration to 32 bit checksums, not just 16.
It was also thought that while 2 byte headers are most common, shouldn't mention be made of 3 and 4 byte headers? Cynthia said due to methodological complications, they might like to strictly focus on the common application, documenting that other header sizes are outside the scope of this document.
It was asked whether the draft should address LMI version issues as they may affect link formation. Jeff asked the whether the draft should address LMI at all? If so, are we looking to come up with related benchmarks? Right now, he considers it a more methodological issue which would be outside the scope of the draft.
Jeff thanked the group for input on this draft and asked that the discussion spill over to the mailing list. He then turned what time he had remaining to the ATM Benchmarking Methodology document.
3. Methodology for ATM Benchmarking.
In a similar manner as the FR presentation, Jeff presentated the motivation, focus, and anticipated milestones of the ATM Benchmarking Methodology work.
The bulk of the discussion departed from addressing the ATM methodology document and centered around the futures slide. Brought up were such things as Packet Over Sonet and Routing Protocol Benchmarking.
The chair asked folks to move the discussion of possible BMWG work to the mailing list.
4. Benchmarking Methodology for LAN Switching Devices
Jerry Perser was next introduced to discuss the LAN Switch Methodology draft.
Jerry's presentation addressed changes made to the draft, its improved organization, as well as some outstanding discussion points.
Much of the discussion, however, revolved around congestion related issues. Was Head of Line Blocking (HOL) a condition or a test? Perser stated that it was a test.
There was confusion over the HOL and the Backpressure indicators as both employed the same methodology. Additionally, it was offered that it wasn't valid to use the same configuration for both indicators. At the end of the discussion, some recommendations were offered:
- Address HOL and Backpressure under the common heading of "Congestion Control."
- Offer insight as to why HOL and backpressure are related methodologically.
There was discussion as to whether it was appropriate to declare a test successfully executed versus passing judgement over a successfully executed test's results. Most thought it was a good thing to identify guidelines that determine whether the results were properly collected.
A question was made as to the appropriateness of psuedo-code in a methodological statement. Some thought it wasn't OK; others thought it was fine. A suggestion was to describe the methodology using prose but offer the pseudo-code in an Appendix.
Jerry indicated that he wasn't happy with the wording of Intended Load as a percentage. He offered a solution. (See slides.)
He'll present the solution to the list.
It was stated that we wanted to move this draft towards last call, so folks should take issues directly to WG mailing list as soon as possible.
In other work, Dubray noted that the Multicast Methodology was revised since the last meeting. He noted that one of the other editors (R. Daniels) did not make his additions to the draft in time for discussion at this meeting. That notwithstanding, Dubray asked if anyone had outstanding issues with the draft. Some comments offered were:
- The Mixed Class Throughput section indicates that the DUT must be set up in one of two ways. It wasn't clear as to the difference between the two approaches. Also, this methodology seems very LAN centric. Could it be made more generic?
- The reference to the 20 nanosecond granularity was questioned. Was it really needed?
- Forwarding Burdened Group Join Delay and Forwarding Burdened Group Latency weren't clear.
- Appendix A offers many examples, but it states no algorithm.
With time running out, folks were requested to mail remaining comments to the WG reflector.
Terminology for Frame Relay Benchmarking; Methodology for ATM Benchmarking
Benchmarking Methodology for LAN Switching Devices