2.4.2 Benchmarking Methodology (bmwg)

NOTE: This charter is a snapshot of the 62nd IETF Meeting in Minneapolis, MN USA. It may now be out-of-date.

Last Modified: 2005-02-21


Kevin Dubray <kdubray@juniper.net>
Al Morton <acmorton@att.com>

Operations and Management Area Director(s):

Bert Wijnen <bwijnen@lucent.com>
David Kessens <david.kessens@nokia.com>

Operations and Management Area Advisor:

David Kessens <david.kessens@nokia.com>

Mailing Lists:

General Discussion: bmwg@ietf.org
To Subscribe: bmwg-request@ietf.org
In Body: subscribe your_email_address
Archive: http://www.ietf.org/mail-archive/web/bmwg/index.html

Description of Working Group:

The major goal of the Benchmarking Methodology Working Group is to make
a series of recommendations concerning the measurement of the
performance characteristics of various internetworking technologies;
further, these recommendations may focus on the systems or services
that are built from these technologies.

Each recommendation will describe the class of equipment, system, or
service being addressed; discuss the performance characteristics that
are pertinent to that class; clearly identify a set of metrics that aid
in the description of those characteristics; specify the methodologies
required to collect said metrics; and lastly, present the requirements
for the common, unambiguous reporting of benchmarking results.

To better distinguish the BMWG from other measurement initiatives in
the IETF, the scope of the BMWG is limited to technology
characterization using simulated stimuli in a laboratory environment.
Said differently, the BMWG does not attempt to produce benchmarks for
live, operational networks. Moreover, the benchmarks produced by this
shall strive to be vendor independent or otherwise have universal
applicability to a given technology class.

Because the demands of a particular technology may vary from
deployment to deployment, a specific non-goal of the Working Group is
to define acceptance criteria or performance requirements.

An ongoing task is to provide a forum for discussion regarding the
advancement of measurements designed to provide insight on the
operation internetworking technologies.

Goals and Milestones:

Done  Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches.
Done  Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC.
Done  Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately.
Done  Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices.
Done  Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft.
Done  Submit initial draft of Benchmarking Methodology for LAN Switches.
Done  Submit Terminology for IP Multicast Benchmarking draft for AD Review.
Done  Submit Benchmarking Terminology for Firewall Performance for AD review
Done  Progress ATM benchmarking terminology draft to AD review.
Done  Submit Benchmarking Methodology for LAN Switching Devices draft for AD review.
Done  Submit first draft of Firewall Benchmarking Methodology.
Done  First Draft of Terminology for FIB related Router Performance Benchmarking.
Done  First Draft of Router Benchmarking Framework
Done  Progress Frame Relay benchmarking terminology draft to AD review.
Done  Methodology for ATM Benchmarking for AD review.
Done  Terminology for ATM ABR Benchmarking for AD review.
Done  Terminology for FIB related Router Performance Benchmarking to AD review.
Done  Firewall Benchmarking Methodology to AD Review
Done  First Draft of Methodology for FIB related Router Performance Benchmarking.
Done  First draft Net Traffic Control Benchmarking Methodology.
Done  Methodology for IP Multicast Benchmarking to AD Review.
Done  Resource Reservation Benchmarking Terminology to AD Review
Done  First I-D on IPsec Device Benchmarking Terminology
Done  EGP Convergence Benchmarking Terminology to AD Review
Done  Resource Reservation Benchmarking Methodology to AD Review
Dec 04  IPsec Device Benchmarking Terminology to AD Review
Apr 05  IGP/Data-Plane Terminology I-D to AD Review
Apr 05  IGP/Data-Plane Methodology and Considerations I-Ds to AD Review
Apr 05  Router Accelerated Test Terminology I-D to AD Review
Apr 05  Net Traffic Control Benchmarking Terminology to AD Review
Apr 05  Methodology for FIB related Router Performance Benchmarking to AD review.
Jul 05  Basic BGP Convergence Benchmarking Methodology to AD Review.
Oct 05  Router Accelerated Test Method. and Considerations I-Ds to AD Review
Oct 05  Hash and Stuffing I-D to AD Review
Dec 05  Net Traffic Control Benchmarking Methodology to AD Review.


  • draft-ietf-bmwg-dsmterm-10.txt
  • draft-ietf-bmwg-benchres-term-05.txt
  • draft-ietf-bmwg-conterm-06.txt
  • draft-ietf-bmwg-fib-meth-03.txt
  • draft-ietf-bmwg-ospfconv-term-10.txt
  • draft-ietf-bmwg-ospfconv-intraarea-10.txt
  • draft-ietf-bmwg-ospfconv-applicability-07.txt
  • draft-ietf-bmwg-ipsec-term-05.txt
  • draft-ietf-bmwg-igp-dataplane-conv-meth-05.txt
  • draft-ietf-bmwg-igp-dataplane-conv-term-05.txt
  • draft-ietf-bmwg-igp-dataplane-conv-app-05.txt
  • draft-ietf-bmwg-acc-bench-term-05.txt
  • draft-ietf-bmwg-acc-bench-meth-02.txt
  • draft-ietf-bmwg-hash-stuffing-02.txt

    Request For Comments:

    RFC1242 I Benchmarking Terminology for Network Interconnection Devices
    RFC1944 I Benchmarking Methodology for Network Interconnect Devices
    RFC2285 I Benchmarking Terminology for LAN Switching Devices
    RFC2432 I Terminology for IP Multicast Benchmarking
    RFC2544 I Benchmarking Methodology for Network Interconnect Devices
    RFC2647 I Benchmarking Terminology for Firewall Performance
    RFC2761 I Terminology for ATM Benchmarking
    RFC2889 I Benchmarking Methodology for LAN Switching Devices
    RFC3116 I Methodology for ATM Benchmarking
    RFC3133 I Terminology for Frame Relay Benchmarking
    RFC3134 I Terminology for ATM ABR Benchmarking
    RFC3222 I Terminology for Forwarding Information Base (FIB) based Router Performance
    RFC3511 I Benchmarking Methodology for Firewall Performance
    RFC3918 I Methodology for IP Multicast Benchmarking

    Current Meeting Report

    Benchmarking Methodology WG (bmwg) TUESDAY, March 8, 2005, 0900-1130 ======================================= CHAIRS: Kevin Dubray <kdubray (at) juniper.net> Al Morton <acmorton (at) att.com> About 30 people attended the session in person, with several more joining the jabber group and others simply listening to the audio stream. The Chairs supplied a URL for all slides provided in advance of the meeting, to foster remote participation. The following minutes were prepared by Kevin Dubray and Al Morton, based on the detailed account supplied by Jeff Dunn as official note-taker, and the audio archive itself. 0. Agenda bashing Tom Alexander proposed to share status slides on the progress of IEEE 802.11 T on benchmarking wireless subnets. His offer was accepted (time permitting) and we deferred the Chairs' WLAN Status slide to the end of the meeting to introduce the topic. 1. Working Group Status (Chairs) Here's the state of WG I-D's and activities at IETF 62: AD/IESG Review none WG Last Call <draft-ietf-bmwg-benchres-term-05.txt>, ended 3/14 w/comments <draft-ietf-bmwg-fib-meth-03.txt>, New Editors, revised <draft-ietf-bmwg-igp-dataplane-conv-term-05.txt> Revised on input <draft-ietf-bmwg-igp-dataplane-conv-meth-05.txt> Revised <draft-ietf-bmwg-igp-dataplane-conv-app-05.txt> Revised <draft-ietf-bmwg-dsmterm-10.txt>, Revised, new editor I-Ds <draft-ietf-bmwg-ipsec-term-03.txt> Revised, LC imminent <draft-ietf-bmwg-acc-bench-term-05.txt> Revised, LC imminent <draft-ietf-bmwg-acc-bench-meth-02.txt> Revised <draft-ietf-bmwg-hash-stuffing-02.txt> Revised Expired BMWG I-Ds <draft-ietf-bmwg-bgpbas-01.txt>, Pending term. progress <draft-ietf-bmwg-benchres-method-00.txt> Pending term prog. New Work proposals. LDP Convergence - need I-D revision. Protection benchmarking - I-D pending? RFC Editor Queue -- <draft-ietf-bmwg-conterm-06.txt> <draft-ietf-bmwg-ospfconv-term-10.txt>* <draft-ietf-bmwg-ospfconv-intraarea-10.txt>* <draft-ietf-bmwg-ospfconv-applicability-07.txt>* * required NORMATIVE reference resolution - resolved after meeting See the BMWG mail archive for recent revision status comments: http://www.ietf.org/mail-archive/working-groups/bmwg/current/ With a revised version for this meeting, the IPsec terminology draft is thought complete by its editors, and will be tested in WGLC. The benchres term draft underwent WGLC recently, and the editors have some comments to address. Another WGLC is expected shortly. All editors were encouraged to consult the "tools" prototype web pages to see the results of idnits checking: http://tools.ietf.org/wg/bmwg/ For example, all drafts are required to have an IANA section, and nits must be resolved before AD-level submission. 2. Revised Milestones. (Dubray) We added a new milestone on Hash and Stuffing draft, and one milestone is overdue (IPsec terminology draft to AD review). Many other milestone dates are drawing near. 3. IGP Data plane convergence benchmark I-Ds. (S.Poretsky) http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-term-05.txt http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-meth-05.txt http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-app-05.txt The 05 versions reflect final comments from IETF 61. Scott presented a version history and the summarized the changes in all drafts. He asked to test readiness for another (possibly final) WG Last Call. There were several comments and questions from the floor: (terminology) - the factor of 5 in equation 3 was suggested in earlier comments - there may be some interesting test cases when using different interface media types, but the current draft requires the same types. This point should not impede progress. (methodology) - There is an IESG mandate to handle IPv6 on par with IPv4 - There should be a test case with a smaller number of routes (to vet config) - Multiple ingress interfaces must share the Forwarding Capacity, otherwise congestion will result - Establish the Maximum Offered Load with a test up-front to set the baseline 4. Tech. for Benchmarking Core Router Accelerated Life Testing (S.Poretsky) http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-term-05.txt http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-meth-02.txt The Terminology & Methodology I-Ds were revised on comments. Scott summarized the changes in these drafts. The Terminology was clarified with a figure, and there are six new test cases in the Methodology (where some additional work is planned). He asked if the group was ready to begin the WGLC process on the Terms, and this seems to be plausible. However, there were comments on the Whitebox terms in Appendix I, where a uniform standard for describing memory is needed (only CPU memory was intended). Regarding the Methodology, there were several comments and questions from the floor: - Suggestions for new test cases: Security, Authentication, Encryption, IPv6 Tunnel, Policy Case with filters and ACLs. - With so many test cases, the memo needs to organize them into groups (e.g., all BGP cases). Tests could be grouped by purpose and split into different drafts. It was suggested to start with the generic test cases for simplicity and follow-up with more specific cases in separate memos. - Section 4.6, method with Route Flap Dampening specifies DEFAULT parameters, but this needs clarification, perhaps a single set of values will lead to simpler comparison. RFC 2439 should be referenced. - Each benchmark collected needs a method of measurement specified, beyond the procedures given in the memo. For example, there is no method to collect "Recovered Sessions Lost". The exact methods to measure time intervals and rates are not defined. 5. Terminology for Benchmarking Network-layer Traffic Control Mechanisms (S.Poretsky) http://www.ietf.org/internet-drafts/draft-ietf-bmwg-draft-ietf-bmwg-dsmterm-10.txt After a series of active Last Calls, a list of editorial issues have been addressed in version 10, and the new Editors believed the work complete. The idnits checker has identified some additional work, and if these edits are completed in version 11, this memo will be ready for (possibly the final) WG Last Call. 6. FIB Methodology (J.Dunn) http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-fib-meth-03.txt Jeff described the approach to bringing this memo to consensus, including the addition of ISO 9646 formalism (test groups). It was clarified that Verdicts only apply to the qualifications on a benchmark, and not the benchmark itself (because BMWG does not establish pass/fail criteria). There must be some quantitative criterion, such as "how long does it take?" This draft will reuse many of the Dataplane convergence definitions. Readership is encouraged here, there has been no comment on the list. One of the next steps is to modify RFC 3222, FIB terminology. Kevin asked that proposals for this be taken to the list. Another future activity might be a RIB to FIB convergence draft. This topic spawned considerable interest, but it is seen to be rather difficult to observe externally. Methods could include observation of SNMP and routing updates. Again, Kevin asked that participants post their questions and suggestions for this topic on the list. 7. IPv6 Benchmarking Introduction (C. Martin) Cynthia led a short discussion of IPv6 benchmarking terminology work. A link to her draft was posted to BMWG mailing list, and the draft will be resubmitted as an individual submission shortly after the meeting. Cynthia planned to treat both Dual-Environment and IPv6-only aspects, but a suggestion to split this work into separate drafts was supported by the attendees and chairs. There are many transition mechanisms, and they are expected to evolve with experience, while IPv6-only configs are a stable target. Also, smaller drafts require less time to review, allow easier course corrections, and tighter scopes may lead to quicker consensus, so the split was agreed. Once the draft is posted in the I-D directories, we can have discussions on the list and likely follow-up with a new work proposal. 8. LDP Convergence (T. Eriksson) draft-eriksson-ldp-convergence-term-02.txt Thomas presented the changes to the current version (from October), new terms may be needed when the methodology is more mature. He then presented some equipment configurations for the methodology in order to get feedback. Tests of the Egress LSR will not see labeled traffic when PHP (Penultimate Hop Popping) is used, and this is the norm for deployment today. There could be two test cases, one with PHP enabled and one with labeled traffic. There was a suggestion to measure LSP flapping, but the authors felt this was better included under the Accelerated Stress testing work, since a single convergence time benchmark is sought. There was a comment that the considerations identified in draft-white-network-benchmark-01 that should be taken account here. There has been commentary that these considerations are useful. Unfortunately, this individual draft had a wider scope than either BMWG or IPPM charters, and there was no new benchmark defined in the text. 9. Automatic Protection Switching Proposal Al gave a brief status of this item: At IETF-61, we discussed this item and there was reasonable support to take-up this work. But there were comments identified that must be dealt with before we proceeded with a work proposal. Then we went into "suspended animation", and when we see the revisions to the drafts we'll take this up at the proposal stage. Scott Poretsky noted that there have been some comments off-list, so the work is progressing, but off-list commentary is not part of our record unless it's brought-in by re-posting the comments with permission, etc. 10. Status of Unofficial Liaison with 802.11"T", and their progress to date. Following discussions in BMWG at IETF-61 (where there was considerable interest expressed in this work), IEEE 802.11 Task Group T met and decided that all of the work proposed in draft-alexander-wlan-meth-00 was within their scope. This was communicated to the BMWG chairs by the IEEE 802 liaison Bernard Aboba, who offered that the work could still be reviewed in BMWG and that an informal liaison could be established if someone volunteered to perform the role as go-between. Fortunately, Tom Alexander (co-author of the methodology draft with Scott Bradner), volunteered and joined the BMWG session in person. Tom presented his perspective on the status of 802.11 Task Group T (see slides). The output of this group will be a Recommended Practice (SHOULD, not mandatory). The membership is broad, and meetings are frequent, with weekly teleconferences, ad hocs, and 10-16 hours scheduled during plenary weeks. There have been many presentations of the methods that vendors are using/selling today, a wide range of approaches. There has been progress on the ground rules for voting, but the first draft terminology will not be ready for next week's meeting as scheduled. The group will first define terminology and then work methodologies. There are two processes that BMWG participants can follow to review and comment on this topic. There is no draft yet available in the members-only section of the www.802wirelessworld.com site, but when there is, the BMWG chairs can request a password and communicate it to interested individuals who will be expected to provide comments in return. Also, contributions can be viewed now by anyone who registers their e-mail on the wirelessworld site. Dorothy Stanley noted that the TG T Chairman's Agenda slides cover site access. If BMWG input is sought early-on, then Tom Alexander will notify BMWG list of the contribution and the topic or issue. For example, Task Group T is beginning their work by defining terminology and has begun to borrow terms from BMWG RFCs, so clarifications on terminology may be sought from BMWG practicioners. Noting that permission is needed to use IETF RFCs in this way, Scott Bradner gave his permission (as editor) to use the text of RFC 2544. There was a question about OAM tool definitions, and Tom answered that this work is in progress under 802.11k. Also, Dorothy Stanley agreed to give a walk-through of the wireless world site immediately following the meeting. 11. Wrap-up (Chairs) Quite a few drafts are headed for WGLC, and folks should expect that we will use the Active Review Templates developed last year, and that *volunteers* will be tapped to provide detailed review. The chairs will investigate an issue-tracker, or bug-tracker for our drafts. They will also consider a charter re-fresh with the ADs. We might be able define a "benchmark" more crisply. There have been some other forms of testing that seem to be creeping into the group, and we need to clarify BMWG's role in this area.


    IGP Data Plane Convergence Benchmarking
    Accelerated Stress Benchmarking
    Terminology for Network Layer Traffic Control Mechanisms
    Methodology for Forwarding Information Base (FIB) based Router Performance
    Terminology for IPv6 Benchmarking
    LDP Data Plane Convergence Benchmarking
    IEEE 802.11T