2.4.1 Benchmarking Methodology (bmwg)

NOTE: This charter is a snapshot of the 64th IETF Meeting in Vancouver, British Columbia Canada. It may now be out-of-date.

Last Modified: 2005-10-03


Kevin Dubray <kdubray@juniper.net>
Al Morton <acmorton@att.com>

Operations and Management Area Director(s):

Bert Wijnen <bwijnen@lucent.com>
David Kessens <david.kessens@nokia.com>

Operations and Management Area Advisor:

David Kessens <david.kessens@nokia.com>

Mailing Lists:

General Discussion: bmwg@ietf.org
To Subscribe: bmwg-request@ietf.org
In Body: subscribe your_email_address
Archive: http://www.ietf.org/mail-archive/web/bmwg/index.html

Description of Working Group:

The major goal of the Benchmarking Methodology Working Group is to make
a series of recommendations concerning the measurement of the
performance characteristics of various internetworking technologies;
further, these recommendations may focus on the systems or services
that are built from these technologies.

Each recommendation will describe the class of equipment, system, or
service being addressed; discuss the performance characteristics that
are pertinent to that class; clearly identify a set of metrics that aid
in the description of those characteristics; specify the methodologies
required to collect said metrics; and lastly, present the requirements
for the common, unambiguous reporting of benchmarking results.

To better distinguish the BMWG from other measurement initiatives in
the IETF, the scope of the BMWG is limited to technology
characterization using simulated stimuli in a laboratory environment.
Said differently, the BMWG does not attempt to produce benchmarks for
live, operational networks. Moreover, the benchmarks produced by this
shall strive to be vendor independent or otherwise have universal
applicability to a given technology class.

Because the demands of a particular technology may vary from
deployment to deployment, a specific non-goal of the Working Group is
to define acceptance criteria or performance requirements.

An ongoing task is to provide a forum for discussion regarding the
advancement of measurements designed to provide insight on the
operation internetworking technologies.

Goals and Milestones:

Done  Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches.
Done  Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC.
Done  Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately.
Done  Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices.
Done  Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft.
Done  Submit initial draft of Benchmarking Methodology for LAN Switches.
Done  Submit Terminology for IP Multicast Benchmarking draft for AD Review.
Done  Submit Benchmarking Terminology for Firewall Performance for AD review
Done  Progress ATM benchmarking terminology draft to AD review.
Done  Submit Benchmarking Methodology for LAN Switching Devices draft for AD review.
Done  Submit first draft of Firewall Benchmarking Methodology.
Done  First Draft of Terminology for FIB related Router Performance Benchmarking.
Done  First Draft of Router Benchmarking Framework
Done  Progress Frame Relay benchmarking terminology draft to AD review.
Done  Methodology for ATM Benchmarking for AD review.
Done  Terminology for ATM ABR Benchmarking for AD review.
Done  Terminology for FIB related Router Performance Benchmarking to AD review.
Done  Firewall Benchmarking Methodology to AD Review
Done  First Draft of Methodology for FIB related Router Performance Benchmarking.
Done  First draft Net Traffic Control Benchmarking Methodology.
Done  Methodology for IP Multicast Benchmarking to AD Review.
Done  Resource Reservation Benchmarking Terminology to AD Review
Done  First I-D on IPsec Device Benchmarking Terminology
Done  EGP Convergence Benchmarking Terminology to AD Review
Done  Resource Reservation Benchmarking Methodology to AD Review
Dec 2004  IPsec Device Benchmarking Terminology to AD Review
Apr 2005  IGP/Data-Plane Terminology I-D to AD Review
Apr 2005  IGP/Data-Plane Methodology and Considerations I-Ds to AD Review
Apr 2005  Router Accelerated Test Terminology I-D to AD Review
Apr 2005  Net Traffic Control Benchmarking Terminology to AD Review
Apr 2005  Methodology for FIB related Router Performance Benchmarking to AD review.
Jul 2005  Basic BGP Convergence Benchmarking Methodology to AD Review.
Oct 2005  Router Accelerated Test Method. and Considerations I-Ds to AD Review
Oct 2005  Hash and Stuffing I-D to AD Review
Dec 2005  Net Traffic Control Benchmarking Methodology to AD Review.


  • draft-ietf-bmwg-dsmterm-11.txt
  • draft-ietf-bmwg-benchres-term-06.txt
  • draft-ietf-bmwg-ipsec-term-07.txt
  • draft-ietf-bmwg-igp-dataplane-conv-meth-08.txt
  • draft-ietf-bmwg-igp-dataplane-conv-term-08.txt
  • draft-ietf-bmwg-igp-dataplane-conv-app-08.txt
  • draft-ietf-bmwg-acc-bench-term-07.txt
  • draft-ietf-bmwg-acc-bench-meth-04.txt
  • draft-ietf-bmwg-hash-stuffing-04.txt
  • draft-ietf-bmwg-acc-bench-meth-ebgp-00.txt
  • draft-ietf-bmwg-acc-bench-meth-opsec-00.txt
  • draft-ietf-bmwg-ipsec-meth-00.txt

    Request For Comments:

    RFC1242 I Benchmarking Terminology for Network Interconnection Devices
    RFC1944 I Benchmarking Methodology for Network Interconnect Devices
    RFC2285 I Benchmarking Terminology for LAN Switching Devices
    RFC2432 I Terminology for IP Multicast Benchmarking
    RFC2544 I Benchmarking Methodology for Network Interconnect Devices
    RFC2647 I Benchmarking Terminology for Firewall Performance
    RFC2761 I Terminology for ATM Benchmarking
    RFC2889 I Benchmarking Methodology for LAN Switching Devices
    RFC3116 I Methodology for ATM Benchmarking
    RFC3133 I Terminology for Frame Relay Benchmarking
    RFC3134 I Terminology for ATM ABR Benchmarking
    RFC3222 I Terminology for Forwarding Information Base (FIB) based Router Performance
    RFC3511 I Benchmarking Methodology for Firewall Performance
    RFC3918 I Methodology for IP Multicast Benchmarking
    RFC4061 I Benchmarking Basic OSPF Single Router Control Plane Convergence
    RFC4062 I OSPF Benchmarking Terminology and Concepts
    RFC4063 I Considerations When Using Basic OSPF Convergence Benchmarks
    RFC4098 I Terminology for Benchmarking BGP Device Convergence in the Control Plane

    Current Meeting Report

    Benchmarking Methodology WG (bmwg) MONDAY, November 7, 2005 1510-1710 Afternoon Session II (Oak) ======================================= CHAIRS: Kevin Dubray <kdubray@juniper.net> Al Morton <acmorton@att.com> MEETING MINUTES/REPORT: Matt Zekauskas appeared in time to be graciously drafted as minute taker, so his detailed notes, along with Kevin's Jabber Log, http://www.xmpp.org/ietf-logs/bmwg@ietf.xmpp.org/2005-11-07.html were edited into the minutes by the Co-Chairs. About 20 people attended the BMWG session. 0. Agenda bashing (we may need to shuffle a few items) See https://datatracker.ietf.org/public/meeting_materials.cgi?meeting_num=64 We bashed the agenda to move-up items that were somewhat controversial, so that AD David Kessens could attend a conflicting meeting. 1. Milestone Status (Chairs) See Slide 6 in the Chairs slide deck for the list of proposed milestones. Based on presumed misses, the Chairs propose to revise milestones by a year. There has been clear progress made, but progress requires more than work by the editors. Some of the 350-plus members of the WG-list must help by providing reviews. Our revised milestones must represent real commitments to complete the work. The meeting produced more changes and revisions, and the new dates will posted for discussion. The chairs noted that the FIB methodology draft has expired again due to lack of interest/support, and they plan to remove this milestone altogether, following notice of intent on the bmwg-list. 2. Revised Charter discussion (Chairs) See slides 7 through 9 in the chairs' deck. The main discussion covered a point raised at the Paris IETF-63 regarding the scope of BMWG coverage. The chairs proposed to focus the WG effort "primarily" on internetworking technologies developed by the IETF. This clearly indicates our emphasis, but also leaves some wiggle room so that we can consider exceptions if they arise, because it is difficult to assess any technology in isolation (IPsec is a classic example). The charter discussion also covered points that are considered beyond the scope of BMWG. Compliance testing is definitely out of scope, BMWG will assume compliance when a manufacturer claims it. The consensus was that classic reliability benchmarks are out-of scope, but also that the sub-IP restoration and current stress testing are reasonably within the charter. There should also be a de-emphasis on niche technologies; it is better to focus on the "important" protocols that many operators use, and the level of importance is a matter for the WG to determine through consensus. 3. New Work Proposal (Chairs) Sub-IP Protection Mechanisms - (Poretsky) a. work item proposal We have drafts that implement a harmonized proposal, but there needs to be enthusiastic support to get this off the ground. b. progress achieved http://www.ietf.org/internet-drafts/draft-poretsky-protection-term-00.txt http://www.ietf.org/internet-drafts/draft-poretsky-mpls-protection-meth-04.txt This work has a long history, but it has evolved into a harmonized proposal, and grown with considerable interest of new participants. It is very much like the current work on IGP-dataplane benchmarking, and reflects the fact that operators are demanding better restoration performance and are investigating sub-IP mechanisms. Only 2 people in the room had read the drafts, so it was not possible to develop consensus to adopt the drafts as a work item. However, many people *have* read these drafts and provided comments, it's just that almost all of them are not present today, so there is active interest. It may be helpful to contact other WG chairs to recruit reviewers and they may join us in bmwg to work this topic: MPLS is a prime example. Scott's next steps are to determine what other WG should be contacted to review this work. David Kessens reminded us that we need more than interest to begin new work -- people must say that they will read, review, and comment to make the result a WG product. About 8 people were attending BMWG for the first time, and 2 of them had engaged in this work item and read the drafts, so there has been some successful recruiting already. 4. Extended Working Group Status (Chairs) BMWG will follow the new WG Chair Shepherding process for all new publication submissions. See: draft-ietf-proto-wgchair-doc-shepherding-05.txt This effectively means one WG chair takes on role of shepherd to assure progress through AD and IESG review and the rest of the publication process. Network-layer Traffic Control Mechanisms Terms (dsmterm) http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-dsmterm-11.txt In his final read before shepherding this draft, Al found many typos that should be corrected and a clarification that the "delay" mentioned in several delay-related vectors is "Forwarding Delay". This draft needs some minor Final Editing before submission to ADs for publication. Hash and Stuffing Draft http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-hash-stuffing-04.txt The authors of this draft have requested a WGLC. Version 04 has only minor changes, so now we need to test for consensus. We will apply our active review process with the review templates, and need review volunteers when starting Last Call. Benchmarking Terminology for Resource Reservation Capable Routers http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-benchres-term-06.txt WGLC completed on this draft without comment, and the plan is to submit this draft for publication. The proposed work area on LDP convergence was briefly discussed during the status discussion, and it was confirmed that this work was on hold until the IGP-Dataplane work moves on. 5. IGP Data plane convergence benchmark I-Ds (Poretsky) http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-meth-08.txt http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-term-08.txt http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-app-08.txt WGLC completed with all positive comments. Some editorial work is needed to clean-up nits and typos before request for publication (reference to Al's comments on the list). Sue Hares has committed to review the drafts, so that should satisfy our need for cross area review. Scott implored the group to read this now, so that the 09 versions can be submitted for publication. There have been no technical issues raised on these drafts for months. 6. Terms and Methods for Benchmarking IPsec Devices (Merike Kaeo) http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-ipsec-term-07.txt http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-ipsec-meth-00.txt WGLC was completed on term version 05 with comments (some yet to be addressed), and version 07 reflects revisions stemming from development of the methodology (available for the first time at this meeting). Merike emphasized that there is lots of confusion about IPsec, so benchmarking for IPsec needs to be extremely clear. There have been many clarifications as the methodology was developed. For example, the new RFC4109 lists updated required and suggested algorithms for IKEv1. The authors would like document reviewers to identify any missing benchmarks and ensure that there is consensus on sections describing test parameters, security context and frame formats. Regarding the lack of coverage for IKEv2, it seemed to be best to complete the work with IKEv1 and manual security associations, but Merike will test this plan with a message to the list. IKEv2 may be addressed in a separate work item, as it may soon become widely implemented and deployed. 7. Techniques for Benchmarking Core Router Accelerated Life Testing. (Poretsky) http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-term-07.txt http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-meth-04.txt http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-meth-ebgp-00.txt http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-meth-opsec-00.txt Terminology & Methodology I-Ds were revised. The Methodology has been split into 3 drafts (general, EBGP, & security). Editor wishes to test readiness for WGLC on terms and general methods drafts and discuss the next steps for the specific-topic methodologies. The presentation included an example implementation of the terms and methodology drafts, prompting very active discussion and detailed feedback, including: o the benchmark Recovery Time would be better if measured with sub-second resolution. o since the benchmarks span multiple dimensions (rate, latency, lost sessions), it may be helpful to discuss the possibility that comparisons may not yield superior performance for one device in every category. However, the guidance must not go down the slippery slope of interpretation or "compliance". The guidance could address the general question, "what do we do with the benchmarks?" o "degraded forwarding rate" is really a derived benchmark, not the same as other rates, and should be named "forwarding rate degradation" o there were many suggestions and requests for more detailed information on the start-up and general configurations. Specifically: - Stats enabled, but what options? (should be similar for both DUTs) - Telnet sessions in number enabled, but how much traffic? (Scott reported this in his slides, but needs to require it in the methodology.) - Debugging - similar need for detail o Line card insertion and removal - agreed to be useful, but difficult to develop a a "standard" procedure that would yield uniform results, and physical insertions and removals would be complex to automate! The chairs said that they believed that the terms and methodology were ready to enter the WGLC process, following revisions to reflect the comments received at this meeting. The Last Call Process will use the Active review templates to ensure detailed reviews and hopefully accelerate the development of these drafts. However, the chairs asked that everyone defer development of the detailed methodologies until we have progressed the general method and terms. This topic was also discussed during the OPSEC working group meeting, and additional comments may be captured there. ACTION ITEMS: + Deal with charter and milestone updates + Restoration work proposal: looking for more volunteers and WGs to review + Merike will post message about IKEv1 vs IKEv2 in the IPsec documents. However, the WG has discussed this, should probably progress the current memos making it clear that they apply only to IKEv1. + WGLC on hash and stuffing + WGLC on accelerated stress term+meth (following revisions discussed) + Publication Requests on benchres, dsmterm w/rev, and igp-dataplane w/rev.


    Chairs' Agenda, Status, Milestones, Charter
    Sub-IP Protection Mechanisms
    IGP Data plane convergence benchmark I-Ds
    Terms and Methods for Benchmarking IPsec Devices
    Techniques for Benchmarking Core Router Accelerated Life Testing