2.4.3 Benchmarking Methodology (bmwg)

NOTE: This charter is a snapshot of the 56th IETF Meeting in San Francisco, California USA. It may now be out-of-date.

Last Modified: 2003-03-10

Kevin Dubray <kdubray@juniper.net>
Alfred Morton <acmorton@att.com>
Operations and Management Area Director(s):
Randy Bush <randy@psg.com>
Bert Wijnen <bwijnen@lucent.com>
Operations and Management Area Advisor:
Randy Bush <randy@psg.com>
Mailing Lists:
General Discussion: bmwg@ietf.org
To Subscribe: bmwg-request@ietf.org
In Body: subscribe your_email_address
Archive: ftp://ftp.ietf.org/ietf-mail-archive/bmwg/
Description of Working Group:
The major goal of the Benchmarking Methodology Working Group is to make a series of recommendations concerning the measurement of the performance characteristics of various internetworking technologies; further, these recommendations may focus on the systems or services that are built from these technologies.

Each recommendation will describe the class of equipment, system, or service being addressed; discuss the performance characteristics that are pertinent to that class; clearly identify a set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of benchmarking results.

Because the demands of a class may vary from deployment to deployment, a specific non-goal of the Working Group is to define acceptance criteria or performance requirements.

An ongoing task is to provide a forum for discussion regarding the advancement of measurements designed to provide insight on the operation internetworking technologies.

Goals and Milestones:
Done  Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches.
Done  Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC.
Done  Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately.
Done  Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices.
Done  Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft.
Done  Submit initial draft of Benchmarking Methodology for LAN Switches.
Done  Submit Terminology for IP Multicast Benchmarking draft for AD Review.
Done  Submit Benchmarking Terminology for Firewall Performance for AD review
Done  Progress ATM benchmarking terminology draft to AD review.
Done  Submit Benchmarking Methodology for LAN Switching Devices draft for AD review.
Done  Submit first draft of Firewall Benchmarking Methodology.
Done  First Draft of Terminology for FIB related Router Performance Benchmarking.
Done  First Draft of Router Benchmarking Framework
Done  Progress Frame Relay benchmarking terminology draft to AD review.
Done  Methodology for ATM Benchmarking for AD review.
Done  Terminology for ATM ABR Benchmarking for AD review.
MAR 01  Router Benchmarking Framework to AD review.
JUL 01  Terminology for FIB related Router Performance Benchmarking to AD review.
NOV 01  Methodology for IP Multicast Benchmarking to AD Review.
NOV 01  Firewall Benchmarking Methodology to AD Review
NOV 01  Net Traffic Control Benchmarking Terminology to AD Review
NOV 01  Resource Reservation Benchmarking Terminology to AD Review
NOV 01  EGP Convergence Benchmarking Terminology to AD Review
DEC 01  First Draft of Methodology for FIB related Router Performance Benchmarking.
FEB 02  First draft Net Traffic Control Benchmarking Methodology.
FEB 02  Resource Reservation Benchmarking Methodology to AD Review
FEB 02  Basic BGP Convergence Benchmarking Methodology to AD Review.
JUN 02  Methodology for FIB related Router Performance Benchmarking to AD review.
NOV 02  Net Traffic Control Benchmarking Methodology to AD Review.
  • - draft-ietf-bmwg-mcastm-11.txt
  • - draft-ietf-bmwg-firewall-08.txt
  • - draft-ietf-bmwg-dsmterm-05.txt
  • - draft-ietf-bmwg-benchres-term-02.txt
  • - draft-ietf-bmwg-conterm-04.txt
  • - draft-ietf-bmwg-fib-meth-01.txt
  • - draft-ietf-bmwg-ospfconv-term-02.txt
  • - draft-ietf-bmwg-ospfconv-intraarea-03.txt
  • - draft-ietf-bmwg-ospfconv-applicability-01.txt
  • - draft-ietf-bmwg-ipsec-term-00.txt
  • Request For Comments:
    RFC1242 I Benchmarking Terminology for Network Interconnection Devices
    RFC1944 I Benchmarking Methodology for Network Interconnect Devices
    RFC2285 I Benchmarking Terminology for LAN Switching Devices
    RFC2432 I Terminology for IP Multicast Benchmarking
    RFC2544 I Benchmarking Methodology for Network Interconnect Devices
    RFC2647 I Benchmarking Terminology for Firewall Performance
    RFC2761 I Terminology for ATM Benchmarking
    RFC2889 I Benchmarking Methodology for LAN Switching Devices
    RFC3116 I Methodology for ATM Benchmarking
    RFC3133 I Terminology for Frame Relay Benchmarking
    RFC3134 I Terminology for ATM ABR Benchmarking
    RFC3222 I Terminology for Forwarding Information Base (FIB) based Router Performance

    Current Meeting Report

    IETF56 Benchmarking Methodology WG Meeting Minutes
    The meeting was chaired by Kevin Dubray and Al Morton.  
    Contributors to the meeting's minutes include Matt Zekauskas, Al Morton, and 
    Kevin Dubray.  Special thanks to Matt for acting as meeting scribe.
    About 75 people attended the BMWG session.  The original agenda:
    0. Administration (Dubray, 15 min)
    1. OSPF convergence I-D’s Last Call (White, 15 min)
    2. Considerations Benchmarking Routing Protocol Network Convergence 
    (White, 10 min)
    3. IPsec Device Benchmarking Term. (Bustos, 20 min)
    4. Traffic Control I-D update (Perser, 10 min)
    5. Core Router Software Accelerated Life Testing (Poretsky, 30 min)
    6. IGP Data Plane Convergence I-Ds. (Dubray, 10 min)
    was approved without modification.
    0. Administration
    Al Morton was welcomed as the new BMWG co-chair.
    It was mentioned that the charter was revised to better reflect the 
    lab-centric, vendor-independent nature of the BMWG's benchmarks.  
    Milestones were brought current.  Interested parties were directed to the 
    mailing list archive as the BMWG web page didn't reflect the Charter 
    changes yet.
    A status summary of BMWG Internet Drafts at the time of the meeting was 
    presented as follows:
    * AD/IESG Review:
        <draft-ietf-bmwg-conterm-04.txt>, revised to reflect IESG input.
    <draft-ietf-bmwg-benchres-term-02.txt>, back in chairs’ domain.
    * I-D Last Call:
        <draft-ietf-bmwg-fib-meth-01.txt>,  Call ended 3/14.
        <draft-ietf-bmwg-dsmterm-05.txt>, Call ends 3/25.
    <draft-ietf-bmwg-ospfconv-term-02,.txt>, Clarifying input
    <draft-ietf-bmwg-ospfconv-intraarea-03.txt>, Clarifying input
    <draft-ietf-bmwg-ospfconv-applicability-01.txt> Clarifying input
    * I-Ds:
         <draft-ietf-bmwg-mcastm-11.txt>, last call, task team, revised
    * RFC Editor:
         <draft-ietf-bmwg-firewall-08.txt>, in RFC Editor queue.
    * Expired BMWG I-Ds:
         <draft-ietf-bmwg-bgpbas-01.txt>, Pending term. progress
    <draft-ietf-bmwg-benchres-method-00.txt> Pending term progress
    * New Work proposals:
         Core Router Software Accelerated Life testing,
         Protection Switch I-Ds (To revise I-D before testing WG support),
         IGP Convergence benchmarking,
    1.  Discussion on OSPF Convergence Benchmarking WG Last Call comments.
    Discussion Goal: Last call is over, but consensus on outstanding issues 
    isn't clear.  Identify outstanding issues.
    Changes to Terminology Draft: Tree-shape discussion moves to the 
    Applicability draft. Remove Duplicate Definitions for terms that are 
    already defined, such as Shortest Path First (SPF). Changes in the 
    Methodology Draft: The single "white box" test (section 8.3) becomes a 
    supplement to (or validation of) the SPF convergence time black box test in 
    section 8.4.
    2.  Considerations in Benchmarking Routing Protocol Network 
    Discussion goal: Test whether the BMWG the right home for this I-D.
    Russ White presented an overview of this draft, what was in/out of scope, 
    and what was left to do next. He described a future investigation of this 
    convergence testing, where a network of 300 routers would be used to 
    evaluate a DUT, then substitute the networks with emulators and see what 
    results are different, if any.
    The memo will continue development as an individual draft, and Russ asked 
    for comments. The WG can decide where this should go when the I-D is more 
    mature (ippm did not seem like an option, since scope is 
    emulation/lab net).
    Presentation slides can be found in the Proceedings (bmwg-2)
    3. IPsec Device Benchmarking Terminology I-D.
    Discussion Goal: Present I-D introduction and discuss current issues.
    Michele gave an overview of the draft in its initial version, 
    including an animated, two-phase tunnel establishment.  
    Presentation slides can be found in the Proceedings. (bmwg-1)
    There was a comment to add tunnel disengagement time and rate to the I-D 
    since it seemed complimentary to tunnel establishment. Discussion 
    revealed that the usual tunnel deletion is by timeout, and the "delete" 
    capability for tunnel participants is optional.
    4. Benchmarking Network-layer Traffic Control Mechanisms Terminology I-D
    Discussion Goal: Communicate latest changes & test progress of current WG 
    Last Call.
    There are several comments so far in Last Call and a few changes to make to 
    the current I-D (including reflecting loss in the jitter 
    measurement).  There will be another last call once the revised I-D is 
    5. Terminology for Benchmark Core Router Software Accelerated Life 
    Testing I-D
    Discussion Goal: Version 00 of this draft seemed to present more 
    functional testing than benchmarks - I-D fit to BMWG was questioned. This 
    draft presumes better WG fit; did the principals succeed?
    Scott Poretsky described the deltas that went into the draft since it was 
    last proposed in Atlanta.  For details, see slides in the 
    Proceedings. (bmwg-3)
    Scott asked whether the I-D was ready to move forward to the ADs. The 
    chairs advised that first we should see if the working group supports 
    adding this I-D to the BMWG charter as a new deliverable.
    There were concerns about whether the ensuing benchmarks could be 
    specified with enough methodological detail so as to produce a uniform 
    yardstick that would yield repeatable results. Scott indicated that he 
    didn't believe detail regarding the associated methodologies would be an 
    issue; moreover, he indicated that the benchmarks would be a good tool for 
    multiple vendor comparisons.
    There appeared to be interest from the attendees to test acceptance on the 
    list of this I-D as a new BMWG project; the chairs indicated they would 
    test acceptance of this proposed BMWG deliverable on the list.
    IGP Data plane convergence benchmark I-Ds.
    Discussion Goal: Regarding "convergence" benchmarking, the WG had 
    restricted scoping of initial convergence benchmarks to the control 
    plane.  However, the IGP convergence I-Ds were originally offered as a 
    replacement to the OSPF convergence I-Ds. Should they replace the OSPF 
    convergence IDs?  If not, is it time for the BMWG to consider explicit data 
    plane convergence?  If yes, what is the IGP drafts' relationship to the 
    OSPF convergence I-Ds: replacement on companion?
    Kevin reminded the group that due to the complexity of 
    characterizing convergence, the WG chose to take a stepwise approach in 
    specifying related benchmarks: demonstrate progress in the control plane, 
    then move onto other areas.
    The group appeared non-committal as to whether the BMWG has achieved 
    critical mass in the control plane convergence deliverables.  That said, 
    there appeared to be a stronger opinion that we should start the data 
    plane work as that is where practical meaning was to be found.
    It was stated that IGP convergence seemed complementary to the current OSPF 
    control plane convergence I-Ds.
    There was talk of producing a master convergence document which sought to 
    unify all that the individual I-Ds were prescribing.  Kevin recounted that 
    the group considered producing a single work when initially 
    discussing the convergence work.  It was thought, at the time, smaller 
    deliverables were the way to go.  He offered that a single, unified 
    document was not undesirable, but it may be prudent to wait until we have 
    momentum and experience with the individual efforts from which to draw.
    The chairs indicated that the would test acceptance of these proposed BMWG 
    deliverables on the list.


    Terminology for Benchmarking IPsec Devices
    Considerations in Benchmarking Routing Protocol Network Convergence
    Core Router Software Accelerated Life Testing