2.4.1 Benchmarking Methodology (bmwg)

NOTE: This charter is a snapshot of the 71st IETF Meeting in Philadelphia, PA USA. It may now be out-of-date.
In addition to this official charter maintained by the IETF Secretariat, there is additional information about this working group on the Web at:

       Additional BMWG Page

Last Modified: 2007-09-26

Chair(s):

Al Morton <acmorton@att.com>

Operations and Management Area Director(s):

Dan Romascanu <dromasca@avaya.com>
Ronald Bonica <rbonica@juniper.net>

Operations and Management Area Advisor:

Ronald Bonica <rbonica@juniper.net>

Mailing Lists:

General Discussion: bmwg@ietf.org
To Subscribe: bmwg-request@ietf.org
In Body: subscribe your_email_address
Archive: http://www.ietf.org/mail-archive/web/bmwg/index.html

Description of Working Group:

The major goal of the Benchmarking Methodology Working Group is to make
a series of recommendations concerning the measurement of the
performance characteristics of various internetworking technologies;
further, these recommendations may focus on the systems or services
that are built from these technologies.

Each recommendation will describe the class of equipment, system, or
service being addressed; discuss the performance characteristics that
are pertinent to that class; clearly identify a set of metrics that aid
in the description of those characteristics; specify the methodologies
required to collect said metrics; and lastly, present the requirements
for the common, unambiguous reporting of benchmarking results.

To better distinguish the BMWG from other measurement initiatives in
the IETF, the scope of the BMWG is limited to technology
characterization using simulated stimuli in a laboratory environment.
Said differently, the BMWG does not attempt to produce benchmarks for
live, operational networks. Moreover, the benchmarks produced by this
WG shall strive to be vendor independent or otherwise have universal
applicability to a given technology class.

Because the demands of a particular technology may vary from
deployment to deployment, a specific non-goal of the Working Group is
to define acceptance criteria or performance requirements.

An ongoing task is to provide a forum for discussion regarding the
advancement of measurements designed to provide insight on the
operation internetworking technologies.

Goals and Milestones:

Done  Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches.
Done  Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC.
Done  Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately.
Done  Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices.
Done  Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft.
Done  Submit initial draft of Benchmarking Methodology for LAN Switches.
Done  Submit Terminology for IP Multicast Benchmarking draft for AD Review.
Done  Submit Benchmarking Terminology for Firewall Performance for AD review
Done  Progress ATM benchmarking terminology draft to AD review.
Done  Submit Benchmarking Methodology for LAN Switching Devices draft for AD review.
Done  Submit first draft of Firewall Benchmarking Methodology.
Done  First Draft of Terminology for FIB related Router Performance Benchmarking.
Done  First Draft of Router Benchmarking Framework
Done  Progress Frame Relay benchmarking terminology draft to AD review.
Done  Methodology for ATM Benchmarking for AD review.
Done  Terminology for ATM ABR Benchmarking for AD review.
Done  Terminology for FIB related Router Performance Benchmarking to AD review.
Done  Firewall Benchmarking Methodology to AD Review
Done  First Draft of Methodology for FIB related Router Performance Benchmarking.
Done  First draft Net Traffic Control Benchmarking Methodology.
Done  Methodology for IP Multicast Benchmarking to AD Review.
Done  Resource Reservation Benchmarking Terminology to AD Review
Done  First I-D on IPsec Device Benchmarking Terminology
Done  EGP Convergence Benchmarking Terminology to AD Review
Done  Resource Reservation Benchmarking Methodology to AD Review
Done  Net Traffic Control Benchmarking Terminology to AD Review
Done  IGP/Data-Plane Terminology I-D to AD Review
Done  IGP/Data-Plane Methodology and Considerations I-Ds to AD Review
Done  Hash and Stuffing I-D to AD Review
Mar 2007  IPv6 Benchmarking Methodology to AD Review
Dec 2007  Net Traffic Control Benchmarking Methodology to AD Review.
Dec 2007  IPsec Device Benchmarking Terminology to AD Review
Dec 2007  Router Accelerated Test Terminology to AD Review
Dec 2007  Router Accelerated Test Methodology to AD Review
Dec 2007  IPsec Device Benchmarking Methodology to AD Review
Dec 2007  Router Accelerated Test Method for EBGP to AD Review
Dec 2007  Router Accelerated Test Method for Operational Security to AD Review
Feb 2008  Terminology For Protection Benchmarking to AD Review
Feb 2008  Methodology For Protection Benchmarking to AD Review
Jul 2009  Basic BGP Convergence Benchmarking Methodology to AD Review.

Internet-Drafts:

  • draft-ietf-bmwg-ipsec-term-10.txt
  • draft-ietf-bmwg-igp-dataplane-conv-meth-15.txt
  • draft-ietf-bmwg-igp-dataplane-conv-term-15.txt
  • draft-ietf-bmwg-igp-dataplane-conv-app-15.txt
  • draft-ietf-bmwg-acc-bench-term-13.txt
  • draft-ietf-bmwg-acc-bench-meth-09.txt
  • draft-ietf-bmwg-ipsec-meth-03.txt
  • draft-ietf-bmwg-protection-term-04.txt
  • draft-ietf-bmwg-protection-meth-03.txt
  • draft-ietf-bmwg-ipv6-meth-05.txt

    Request For Comments:

    RFCStatusTitle
    RFC1242 I Benchmarking Terminology for Network Interconnection Devices
    RFC1944 I Benchmarking Methodology for Network Interconnect Devices
    RFC2285 I Benchmarking Terminology for LAN Switching Devices
    RFC2432 I Terminology for IP Multicast Benchmarking
    RFC2544 I Benchmarking Methodology for Network Interconnect Devices
    RFC2647 I Benchmarking Terminology for Firewall Performance
    RFC2761 I Terminology for ATM Benchmarking
    RFC2889 I Benchmarking Methodology for LAN Switching Devices
    RFC3116 I Methodology for ATM Benchmarking
    RFC3133 I Terminology for Frame Relay Benchmarking
    RFC3134 I Terminology for ATM ABR Benchmarking
    RFC3222 I Terminology for Forwarding Information Base (FIB) based Router Performance
    RFC3511 I Benchmarking Methodology for Firewall Performance
    RFC3918 I Methodology for IP Multicast Benchmarking
    RFC4061 I Benchmarking Basic OSPF Single Router Control Plane Convergence
    RFC4062 I OSPF Benchmarking Terminology and Concepts
    RFC4063 I Considerations When Using Basic OSPF Convergence Benchmarks
    RFC4098 I Terminology for Benchmarking BGP Device Convergence in the Control Plane
    RFC4689 I Terminology for Benchmarking Network-layer Traffic Control Mechanisms
    RFC4814 I Hash and Stuffing: Overlooked Factors in Network Device Benchmarking
    RFC4883 I Benchmarking Terminology for Resource Reservation Capable Routers

    Meeting Minutes


    Slides

    BMWG Agenda, Status, Milestones
    IGP Dataplane Convergence Time Benchmarking
    LDP Convergence Benchmarking
    Accelerated Stress Benchmarking
    Sub-IP Protection Benchmarking
    SIP Device Benchmarking