Benchmarking Methodology (bmwg)

In addition to this official charter maintained by the IETF Secretariat, there is additional information about this working group on the Web at:

       Additional BMWG Page

Last Modified: 2008-08-21

Additional information is available at tools.ietf.org/wg/bmwg

Chair(s):

  • Al Morton <acmorton@att.com>

    Operations and Management Area Director(s):

  • Dan Romascanu <dromasca@avaya.com>
  • Ronald Bonica <rbonica@juniper.net>

    Operations and Management Area Advisor:

  • Ronald Bonica <rbonica@juniper.net>

    Mailing Lists:

    General Discussion: bmwg@ietf.org
    To Subscribe: bmwg-request@ietf.org
    In Body: subscribe your_email_address
    Archive: http://www.ietf.org/mail-archive/web/bmwg/index.html

    Description of Working Group:

    The major goal of the Benchmarking Methodology Working Group is to make
    a series of recommendations concerning the measurement of the
    performance characteristics of various internetworking technologies;
    further, these recommendations may focus on the systems or services that
    are built from these technologies.

    Each recommendation will describe the class of equipment, system, or
    service being addressed; discuss the performance characteristics that
    are pertinent to that class; clearly identify a set of metrics that aid
    in the description of those characteristics; specify the methodologies
    required to collect said metrics; and lastly, present the requirements
    for the common, unambiguous reporting of benchmarking results.

    To better distinguish the BMWG from other measurement initiatives in the
    IETF, the scope of the BMWG is limited to technology characterization
    using simulated stimuli in a laboratory environment.  Said differently,
    the BMWG does not attempt to produce benchmarks for live, operational
    networks. Moreover, the benchmarks produced by this WG shall strive to
    be vendor independent or otherwise have universal applicability to a
    given technology class.

    Because the demands of a particular technology may vary from deployment
    to deployment, a specific non-goal of the Working Group is to define
    acceptance criteria or performance requirements.

    An ongoing task is to provide a forum for discussion regarding the
    advancement of measurements designed to provide insight on the operation
    inter-networking technologies.

    In addition to its current work plan, the BMWG is explicitly tasked to
    develop benchmarks and methodologies for the following technologies:

    * MPLS Forwarding: Develop specific methods to characterize the
    latency and forwarding performance of MPLS devices, extending the
    fundamental recommendations of RFC 1242 and RFC 2544 to this networking
    technology.

    * SIP Networking Devices: Develop new terminology and methods to
    characterize the key performance aspects of network devices using
    SIP, including the signaling plane scale and service rates while
    considering load conditions on both the signaling and media planes. This
    work will be harmonized with related SIP performance metric definitions
    prepared by the PMOL working group.

    Goals and Milestones:

    Done  Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches.
    Done  Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC.
    Done  Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately.
    Done  Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices.
    Done  Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft.
    Done  Submit initial draft of Benchmarking Methodology for LAN Switches.
    Done  Submit Terminology for IP Multicast Benchmarking draft for AD Review.
    Done  Submit Benchmarking Terminology for Firewall Performance for AD review
    Done  Progress ATM benchmarking terminology draft to AD review.
    Done  Submit Benchmarking Methodology for LAN Switching Devices draft for AD review.
    Done  Submit first draft of Firewall Benchmarking Methodology.
    Done  First Draft of Terminology for FIB related Router Performance Benchmarking.
    Done  First Draft of Router Benchmarking Framework
    Done  Progress Frame Relay benchmarking terminology draft to AD review.
    Done  Methodology for ATM Benchmarking for AD review.
    Done  Terminology for ATM ABR Benchmarking for AD review.
    Done  Terminology for FIB related Router Performance Benchmarking to AD review.
    Done  Firewall Benchmarking Methodology to AD Review
    Done  First Draft of Methodology for FIB related Router Performance Benchmarking.
    Done  First draft Net Traffic Control Benchmarking Methodology.
    Done  Methodology for IP Multicast Benchmarking to AD Review.
    Done  Resource Reservation Benchmarking Terminology to AD Review
    Done  First I-D on IPsec Device Benchmarking Terminology
    Done  EGP Convergence Benchmarking Terminology to AD Review
    Done  Resource Reservation Benchmarking Methodology to AD Review
    Done  Net Traffic Control Benchmarking Terminology to AD Review
    Done  IGP/Data-Plane Terminology I-D to AD Review
    Done  IGP/Data-Plane Methodology and Considerations I-Ds to AD Review
    Done  Hash and Stuffing I-D to AD Review
    Done  IPv6 Benchmarking Methodology to AD Review
    Sep 2008  IPsec Device Benchmarking Terminology to IESG Review
    Sep 2008  IPsec Device Benchmarking Methodology to IESG Review
    Dec 2008  Net Traffic Control Benchmarking Methodology to AD Review.
    Dec 2008  Router Accelerated Test Terminology to IESG Review
    Dec 2008  Router Accelerated Test Methodology to IESG Review
    Feb 2009  Terminology For Protection Benchmarking to AD Review
    Feb 2009  Methodology For Protection Benchmarking to AD Review
    Apr 2009  Methodology for MPLS Forwarding to AD Review
    Jun 2009  Terminology for SIP Device Benchmarking to IESG Review
    Jun 2009  Methodology for SIP Device Benchmarking to IESG Review
    Dec 2009  Router Accelerated Test Method for EBGP to IESG Review
    Dec 2009  Router Accelerated Test Method for Operational Security to IESG Review
    Jul 2010  Basic BGP Convergence Benchmarking Methodology to AD Review.

    Internet-Drafts:

    Benchmarking Methodology for Link-State IGP Data Plane Route Convergence (48516 bytes)
    Terminology for Benchmarking Link-State IGP Data Plane Route Convergence (41039 bytes)
    Considerations for Benchmarking Link-State IGP Data Plane Route Convergence (15194 bytes)

    Request For Comments:

    Benchmarking Terminology for Network Interconnection Devices (RFC 1242) (22143 bytes)
    Benchmarking Methodology for Network Interconnect Devices (RFC 1944) (66061 bytes) obsoleted by RFC 2544
    Benchmarking Terminology for LAN Switching Devices (RFC 2285) (43130 bytes)
    Terminology for IP Multicast Benchmarking (RFC 2432) (29758 bytes)
    Benchmarking Methodology for Network Interconnect Devices (RFC 2544) (66688 bytes) obsoletes RFC 1944
    Benchmarking Terminology for Firewall Performance (RFC 2647) (45374 bytes)
    Terminology for ATM Benchmarking (RFC 2761) (61219 bytes)
    Benchmarking Methodology for LAN Switching Devices (RFC 2889) (73251 bytes)
    Methodology for ATM Benchmarking (RFC 3116) (294857 bytes)
    Terminology for Frame Relay Benchmarking (RFC 3133) (44182 bytes)
    Terminology for ATM ABR Benchmarking (RFC 3134) (29542 bytes)
    Terminology for Forwarding Information Base (FIB) based Router Performance (RFC 3222) (25483 bytes)
    Benchmarking Methodology for Firewall Performance (RFC 3511) (67916 bytes)
    Methodology for IP Multicast Benchmarking (RFC 3918) (64652 bytes)
    Terminology for Benchmarking BGP Device Convergence in the Control Plane (RFC 4098) (66845 bytes)
    Benchmarking Basic OSPF Single Router Control Plane Convergence (RFC 4061) (32706 bytes)
    OSPF Benchmarking Terminology and Concepts (RFC 4062) (15784 bytes)
    Considerations When Using Basic OSPF Convergence Benchmarks (RFC 4063) (23401 bytes)
    Terminology for Benchmarking Network-layer Traffic Control Mechanisms (RFC 4689) (62369 bytes)
    Hash and Stuffing: Overlooked Factors in Network Device Benchmarking (RFC 4814) (59272 bytes)
    Benchmarking Terminology for Resource Reservation Capable Routers (RFC 4883) (54205 bytes)
    IPv6 Benchmarking Methodology for Network Interconnect Devices (RFC 5180) (41712 bytes)
    IETF Secretariat - Please send questions, comments, and/or suggestions to ietf-web@ietf.org.

    Return to working group directory.

    Return to IETF home page.