Current Meeting Report
Jabber Logs

2.4.3 Benchmarking Methodology (bmwg)

NOTE: This charter is a snapshot of the 55th IETF Meeting in Altanta, Georgia USA. It may now be out-of-date.

Last Modifield: 05/23/2002

Kevin Dubray <>
Operations and Management Area Director(s):
Randy Bush <>
Bert Wijnen <>
Operations and Management Area Advisor:
Randy Bush <>
Mailing Lists:
General Discussion:
To Subscribe:
In Body: subscribe your_email_address
Description of Working Group:
The major goal of the Benchmarking Methodology Working Group is to make a series of recommendations concerning the measurement of the performance characteristics of various internetworking technologies; further, these recommendations may focus on the systems or services that are built from these technologies.

Each recommendation will describe the class of equipment, system, or service being addressed; discuss the performance characteristics that are pertinent to that class; clearly identify a set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of benchmarking results.

Because the demands of a class may vary from deployment to deployment, a specific non-goal of the Working Group is to define acceptance criteria or performance requirements.

An ongoing task is to provide a forum for discussion regarding the advancement of measurements designed to provide insight on the operation internetworking technologies.

Goals and Milestones:
Done  Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches.
Done  Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC.
Done  Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately.
Done  Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices.
Done  Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft.
Done  Submit initial draft of Benchmarking Methodology for LAN Switches.
Done  Submit Terminology for IP Multicast Benchmarking draft for AD Review.
Done  Submit Benchmarking Terminology for Firewall Performance for AD review
Done  Progress ATM benchmarking terminology draft to AD review.
Done  Submit Benchmarking Methodology for LAN Switching Devices draft for AD review.
Done  Submit first draft of Firewall Benchmarking Methodology.
Done  First Draft of Terminology for FIB related Router Performance Benchmarking.
Done  First Draft of Router Benchmarking Framework
Done  Methodology for ATM Benchmarking for AD review.
Done  Progress Frame Relay benchmarking terminology draft to AD review.
Done  Terminology for ATM ABR Benchmarking for AD review.
MAR 01  Router Benchmarking Framework to AD review.
JUL 01  Terminology for FIB related Router Performance Benchmarking to AD review.
NOV 01  Methodology for IP Multicast Benchmarking to AD Review.
NOV 01  Firewall Benchmarking Methodology to AD Review
NOV 01  Net Traffic Control Benchmarking Terminology to AD Review
NOV 01  Resource Reservation Benchmarking Terminology to AD Review
NOV 01  EGP Convergence Benchmarking Terminology to AD Review
DEC 01  First Draft of Methodology for FIB related Router Performance Benchmarking.
FEB 02  First draft Net Traffic Control Benchmarking Methodology.
FEB 02  Resource Reservation Benchmarking Methodology to AD Review
FEB 02  Basic BGP Convergence Benchmarking Methodology to AD Review.
JUN 02  Methodology for FIB related Router Performance Benchmarking to AD review.
NOV 02  Net Traffic Control Benchmarking Methodology to AD Review.
  • - draft-ietf-bmwg-mcastm-08.txt
  • - draft-ietf-bmwg-firewall-05.txt
  • - draft-ietf-bmwg-dsmterm-03.txt
  • - draft-ietf-bmwg-bgpbas-01.txt
  • - draft-ietf-bmwg-conterm-03.txt
  • - draft-ietf-bmwg-fib-meth-00.txt
  • - draft-bmwg-ospfconv-term-00.txt
  • - draft-bmwg-ospfconv-intraarea-00.txt
  • Request For Comments:
    RFC1242 I Benchmarking Terminology for Network Interconnection Devices
    RFC1944 I Benchmarking Methodology for Network Interconnect Devices
    RFC2285 I Benchmarking Terminology for LAN Switching Devices
    RFC2432 I Terminology for IP Multicast Benchmarking
    RFC2544 I Benchmarking Methodology for Network Interconnect Devices
    RFC2647 I Benchmarking Terminology for Firewall Performance
    RFC2761 I Terminology for ATM Benchmarking
    RFC2889 I Benchmarking Methodology for LAN Switching Devices
    RFC3116 I Methodology for ATM Benchmarking
    RFC3133 I Terminology for Frame Relay Benchmarking
    RFC3134 I Terminology for ATM ABR Benchmarking
    RFC3222 I Terminology for Forwarding Information Base (FIB) based Router Performance

    Current Meeting Report

    BMWG Minutes
    The meeting was chaired by Kevin Dubray.  Contributors to the 
    meeting's minutes include Scott Poretsky, Al Morton, and Kevin Dubray.
    About 35 people attended the BMWG session.  The original agenda,
    0. Administration
    1. Benchmarking Network-layer Traffic Control Mechanisms
       Attempt to resolve contentious issues regarding the definition of the 
    term "congestion." Assess I-D's last call suitability.
    2. Status update of OSPF benchmarking I-Ds:
    3. Followup to a proposed, new BMWG work item: "SONET/SDH APS 
    Performance Benchmarking."
    4. New work proposal: Core Router Accelerated Life Testing
    was approved, with the modification that the chair would attempt to 
    briefly address item 2, the OSPF benchmark I-Ds, as the I-D's 
    principals were unable to attend this session.
    0. Administration
    A status summary of BMWG Internet Drafts was presented as follows:
    *AD/IESG Review:
    <draft-ietf-bmwg-benchres-term-02.txt>, in review
      <draft-ietf-bmwg-conterm-03.txt>, editing required
    *I-D Last Call:
      <draft-ietf-bmwg-firewall-06.txt>, ready to forward to ADs.
      <draft-ietf-bmwg-mcastm-09.txt>, revised
      <draft-ietf-bmwg-dsmterm-04.txt>, revised
    <draft-ietf-bmwg-ospfconv-term-01.txt>, revised
    <draft-ietf-bmwg-ospfconv-intraarea-02.txt>, revised
    <draft-ietf-bmwg-ospfconv-applicability-00.txt> initial
      <draft-ietf-bmwg-fib-meth-00.txt>, expired
    Kevin Dubray made a note that we'd try to get the FIB methodology draft 
    'unexpired' and in last call, as there was not a lot of commentary on it.
    Regarding the BGP convergence terminology work, it was noted that the 
    AD/IESG review had presented substantive commentary.  It was unclear as to 
    where the principals were in addressing the draft.  Elwyn Davies replied 
    that it was on the radar and he would help coax the production of a 
    follow-up I-D.
    There was concern raised over the progression (i.e., slow progress of the 
    multicast methodology I-D. Dubray deferred the discussion of this topic 
    until later in the meeting.
    There was a re-ordering of the agenda slightly to talk about the OSPF 
    benchmarking work, before tackling the subsequent meeting items.
    Status update of OSPF benchmarking I-Ds.
    Due to other WG meeting conflicts, the OSPF I-D editors were unable to 
    attend the BMWG session. Dubray said that the editors contacted him prior to 
    the meeting and indicated that the terminology and methodology were 
    nearing readiness for WG last call.  There appeared to be agreement in the 
    room along those lines.
    Kevin also asked folks to pay particular attention to the 
    corresponding "applicability" statement, as it represented a new type of 
    document for the BMWG - trying to identify contexts where the 
    benchmarks had utility and where they did not.
    Benchmarking Network-layer Traffic Control Mechanisms
    Jerry Perser led a discussion on the current state of the 
    terminology I-D. The slides can be found in the proceedings.  A 
    discussion on the I-D's definition of "Congestion" ensued.
    Jerry presented a viewpoint on the nature of delay and congestion.
    In the I-D, Congestion was tightly coupled to lossful events. It was 
    argued that congestion can mean more than this. If the I-D wanted to stick to 
    this definition, it should not have a title so encompassing. It was 
    recommended that definition be called something like 
    "Loss-indicated Congestion" to prevent confusion.
    With respect to delay, it was discussed that delay may also be an 
    indicator of congestion, but it was not necessarily a 
    deterministic one. So, it was recommended to add a paragraph to the 
    Congestion discussion on why delay is not a deterministic indicator of 
    congestion and add a paragraph to the Delay discussion on what's 
    different from RFC 2679. In delay calculations, Jerry made note that the 
    inclusion of packet insertion time generally concerned him.
    It was also thought to be useful to add a paragraph comparing the notion of a 
    "Loss Vector" to RFC 2680.
    Proposed BMWG work item: "SONET/SDH APS Performance 
    Takumi Kimura and Jerry Perser were present to lead a discussion 
    regarding the I-D. Presenters' slides can be found in the 
    Following the general presentation, a discussion was had regarding 
    whether this work is a good fit for the BMWG.  A question was asked as to 
    whether this work was better suited for sub-IP types of 
    organizations, like the IEEE.  It was countered that providers are asking 
    vendors to quantify how sub-IP technologies' failover or recovery affect the 
    IP layer related reachability and forwarding.
    It was thought the work was OK, if it kept its focus on IP affected 
    responses.  It was thought the title should be changed to reflect said 
    Since protection could happen at various layers (or sub-layers), such as 
    SONET or MPLS, there was a discussion on how best to tackle this.  The 
    group gravitated to the idea that a single, common terminology document 
    with subsequent methodology documents for individual recovery 
    mechanisms (e.g., one for MPLS, one for APS, etc.) might be the way to go.
    A specific comment directed towards the draft thought it would be a good 
    thing to lose the use of the term "service". I.e., reformulate the notion of 
    "Loss of Service," or "Loss of Service Time."
    Proposed BMWG work item: Core Router Accelerated Life Testing
    Scott Poretsky gave an overview of the proposed work item.  His slides are in 
    the proceedings.
    Many ISP folks in the room indicated they do forms of this style of 
    It was also thought that while the tests addressed in the I-D were 
    interesting, there was a question of "fit" for the work in the BMWG.  Some 
    issues raised:
          The tests seem to border more on functional tests than 
    benchmarks. (The IETF has seemed to shy away from specifying 
    functional assessments.)
          The tests seem to rely on white box instrumentation.  (The BMWG has 
    traditionally agreed that white box tests are out of scope for the WG.)
          The bounding of the test domain. What do you include?  Where do you 
          The difficulty "standardizing" an operational model on which to 
          Parallels were drawn to the failed Router Benchmarking Framework 
    While there appeared to be general interest in the work from the 
    attendees, the chair counsel that work might have to be more tailored 
    along traditional BMWG lines to be considered.
    With the discussion on the Router Life Testing concluded, discussion 
    returned to the Multicast Benchmarking Methodology I-D.   There was some 
    frustration expressed as to the degree of progress the draft was making. The 
    chair replied that, currently, the only topic being actively discussed on 
    the mailing list was the encapsulation family of metrics.  It was 
    countered that the methodology I-D seemed disconnected from its 
    terminology parent, RFC 2432. Kevin made the appeal to get issues and 
    suggested fixes to the list so we can bring this work item to a quick and 
    useful conclusion.


    Terminology for Benchmarking Network-layer Traffic Control Mechanisms - draft 4
    Benchmarking Terminology for Protection Performance
    Core Router Software Accelerated Life Testing