Benchmarking Methodology WG (bmwg)
MONDAY, November 7, 2005
1510-1710 Afternoon Session II (Oak)
CHAIRS: Kevin Dubray
Matt Zekauskas appeared in time to be graciously drafted as minute taker,
so his detailed notes, along with Kevin's Jabber Log,
were edited into the minutes by the Co-Chairs.
About 20 people attended the BMWG session.
0. Agenda bashing (we may need to shuffle a few items)
We bashed the agenda to move-up items that were somewhat controversial,
so that AD David Kessens could attend a conflicting meeting.
1. Milestone Status (Chairs)
See Slide 6 in the Chairs slide deck for the list of proposed milestones.
Based on presumed misses, the Chairs propose to revise milestones by a year.
There has been clear progress made, but progress requires more than
work by the editors. Some of the 350-plus members of the WG-list
must help by providing reviews. Our revised milestones must represent
real commitments to complete the work. The meeting produced more changes and
revisions, and the new dates will posted for discussion.
The chairs noted that the FIB methodology draft has expired again
due to lack of interest/support, and they plan to remove this milestone
altogether, following notice of intent on the bmwg-list.
2. Revised Charter discussion (Chairs)
See slides 7 through 9 in the chairs' deck.
The main discussion covered a point raised at the Paris IETF-63
regarding the scope of BMWG coverage. The chairs proposed to
focus the WG effort "primarily" on internetworking technologies
developed by the IETF. This clearly indicates our emphasis, but
also leaves some wiggle room so that we can consider exceptions
if they arise, because it is difficult to assess any technology
in isolation (IPsec is a classic example).
The charter discussion also covered points that are considered beyond
the scope of BMWG. Compliance testing is definitely out of scope,
BMWG will assume compliance when a manufacturer claims it.
The consensus was that classic reliability benchmarks
are out-of scope, but also that the sub-IP restoration and current
stress testing are reasonably within the charter. There should also
be a de-emphasis on niche technologies; it is better to focus on the
"important" protocols that many operators use, and the level of
importance is a matter for the WG to determine through consensus.
3. New Work Proposal (Chairs)
Sub-IP Protection Mechanisms - (Poretsky)
a. work item proposal
We have drafts that implement a harmonized proposal, but there
needs to be enthusiastic support to get this off the ground.
b. progress achieved
This work has a long history, but it has evolved into a harmonized
proposal, and grown with considerable interest of new participants.
It is very much like the current work on IGP-dataplane benchmarking,
and reflects the fact that operators are demanding better restoration
performance and are investigating sub-IP mechanisms. Only 2
people in the room had read the drafts, so it was not possible to
develop consensus to adopt the drafts as a work item. However, many
people *have* read these drafts and provided comments, it's just that
almost all of them are not present today, so there is active interest.
It may be helpful to contact other WG chairs to recruit reviewers and they
may join us in bmwg to work this topic: MPLS is a prime example.
Scott's next steps are to determine what other WG should be contacted
to review this work. David Kessens reminded us that we need
more than interest to begin new work -- people must say that they
will read, review, and comment to make the result a WG product.
About 8 people were attending BMWG for the first time, and 2 of them
had engaged in this work item and read the drafts, so there has been
some successful recruiting already.
4. Extended Working Group Status (Chairs)
BMWG will follow the new WG Chair Shepherding process for all
new publication submissions. See:
This effectively means one WG chair takes on role of shepherd to
assure progress through AD and IESG review and the rest of the
Network-layer Traffic Control Mechanisms Terms (dsmterm)
In his final read before shepherding this draft, Al found many typos that should
be corrected and a clarification that the "delay" mentioned in
several delay-related vectors is "Forwarding Delay".
This draft needs some minor Final Editing before submission to ADs for publication.
Hash and Stuffing Draft
The authors of this draft have requested a WGLC. Version 04
has only minor changes, so now we need to test for consensus. We will
apply our active review process with the review templates, and need
review volunteers when starting Last Call.
Benchmarking Terminology for Resource Reservation Capable Routers
WGLC completed on this draft without comment, and the plan is to
submit this draft for publication.
The proposed work area on LDP convergence was briefly discussed
during the status discussion, and it was confirmed that this work
was on hold until the IGP-Dataplane work moves on.
5. IGP Data plane convergence benchmark I-Ds (Poretsky)
WGLC completed with all positive comments. Some editorial work
is needed to clean-up nits and typos before request for publication
(reference to Al's comments on the list).
Sue Hares has committed to review the drafts, so that should satisfy
our need for cross area review. Scott implored the group to read this
now, so that the 09 versions can be submitted for publication.
There have been no technical issues raised on these drafts for months.
6. Terms and Methods for Benchmarking IPsec Devices (Merike Kaeo)
WGLC was completed on term version 05 with comments (some yet to be
addressed), and version 07 reflects revisions stemming from development
of the methodology (available for the first time at this meeting).
Merike emphasized that there is lots of confusion about IPsec,
so benchmarking for IPsec needs to be extremely clear.
There have been many clarifications as the methodology was developed.
For example, the new RFC4109 lists updated required and suggested
algorithms for IKEv1.
The authors would like document reviewers to identify any missing
benchmarks and ensure that there is consensus on sections describing
test parameters, security context and frame formats.
Regarding the lack of coverage for IKEv2, it seemed to be best to complete
the work with IKEv1 and manual security associations, but Merike will
test this plan with a message to the list. IKEv2 may be addressed
in a separate work item, as it may soon become widely implemented
7. Techniques for Benchmarking Core Router Accelerated Life Testing.
Terminology & Methodology I-Ds were revised. The Methodology has been split
into 3 drafts (general, EBGP, & security). Editor wishes to test
readiness for WGLC on terms and general methods drafts and discuss the
next steps for the specific-topic methodologies.
The presentation included an example implementation of the terms
and methodology drafts, prompting very active discussion and detailed
o the benchmark Recovery Time would be better if measured with
o since the benchmarks span multiple dimensions (rate, latency,
lost sessions), it may be helpful to discuss the possibility that
comparisons may not yield superior performance for one device
in every category. However, the guidance must not go down the
slippery slope of interpretation or "compliance". The guidance
could address the general question, "what do we do with the benchmarks?"
o "degraded forwarding rate" is really a derived benchmark, not the
same as other rates, and should be named "forwarding rate degradation"
o there were many suggestions and requests for more detailed
information on the start-up and general configurations. Specifically:
- Stats enabled, but what options? (should be similar for both DUTs)
- Telnet sessions in number enabled, but how much traffic? (Scott
reported this in his slides, but needs to require it in the methodology.)
- Debugging - similar need for detail
o Line card insertion and removal - agreed to be useful, but difficult
to develop a a "standard" procedure that would yield uniform results, and
physical insertions and removals would be complex to automate!
The chairs said that they believed that the terms and methodology
were ready to enter the WGLC process, following revisions to reflect
the comments received at this meeting. The Last Call Process will
use the Active review templates to ensure detailed reviews and
hopefully accelerate the development of these drafts. However,
the chairs asked that everyone defer development of the detailed
methodologies until we have progressed the general method and terms.
This topic was also discussed during the OPSEC working group meeting,
and additional comments may be captured there.
+ Deal with charter and milestone updates
+ Restoration work proposal: looking for more volunteers and WGs to review
+ Merike will post message about IKEv1 vs IKEv2 in the IPsec documents.
However, the WG has discussed this, should probably progress the current
memos making it clear that they apply only to IKEv1.
+ WGLC on hash and stuffing
+ WGLC on accelerated stress term+meth (following revisions discussed)
+ Publication Requests on benchres, dsmterm w/rev,
and igp-dataplane w/rev.