Benchmarking Methodology WG (bmwg)
Tuesday, July 15, 2003, 1300-1400 and 1415-1515
CHAIRS Kevin Dubray <email@example.com>
Al Morton <firstname.lastname@example.org>
Reported by Al Morton, based on the information generously compiled by Sue
Hares as official note-taker.
About 66 people attended one or more of the BMWG sessions.
1. Working Group Status (Morton)
Status of BMWG I-Ds at close of IETF-57
<draft-ietf-bmwg-conterm-05.txt>, revised to reflect IESG input, WG LC
I-D Last Call
<draft-ietf-bmwg-fib-meth-01.txt>, Call ended 3/14.
<draft-ietf-bmwg-dsmterm-07.txt>, Call ended 7/15 with comment.
<draft-ietf-bmwg-mcastm-13.txt>, Call ended 7/5, to ADs soon
<draft-ietf-bmwg-ospfconv-term-05.txt>, Call ended 7/1 with
<draft-ietf-bmwg-ospfconv-intraarea-04.txt>, Call ended 7/1
<draft-ietf-bmwg-ospfconv-applicability-03.txt> Call ended 7/1
<draft-ietf-bmwg-dsmmeth-00.txt>, coming soon
<draft-ietf-bmwg-ipsec-term-01.txt>(draft 07/2003, need readers)
<draft-ietf-bmwg-benchres-term-03.txt>, back in WG
<draft-ietf-bmwg-acc-bench-term-00.txt> New 06/2003
<draft-ietf-bmwg-igp-dataplane-conv-term-00.txt> New 06/2003
<draft-ietf-bmwg-igp-dataplane-conv-meth-00.txt> New 06/2003
<draft-ietf-bmwg-igp-dataplane-conv-app-00.txt> New 06/2003
2. Discussion on OSPF Convergence Benchmarking WG Last Call
comments. Last call is over, but a few comments need discussion.
Identification of remaining issues. (Morton)
There was one remaining comment from the Last Call ending July 1, asking to
remove definitions of "Internal Measurements" and "External
Measurements". Following a short discussion, the commentor
(S.Proetsky) agreed that the definitions should remain, but that the terms
most frequently used in BMWG drafts and discussion are "White Box" and
"Black Box" measurements. The term names should be changed to add the
"Boxes" as synonyms and any resulting changes to the definition text. This
potential resolution will be discussed with authors and on the list.
3. Benchmarking Network-layer Traffic Control Mechanisms
Terminology -- (J. Perser et al.)
Jerry Perser communicated the latest changes and offered one last chance to
comment during WG Last Call. There was one comment on the "Channel
Capacity" term and its definition from Tony DeLaRosa. This comment was
resolved during a face-face meeting later in the week.
Also, Future Draft on Benchmarking Traffic Control Mechanisms
Scott Poretsky gave a brief introduction and identified areas for
comment and contribution during development. The purpose statement is to
provide methodologies to benchmark devices "capable of delivering the
specific packet forwarding treatment indicated by the DS field value"
[RFC2474]. The Benchmark is a Devices ability to meet configured PHBs
(the Expected Vector), not on specific QoS mechanisms. This makes it
possible to compare performance of routers implementing different
mechanisms. A short discussion revealed that measurement time scale will be
important for EF evaluation.
4. Terminology for Benchmarking BGP convergence in the Control Plane
This draft was recently revised, and there will be a short WG Last Call
before returning it to the ADs.
5. IPsec Device Benchmarking Terminology I-D -- (M. Kaeo, et al.)
The text of IPsec Terminology was completed in this version (01). Merike
asked BMWG if there were any terms missing, any problems with
Definitions, and if the group believed this draft was ready for WG Last
Call. There were several readers: one offering an endorsement, and two
questions on the scope (NAT devices and IKEv2 are out of scope, however
NAT-Transversal is included in the scope of this effort). The current
draft defined IMIX (Internet Mix for traffic synthesis), but did not
include the reference that Michele Bustos provided on the list. There was a
comment suggesting that this is a good definition to include, with the
caveat that no one mix of packet sizes represents Internet usage. There
will be a revision to make these points clear, and then a WG Last Call.
6. IGP Data plane convergence benchmark I-Ds -- (S.Poretsky)
Scott Poretsky gave a brief introduction of the new work item. All three
drafts are progressing with comments. There were several bmwg-list
suggestions for terminology changes, and they were illustrated in the
viewgraphs. Discussion of the methodology identified the need for a note to
cover forwarding rate change after convergence, and another suggested
specific reporting formats that remove the effect of delay timers. There was
also a discussion of measurement sampling rate and its affect on the
accuracy of loss-based convergence time measurements when
convergence approaches the sampling interval.
Tuesday, July 15, 2003, 1415-1515
7. Resource Reservation Terminology benchmark I-Ds -- (Fehér Gábor Gume)
Description of changes based on AD review, primarily to remove Diffserv
material, sync-up with current NSIS drafts, and remove non-standard
examples (e.g., Boomerang).There were still a few places where comments
could be more thoroughly adopted, and the draft will be revised again
shortly. Gábor made an appeal for more readership and review on
8. Terminology for Benchmarking Core Router Accelerated Life Testing. (S.
Poretsky et al.).
Comments asked for examples of Management Plane Failures, and
identified problems with allowing complete flexibility in test
configurations (results comparisons will likely be confounded by
differences). The level of readership and review is not yet clear for this
9. Proposed Milestone Revisions -- (Morton)
The review of milestones revealed a growing list, and a need to achieve
"Done" on a few more before adding new work. Editors and list members
should comment on this proposal (on the list).
Done First I-D on IPsec Device Benchmarking Terminology
Aug 03 Methodology for IP Multicast Benchmarking to AD Review.
Nov 03 Resource Reservation Benchmarking Terminology to AD Review
Aug 03 Net Traffic Control Benchmarking Terminology to AD Review
Dec 03 Methodology for FIB related Router Performance
Benchmarking to AD review.
Dec 03 EGP Convergence Benchmarking Terminology to AD Review
Mar 04 Resource Reservation Benchmarking Methodology to AD Review
Dec 03 Basic BGP Convergence Benchmarking Methodology to AD Review.
Dec 03 Net Traffic Control Benchmarking Methodology to AD Review.
Dec 03 IPsec Device Benchmarking Terminology to AD Review
Dec 03 AD review on IGP/Data-Plane Terminology I-D
Mar 04 AD review on IGP/Data-Plane Methodology and
Nov 03 AD review on Router Accelerated Test Terminology I-D
Jul 04 AD review on Router Accelerated Test Methodology and
10. New work proposal Automatic Protection Switching Benchmark
Terminology (T.Kimura, J.Perser)
Related Individual Draft on MPLS Protection Benchmarking
Methodology (S.Poretsky, et al.)
The Goal has been articulated as follows:
The objective of this effort is to produce a terminology and
methodology set of drafts that specifies the performance
benchmarking sub-IP layer resiliency and protection technologies. There is a
common terminology draft and multiple methodology drafts for the
technologies. The methodology drafts will include (but not limited to)
Automatic Protection Switching (APS) for SONET/SDH, Fast Reroute for
Multi-Protocol Label Switching (MPLS), and Resilient Packet Ring (RPR)
standardized in IEEE.
Discussion revealed that the previous direction to have a common
terminology draft for this work will be stretched by the plethora of Layer 2
terms, and so this issue must be worked further to see if a common
terminology is possible. The best way forward may be to link the
technology-specific terms to the appropriate IP layer terms. Recovery time is
key concept: accurate measurement takes all 5 recovery types into
account: Lost packets, Induced delays, Duplicate packets,
Out-of-order packets, and Errored packets. Clearly, measuring packet loss
alone to assess Recovery Time will not be sufficient in some cases. The
general assumption here is that the protection recovery times will be
sufficiently fast to avoid triggering IGP reconvergence, in response to
Craig White's inquiry on this interaction. Craig also commented that other
transport components interact with these protection mechanisms, and Tony
DeLaRosa added that a System Under Test may be needed to accommodate this
point. Matt Mathis suggested using general terms for the paths, such as
primary and secondary (and tertiary), and that we should invent some term
for the triggered failure, like "induced failure." Also, a generic IP
Protection Methodology may be useful to benchmark unspecified
protection mechanisms. The discussion concluded that the proposal
requires more work before review by the WG.