[bmwg] Review/comments on version 2 of IPv6 benchmarking draft

["Bill Cerveny" <bill-bmwg at wjcerveny.com>]

Below are my comments on version 2 of the IPv6 Benchmarking Methodology
draft.  Most recommended changes are grammatical in nature.

Bill Cerveny

Last Call Review Template

I-D Title(s): IPv6 Benchmarking Methodology
Filename(s):draft-ietf-bmwg-meth-02.txt
Reviewer Name: Bill Cerveny
Date: August 10, 2007

Please organize your comments in the following categories below.

Review Summary:

Overall:

   * Does/Do the draft(s) provide clear identification of the
     scope of work? E.g., is the class of device, system, or
     service being characterized clearly articulated.

Yes


   * If a terminology memo, are the measurement areas clearly
     defined or otherwise cited?  

Yes

Is the working set of
     supporting terminology sufficient and correct?  
Yes

To your
     knowledge, are the areas of the memo that may conflict
     with other bodies of work? Are there any measurements or
     terminology that are superfluous?  Are any missing?

No
   * If a methodology memo, does the methodology AND its
     corresponding terminology adequately define a benchmarking
     solution for its application area? Do the methodologies present
     sufficient detail for the experimental control of the benchmarks?

Yes


   * If neither a terminology or methodology, does the offered
     memo offer complementary information important to the use
     or application of the related benchmarking solution?


   * Do you feel there are undocumented limitations or caveats to
     the benchmarking solution being proposed?  If so, please
     describe.

No


   * Does the memo attempt to define acceptance criteria for
     any of the benchmark areas?


Technical Content:  (Accuracy, Completeness of coverage)

   Are definitions accurate? Is the terminology offered relevant?

Yes

   To your knowledge, are there technical areas that are erroneous?
   Are there questionable technical areas that need to be re-examined
   or otherwise scrutinized.

Not to my knowledge

   Does the solution adequately address IPv6?

Yes


   Do you feel the memo(s) being offered are technically mature enough
   for advancement to informational RFC?

Yes


Clarity and Utility:

  If you had a need, would you utilize the benchmarking solutions
  advocated by this and its related memos?  If not, why?

Yes


Conformance to BMWG principles: (see BMWG charter)
http://www.ietf.cnri.reston.va.us/html.charters/bmwg-charter.html


  Do you have confidence that the benchmarks, as explicitly
  defined, will yield consistent results if repeated on the
  same device (DUT/SUT), multiple times for a given test condition.
  If not, cite benchmark(s) and issue(s).

Yes


  Do you have confidence that the benchmarks, if executed for a
  given test condition, utilizing the documented methodology
  on multiple test infrastructure (e.g., test equipment), would
  yield correct and consistent results on the same DUT/SUT?
  (Said differently, are the benchmark's methodology written
  with enough exacting detail, that benchmark implementation
  differences do not yield a difference in the measured quantities?)
  If not, cite benchmark(s) and issue(s).
 
Yes


  Do you feel that the benchmarks form a basis of comparison between
  implementations of quantity being characterized? (I.e., are the
  benchmarks suitable for comparing solutions from different vendors.)

Yes


  If not, cite benchmarks and issues.


  For those benchmarks cited above, do you feel that the benchmarks,
  as specified, have universal applicability for the given
  behavior being characterized?  (i.e., benchmarks might not form
  a basis for cross-vendor comparison, can be used universally
  in a different role.)



Editorial Comments:
(includes any deficiencies noted w.r.t. I-D Nits, spelling, & grammar)

Bill's preface: Some of the changes are merely my preference ... for
example I tried to delete use of words I thought were unnecessary, such
as "that" or "the"


Page header: Title of document is "IPv6 Benchmarking Methodology".
However, page headers say "IPv6 Performance Benchmarking"

Abstract:

Consider changing first sentence to something like (removed
capitalization,
reworded):

The benchmarking methodologies defined in RFC2544 [8] are IP version
independent.  However, RFC2544 does not address some of the unique
attributes
of IPv6.

Next sentence (added punctuation):

This document provides additional benchmarking guidelines, which in
conjunction with RFC2544, lead to a more complete and realistic
evaluation
of the IPv6 performance of network elements.

Section 1:

Second paragraph, last sentence (rewording, shortened)

These recommendations are defined within the RFC2544 framework and
complement the test and result analysis procedures as described in
RFC2544.

Section 3:

I think you need to change references to section 7, not section 6.

s/for each evaluated device it is/for each evaluated device, it is/
(punctuation)

s/Test execution and the results/Test execution and results/

Section 4:

s/are the same as the ones/are the same as those/

s/Single-port testing is used in measuring per interface/Single-port
testing measures per-interface/

s/multi-port testing is used to measure/multi-port testing measures/

s/data that might be recommended to be collected/data collected/

s/scenarios assume the test traffic/scenarios assume test traffic/

s/and the IPv6 source and destination/and IPv6 source and destination/

Section 5:

s/The traffic used/Traffic used/

s/traffic in all of its aspects/traffic/

The sentence "Ethernet in all of its types has become the most commonly
deployed
interface ..." seems to imply that Ethernet is an interface -- consider
rewording.

s/selection of addresses for the test traffic/selection of addresses for
test traffic/

s/into either one of the following/into either of the following/

s/additional data with respect to the performance/additional data
regarding the performance/

s/The IPv6 source and destination addresses for the test/IPv6 source and
destination addresses for test/

s/If the network element/If network element/

s/the tests SHOULD focus/tests SHOULD focus/

s/The IPv6 source and destination addresses/IPv6 source and destination
addresses/

s/They can be selected from/Extension header types can be selected from/

s/Encapsulation Security Payload header/Encapsulating security payload
(ESP) header/

s/Considering that extension/Since extension/

s/Unlike the other/Unlike other/

s/the DUT but/the DUT, but/

s/Hop-by-hop/hop-by-hop/ (multiple instances)

s/extension headers but/extension headers, but/

s/traffic with the Hop-by-hop/traffic with hop-by-hop/

s/The tests with traffic/Tests with traffic/

s/The chain should also exclude/This chain should exclude/

s/chain recommended to be used in testing/chain recommended for testing/

s/smallest frame size both types of traffic/smallest frame size of both
types of traffic/

s/it is likely that the/it is likely the/

s/extension headers related performance impact/performance impact
related to extension headers/

Section 6:

Consider rewording first paragraph, second sentence as follows:

The conditions defined in RFC2544 section 11 are common to IPv4 and
IPv6, except that IPv6
does not employ layer 2 or 3 broadcast frames.

lower-case: Frames -> frames

s/The filter definitions however must/The filter definitions must/

Regarding statement "which are an important architecture component of
IPv6." at end of last sentence
in section 6.2 ... hasn't this already been established earlier?

s/(DA) are IPv6./(DA) are IPv6 addresses/

s/must parse through the extension headers in order to reach/must parse
through extension headers to reach/

s/for example, the upper/for example, upper/

s/For these reasons, to/To/

New paragraph before "The upper layers protocols listed above ..."

s/recommended selection however they/recommended selection. However,
they/

Not sure if the below changes what you were attempting to say, but it is
unclear as is in my opinion:

Suggested changed first paragraph of 6.2.2:

Based on RFC2544 recomendations, two types of tests are executed when
evaluating performance in the presence of modifiers: One with a single
filter and another with 25 filters.  The example filters are illustrated
using the IPv6 documentation prefix [10]
2001:DB8::.

I'm puzzled about your heading of "Example of 25 filters" ... is it
intended there will be 25 filter lines in the final document?

Section 7:

Second paragraph, second sentence, consider changing to:

The tests recommended by RFC2544 MUST be repeated for both IPv6 traffic
without extension headers and for IPv6 traffic with one or multiple
extension headers. ("both" might
be deletable here)

Phrase "... performance of platforms forwarding both types of traffic."
Perhaps we will always be testing "forwarding" but would "processing" be
a more appropriate
word if there are cases where the DUT is not forwarding traffic??

s/it is important for IPv6 enabled platforms to not/it is important that
IPv6-enabled platforms not/

s/to cover the co-existence/to cover co-existence/

Sentence "A minimum requirement is to cover ..." seems really awkward to
me.  Consider rewording.

s/The subsequent sections describe each/Subsequent sections each
describe/

Lower-case: Back-to-Back -> back-to-back

Section 10:

s/This document is addressing/This document addresses/

_______________________________________________
bmwg mailing list
bmwg at ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg