[bmwg] Review: WGLC: draft-ietf-bmwg-mpls-forwarding-meth-00

["Rajiv Papneja" <rpapneja at isocore.com>]

Hello Al,

Here is my review in response to the BMWG initial WGLC for MPLS Forwarding
Benchmarking Methodology co-authored by Aamer Akhter and Rajiv Asati. 

I-D Title(s): MPLS Forwarding Benchmarking Methodology
Filename(s): draft-ietf-bmwg-mpls-forwarding-meth-00.txt
Reviewer Name: Rajiv Papneja
Date: October 13, 2008


Review Summary:

Let me start by saying that since inception of this effort, I have been
supporter of this draft as it provides guidelines for test engineers for
measuring the MPLS Forwarding performance within the bounds of RFC 2544.  


Overall:

   * Does/Do the draft(s) provide clear identification of the
     scope of work? E.g., is the class of device, system, or
     service being characterized clearly articulated.

Yes, I believe so. It does define the scope of work, and also relates it as
an extension of the RFC 2544. 

   * If a terminology memo, are the measurement areas clearly
     defined or otherwise cited?  Is the working set of
     supporting terminology sufficient and correct?  To your
     knowledge, are the areas of the memo that may conflict
     with other bodies of work? Are there any measurements or
     terminology that are superfluous?  Are any missing?


   * If a methodology memo, does the methodology AND its
     corresponding terminology adequately define a benchmarking
     solution for its application area? Do the methodologies present
     sufficient detail for the experimental control of the benchmarks?

Test methodology section suggests a common topology to be used. Recommends
test considerations including the initial conditions before starting the
test. Also defines the control plane requirements before forwarding tests
could be executed, for example support of IGP, and label distribution
mechanisms. It is very important for the consistency of the test results
that the initial conditions are correctly met. The methodology provides
recommended values frame sizes, trial run time, and TTL handling.  

   * If neither a terminology or methodology, does the offered
     memo offer complementary information important to the use
     or application of the related benchmarking solution?


   * Do you feel there are undocumented limitations or caveats to
     the benchmarking solution being proposed?  If so, please
     describe.

I do not believe so.

   * Does the memo attempt to define acceptance criteria for
     any of the benchmark areas?

There is no acceptance criterion. However, this benchmark presents a
methodology for comparing performance of various MPLS enabled forwarding
devices.

Technical Content:  (Accuracy, Completeness of coverage)

   Are definitions accurate? Is the terminology offered relevant?

Yes, and are elaborated where necessary.


   To your knowledge, are there technical areas that are erroneous?
   Are there questionable technical areas that need to be re-examined
   or otherwise scrutinized.

I do not think so.

   Does the solution adequately address IPv6?

NA

   Do you feel the memo(s) being offered are technically mature enough
   for advancement to informational RFC?


I believe so.

Clarity and Utility:

  If you had a need, would you utilize the benchmarking solutions
  advocated by this and its related memos?  If not, why?

Yes, we have already used these in some of our tests

Conformance to BMWG principles: (see BMWG charter)
http://www.ietf.cnri.reston.va.us/html.charters/bmwg-charter.html


  Do you have confidence that the benchmarks, as explicitly
  defined, will yield consistent results if repeated on the
  same device (DUT/SUT), multiple times for a given test condition.
  If not, cite benchmark(s) and issue(s).

As noted earlier, the initial conditions and requirements are laid out
clearly in the document. I believe that the results obtained using the
benchmarks defined would yield consistent results.

  Do you have confidence that the benchmarks, if executed for a
  given test condition, utilizing the documented methodology
  on multiple test infrastructure (e.g., test equipment), would
  yield correct and consistent results on the same DUT/SUT?
  (Said differently, are the benchmark's methodology written
  with enough exacting detail, that benchmark implementation
  differences do not yield a difference in the measured quantities?)
  If not, cite benchmark(s) and issue(s).

Yes

  Do you feel that the benchmarks form a basis of comparison between
  implementations of quantity being characterized? (I.e., are the
  benchmarks suitable for comparing solutions from different vendors.)

Yes

  If not, cite benchmarks and issues.


  For those benchmarks cited above, do you feel that the benchmarks,
  as specified, have universal applicability for the given
  behavior being characterized?  (i.e., benchmarks might not form
  a basis for cross-vendor comparison, can be used universally
  in a different role.)


Yes


_______________________________________________
bmwg mailing list
bmwg at ietf.org
https://www.ietf.org/mailman/listinfo/bmwg