BMWG,

When a BMWG Draft begins the Last Call Process, the WG-level
quality of the draft improves when there is an active
review, as opposed to a tacit approval or minimal discussion.
To this end, we are going to try adding a step in the Last Call
Process for new drafts entering this phase of development.
We are defining the "Last Call Process" as one or more WG Last Calls
prior to forwarding the draft(s) up to AD-level review.
We alerted BMWG to the need for more active review at the IETF-58,
as captured in the minutes from that meeting.

The primary change is a requirement that approximately four WG members
complete a review template for the draft (or set of drafts),
upon entry to the Last Call Process. The WG reviewers should be
outside the body of active contributors (editors + authors).
The goal is to produce a quality review with valuable feedback
on the draft(s), and not to set a minimum number of people willing
to complete the review template. Directed feedback,
regardless of quantity, is a better condition by which BMWG can
more clearly assess a draft's readiness to advance.

The review template is appended below. Reviewers will complete
the template and send it to the list. Ideally, the reviewers
will have varied backgrounds, perspectives, and expertise
(test/network equipment developer/designer, network operator,
and subject matter expert). The chairs will ask for volunteers
and also do some recruiting for reviewers.

The use of reviewers does NOT SUPPLANT the need for the
general BMWG community to review and to comment on WG deliverables.
It is hoped that use of directed reviews AND community input will
improve the quality of BMWG work items. We also seek to increase our
WG "throughput", and make more room for new work proposals.

In BMWG's recent history, some drafts have spent several cycles
in the WG Last Call Process.  Our intent is to make substantial
improvements to the draft(s) early in the process (when such
changes are needed).  Therefore, subsequent WG Last Calls would
proceed as they do today, but there should be fewer cycles needed.

Our plan is to take one pass through this process, then
take a WG-consensus as to whether it is worth continuing, possibly
with modifications, or just return to the free-form Last Call.

We hope that you will make the time to participate
in the review program, whether you volunteer or respond
to a request to help.

Kevin, Al
BMWG Co-Chairs


Last Call Review Template

I-D Title(s):
Filename(s):
Reviewer Name:
Date:

Please organize your comments in the following categories below.

Review Summary:

Overall:

    * Does/Do the draft(s) provide clear identification of the
      scope of work? E.g., is the class of device, system, or
      service being characterized clearly articulated.


    * If a terminology memo, are the measurement areas clearly
      defined or otherwise cited?  Is the working set of
      supporting terminology sufficient and correct?  To your
      knowledge, are the areas of the memo that may conflict
      with other bodies of work? Are there any measurements or
      terminology that are superfluous?  Are any missing?


    * If a methodology memo, does the methodology AND its
      corresponding terminology adequately define a benchmarking
      solution for its application area? Do the methodologies present
      sufficient detail for the experimental control of the benchmarks?


    * If neither a terminology or methodology, does the offered
      memo offer complementary information important to the use
      or application of the related benchmarking solution?


    * Do you feel there are undocumented limitations or caveats to
      the benchmarking solution being proposed?  If so, please
      describe.


    * Does the memo attempt to define acceptance criteria for
      any of the benchmark areas?


Technical Content:  (Accuracy, Completeness of coverage)

    Are definitions accurate? Is the terminology offered relevant?


    To your knowledge, are there technical areas that are erroneous?
    Are there questionable technical areas that need to be re-examined
    or otherwise scrutinized.


    Does the solution adequately address IPv6?


    Do you feel the memo(s) being offered are technically mature enough
    for advancement to informational RFC?



Clarity and Utility:

   If you had a need, would you utilize the benchmarking solutions
   advocated by this and its related memos?  If not, why?


Conformance to BMWG principles: (see BMWG charter)
http://www.ietf.cnri.reston.va.us/html.charters/bmwg-charter.html


   Do you have confidence that the benchmarks, as explicitly
   defined, will yield consistent results if repeated on the
   same device (DUT/SUT), multiple times for a given test condition.
   If not, cite benchmark(s) and issue(s).


   Do you have confidence that the benchmarks, if executed for a
   given test condition, utilizing the documented methodology
   on multiple test infrastructure (e.g., test equipment), would
   yield correct and consistent results on the same DUT/SUT?
   (Said differently, are the benchmark's methodology written
   with enough exacting detail, that benchmark implementation
   differences do not yield a difference in the measured quantities?)
   If not, cite benchmark(s) and issue(s).


   Do you feel that the benchmarks form a basis of comparison between
   implementations of quantity being characterized? (I.e., are the
   benchmarks suitable for comparing solutions from different vendors.)


   If not, cite benchmarks and issues.


   For those benchmarks cited above, do you feel that the benchmarks,
   as specified, have universal applicability for the given
   behavior being characterized?  (i.e., benchmarks might not form
   a basis for cross-vendor comparison, can be used universally
   in a different role.)



Editorial Comments:
(includes any deficiencies noted w.r.t. I-D Nits, spelling, & grammar)



_______________________________________________
bmwg mailing list
bmwg@ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg