[bmwg] comments on bmwg-acc-bench-term-12

[Al Morton <acmorton at att.com>]

Scott, Shankar,

Here are some comments on the bmwg-acc-bench-term-12 draft.

Al
(as bmwg participant)

General: The most important comment is at the end!

First Line:
The IPR statement should be indented 3 spaces, like the
rest of the text in the draft.  This comment applies to
*lots* of drafts where Scott is a co-author...

S 1 2nd para
"...These are the Control Plane, Data Plane,
Management Plane and Security Plane  Multiple benchmarks are measured ..."
                    missing period ^

S 2, Table 1

   III. Recovery Phase   II. Instability Phase    I. Startup Phase
   <-----------------<---<-------------------<----<--------------<
    Remove Instability   Achieve Configuration    Apply Startup
    Conditions           Set "and Apply            Conditions
Add - - - - - - - - - >  Instability Conditions"
words in quotes to Phase II


S 3.1.1
Some lines in Figure 1 print-out to the left of their intended
positions.  I know, it looks fine on the screen...

S 3.1.2
OLD 3.1.2 Configuration Sets
      Definition:
        The offered load, features, and scaling limits used during the
        Accelerated Stress Benchmarking.
NEW
        The specifications of the offered load, features, and ...
            ^^^^^^^^^^^^^^^^^^^^^

S 3.1.3
...
      Discussion:
        Startup Conditions may cause stress on the DUT and produce
        failure.  ...
Comment:  Really?  Why would the Startup cause failures?
How would we measure the "stable aggregate forwarding rate" when
things are failing?

It seems to me that if the startup conditions cause failure,
then they are too stressful for a particular DUT and the failure
should be noted.  But, I don't think you can proceed on to
add more Instability conditions, you don't have a stable baseline.

I agree that this is a possible outcome during the benchmarking,
but it needs to treated differently than other cases, IMO.
This single sentence noting the possibility doesn't seem to
deal with the possibility satisfactorily.

S 3.1.4
Suggest rewording the definition and discussion as follows:

   3.1.4 Instability Conditions

      Definition:
        Stressful test conditions that are introduced during the
        Accelerated Stress Benchmark to produce instability and
        stress the DUT.

      Discussion:
        Instability Conditions are applied to the DUT after it has
        been benchmarked under the Startup Conditions.  Instability
        Conditions occur for the Control Plane, Data Plane, Management Plane,
        and Security Plane.

S 3.1.5
Suggest renaming this term:
   3.1.5 Stable Aggregate Forwarding Rate
         ^^^^^^
     Definition:
        Sum of forwarding rates for all interfaces on the
        DUT during the Startup Phase.
(because this term applies to the start-up phase)

S 3.1.6
Suggest renaming this term:
   3.1.6 Discontinued Sessions  (was: Controlled Session Loss)

     Definition:
        Control Plane sessions that are intentionally brought
        down during the Stress test.

Because "loss" sounds unintentional, as in packet loss.

S 3.3.1
   3.3.1 Startup Phase

     Definition
        The portion of the benchmarking test in which the
        Startup Conditions are generated with the DUT.  This
        begins with the attempt to establish the first session
        and ends when the last Control Plane session is
        established.
...
     Issues:
        The 'last control plane session is established' may not
        be a sufficient indicator that steady-state is achieved
        and Instability Conditions can be applied to begin the
        Instability Phase.

If this Issue is true, then the Startup phase doesn't have a reliable
end definition.

I think the entire Startup Phase needs to explicitly adopt the
notion that it is a steady/stable phase.  If stability cannot
be achieved during startup, then this invalid condition must be
noted and the process does not proceed.

So,  if this is agreeable to others, then there are probably lots
of places where this idea that "Startup is Stable" should be made
explicit in the draft.

(and that's where I've stopped in this review)


_______________________________________________
bmwg mailing list
bmwg at ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg