RE: [bmwg] comments on bmwg-acc-bench-term-12

["Poretsky, Scott" <sporetsky at reefpoint.com>]

Thanks Al!  This is an excellent point you raise since it is quite
possible that the conditions for the startup phase could produce failure
in the DUT.  In this case the test should not continue and the Startup
Configuration Set must be made less aggressive.  The documents will be
updated to reflect this.  Any further comments from you or other BMWG
participants would be most appreciated since we have completed a WGLC on
this work item.

Scott

-----Original Message-----
From: Al Morton [mailto:acmorton at att.com] 
Sent: Saturday, August 11, 2007 11:52 AM
To: bmwg at ietf.org
Subject: [bmwg] comments on bmwg-acc-bench-term-12

Scott, Shankar,

Here are some comments on the bmwg-acc-bench-term-12 draft.

Al
(as bmwg participant)

General: The most important comment is at the end!

First Line:
The IPR statement should be indented 3 spaces, like the
rest of the text in the draft.  This comment applies to
*lots* of drafts where Scott is a co-author...

S 1 2nd para
"...These are the Control Plane, Data Plane,
Management Plane and Security Plane  Multiple benchmarks are measured
..."
                     missing period ^

S 2, Table 1

    III. Recovery Phase   II. Instability Phase    I. Startup Phase
    <-----------------<---<-------------------<----<--------------<
     Remove Instability   Achieve Configuration    Apply Startup
     Conditions           Set "and Apply            Conditions
Add - - - - - - - - - >  Instability Conditions"
words in quotes to Phase II


S 3.1.1
Some lines in Figure 1 print-out to the left of their intended
positions.  I know, it looks fine on the screen...

S 3.1.2
OLD 3.1.2 Configuration Sets
       Definition:
         The offered load, features, and scaling limits used during the
         Accelerated Stress Benchmarking.
NEW
         The specifications of the offered load, features, and ...
             ^^^^^^^^^^^^^^^^^^^^^

S 3.1.3
...
       Discussion:
         Startup Conditions may cause stress on the DUT and produce
         failure.  ...
Comment:  Really?  Why would the Startup cause failures?
How would we measure the "stable aggregate forwarding rate" when
things are failing?

It seems to me that if the startup conditions cause failure,
then they are too stressful for a particular DUT and the failure
should be noted.  But, I don't think you can proceed on to
add more Instability conditions, you don't have a stable baseline.

I agree that this is a possible outcome during the benchmarking,
but it needs to treated differently than other cases, IMO.
This single sentence noting the possibility doesn't seem to
deal with the possibility satisfactorily.

S 3.1.4
Suggest rewording the definition and discussion as follows:

    3.1.4 Instability Conditions

       Definition:
         Stressful test conditions that are introduced during the
         Accelerated Stress Benchmark to produce instability and
         stress the DUT.

       Discussion:
         Instability Conditions are applied to the DUT after it has
         been benchmarked under the Startup Conditions.  Instability
         Conditions occur for the Control Plane, Data Plane, Management
Plane,
         and Security Plane.

S 3.1.5
Suggest renaming this term:
    3.1.5 Stable Aggregate Forwarding Rate
          ^^^^^^
      Definition:
         Sum of forwarding rates for all interfaces on the
         DUT during the Startup Phase.
(because this term applies to the start-up phase)

S 3.1.6
Suggest renaming this term:
    3.1.6 Discontinued Sessions  (was: Controlled Session Loss)

      Definition:
         Control Plane sessions that are intentionally brought
         down during the Stress test.

Because "loss" sounds unintentional, as in packet loss.

S 3.3.1
    3.3.1 Startup Phase

      Definition
         The portion of the benchmarking test in which the
         Startup Conditions are generated with the DUT.  This
         begins with the attempt to establish the first session
         and ends when the last Control Plane session is
         established.
...
      Issues:
         The 'last control plane session is established' may not
         be a sufficient indicator that steady-state is achieved
         and Instability Conditions can be applied to begin the
         Instability Phase.

If this Issue is true, then the Startup phase doesn't have a reliable
end definition.

I think the entire Startup Phase needs to explicitly adopt the
notion that it is a steady/stable phase.  If stability cannot
be achieved during startup, then this invalid condition must be
noted and the process does not proceed.

So,  if this is agreeable to others, then there are probably lots
of places where this idea that "Startup is Stable" should be made
explicit in the draft.

(and that's where I've stopped in this review)


_______________________________________________
bmwg mailing list
bmwg at ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
bmwg at ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg