-----Original Message-----
From: Poretsky, Scott 
Sent: Saturday, October 21, 2006 7:43 AM
To: 'Tom Alexander'; shankar.rao@qwest.com
Cc: 'Al Morton'
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts

Thanks Tom!  Responses for the comments on the Terminology are below.

Scott
####################
------------------------------------------------------------------------

1. Introduction

   Routers in an operational network are simultaneously configured with 
   multiple protocols and security policies while forwarding traffic and

   being managed.  To accurately benchmark a router for deployment it is

   necessary to test that router in operational conditions by 
   simultaneously configuring and scaling network protocols and security

   policies, forwarding traffic, and managing the device.  It is helpful

   to accelerate these network operational conditions so that the 
   router under test can be benchmarked with faster test duration.  
   Testing a router in accelerated network conditions is known as 
   Accelerated Stress Benchmarking.  

[Tom A: The above paragraph is essential to understanding the rationale
for the rest of the draft, but is somewhat confusing. Suggest rewriting
above paragraph as:
"Routers in an operational network are configured with multiple
protocols
and security policies while simultaneously forwarding traffic and 
being managed.  To accurately benchmark a router prior to deployment, it
is necessary to test that router under operational conditions by 
configuring and scaling network protocols and security policies, and
simultaneously forwarding traffic as well as managing the device.  

Scott> Added the 3 words changes above

[Tom A:
configuration and management represent relatively infrequent activities
during actual operation 

Scott> This statement is incorrect.

[Tom A:
(compared to traffic forwarding), it is useful
to artificially overstress (accelerate) these network operational
conditions so that the router under test can be benchmarked with lower
test durations.  

Scott> Partially agree.  Now using the sentence "It is useful to
accelerate these network operational conditions so that the router under
test can be benchmarked with a shorter test duration."


------------------------------------------------------------------------

   This document provides the Terminology for performing Stress 
   Benchmarking of networking devices.  The three phases of the Stress 
   Test: Startup, Instability and Recovery are defined along with the
   benchmark and configuration terms associated with the each phase.
   Benchmarks for stress testing are defined using the Aggregate 
   Forwarding Rate and control plane Session Count during each phase 
   of the test.  For each plane, the Configuration Set, Startup 
   Conditions, and Instability Conditions are defined.  Also defined are

   the Benchmark Planes fundamental to stress testing configuration, 
   setup and measurement.  These are the Control Plane, Data Plane, 
   Management Plane and Security Plane  Multiple benchmarks are made 
   for each Benchmark Plane during each Phase.  Benchmarks can be 
   compared across multiple planes for the same DUT or at the same 
   plane for 2 or more DUTS.  These benchmarks White Box benchmarks 
   are provided in Appendix 1 for additional DUT behavior 
   measurements.  The terminology is to be used with the companion 
   methodology document [4].  The sequence of phases, actions, and 
   benchmarks are shown in Table 1.

[Tom A: Suggest rewording above sentence beginning with "These
benchmarks
White Box benchmarks ..." as follows:
"Benchmarks of internal DUT characteristics such as memory and CPU
utilization (also known as White Box benchmarks) are described in
Appendix 1, to allow additional characterization of DUT behavior."]

Scott> Agree
------------------------------------------------------------------------

   Table 1. Phase Sequence and Benchmarks 
   III. Recovery Phase   II. Instability Phase    I. Startup Phase
   <-----------------<---<-------------------<----<--------------<
    Remove Instability   Achieve Configuration    Apply Startup
    Conditions           Set                      Conditions

    Benchmark:           Benchmark:               Benchmark:
    Recovered Aggregate  Unstable Aggregate       Stable Aggregate 
    Forwarding Rate      Forwarding Rate          Forwarding Rate

                         Degraded Aggregate 
                         Forwarding Rate          

                         Average Degraded 
                         Forwarding Rate

    Recovered Latency    Unstable Latency         Startup Latency

    Recovered Uncontrolled Recovered Uncontrolled Stable Session Count 
    Sessions Lost          Sessions Lost

    Recovery Time

[Tom A: Is there some reason why the above table reads from right to
left? It would be more logical for it to read from left to right.]

Scott> Yes.  The BMWG has discussed this on previous occasions.  Test 
Equipment displays time on a scale that scrolls right to left.  We 
want to match that.
 
------------------------------------------------------------------------

3. Term definitions
   3.1 General Terms
   3.1.1 Benchmark Planes

      Definition:  
      The features, conditions, and behavior for the Accelerated Stress 
      Benchmarking.

      Discussion:
      There are four Benchmark Planes: Control Plane, Data Plane, 
      Management Plane, and Security Plane as shown in Figure 1. The
      Benchmark Planes define the Configuration, Startup Conditions, 
      Instability Conditions, and Failure Conditions used for the test.

[Tom A: The last sentence does not make sense. Benchmark planes cannot
define the conditions; instead, conditions are defined for each plane.
Suggest rewording as:
"Configuration, Startup Conditions, Instability Conditions, and
Failure Conditions used for each test are defined for each of these
four Benchmark Planes."]

Scott> Agree

------------------------------------------------------------------------

    3.1.2 Configuration Sets

      Definition:  
        The features and scaling limits used during the Accelerated
Stress 
        Benchmarking.

      Discussion:
        There are four Configuration Sets: Control Plane Configuration
Set, 
        Data Plane Configuration Set, Management Plane Configuration
Set, 
        and Security Plane Configuration Set. 

[Tom A: The above discussion does not help me understand what a
Configuration Set is. Please consider adding text to clarify. A
couple of examples of Configuration Sets would be MOST helpful.]

Scott> Examples are in the methodology.  Al requested to have one
mandatory minimum configuration set.  I will add that here to also 
satisfy your request for an example.

------------------------------------------------------------------------

   3.1.6 Controlled Session Loss

     Definition: 
        Control Plane sessions that are intentionally brought
        down during the Stress test.

      Discussion:
        The test equipment is able to control protocol 
        session state with the DUT.

[Tom A: The above discussion specifies some capability of the
test equipment, and does not clarify what Controlled Session
Loss is. Suggest rewording as follows:
"Controlled Session Loss is performed during the test in order
to stress the DUT by forcing it to tear down Control Plane
sessions while handling traffic. It is assumed that the test
equipment is able to control protocol session state with the
DUT and is therefore able to introduce Controlled Session Loss."]

Scott> Agree
------------------------------------------------------------------------
 
   3.2.3 Management Plane

      Definition:
        The Management features and tools used for the 
        Accelerated Stress Benchmarking.


Poretsky and Rao                                               [Page 8]
 
   
INTERNET-DRAFT        Terminology for Accelerated          June 2006
                         Stress Benchmarking  

      Discussion:
        A key component of the Accelerated Stress Benchmarking is the 
        Management Plane to assess manageability of the router 
        under stress.  The Management Plane defines the Configuration, 
        Startup Conditions, and Instability Conditions of the 
        management protocols and features.  The Management Plane 
        includes SNMP, Logging/Debug, Statistics Collection, and 
        management configuration sessions such as telnet, SSH, and 
        serial console.  SNMP Gets SHOULD be performed continuously.  
        Management configuration sessions should be open 
        simultaneously and be repeatedly open and closed.  Open 
        management sessions should have valid and invalid 
        configuration and show commands entered.

[Tom A: It is difficult, in the above discussion, to determine where
terminology ends and methodology takes up. It appears that the
discussion is trying to specify methodology together with terminology.
Suggest moving the last 2 sentences (beginning with "SNMP Gets ..."
into the methodology.]

Scott> Agree

------------------------------------------------------------------------

   3.2.4 Security Plane

      Definition:
        The Security features used during the Accelerated Stress 
        Benchmarking.

      Discussion:
        The Security Plane defines the Configuration, Startup 
        Conditions, and Instability Conditions of the security 
        features and protocols.  The Security Plane includes the 
        ACLs, Firewall, Secure Protocols, and User Login.  Tunnels 
        for those such as IPsec should be established and flapped.  
        Policies for Firewalls and ACLs should be repeatedly added 
        and removed from the configuration via telnet, SSH, or 
        serial management sessions.

[Tom A: Same issue as for Management Plane: the above discussion
mixes terminology and methodology. Suggest moving the last 2
sentences (starting with "Tunnels for ...") to the methodology.

Scott> Agree.
------------------------------------------------------------------------

  3.3.5.1 Management Plane Configuration Set

      Definition:
        The router management features enabled for the 
        Accelerated Stress Benchmark.

      Discussion:
        A key component of the Accelerated Stress Benchmark is the 
        Management Configuration Set to assess manageability of the 
        router under stress.  The Management Configuration Set defines 
        the management configuration of the DUT.  Features that are 
        part of the Management Configuration Set include access, SNMP, 
        Logging/Debug, and Statistics Collection, and services such as 
        FTP, as shown in Figure 3. These features should be enabled
        throughout the Stress test. SNMP Gets should be made 
        continuously with multiple FTP and Telnet sessions operating 
        simultaneously.  FTP sessions should be opened and closed at 
        varying intervals and get and put files while open.  Telnet 
        sessions should be opened and closed at varying intervals and 
        enter valid and invalid show and configuration commands while 
        open.

[Tom A: Same issue as before - methodology is mixed in with the
terminology. Suggest moving the last 4 sentences (starting with
"These features should be enabled ...") to the methodology.]

Scott> Agree. 
------------------------------------------------------------------------

Appendix 1.  White Box Benchmarking Terminology
   Minimum Available Memory
      Definition: 
        Minimum DUT Available Memory during the duration of the 
        Accelerated Stress Benchmark.   

      Discussion:
        It is necessary to monitor DUT memory to measure this 
        benchmark.                      

[Tom A: It is nice that one has to monitor DUT memory to measure
this benchmark, but what is the significance of this benchmark?
Please explain. Even a sentence like "This benchmark enables the
assessment of reserve capacity in the DUT, as well as capacity
degradation over repeated trials" would be fine.]

Scott>  Used the sentences, "This benchmark enables the assessment of
resources in the DUT.  It is necessary to monitor DUT memory to measure
this benchmark."
Please note that when benchmarking "Minimum Available Memory" it is
necessary
to monitor DUT memory .  			
------------------------------------------------------------------------

   Maximum CPU Utilization
      Definition: 
        Maximum DUT CPU utilization during the duration of the 
        Accelerated Stress Benchmark.
        
      Discussion:
        It is necessary to monitor DUT CPU Utilization to measure
        this benchmark.         

[Tom A: Same comment as for previous White Box benchmark.]

Scott> Used the sentences, "	This benchmark enables the assessment of

resources in the DUT.  It is necessary to monitor DUT CPU Utilization 
to measure this benchmark."  Please note that when benchmarking "Maximum
CPU Utilization" it is necessary to monitor the DUT CPU.  



-----Original Message-----
From: Tom Alexander [mailto:tom@veriwave.com] 
Sent: Saturday, October 21, 2006 3:47 AM
To: Poretsky, Scott; shankar.rao@qwest.com
Cc: 'Al Morton'
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts

Scott, Shankar,

I read the accelerated benchmarking drafts and have some comments, in
the
enclosed files. As I'm a relative newbie in BMWG I didn't feel
comfortable
sending them to the whole list (when you're about to put your foot in
your
mouth, keeping the audience small is a good idea!) so I'm sending them
directly to you instead.

Note that I've excerpted pieces of the drafts and added my comments
below,
using the general syntax "[Tom A. -  ..... ]".

Best regards,

- Tom


_______________________________________________
bmwg mailing list
bmwg@ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg