[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [bmwg] comments on draft-ietf-bmwg-sip-bench-term-00
At 04:58 PM 5/6/2009, Al Morton wrote:
I have the following comments/suggestions on term-00.
Al
(as a bmwg participant)
<snip>
-------------------------------------------------------
I also have some comments to type up on section 3.4, ...
I'm interested in some feedback from the WG, especially the
author team, before proceeding to suggest more changes like
the ones below.
In section 3.4:
- There are several benchmarks which attempt to discern
the largest value from among a set of values.
- The values themselves may be time averages (attempts per second)
or simple counts, and metric definitions are needed to
produce the values.
IMO, the terms and definitions should clearly distinguish between
these two aspects, because it helps them to be clear and
unambiguous.
<rant>
When I step back for a moment and take a fresh look at term names,
it's always been a rough point that we have metrics with simple
names, like "Throughput", which is actually the maxima of
offered load under lossless conditions, obtained over multiple iterations.
It's really "THE Throughput Benchmark" [RFC1242, RFC2544].
Just saying "throughput" and ignoring all the other existing
definitions of this term can certainly lead to confusion
beyond bmwg-list, or beyond the benchmarking community in general.
We might avoid some confusion with SIP device benchmarking now,
if we try. We might also produce some SIP metrics that are useful
*beyond benchmarking*.
</rant>
So, I propose to organize all of section 3.4 to have
pairs of sections: a metric definition, and then the benchmark
based on that metric.
OLD
3.4 Benchmarks
3.4.1. Registration Rate
Definition:
The maximum number of registrations that can be successfully
completed by the DUT/SUT in a given time period.
Discussion:
This benchmark is obtained with zero failure in which 100% of the
registrations attempted by the Emulated Agent are successfully
completed by the DUT/SUT. The maximum value is obtained by
testing to failure. This means that the registration rate
provisioned on the EA is raised progressively until a registration
attempt failure is observed.
Measurement Units:
registrations per second (rps)
Issues:
None.
See Also:
SUGGEST:
3.4 Metrics and Benchmarks
3.4.1. Registration Rate
Definition:
The number of registrations successfully
established by the DUT/SUT in a given time period.
^^^^^^^^^^^
Discussion:
| The time period is defined at the EA as the time basis
| for outgoing registrations that are eventually successful.
| However, time is also needed for the EA to evaluate the success
| or failure of each registration attempt, thus there are really
| two time intervals involved.
Measurement Units:
registrations per second (rps)
Issues:
None.
See Also:
NEW:
3.4.2. Maximum Registration Rate Benchmark
Definition:
The maximum Registration Rate accompanied by zero Ineffective
Registration Attempts [Malas] in a given time period.
Discussion:
This benchmark is obtained with zero failure in which 100% of the
registrations attempted by the Emulated Agent are successfully
completed by the DUT/SUT. The maximum value is obtained by
testing to failure over multiple trials at fixed Registration Rates.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This means that the registration rate
provisioned on the EA is raised incrementally until a registration
^^^^^^^^^^^^^
attempt failure is observed, and then decremented to find the
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
highest successful Registration Rate in a repeatable trial.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Measurement Units:
registrations per second (rps)
Issues:
None.
See Also:
where [Malas] is draft-ietf-pmol-sip-perf-metrics-