PMOL,
While reviewing the SIP metrics draft, I found a few places
where the framework draft needed further development.
I suggest some text in several sections below. I'm not
exactly sure where most of them fit in the outline, but
section 4.2 is strawman text for an empty section.
Al
(as a participant)
3.x. Organization of Results
The IPPM Framework [RFC2330] organizes the results of metrics into
three related notions:
1. singleton, an elementary instance, or "atomic" value
2. sample, a set of singletons with some common properties and some
varying properties.
3. statistic, a value derived from a sample through deterministic
calculation, such as the mean.
Many metrics can use this organization for the results, with or
without the term names used by IPPM working group. Section 11 of
[RFC2330] should consulted for further details.
3.y. Parameters, the variables of a metric
Metrics are completely defined when all options and input variables
have been identified and considered. These variables are sometimes
left unspecified in a metric definition, and their general name
indicates that the user must set them and report them with the
results. Such variables are called "parameters" in the IPPM metric
template. The scope of the metric, the time at which it was
conducted, the settings for timers and the thresholds for counters
are all examples of parameters.
All memos defining performance metrics SHOULD identify ALL key
parameters for each metric.
3.z. Packet Measurement Details
Another key aspect of the IPPM Framework [RFC2330] is the discussion
of "wire time". Internet hosts require physical resources to observe
packet traffic, and they can contribute to the performance observed.
Further, the observations SHOULD be related to some instant of time
(such as a specific bit of a packet), because a packet contains many
bits and exists on physical media long enough to introduce some
ambiguity. Section 10.2 of [RFC2330] should be consulted.
3.a. Identifying and Categorizing the Audience
Many of the aspects of metric definition and reporting, even the
selection or determination of the essential metrics, depend on who
will use the results, and for what purpose. The question, "How will
the results be used?" usually yields important factors to consider
when developing performance metrics.
All memos defining performance metrics SHOULD identify the primary
audience and its associated requirements. The audience can influence
both the definition of metrics and the methods of measurement.
The key areas of variation between different metric users include:
o Suitability of passive measurements of live traffic, or active
measurements using dedicated traffic
o Measurement in laboratory environment, or on a network of deployed
devices
o Accuracy of the results
o Access to measurement points and configuration information
o Measurement topology (point-to-point, point-to-multipoint)
o Scale of the measurement system
o Measurements conducted on-demand, or continuously
o Required reporting formats
4.2. Proposal Approval
The process depends on the form that the PM Entity ultimately takes,
Directorate or working group.
In all cases, the proposal will need to achieve consensus, in the
corresponding protocol development working group (or alternatively,
an "Area" working group with broad charter), that there is interest
and a need for the work.
IF the PM Entity is a Directorate,
THEN Approval SHOULD include the following steps
o consultation with the PM Directorate, using this framework memo
o consultation with Area Director(s)
o and possibly IESG approval of a new or revised charter for the
working group
IF the PM Entity is a Working Group, and the protocol development
working group decides to take up the work under its charter,
THEN the approval is the same as the PM Directorate steps above, with
the possible additional assignment of a PM Advisor for the work item.
IF the PM Entity is a Working Group, and the protocol development
working group decides it does not have sufficient expertise to
take-up the work, or the proposal falls outside the current charter,
THEN
Approval SHOULD include the following steps
o identification of protocol expertise to support metric development
o consensus in the PM working group that there is interest and a
need for the work, and that a memo conforming to this framework
can be successfully developed
o consultation with Area Director(s)
o IESG approval of a revised charter for the PM working group
_______________________________________________
PMOL mailing list
PMOL at ietf.org
http://www.ietf.org/mailman/listinfo/pmol