Howdy.

I have been selected as the Applications Area Directorate reviewer for 
this draft (for background on appsdir, please see
 
http://trac.tools.ietf.org/area/app/trac/wiki/ApplicationsAreaDirectorate ).

Please resolve these comments along with any other Last Call comments 
you may receive. Please wait for direction from your document shepherd 
or AD before posting a new version of the draft.


Document:

    draft-ietf-bmwg-2544-as-03

Title:

    RFC 2544 Applicability Statement: Use on Production Networks
    Considered Harmful

Reviewer:

    D. Crocker <dcrocker@bbiw.net>

Review Date:

    15 May 2012

Summary:

    This draft seeks to constrain use of a testing specification.  It is 
motivated by an established need (problem).  The document is 
well-organized and generally clear about the issues it covers.  It could 
benefit from some minor refinement.

    The draft is explicitly in response to a pattern of misapplication 
of RFC 2544.  As such, I suggest it be tailored a bit for readers who 
need some extra help in getting the main points of the document.  The 
changes would be minor  -- there would be no changes to the substance of 
the document -- but would make scanning the outline and 
less-than-diligent reading of the text a bit more productive for 
relatively casual readers.


Detailed comments:


> Abstract
>
>    Benchmarking Methodology Working Group (BMWG) has been developing key
>    performance metrics and laboratory test methods since 1990, and
>    continues this work at present.  Recent application of the methods
>    beyond their intended scope is cause for concern.  This memo
>    clarifies the scope of RFC 2544 and other benchmarking work for the
>    IETF community.

Given the word 'harm' in the title, I suggest that the abstract contain 
a one-sentence summary of the potential harms and/or available benefits 
being targeted.  It helps to have an abstract contain a concrete sense 
of the document's concerns and findings.


> 1.  Introduction
>
>    This memo clarifies the scope of RFC 2544 [RFC2544], and other
>    benchmarking work for the IETF community.

The word benchmarking in the document title might make focus and scope 
obvious, but it is still polite to the more casual reader to introduce 
early references with a summary explanation.  For example, drawing from 
RFC 2544:

    ...the scope of [RFC 2544] which "discusses and defines a number of 
tests that may be used to describe the performance characteristics of a 
network interconnecting device."

This nicely summarizes that document without re-using the benchmarking term.


>    Benchmarking Methodologies (beginning with [RFC2544]) have always
>    relied on test conditions that can only be produced and replicated
>    reliably in the laboratory.  Thus it was surprising to find that this
>    foundation methodology was being cited in several unintended
>    specifications [Y.1731] and products performing applications such as:

I'd claim it is not at all surprising.  However it /is/ unfortunate. 
(This is meant as a serious point.  "surprising" focuses on the actors. 
  "unfortunate" focuses on the actions.)

The document does not really discuss the downsides of this inappropriate 
use, except deeply buried in the main text.  While Section 3 talks about 
controls, there is nothing that explores what bad results are likely 
with misapplication.  It will help to summarize the downsides early and 
directly.


> 3.  The Concept of an Isolated Test Environment
>
>    An Isolated Test Environment (ITE) used with [RFC2544] methods (as
>    illustrated in Figures 1 through 3 of [RFC2544])has the ability to:

    ])h  ->  ]) h


>
>    o  contain the test streams to paths within the desired set-up
>
>    o  prevent non-test traffic from traversing the test set-up
>
>    These features allow unfettered experimentation, while at the same
>    time protecting equipment management LANs and other production

"equipment management LANs"?  What does this term mean?


>    networks from the unwanted effects of the test traffic.
>
>
> 4.  Why RFC 2544 Methods are intended for ITE
>
>    The following sections discuss some of the reasons why RFC 2544
>    [RFC2544] methods were intended only for isolated laboratory use, and
>    the difficulties of applying these methods outside the lab
>    environment.
>
> 4.1.  Experimental Control, Repeatability, and Accuracy

Might be worth considering shortening this to

    4.1 Accuracy

For anyone needing to hear the sermon of this document, short pithy 
titles are likely to be more helpful.

If something cannot be repeated, it isn't accurate.  Also Experimental 
Control is an abstraction likely to have little impact on casual readers.


>    All of the tests described in RFC 2544 assume that the tester and
>    device under test are the only devices on the networks that are
>    transmitting data.  The presence of other unwanted traffic on the
>    network would mean that the specified test conditions have not been
>    achieved.

Given the first sentence, the second is a tautology.  I don't understand 
what it contributes.

Also, I assume you say 'assume' because this is not stated in 2544.  As 
such, the "specified test condition" wasn't specified.

I suggest replacing the second sentence with something here like:

    This condition is a requirement.


>    Assuming that the unwanted traffic appears in variable amounts over
>    time, the repeatability of any test result will likely depend to some
>    degree on the unwanted traffic.

Assuming...time,  ->  Given that real-world traffic varies statistically 
over time and affects overall system performance,


>    The presence of unwanted or unknown traffic makes accurate,
>    repeatable, and consistent measurements of the performance of the
>    device under test very unlikely, since the actual test conditions
>    will not be reported.

the actual test conditions -> the complete details of the test conditions


>    For example, the RFC 2544 Throughput Test attempts to characterize a
>    maximum reliable load, thus there will be testing above the maximum
>    that causes packet/frame loss.  Any other sources of traffic on the
>    network will cause packet loss to occur at a tester data rate lower
>    than the rate that would be achieved without the extra traffic.
>
> 4.2.  Containment of Implementation Failure Impact

As with the suggestion for the title of 4.1, I suggest a more 
in-your-face title.  For example:

   4.2  Damage


>    RFC 2544 methods, specifically to determine Throughput as defined in
>    [RFC1242] and other benchmarks, may overload the resources of the
>    device under test, and may cause failure modes in the device under
>    test.  Since failures can become the root cause of more wide-spread
>    failure, it is clearly desirable to contain all test traffic within
>    the ITE.
>
>    In addition, such testing can have a negative affect on any traffic

    affect -> effect


>    which shares resources with the test stream(s) since, in most cases,

    which -> that


>    the traffic load will be close to the capacity of the network links.
>
>    Appendix C.2.2 of [RFC2544] (as adjusted by errata) gives the private
>    IPv4 address range for testing:
>
>    "...The network addresses 198.18.0.0 through 198.19.255.255 have been
>    assigned to the BMWG by the IANA for this purpose.  This assignment
>    was made to minimize the chance of conflict in case a testing device
>    were to be accidentally connected to part of the Internet.  The
>    specific use of the addresses is detailed below."
>
>    In other words, devices operating on the Internet may be configured
>    to discard any traffic they observe in this address range, as it is
>    intended for laboratory ITE use only.  Thus, testers using the
>    assigned testing address ranges MUST NOT be connected to the
>    Internet.
>
>    We note that a range of IPv6 addresses has been assigned to BMWG for
>    laboratory test purposes, in [RFC5180].  Also, the strong statements
>    in the Security Considerations Section of this memo make the scope
>    even more clear; this is now a standard fixture of all BMWG memos.

The document lists the addresses assigned for IPv4.  It might be worth 
listing the ones for IPv6.


>
> 5.  Advisory on RFC 2544 Methods in Real-world Networks
>
>    The tests in [RFC2544] were designed to measure the performance of
>    network devices, not of networks, and certainly not production
>    networks carrying user traffic on shared resources.  There will be
>    unanticipated difficulties when applying these methods outside the
>    lab environment.

I suggest placing the first sentence, and possibly the second, in the 
Abstract and the Introduction.  It is the simple, direct and very clear 
premise of of this document.


>
>    Operating test equipment on production networks according to the
>    methods described in [RFC2544], where overload is a possible outcome,
>    would no doubt be harmful to user traffic performance.  These tests
>    MUST NOT be used on production networks and as discussed above, the
>    tests will never produce a reliable or accurate benchmarking result
>    on a production network.

Except for the use of normative language, the above paragraph appears to 
be wholly redundant with earlier text.


>    [RFC2544] methods have never been validated on a network path, even
>    when that path is not part of a production network and carrying no
>    other traffic.  It is unknown whether the tests can be used to
>    measure valid and reliable performance of a multi-device, multi-
>    network path.  It is possible that some of the tests may prove valid
>    in some path scenarios, but that work has not been done or has not
>    been shared with the IETF community.  Thus, such testing is contra-
>    indicated by the BMWG.

Ditto.


>
> 6.  What to do without RFC 2544?

Perhaps instead:

    6.  Performance measurement without RFC 2544

(Yes, I'm suggesting targeting /very/ casual readers for such titles...)


>    The IETF has addressed the problem of production network performance
>    measurement by chartering a different working group: IP Performance
>    Metrics (IPPM).  This working group has developed a set of standard
>    metrics to assess the quality, performance, and reliability of
>    Internet packet transfer services.  These metrics can be measured by
>    network operators, end users, or independent testing groups.  We note
>    that some IPPM metrics differ from RFC 2544 metrics with similar
>    names, and there is likely to be confusion if the details are
>    ignored.
>
>    IPPM has not yet standardized methods for raw capacity measurement of
>    Internet paths.  Such testing needs to adequately consider the strong
>    possibility for degradation to any other traffic that may be present
>    due to congestion.  There are no specific methods proposed for
>    activation of a packet transfer service in IPPM.
>
>    Other standards may help to fill gaps in telecommunication service
>    testing.  For example, the IETF has many standards intended to assist
>    with network operation, administration and maintenance (OAM), and
>    ITU-T Study Group 12 has a recommendation on service activation test
>    methodology.
>
>    The world will not spin off axis while waiting for appropriate and
>    standardized methods to emerge from the consensus process.

The bottom line answer that this section provides to the question of the 
section's title is really "go figure it out".

That's not very helpful and the last sentence is likely to be 
irritating.  It will be especially irritating for anyone who finds out 
that they are being told to wait on IPPM but that IPPM has been at work 
for 15 years -- it was chartered in 1997 -- and hasn't produced 
standards useful for this.

This section should contain concrete and useful pointers to alternative 
or it should say that there are no standardized alternatives or it 
should be removed.


> 7.  Security Considerations
>
>    This Applicability Statement is also intended to help preserve the

Having an opening sentence to a section say "also" is confusing.  What 
came before?


>    security of the Internet by clarifying that the scope of [RFC2544]
>    and other BMWG memos are all limited to testing in a laboratory ITE,
>    thus avoiding accidental Denial of Service attacks or congestion due


d/
-- 
  Dave Crocker
  Brandenburg InternetWorking
  bbiw.net