Benchmarking Methodology WG (bmwg)

MONDAY, November 14, 2011

0900-1130 Morning Session I

Room 101A OPS bmwg

CHAIR(s): Al Morton acmorton@att.com

INTRODUCTION

BMWG met at IETF-82 with 20 people attending and at least 5 more participating remotely. Al Morton chaired the Meeting and prepared these minutes, based on detailed notes from Chris Inacio as official Note-taker (using the new Etherpad tool). Chris and Mike Hamilton monitored Jabber, and Mike added some notes in Etherpad when Chris was speaking. The meeting was broadcast via one-way audio on the IETF audio stream. The chair requested a 2 hour session, and used 2:15 of the allotted 2:30 time.

This report is divided in two parts: an executive summary with action items, and detailed minutes of the meeting.

SUMMARY

Brief status, the IGP-convergence drafts are in AUTH48

and making progress. All points are reviewed by Authors

and we are waiting for publication.

IP Flow Export Benchmarking has been updated following a productive

Third WGLC. The chair has identified 5 open issues for list discussion

and there will be a 4th WGLC when this is complete (target end of Month).

There was also a productive discussion of issues on the new draft on

"Restoration and BGP Convergence of Contemporary Routers"

methodology. The coverage of BGP Data Plane convergence

will be addressed in single draft, requiring the key issue that

if forwarding can be repaired without any control plane action,

that feature MUST be disabled for this benchmarking.

The main next step is to get Operator feedback on the

metrics proposed in these drafts, a rep from RIPE NCC was

present and now need to get feedback from NANOG and APNIC.

The Content-Aware (CA) authors revised the methodology draft to address

the first round of comments from chair and others, Mike Hamilton reported.

A "boatload" of comments from Tom Alexander remain open on

both the terms and methodology. The chair has reviewed these comments

and suggested several ways forward to resolution.

There is a plan to generate pseudo-random and malformed traffic,

using algorithms that can be standardized.

The existing CA work might be augmented by a new proposal on Security

Effectiveness Benchmarking, as proposed by new attendee Kenneth Green.

The point is to measure the function of culling

malicious traffic from desired traffic, and not forwarding performance

in the presence of malicious traffic, as the current CA method is doing.

Thus, the Security Effectiveness and CA work are complementary.

Al made arrangements to meet off-line with the Security Effectiveness

and Content Aware authors to provide comments and coordinate the work.

Al presented the IMIX Genome project, and made a point to resolve Ilya

Varlashkin's comments off-line.

Al briefly presented the RFC 2544 Applicability Statement,

which resolves comments from David Newman, Curtis Villamizar, and Bill Cerveny.

the WG may be done with now (so test with WGLC).

WGLC is also expected on the protection methodology draft.

On the other hand, LDP convergence work needs to get going

(it's been on the charter for more than a year now),

or it risks being dropped.

There was a very brief description of work on Benchmarking time

synchronization devices from David Moran - which will be discussed

in more detail in TICTOC WG on Thursday.

ACTION ITEMS:

· WGLC on Protection Methodology Draft

· WGLC on IP Flow Export Methodology Draft

· WGLC on RFC2544 Applicability Statement Draft

· Obtain Operator feedback on methods and metrics in BGP drafts

· Address comments on the list, or with updates to drafts (Everybody).

DETAILED NOTES

Welcome to Etherpad Lite!

0. Agenda Bashing

1. WG Status and Milestones

Approved:
Benchmarking Link-State IGP Data Plane Route Convergence
State:   nearing the end of AUTH48

ci: Ron Bonica looking for the authors of the dataplane convergence drafts so that the 48-hour time is met.

'Testing Eyeball Happiness' Approved as an Informational RFC

Drafts not presented at this meeting: (need reviews as noted)

ci: Al: hoped that LDP documents would be resurrected by now, but did want IGP to be further to along for LDP to go forward.
ci: Rajiv via jabber: will get revived soon.

Draft Preparation Discussion Summary

ci: Ilya, use version 4.9 of the XML Mind tools for compatibility reasons.  Version 5 is not compatible with the XML2RFC plugin.


RFCs on the Standards Track
   Brief update on IPPM progress and implications for BMWG

ci: Al: Is there any Interest in pursuing a move to the Standards Track here in BMWG??
ci: Kenneth Green: How do you differentiate the variances in the implementations, versus variances in the DUT?
ci: Al: The implementations shouldn't produce different results if they implement the (well specified) standard appropriately.

ci: Ron Bonica: question about being ready for STD's track

ci: Mike Hamilton: 4 implementations of RFC 2544 side-by-side that have fairly significantly different outputs, experience note.

ci: Mike Hamilton: Do we want to be beholden to a 3rd party of an implementation for anything on STD's track.
ci: Al: in that case, then the STD isn't tight enough, excluding the case where the 3rd party is incapable of implementing the spec.


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

2. IP Flow Information Accounting and Export Benchmarking Methodology
   Presenter: Al for Jan Novak
The chair has identified 5 open issues for list discussion and there will be a 4th WGLC when this is complete (target end of Month).

3. Basic BGP Convergence Benchmarking Methodology status
   Presenter: Ilya Varlashkin and Dean Lee

ci: 3 people have read Ilya's draft.

ci: Dean Lee: A lot of experience doing these two methodologies; one looking at only the data plane, one attempting to look at the control plane.  Very good to have both methodologies.

ci: Al emphasizing Ilya's point that data plane convergence may not necessarily indicate control plane convergence.

ci: Ilya wants actual feedback, volunteers?
ci: Al: in the current text of the charter we are required to get operators feedback.
ci: Al: We should be presenting this at the operators working groups.

ci: Dean: He's been getting constant requests from carriers/operators on how to do these types of measurements.  We should be able to ask those carriers for feedback.
ci: Al: recommends to do it!

ci: Ilya: can present this to the RIPE community.
ci: Al: need volunteers to present at NANOG and APNIC.  This is what Ron envisioned within the WG charter language.

ci: Ilya requests any feedback now:
ci: Al: Would like to get more configuration definition in the RFC
ci: Ilya already had this in his list within his presentation
ci: Al: IGP & BGP interactions to consider in this measurement.
ci: Al: always want a default config so that a comparison between work at Lab A and Lab B is possible.

ci: Craig White: Worried about potential vendor tweaks in order to accelerate performance with respect to this testing methodology / config.  Would really like to have the independant lab measurement capability exist (Lab A, B, C can compare results.)
ci: Al: Concurs, although the saving grace may be allowing operator configurations in the mix.
ci: Ilya: Goal of test design, including the number of devices within the test design, are there to help mitigate the problems of a vendor designing to do well on the test / avoid test bias.

4. Benchmarking Methodology for Content-Aware Network Devices
   Presenter: Mike Hamilton

Mike: Would like to use an algorithm in the specification based on an open source system.  Would like information on the IETF policy.
Al: Would depend on the license of the algorithm; would have to investigate.

Kenneth Green: Worried about implementation vs. specification, basing on open source.
Mike: would just like to reference / use their algorithm, not their implementation.

Al: - many reactions after reading the drafts and Tom Alexander’s comments on the plane ride…
Al: Change wording from TCP throughput to TCP bulk transfer capacity.
Al: terminology updates are needed; Mike agrees, but wants to fix content and then get back to terminology.
Al: client/server metric differences; need to get tighter on the definitions on places for specifying measurement points.
Al: May want to reference ITU-T Recommendation Y.1560 possibly, which talks about TCP setup times.
Al: Only *one* introduction should exist for the documents.

Impromptu Agenda Bash – Continue on the related topic/draft ahead of Al’s presentations:

x. Security Effectiveness Benchmark
   Presenter: Kenneth Green
ci: Ilya: wants the addition of performance of passing good traffic when the network is under attack.
ci: Mike: Willing to work with Kenneth on making sure to get this covered in at least one of the documents.

MH: Chris asking clarification on types of devices.  Incredibly difficult to distinguish/define legal/illegal.
MH: Kenneth agrees and stated earlier about difficulty of enumerating these categories.
MH: Chris talked about SCAP or other definitions.  

Al: mentioned discussing the terminology topic with Steve Bellovin before the Reception, There is no special status for the term “evil” in the IETF.

ci: Ilya: Can add more & more evil traffic and watch the good-put performance decrease.

5. IMIX Genome
   Presenter: Al

ci: Al & Ilya will work the “Compression” issue offline, and considered name change of “Genome”, but will leave as-is.


6. RFC 2544 Applicability Statement: 
   Presenter: Al

Note the name change to “Production Networks” from “Real-World Networks”

New Work Proposals:

y. Benchmarking Time Synchronization (David Moran)
ci: Work on measuring differences in time synchronization will be talked about in ticktock WG.

z. Software Update Benchmarking Brief Update

Al: Authors expect to have a draft soon.

LAST. AOB