BMWG – Prague IETF 80 Meeting Minutes

TUESDAY, March 29, 2011 0900-1130 




 
 
CHAIR(s): Al Morton

Meeting Room: Karlin II & III
Notes taken by: Fernando Calabria

INTRODUCTION

BMWG met at IETF-80 with 34 people attending and 3 more participating remotely, including AD Advisor Ron Bonica.

Al Morton chaired the Meeting and prepared these minutes, based on detailed notes from Fernando Calabria as official Note-taker.

Matt Zekaukas monitored Jabber + contributed some additional notes.

The meeting was broadcast via one-way audio on webex, and the usual IETF audio stream.

This report is divided in two parts: an executive summary with action items, and detailed minutes of the meeting (from Fernando + Matt).

SUMMARY

During a brief status, the IGP-convergence draft update addressing *ALL* IESG Discusses, past and present, was raised as an action item and blocking point for progress on several other work items.

IP Flow Export Benchmarking has just completed a productive WGLC, with many new comments. Jan Novak will address the coverage of non-forwarding flow monitoring devices according to the model proposed by Brian Trammel.

ACTION: WGLC after revision.

The SIP device terms and methodology were updated for this meeting, and Carl Davids presented the results of student implementation of the methods. This prompted a very lively discussion, with many improvements suggested for the drafts. As always, performing the methodology results in advanced learning on the topic.

Also, now that SIP Metrics RFC is approved, this draft MUST learn from that experience and avoid the same DISCUSS tarpit.

ACTION: 2^nd WGLC after revision (acm confirms this is the 2^nd, previous WGLC in March 2010 on version 01).

There was also a productive discussion of issues on the BGP convergence methodology, prompted by several members reading the drafts and making comments on the list! The coverage of both Control and Data plane must be made clear(er) and the need for testing Graceful Restart must be justified in the future.

The topic of emulating the "real-world" comes-up frequently, in routing tables, traffic mixes, and other environment variables.

ACTION for all work to anticipate all possible variables in testing scenarios, but not to recommend any particular conditions as "standard real-world" - the world is different for every user.

The Content-Aware work now has a terminology draft to accompany the methodology, so development is proceeding. Feedback asked to address the possibility of payload compression and partially cache-able traffic.

There was a discussion of a new work proposal to follow-up on the RESET draft (now RFC6201), measuring other forms of interruption, such as device upgrade, software patch, system crash, In-service Software Update. This topic is agreed to highly useful info to operators, but can a standard procedure (with repetition) be constructed? Some thought not, but others think yes, and the next step is to write a draft and BMWG will discuss further from there. Also, need to see if the charter allows single vendor tests (SOB thinks yes, ACM not so sure). The proposed Scope is to prepare a standard procedure for all vendors to follow, but likely not for direct comparisons *between* vendors.

ACTION: Proponents to prepare a draft to advance this discussion.

ACTION for ALL: Publish Updated Drafts during the Interim period – greater likelihood of review than publication at the deadline…

------------------------------------------------------------------------------------------------------------------------------------------

Agenda:

0. Agenda Bashing
 
 1. WG Status and Milestones
 http://datatracker.ietf.org/wg/bmwg/
 
 Approved:
 
 RESET benchmarks: Update to RFC 2544
 http://tools.ietf.org/html/draft-ietf-bmwg-reset-02
 Approved, RFC EDITOR, AUTH48
 
 Status of Drafts not presented at this meeting:
 
 Benchmarking Link-State IGP Data Plane Route Convergence
 http://tools.ietf.org/html/draft-ietf-bmwg-igp-dataplane-conv-term-23
 http://tools.ietf.org/html/draft-ietf-bmwg-igp-dataplane-conv-meth-23
 Need to resolve of IESG DISCUSSes, now believed to be fully addressed.
 
 Methodology for benchmarking MPLS protection mechanisms,
 http://tools.ietf.org/html/draft-ietf-bmwg-protection-meth
 Need to consider IGP feedback and revise/submit
 
 Benchmarks For Data Center Bridging Devices
 expressions of interest to review,
 Should BMWG adopt this item?
 http://tools.ietf.org/html/draft-player-dcb-benchmarking-03
 
 Happy Eyeballs Benchmarking
 Anything left to discuss here? Topic also raised in v6ops (Thursday)
 http://tools.ietf.org/html/draft-baker-bmwg-testing-eyeball-happiness-04
 
 
 Chartering Status:
 Approved Charter text and Milestones
 
 
 2. IP Flow Information Accounting and Export Benchmarking Methodology
 Draft adopted as chartered item.
 Extended Last Call closed March 28
 Review and Resolve comments
 http://tools.ietf.org/html/draft-ietf-bmwg-ipflow-meth-00
 Presenter: Jan Novak
 
 
 3. SIP Benchmarking Drafts:
  Brief updates, status, and implementation experience!
 http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-term
 http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-meth
 Comments?
 Presenter: Carol Davids
 
 
 4. Basic BGP Convergence Benchmarking Methodology status
 "Ancient" work item, with renewed interest.
 >>> Held-over from last meeting, due to schedule conflict
 Related Draft:
 http://tools.ietf.org/html/draft-papneja-bgp-basic-dp-convergence-01
 Presenter: Bhavani Parise
 
 
 5. Benchmarking Methodology for Content-Aware Network Devices
 Extension of BMWG Firewall RFCs
  New Terminology Draft
 Should BMWG adopt this item?
 GOALS: To discuss the updated drafts and discuss future of drafts
 Related Drafts
 http://tools.ietf.org/html/draft-hamilton-bmwg-ca-bench-meth-06
 http://tools.ietf.org/html/draft-hamilton-bmwg-ca-bench-term-00
 Presenter: Mike Hamilton
 
 
 Other New Work Proposals:
 
 a. Discussion of Software Update Time Benchmarking
 Possible transfer of the proposal to another WG...
 Presenters: Ilya and/or Fernando?
 
 b. IMIX Genome
 Related Draft:
 http://tools.ietf.org/html/draft-morton-bmwg-imix-genome-01
 Review of revisions based on comments and next steps
 Presenter: Al
 
 c. RFC 2544 Applicability Statement:
 Use on Real-World Networks Considered Harmful
 Initial Review of the draft with the working group, following proposal@IETF-79
 Related Draft:
 http://tools.ietf.org/html/draft-chairs-bmwg-2544-as-00
  Presenter: Al
 
 d. Power usage while Networking
 http://tools.ietf.org/html/draft-manral-bmwg-power-usage-01
 Comments that the spec is similar to the TEEER spec and authors
 should try to merge the two before proceeding.
 --- How was this resolved???

 Brief Status: Al for the authors
 

Meeting started at 9 32 AM

Al Welcomes all local and remote participants, Ron Bonica remote from the US

Intellectual property Rights (IPR) remark – Thoughts for the Japanese community

0 – Agenda bashing

All items listed – Al questioning for any additional items to the proposed agenda (no additional items)

1 – WG status and milestones

All Slide 5 items listed – BMWG activity overview
Device Reset characterization draft approved – RFC 6201
Al’s BMWG web page introduction
Only one first time attendee, Al welcoming to the group encouraging participation – activity in the list
Listed all non-active documents, current no comment on Fred’s Baker eyeball-happiness
Detail on “Standard paragraph – Intro / security
Charter updated and items in red on slide
No questions from Participants (local /remote)

2-IP Flow Information Accounting and Export Benchmarking Methodology
 
Jan Novak 
Describes Document Goals 
Test topology discussed in the draft – two planes 
Few additional pending items - remove reference to Sflow – remove specific HW and SW platforms discussion )not longer applicable) 
Jan questioning the group on the need to have an additional terminology document (answer no) 
Few additional comments on the draft – last called ended last night, so Al asking Jan to address Paul Aikten’s comments as soon 
as they are available…
More comments from Brian Tramell on plane characterization and Netflow for not just only forwarding devices applicability 
- Jan to bounce back some ideas / modifications
 

 
3- SIP Benchmarking Drafts
Carol Davids
Terminology and methodology documents updated 03 rel 
In summary:
SIP devices may be proxy ones
Modified Figures 3 & 8 
Few changes in methodology and minor edits
Experimental Data collected by deploying described methodology in the IIT Real time communications Lab 




Early – Preliminary results presented 

Al question about variability in results - Yes substantial variability based on the Tool deployed (Kamailio vs. Asterik)

Scott Bradner questioning about 50% step rate (50% of total or step) – response is 50% of step

Darell suggest removing the reference to specific test Tool – agreed

General question in regards to the nature of the first error (registration error / timeout? ) - first error not characterize , any error will trigger to stop the test / set scale

Vijay Gurbani in regards to the reference to tools that open source tools will be fixed somehow

Scott Bradner suggested to do not stop test at the first error , use another parameter like time to stop the test as “catatonic” behaviors of devices cannot be characterized by just stopping when the first error is detected

Document is not completed – next steps

Al on good feedback received on today’s session to be incorporated as well as comments from students collaborating with the test

General comment on back to back SUT / DUT configuration and the value of adding additional devices in the test topology - Original idea was to include additional mid-points

 
4- Basic BGP Convergence Benchmarking Methodology
 
Bhavani Parise 
Status on current draft 
Scope of the document to cover FIB V4 and v6 – iBGP and eBGP setups and a 4 router topology
RIB IN/OUT convergence for all test cases 
Next Steps 
Question from Ilya Varlashkin and Srah Banks in regards to general scope (control and or data plane convergence ) or both 
Additional questions on the terminology to be used 
Comment from Fernando Calabria on need to address EoR treatment on Tester and DUT 
Received feedback / comments to be included in new rel of draft 

 
5- Benchmarking Methodology for Content-Aware Network Devices
 
Mike Hamilton

First draft- work in progress

Outstanding question on composition of traffic and IMIX definition

Different requirements from different customers, document to provide a framework for the customer to identify and present their specific needs / traffic constraints

Next Steps – Request feedback

Al encouraging WG to review document and as a general practice to submit documentation and comments in advance

 
Barry Constantine, jdsu. first cut of traffic mix?
doesn't talk bout what mix is, but more example of things necessary to specify
to reproduce. not what a good mix is.
question: plan on taking into account level of compression of payload? because of wan accelerators or... 
send things because want to see perf of device...
 
Mike: yes, as they expand they think will that fall out.
Totally random traffic, totally repeatable traffic and then stuff in between...
up to end user to know what needs to be done. 

 
6 - Work Proposals 
 
a- Operation related event benchmarking 
Fernando Calabria
 
Explain current situation and listed some specific Operational related events not cover in current device reset RFC 

ISSU – software upgrades and operational related benchmarking

Possible move of this work to the OPSAWG

Scott Bradner comment s regarding complexity of the task / methodology to be able to be repetitive due to possible number of iterations

Ilya – Sarah and Fernando believe that a common methodology can be laid down to cover these specific events

<more detail from Matt Z follows>

al: trying to find scope, and then people to do work.

sob, also co-chair of opsarea. don’t' know enough to say much, but I don't know how to do it. how to figure out how long service operation from update, have to apply multiple times, which can't do. have to reset environment to get repeatability. not fun to do, takes forever.

don't know how to do in test lab in way that is repeatable.

do believe can bring methodology together to make it efficient

sob: also different patches applied in different order might have different results

how compare different products

patches & upgrades: yes. some may be SA, some may not, re dependences, can be complex. but same applies to crashes or other errors. will be challenge. what think... benchmarking of established standard like in-service software update is something that can be done, and is of value. know that if platform is ISSU, expect no outages.

sarah: at end of day, want methodology to allowed repeated tests. but getting asked for it across industry, so needed. how choose to measure is thing; and if repeatable; and can do continuously the same way.

customers ask: when I get patch, what happens. that's exactly what this does

but very vendor specific performance. and is it fair to compare between vendors when they are never going to have the same implementation or same thing definable as a patch.

s: could argue that rib/fib...

in the end, did I test same way on two vendors, and if get different results... test in same way all can do.

sob: different approach. not sure valuable to test vendor a vs vendor b. but vendor a w/in what happens with page with this methodology. tested with methodology what happens. not comparing vendors, but giving customers info.

Fernando: another wrinkle, if say support In-service upgrade, want something to verify if that is true.

? from op perspective, how long this device will be out of service. if can compare vendor a vs vendor b... how long, if two redundant boxes, but one sw upgrade, how long w/o redundancy. if 2 min.. or 6 hr.. or ...

al, as participant: no draft for this...

if shoot for a standard methodology that can be applied to multiple vendors

then as step 2, is it fair to compare vendor a vs vendor b using methodology.

but not sure can have initial conditions the same to do that. but same methodology

ali karimi: vendors have methodology for software upgrade. if ask them, will provide. if ask what is impact, give it to you. don't understand... should be methodology, but vendors will be different because different technology sw/hw

sarah: of course, but methodology will make device look best for a vendor, so not apples to apples.

ali: give me meth and impacts, or I wont' touch product.

fernando: exception to rule.

customers will test themselves even give that.

it's their due diligence to do that.

ali: if customer hasn't asked you, those are not tech people.

sarah: want apples to apples even w/in vendor.

al: next step is draft. let's see it. will give time to this in bmwg.

once see draft, think about non-bmwg home for this, because we have to do vendor

comparisons here. this may not be capable for that.

sob: don’t' remember being in charter that restricted to vendor comparisons. it's the gist, but..

sarah: for me, would be willing to take off table if that helps, doesn't have to be vendor a /vendor b comparison

al will re-read charter.

 
Draft to be prepared… 

 
b – IMIX Genome
Al Morton 
 
Work received positive feedback
Definition as a reporting spec 
 

 
c – Power usage in network devices
 
Work in EMANWG 
Al mentioned existing Telecordia (Bellcore) standard – spec 
 
Bruce Nordman (co-chair of EMAN WG)
work on energy efficiency. ATIS has been working on this, esp. sw & routers
have decided to stop doing small devices, because energy star in US have procedures 
all of this is bench testing
if looking things in actual use, would be complement. 
record prots, speeds, actual traffic, and experience does not exist.
 
al: our scope is lab only
don't do live net testing... and measure in live net.
The current proposal may overlap with ATIS work
 
recommendation to look at ATIS doc...
 
bruce: suggests to talk by phone to chair of ATIS committee
 
sob: looked at charter, not vendor a vs vendor b
on live networking... tests in labs, can be applied elsewhere as long as not
modifying network traffic or behavior of device.
 

 
d- RFC 2544 Applicability Statement:
 
Al Morton 
Draft written by present and past WG chairs 
Scope of RFC 2544 is made clear – Lab only, as in Isolated Test Environment - ITE
All about accuracy and repeatability of results – 
Vendors should not ‘hide’ modified methods behind 2544, citing it as their standard reference.
This memo says the 2544 tests MUST NOT be executed in live networks (no place to hide now).
Bring 2544 back to a controlled lab environment.
 
One comment: Matt Z said “ship it.”

7- Adjournment

The meeting was adjourned at 11:36 AM