BMWG Minutes – Benchmarking Methodology WG (bmwg)
THURSDAY, July 28, 2011
1300-1500 Afternoon Session I
Room 2101 OPS bmwg
CHAIR(s): Al Morton <acmorton@att.com>
INTRODUCTION
BMWG met at IETF-81 with 24 people attending and at least 4 more participating remotely.
Al Morton chaired the Meeting and prepared these minutes, based on detailed notes from Sarah Banks as official Note-taker.
Chris I. monitored Jabber + read the many comments at the mic.
The meeting was broadcast via one-way audio on the IETF audio stream.
This report is divided in two parts: an executive summary with action items, and detailed minutes of the meeting (from Sarah).
SUMMARY
During brief status, the IGP-convergence draft update addressing *ALL* IESG Discusses, past and present, has been reviewed by Stewart Bryant and he agrees with the changes and adds a few minor comments to fix. Only the -meth draft has a remaining DISCUSS from Adrian Farrel, and he has been pinged and ACK’d that this is on his list.
IP Flow Export Benchmarking has been updated following a productive WGLC.
ACTION: WGLC on this version after the meeting. (DONE)
The SIP device terms and methodology were updated for this meeting. Vijay Gurbani presented the response to all issues, and the drafts will be uploaded soon.
There was also a productive discussion of issues on the BGP convergence methodology, prompted by experiments discussed on the list and several members discussing the results and their implications. The coverage of both Control Plane convergence and Forwarding-plane restoration be addressed in two drafts for now, addressing the issue that sometimes forwarding can be repaired without any control plane action.
The Content-Aware work revised the methodology draft extensively. Feedback was asked to address the possibility of malicious and malformed traffic, and it was agreed to make this testing a subset of the main tests for performance. The meeting agreed to adopt these drafts as WG drafts to address the chartered work item/milestones.
ACTION: Al to contact the Leadership of the EEMBC – Embedded Electronics Multicore Benchmarking Consortium on this topic and suggest collaboration.
Al presented the IMIX Genome project, and it was adopted by the WG draft under the current charter.
Finally, Al briefly presented the RFC 2544 Applicability Statement, prepared by all current and former chairs of BMWG. There was one
comment to make this more broad to cover all of BMWG's RFCs, but that might best be covered with another draft.
Ron Bonica called the consensus that both these drafts would be adopted by the WG.
DETAILED MINUTES
Introduction
Blue sheet
3-4 new attendees in the room
IPR
BMWG activity – some improvement with AD’s on the dataplane IGP convergence drafts in a positive light.
New RFC published – RFC6201 (device reset characterization)
http://home.comcast.net/~acmacm/BMWG/
We have a standard paragraph that goes in the introduction – please use it!
Item 2: Paul will take a look at the draft, Brian Trammell will review as well.
SIP Performance Benchmarking draft:
Vijay presenting
Item 1: resolved comments from
Item 2: Resolution to not record causes of failures; expect users to resolve this prior to benchmarking
Item 3: Resolved comments, adding a new section to allow duplication of tests (and results)
Item 4: Resolution: clarified the test procedure
Item 5: Resolution to provide for a concatenated system.
Skipped several slides with editing specific comments
Next steps: will provide a draft and have an open list discussion on changes.
Benchmarking content aware network devices
Mike Hamilton Presenting
Covered changes to the draft since last revision
-07 is the latest rev
All comments have been addressed
Major mods to the methodology – focused on RFC3511, but moved this towards metrics from L7 instead – this is the focus of the change throughout the draft
Question from Ilya: what will happen if you benchmark WAN accelerators; are they qualified for this?
Answer: They’re not explicitly covered here, but there’s nothing barring them from being covered.
Ilya V to read draft and reach out to Mike and Sarah with feedback.
EEMBC – embedded electronics multicore benchmarking consortium? They reached out to Mike and are trying to figure out how they fit into this draft/work area.
Al to reach out to these members and let them know we’re interested in working with them and looking forward to their feedback.
Outstanding questions slide
From Al;
- In example provided in appendix, perhaps re-label the columns
- Rewording of the columns in general
- Don’t call it a sample traffic pattern – call it an example traffic pattern
- Reasonable start; more folks to review
- perhaps provide more options to tweak
- clarification around updatable; Mike believe this is moot now, as the person running the test will fill in their own values.
- security – perhaps leave it off the table and put it in the scope that it’s deferred
- malformed traffic – again, like security, leave it out of scope and come back and revisit it later.
Q: why not test without malformed for performance then measure when the box fails if malformed present?
A: it’s not out of bounds; the thought might have been that you’d run the same tests without and then with malformed traffic; this might have been lost in text somewhere during the revision process
From Al: nail down metrics referred to in the scope, that gives a solid view of coverage
- When you get into the procedures, there are some conflicts between MUST and SHOULD, fix them
-
Clarification:
security means malicious? Mike: yes
-
Sample has to be
an example –already covered
-
Might provide some
references or overall guideline for users to determine what their traffic mix
should be
Ilya V Q: with 2544 we don’t send rogue packets and try to assess performance – don’t do it here.
Al: that’s true, it’s worthwhile discussing.
Mike: saw some cases where malformed packets caused catastrophic performance impact, seemed useful to check.
Bill Cerveny (Arbor): hadn’t read draft, sentence should be omitted about why an exhaustive list can’t be provided in the scope.
Al: perhaps it needs to be re-ordered – give the examples first and then say the list can’t be exhaustive…
Al bestowed a marked up copy of the draft to Mike with feedback. Thanks!
BGP Dataplane Convergence
Greg Cauchie presenting
Provided status
- A new draft is coming, a second one.
-
Covered Ilya’s comments on the lists – what are we trying to
measure?
-
Solution:
splitting the doc into 2 drafts? Make a new one? Want feedback here
-
Ilya comment: splitting the doc into 2 would
speed up work. Al concurs
-
Ily expanded a comment that the new draft
would focus on the downstream devices not seeing a failure, such that it’s up
to the DUT to restore traffic ASAP.
Next Steps:
Split into 2 docs?
Ilya will be on the email list and
in
Next draft before
Al prefers drafts in XML, not (evil) word!
Rob: good bit of work to work on, supports the direction this work is going
Work Proposal Summary Slide
SW Update Draft? – Sarah: It’s coming; some confusion on scope, had discussion with customers and they want it,
will present a draft in
Power: not much progress here
IMIX – Al can continue working on this provided this continues as a working group item.
Happy Eyeballs – done, going individual, no action for this working group
2544 Applicability Statement – Al can continue working on them as co-author on them provided they become working group items and Ron Calls consensus
IMIX Genome – work item proposals
Al Presenting
Covered status and comments from
Ilya V Q: If the mix is long, can it somehow be shortened?
Al: wants more details, will take it offline and to the email list.
8 people have read the draft
Al asking for a hum – got it
Ron Bonica called consensus (TB ratified on list)
2544 AS preso
Al presenting
4 authors, all bmwg chairs, wrote this draft
Ilya V Q: People test with real LIVE traffic today and 2544.
Al: Yes, but if they were doing what 2544 says to do, they’d be hurting the Internet traffic
Scope is clear; nothing outside of the test bed in; nothing inside of the test setup gets out. It’s meant for the isolated test bed environment.
Comments
Bill: you are cloning text from 2544 that is an error – you specify the address block and this is incorrect
Bill: Section 4.1 you talk about accuracy being compromised; might want to include repeatability and consistency
Bill: comment about devices on net; what about production networks?
Al: the concern here, from Scott, was that results are not well understood when you tandem multiple devices and drive them to overload.
Bill: typo, will provide to Al
Ilya Q: What exactly is the goal – to discourage people doing silly things? Or to encourage every author to include section in the draft? So, a BCP in every document in bmwg?
Al: about 8 people read/understood, asking for hums for consensus
Ron: consensus obtained – it’s a WG draft
All Other Business
Last item: a question on whether or not draft-hamilton as a working group item (forgot to ask earlier).
Al: asking for consensus to adopt content aware draft(s) as WG drafts
Al: consensus obtained
end of meeting