IETF-77 BMWG Session Minutes

22 March 2010 1300 PDT

WG chair: Al Morton


These minutes are divided in three sections: Summary, Action Items and Detailed Notes.


The BMWG met at IETF-77 with 23 people in attendance, and 10 more using webex conferencing to join remotely (and in one case, to present slides). This meeting report was prepared by Al Morton, based on detailed minutes provided by David Newman, as official note taker, and with Mike Hamilton filling in for David during his presentation and at other times.




The IPsec drafts should be updated soon, according to the lead editor (to resolve remaining DISCUSSes).


The IGP dataplane convergence drafts have been updated based on implementation/testing experience via comments in response to the WGLC in Dec 2009. The drafts have been updated and the will be another short WGLC before returning these drafts to pub-requested state.


The Sub-IP drafts completed WGLC in January.  The terms draft will go to to pub-requested state, while the methodology will wait for IGP-dataplane IESG review to attempt to derive benefits from comments on the similar methods.


The SIP device benchmarking drafts have made good progress, and are in WGLC till March 31st. There are many comments to be incorporated and more readership is needed. The plan is to reach publication-request by July 2010.


The RESET performance work proposal did not yet have enough readership to adopt as a work item, but there is support for the work and no need for a charter update to add the work.


The Flow-Monitoring device Benchmarking drafts were updated to address comments, and there was meeting consensus on the work item paragraph to add to the charter (already reviewed on the list).


The Data Center Bridging work has some industry support, but we need to liaise with IEEE and a letter must be prepared and approved to do that.


Lots of good discussion on the traffic load characteristics for the Content-Aware Device Benchmarking proposal, and an updated draft is expected. BMWG has an open mind w.r.t. new traffic descriptions, with the proviso that results also need to be interpretable and repeatable.


Work on BGP Convergence Benchmarking is focusing on the dataplane for the current work, and may require updates to the terminology RFC.



Question to List to pick-up RESET draft on the Charter – small number of readers/supporters present at the meeting (conclusion that no re-chartering is necessary to begin this work). (Done)

Prepare a Liaison to IEEE 802.1, pointing to the Data Center Bridging draft and the proposal to update RFCs 2544 and 2889 to address the Per-Flow Control capabilities of IEEE 802.1Qbb.(Done)

The Sub-IP Protection Terminology Draft has reached WG Consensus – Doc Shepherding Write-up In-Progress. (Done)

Start a WGLC on the IGP Dataplane Convergence Drafts (Done).

Pursue Re-chartering: When Revising Milestones, include the new dates suggested by the SIP benchmarking authors (June/July 2010 AD-review).

WG participants interested in the SIP Device Benchmarking work should schedule some time to do a detailed review of revised drafts in April 2010, during 2nd WGLC. This will be Cross-posted to SIPCORE WG.

Think hard on traffic for content aware devices – Everybody!

No one has been willing to help on chartered items where the drafts have LONG EXPIRED, such as Accelerated Stress Benchmarking, and this lack of progress & interest will be recognized during re-chartering. Anyone with concerns should contact the Chair immediately.






IPR Policy statement covered before all else.


Agenda bashing (none)


WG status

6 WG documents are active (2 each on dataplane convergence, protection and sip benchmarking; last of these are in wglc)


dataplane convergence applicability ID recently expired


IPSec docs are in IESG processing, possibly done with tweaks from Merike and Tim as soon as today (or perhaps this week) ??


No docs in RFC-editor’s queue (MPLS forwarding is now RFC 5695)


4 documents are proposed work items


Reminder from Al to use standard bmwg scope disclaimer in all documents, it ensures that our BMWG charter limits are accurately reflected in the text.



Link-state IGP dataplane convergence (Kris Michelsen, remote presenter)

Summarize main changes in -20 (from comments during WGLC)

            1. Added 2 optional test cases

                        1-to-N path convergence (local, remote failure cases)

            2, Added accuracy intervals

                        More specifics on calculations for loss-derived and route-specific loss-derived methods

            3. Added packet delay variation evaluation for the rate-derived methodologies

                        Explains calculations for expected packet counts (uses relationship between PDV and packet sampling interval [PSI])


4. Explains when convergence occurs in non-ECMP and ECMP cases



Al: Got many comments from a new reader/implementor after end of WGLC, and it looks like these comments have been addressed. Asks for additional comments from the room; none from attendees.


Al proposes another short WGLC, starting today and running two weeks. Kris may have 1 or 2 more additions after WGLC ends, such as substitution of PDV for jitter. Al believes we can go to IESG after this last call.



Protection performance drafts (Al Morton, presenter)

 Al: These drafts have completed WGLC at end of January 2010. Al believes there is working group consensus on moving ahead, but in terms of schedule also thinks they should move ahead after the IGP drafts. This is in part because the IGP drafts have some methodologies in common, and Al would like to hear from IESG on these documents “for safety reasons.” This order would prevent overloading the IESG with work that may or may not be ready to go from the IESG’s standpoint.


Remote comment from Rajiv: Would prefer to see protection IDs go forward independent of IGP convergence drafts.


Comment from Ron Bonica, as AD: Would prefer that documents be staggered because what authors learn from one document/documents can be applied to the other. Authors don’t want to submit a bunch of documents and have them all bounce at the IESG.


Remote comment from Bavani (and agreed to by Rajiv): Proposes that terminology ID could proceed independently


Comment from Al: This seems like a good compromise. We’ll proceed with the terminology. There was nodding in the room and no further comments.


Comment from Al: In terminology draft, at least two equations have unbalanced parentheses. Since this will cause problems at IESG, this should be cleaned up in AD review.


Question from Al for Kris: Can you quickly make final changes after WGLC to these drafts? Kris says yes.



SIP benchmarking terminology/methodology updates and status (Carol Davids, presenter)

WGLC in progress, IDs need review and comments. Masters students at IIT working to implement open-source SIP servers to test IDs

Presented summary of terminology contents

            Definitions in four sections: Protocol components, test components, test setup parameters, benchmarks

            Fleshed out reporting format and reporting requirements (comment from Al: be sure to include units of measurement in stating reporting format)

Update on terminology changes

            Updated definitions and test cases, based on IIT testing experience


Comment from Al: There is a diagram to show testers back-to-back in terminology, but that topology is not included in the methodology. Carol agrees that is missing, and will add it.


Comment from Vijay (coauthor): There are parts of the methodology that talk about the proxy. For media-related tests, we might want to look at end-points themselves.


Comment from Al: PMOL draft is now in IESG Review, and has collected several DISCUSSes (blocking comments). Vijay is working as document shepherd. Al will steer the current IESG comments to the authors for incorporation in BMWG drafts, as appropriate.



Milestones review

Five sets of docs “in the red”: Net traffic control, Router accelerated benchmarks, protection benchmarking, SIP, accelerated EBGP tests, Op Sec. All the Old, overdue, and expired items  have seem to have entered the DEAD state – they have been expired for a long time. There has been no response to requests to help-out on these toipics.


Comment from Vijay and Carol: June-July is reasonable for next versions of SIP IDs


Al: Can move terminology on protection benchmarking forward now


Al considering rechartering to clean up old work (Net traffic control, Router accelerated benchmarks, all long expired) and possibly adding new items


Basic BGP convergence is on the agenda for discussion today, this is a very old work item.


Work proposal summary matrix has currently has five items

            WG has seen Significant improvement in IP flow export

            Content aware devices (needs significant support at meetings)

            LDP convergence (waiting behind IGP, not clear if there is still interest to do this)

            Data Center Bridge benchmarking

            Router reset



Reset benchmarks (Al Morton, presenter, on behalf of authors)

Review ID’s motivation: “All resets are not created equal” – R. Bonica

Review of core methodology for hardware, software resets and reporting formats

Presented changelog.  David Newman provided extensive comments on the list, and also supports the work to continue.

Al calls for comments and review

Not enough reviewers at the meeting to make go/no go determination; Al will ask for input on the email list (Action).



IP Flow Information Accounting and Export Benchmarking (Al Morton, presenter, with Jan Novak’s remote agreement due to audio quality issues)

Review document goals and key concepts of flow monitoring measurement

Review main changes

            Merged normal and overflow caches into 1

            Moved misc. tests into Appendix B

            Clarified test setup with multiple observation points

            Expanded 2544 measurements

Proposed scope items are to develop terms and methodology, state goal of maximum IP flow rate, and assess forwarding rate performance of DUT/SUT


Comment from Chris Elliott: Chris volunteers himself and/or Benoit to help move this forward.



DCB Benchmarking (David Newman, Presenter)

There is interest among the bridge builders and the test equipment manufacturers to make the proposed measurements. Classic loss-limited Throughput of RFC 2544 will not work. There was agreement to pursue discussing the possible overlap with IEEE 802.1 via liaison. (ACTION)


Content-aware Benchmarking (Mike Hamilton, presenter)

Characterized 03 as a maintenance release

Removal of 2544/3511 wording and “realism” from document, but “underneath the hood” the ID is still aimed at predicting performance in production networks. “Realism was removed but underlying concept is still there.”

Explicitly calls out exclusion of capture/replay test traffic.

Believe they have added Enhanced repeatability to test.


Comment from Al on “realism removed”: Traffic characterization beyond fixed packet sizes is there because it can be well described and understood. This is less true of “real” traffic. “As long as the tests of a good metric are in your mind, that’s probably the most we can ask for at this point.” Al asks for presentation of evidence: Some explanation of test traffic and test results is often compelling and/or revealing.


Comment from Sarah: Something to be said about not having fixed packet sizes. So long as you denote what traffic mixes you use on that day, that’s about the best you can do. There is for me the notion of having some realistic traffic content. She’s had this request from multiple employers current and past. She is strongly in favor of bringing real traffic back.


Response from Al: We’ve only standardized fixed packet sizes because we understand what these mean, but don’t prevent vendors of test equipment from doing “Imix” or other kinds of traffic loads.


Comments from Brett Wolmarans: Offers to help flesh out test cases.  Al’s response – please take part on the list.


Comment from Joel Jaeggli: Would love to have an annex with repeatable tests described. Customers have two devices and want to decide if they’re comparable. They can do this with RFC 2544, which is straightforward. This is more nebulous. A benchmark may not reflect operational reality.


Al Morton: So that’s the thing to focus on: Document repeatable and representative traffic, and then decide whether that produces repeatable and reproducible results.


Joel: So the point where you know something is where you know you have repeatable results.


Comment from George Uttice (sp?): As we’re talking about realism, the question is whether there’s anything that’s closer to realism than a fixed-packet test. Anything that’s closer to “real” is better. Also, because vendors will tune for benchmarks, we don’t want to issue a non-meaningful benchmark.


Comment from Brett Wolmarans: Challenge is to isolate the DUT. We all need to follow a common, good methodology.


Comment from Mike: Less concern about the methodology itself and more about the offered load. Wonders if separate offered load effort would make sense first.


Al’s response: Try to keep them together for now, since methodology has a dependency.



Basic BGP convergence benchmarking (Al Morton, presenter, with Rajiv’s remote approval)

Review of background/motivation/current status of ID and review of document scope

Al asks whether route reflectors should be included in scope, and asks for discussion on the list

Review of test methodologies (a relative large set)

Review of next steps (publication of first draft in April, call for review and comments, and call for input from service providers)

Open question regarding control vs. data plane benchmarking (David Newman, as attendee is in favor of adding data-plane component)



Wrapping up

IGP-Dataplane drafts to WGLC

Move protection terminology forward Doc Shepherd Write-up

AM’s comments to SIP authors

SIP docs in June/July

Open question on reset, to discuss on list

Agree on paragaph on flow/export list – future Re-charter discussion.

Need liaison letter to IEEE for DCB

Think hard on traffic for content aware devices – Everybody!


Meeting concluded at 1500 PDT, on the dot!