Benchmarking Methodology WG (bmwg) THURSDAY, March 26, 2009 1300-1500 Afternoon Session I Franciscan A CHAIR(s): Al Morton <acmorton@att.com> These minutes were prepared by Al Morton, based on detailed minutes provided by Aamer Akhter as official note-taker, with Matt Zekauskas filling-in while Aamer presented his draft and on a few other topics. SUMMARY BMWG met with 23 people attending and 4 people participating via unicast audio and Jabber. We have reached agreement to add an author to the IGP Datapalane convergence drafts, and hopefully can resolve comments with this very direct form of participation (in 1 month). The MPLS Forwarding and Protection Switching Benchmarking Drafts completed WGLC, and both have been revised to reflect the comments. MPLS Forwarding needs another short WGLC and then on to Publication Request. The IPsec drafts will re-appear shortly, and after a short WGLC should also go to Publication Request state. There was considerable discussion of SIP benchmarking drafts, principally the terminology needs to be coordinated with the SIP metrics in PMOL. Also, SIP overload test specifics (503-only, or 503 with retry) can be enhanced. The working group agreed that the IPFIX benchmarking proposal really improved, but the benchmarks still need to be more clearly defined. Also, the applicability to specific roles (devices that export or collect IPFIX records) needs to be clarified - both routers and standalone devices perform in these roles. Finally, there was a new proposal to expand the Firewall benchmarking to general Content-Aware network devices. There was a volunteer to help the author with the draft development, indicating that there was interest in this proposal as well. DETAILED MINUTES: AGENDA: 0. Agenda bashing (if we need to shuffle a few items) See https://datatracker.ietf.org/meeting/74/materials.html for Agenda updates and Slides Check the BMWG mail archive for comments: http://www.ietf.org/mail-archive/working-groups/bmwg/current/ Added short discussion on IPsec drafts by author Merike Kaeo. 1. Working Group Status (Chair) Al went over the agenda and BMWG activity slides. Some new activity, some expired items.* Nothing in the RFC editor queue at the moment. There are some new standard sections for BMWG memos - authors should add these... 2. IGP Data Plane Convergence Discussion GOALS: *Final* resolution of Kris and Peter's comments http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-meth-17.txt http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-term-17.txt http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-app-17.txt Presenter: Scott Poretsky Scott went over the presentation, update on the changes. Clarifying the terminology in the drafts and properly mapping the states in the testing to these terms. Kris Michielsen, participating via jabber, agreed to provide text to help address his recent comments on the -17 versions, and will become a co-author on the terms and methodology drafts. Hopefully, another version will be available in a month. Aamer Akhter: Will the route-specific convergence time benchmark allow the determination of the shape of the convergence curve (eg when 10% of routes are forwarding, 75% of routes etc). Scott & Al: yes, it's possible to calculate the distribution of convergence times, depending on the number of specific routes tested. In short, metrics are there to do this. 3. Protection Mechanisms Discussion GOALS: The DUT/SUT point was controversial, bringing-out opinions from old-timers, so this needs discussion. Also, there is a time requirement for stability in the methodology (to determine that a failover is successful and complete) 30 seconds. Drafts: http://www.ietf.org/internet-drafts/draft-ietf-bmwg-protection-term-06.txt http://www.ietf.org/internet-drafts/draft-ietf-bmwg-protection-meth-05.txt Presenter: Scott Poretsky Scott went over the presentation, update on changes. Noted that the scope has been narrowed, and definitions have been updated. Aamer Akhter: on note #6 (forwarding performance) it would be good to reference and align with the MPLS-forwarding-behavior for IP flows. Al passed on some editorial comments in the form of marked-up copy. Stefan Olafson: Is the case of NDR (no drop rate) during (and after) the convergence event covered within the document? Scott: Yes there is a test case covering that. Al: NDR isn't one of our defined terms in BMWG, it's the offered load under loss-less operation = Throughput. This is more a comment for the MPLS forwarding draft. The authors of the protection methodology draft are asked to see how they can reference the MPLS-forwarding draft, because the benchmarks are collected at the (max loss-less) Throughput offered load. 4. MPLS Forwarding Benchmarking Methodology for IP Flows Discussion Dicussion GOALS: Share, discuss and acknowledge the updates made after the last WGLC. Drive towards the IESG submission. Draft file name, or preferably the complete URL: draft-ietf-bmwg-mpls-forwarding-meth-02 Presenter: Aamer Akhter Aamer went over the presentation: everything starts as IP flows treat mpls as tunnel event. Al had some editorial comments in the form of marked-up copy, and put some comments on the list during the plenary last night. There were no other comments, so another WGLC (expected to be short). 5. SIP Performance Benchmarking Discussion GOALS: Introduce new WG item at -00. Request input/comments. Related Drafts: http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-term-00 http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-meth-00 Presenter: Scott Poretsky Scott when thru the document, updated on changes. Al: spell out everything you can in reporting format (slide 6). Darryl Malas: does the 'media protocol' metric mean codec or RTP? Scott: that is meant to be RTP. Darryl Malas: Suggest adding codec as well as this will significantly change the setup rate. Dan R: This does not seem aligned with the PMOL work, and there must be consistency between these two drafts. Darryl: There has been some work to bring the two drafts together, but there are some different metrics in each - need some suggested names for the benchmarking drafts. Dan R: This is confusing, if they are different items then they should have different names. Darryl: We've adjusted the names of metrics on the drafts several times, and may need Carol: given the late nature of the PMOL document. Can we just change the term 'ratio' Al: no. the ratio is the right word for what we are doing there. They are different metrics. Carol: Right, so we are talking about different things. Perhaps there should be explanation (in the benchmarking draft ) on how the metrics themselves are different. The PMOL work is on metrics outside the lab, and BMWG is about testing in the lab. Scott: I agree that a separate section (Appendix?) would be ideal. Darryl: Session overload capacity: There is a design team looking at overload in SIPPING. It is not clear to me how the overloading draft may be related to the BMWG draft. There are number of mitigation strategies (eg 503 and 503 with retry) that could be done, and the method that BMWG draft will need to capture these. Scott: The idea here is to have a mechanism agnostic method of observing the overload condition. This could be used to compare the performance between different mechanisms. Can you take a look at the test case to see if that test procedure would provide a mechanism-agnostic method? (Daryl nods) 6. IPsec Benchmarking Drafts Merike Kaeo (where are we on this, still in XML-Hell??) Merike provided a status update on both new meth and terminology draft (see list of items in the meeting materials). Scott: Based on experience with the IGP draft: if the draft is based only on IKEv1 then the title should represent that. Merike: It should be in the scope Scott: This was the case with the IGP convergence draft as well (link-state limitation was in the scope section), but it is much more clear to say this in the title of the draft. Merike: really want co-author for ikev2 and a co-author that is implementor, especially. Please send anyone my way! 7. Milestone Status and New Proposal Summary (Chair) Net traffic control draft is lost, so we may drop milestone, unless someone willing to pick up. Scott: looking for an author for this for a while. Daryl: while sympathetic to red on milestones, curious about upcoming ones. Only one real familiar with is SIP. Seems unrealistic that everything done by stockholm, Milestone set a year ago. ******* New Work Proposals ********* 8. IP Flow Information Accounting and Export Benchmarking Methodology Discussion GOALS: progress to WG document Draft file name, or preferably the complete URL: http://www.ietf.org/internet-drafts/draft-novak-bmwg-ipflow-meth-02.txt Presenter: Benoit Claise Scott: what is being benchmarked? Conclusion: The NDR for specific IPFIX collection and export profiles. There may be a "cost" in enabling a specific IPFIX profile on a device. This draft is to help identify that cost. Scott: it is unclear if the exporter is being benchmarking or the collector Benoit: it is the exporter Dan R: there should also be investigation of the collector as well. Brian Tramell: the draft currently assumes that an intermediate system (eg router) is the exporter. It is not a major modification, but the case of an IPFIX-dedicated probe needs to be allowed as well. Benoit: we need to have specific profiles (as there are many permutations) to capture what operators are trying to grab via IPFIX. Comment: not enough to report export rate, accuracy is imporant too. Scott: We should be able to capture sampled as well as non-sampled cases at various speeds. There are IPFIX exporters that are able to capture at 10Gig line rate w/o sampling. 9. Content-Aware Device testing methodology Discussion GOALS: 1. Discuss whether current methodologies are sufficient for testing the latest generation of network devices. 2. Understand whether there is support for creating a new methodology that focuses more on realistic application traffic rather than blasting ones/zeros/HTTP. Draft file name, or preferably the complete URL: draft-hamilton-bmwg-ca-bench-meth-00.txt (assuming it gets posted) Presenter: Mike Hamilton Mike went thru the presentation. Brett Wolmarans: Would like to help with this draft. Have some questions on the charter of the BMWG (lab testing vs. live testing). charter of bmwg. Al mentioned at begining. not live, testing in lab. a lot of refs to "realism" "realistic". Al thinks it needs to be developed more to see how apply to a lab test. Aamer: most important part, needs to be repeatable. Brett: take a look... refer to 2647 2511. 2647 is term doc followed by meth. may have helped with some things earlier. Malicious is subjective. Defining terms here would be better, so need a terminology doc. Mike: agree. but time constraints, did consider that route. completely sep term doc. Tried to use firewall term/meth as much as possible. Doesn't neccessarily correlate, maybe a new doc is way tto go. Brett W: why did u use throughput vs goodput. Mike: I'm happy to move to goodput. Al: thanks Mike for putting this together. Al had some comments on a marked-up paper version - gave to Mike. Meeting concluded on time, as Al thanked everyone for another good BMWG session! *** ***