Benchmarking Methodology WG (BMWG) THURSDAY, November 8, 2018 0900-1100 (UTC+7) Morning Session I Room OPS bmwg Remote Participation: http://www.ietf.org/meeting/103/index/index.html http://www.ietf.org/meeting/103/remote-participation.html -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Summary: BMWG met with 19 people present and 2 remote participants. Al Morton chaired the meeting, and Sudhin Jacob took the minutes. The Call for adoption on Next-Gen FIrewall Draft has just completed, three people offered support (and review) at the meeting, in addition to 5 or more on the list before the meeting). The WG chairs will consider the WG input and determine the outcome of the call. The WG EVPN and EVPN-PBB draft has made much progress addressing comments. The WG chairs will consider the WG input and determine if a WG Last Call is currently a good next step (not withstanding the many comments delivered at the meeting today, which would be considered WGLC comments during the comment period, as appropriate). There are also two new EVPN-related benchmarking proposals available as new drafts. The WG used the session for considerable discussion of buffer-size measurement topics, including the present draft to update RFC 2544, which has benefited from previous comments and additional experimentation. The WG chairs will consider the WG input and determine if a call for WG Adoption is a reasonable next step. There were several other new proposals in draft form, and a discussion of the cross-over between benchmarking test methods and Internet access methods, prompted by a Liaison from ITU-T SG 12 that describes a plan to evaluate these methods in a scientific way. Initial Lab results were shared with the WG, and there was good interest and discussion. The meeting ended precisely on-time! -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 0. Agenda Bashing 1. WG Status (Chairs) (Brief status below) There are many proposals for new work... No comments on the working group status. 2. Charter and Milestones (Chairs) ETSI NFV GS on NFVI Benchmarking Normative Specification Published https://docbox.etsi.org/ISG/NFV/Open/Drafts/TST009_NFVI_Benchmarks We had a draft of this spec available for comment in BMWG during the first half of the year. Now BWMG can appreciate what was accomplished here, in our field of interest. 3. Benchmarking Methodology for Network Security Device Performance Presenter: Samaresh Nair Related RFC: https://tools.ietf.org/html/rfc3511 Related Draft: https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-05 Related Web page: https://www.netsecopen.org/ https://www.netsecopen.org/about Sudhin Jacob support adoption. Also, Mike Ackermann and Barak Gafni added support as tghe meeting closed. Al: After the POC testing, what was something you learned from testing program, something specific in the draft? This is a request to share the results. Samaresh said he will share the results/answers in the future. He will provide info on the changes that are made. ACTION Chairs - call consensus on adoption. 4. Benchmarking Methodology for EVPN and PBB-EVPN Presenter: Sudhin Jacob Related Draft: https://tools.ietf.org/html/draft-ietf-bmwg-evpntest-00 Barak do you intend to test Cntrl and dataplane? SJ - yes, both, BGP type 2 routes MAC learning Barak - do you include both layer 2 and layer 3 prefixes? suggest to add type 5 scaling - believes this is needed. SJ: Had this comment, Decided not to add. Also not multicast. Maybe add another draft to cover these points. Barak - what underlays are included? MPLS? others? SJ: overlay is like a container, we measure what is in the container. Barak - underlay configuration may affect the overlay performance - when you go to the lab, it matters! SJ: Type 6 and 7 multicast considered for new draft? ACTION - Chairs consider WGLC for this draft. Related, NEW EVPN proposals: 5. Benchmarking Methodology for EVPN VPWS Presenter: Sudhin Jacob Related Draft: https://datatracker.ietf.org/doc/draft-kishjac-bmwg-evpnvpwstest/ Al asked ... ansewer: Scale and SOAK tests do test the dataplane Barak very important to test with number of peers and number of underlay destinations SJ VNI and others: there are in-scale tests, Overlay VRFs are considered. SJ will look at Type 5 for this 6. Benchmarking Methodology for EVPN Multihoming Restor. & Mass Withdrawal Presenter: Al Morton Related Drafts: https://tools.ietf.org/html/draft-morton-bmwg-multihome-evpn-00 https://tools.ietf.org/html/rfc8317 SJ has a comment on configuration parameters for mass withdrawl Al make some suggestions, we'll be glad to incorporate. Barak - how is trafic split between PE1 and PE2? we'll find oyut Continuing Proposals (topics may be added): 7. Updates for Back-to-back Frame Benchmark & OPNFV Plugfest/VSPERF Testing Presenter: Al Morton Related Draft: https://tools.ietf.org/html/draft-morton-bmwg-b2b-frame-03 OPNFV Plugfest (Fraser Release) VSPERF Testing Slides Background Slides: https://wiki.opnfv.org/download/attachments/10293193/VSPERF-Dataplane-Perf-Cap-Bench.pptx?api=v2 https://wiki.opnfv.org/display/vsperf/Traffic+Generator+Testing Al made the point that Yoshiaki's input had been incorporated in this version. Further, that all open issues with respect to search algorithms are now closed, through reference to ETSI GS NFVTST009 on NFVI Benchmarking. No other comments. Al said will consider the details for adoption of this draft. ACTION Chairs - consider call for WG adoption. 8. New Buffer assessment method for RFC 8239 Data Center Benchmarking Author: Yoshiaki Itou Mail List References: https://mailarchive.ietf.org/arch/msg/bmwg/lKiImpq8RlNapD8CVRG1dRZlMZ8 https://mailarchive.ietf.org/arch/msg/bmwg/elCgGpsB-TH1zCwaRhzM7B2mW4g Barak there are issues in 100g ethernet port testing, Barak suggested to decouple the pipeline versus buffer. Al there a lot of good benchmarks. Al suggested that Barak write down the ideas. Al said Barak is bringing a lot of expertise to the WG - much appreciated! Al if it 12 and 12 ok, it seems to be the linear case. Barak once you operate with pauses on some vendors SWE, it will change the architecture of buffers, and you will be getting wrong results. Barak question I like previous method pushing the two ports to one. Al 1 percent over-subscription show different results, 24 and 58 frames, so there must be some buffer which is not filled by the 1 percent flow. Barak the test/results have to be based internal architecture of switch (?? whitebox ??). NEW Proposals: 9. New Adaptive Search Algorithm from FD.io CSIT Presenter: Vratko Polak, Maciek Konstantynowicz Related Draft: https://tools.ietf.org/html/draft-vpolak-mkonstan-mlrsearch-00 Al requested to add bmwg in the draft name. Agreed! There are also improved versions of binary search, comparison would be good. The WG may continue to update RFC 2544 in a piece-wise way. 10.New Long-term tests for Loss characterization from FD.io CSIT Presenter: Vratko Polak, Maciek Konstantynowicz Related Draft: https://tools.ietf.org/html/draft-vpolak-plrsearch-00 Same comment about filename. Al says there are improvements coming to RFC 2544 like long term loss measurement. We should compare the efficiency of the different techniques, as they emerge. BMWG Communications: 11.Liaison from ITU-T SG 12: Presenter: Al Morton https://datatracker.ietf.org/liaison/1602/ Michel UDP cap limits. Wolfgang from DT told that many CPE has TCP prioritization. Doug Grinkemeyer has indicated interest to join the work. LAST. AOB Barak: said consider may be to extend the work on the RFC to measure buffer size. Fairness between two ports is important, and he commented this has interest in the industry. Al: There is also the QoS aspect, within multi-port fairness. Al: You said there is interest the industry. are you willing to bring-n tests to characterize the fairness? are you interested in bringing a new draft doccument? Barack said he will try. AD Ignas with the last word: We need to face the reality many topics discussed in the ietf is not known the industry. Need to achieve some visibility at Operator's forums, RIPE and others. Last point: Mike i may have missed the next generation firewall discussion, I am interested in that. Do I talk to you? Al I suggest you read the Internet draft. Al 3 people shown interest in this at the meeting, and about 5 on the list. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-