Benchmarking Methodology WG (BMWG) March 27, 2012 1300-1500 Room 212/213 Audio CH 1 BMWG met with 23 people attending in-person, while several authors participated remotely using Unicast Audio and Jabber. These minutes are divided into two sections: an Executive Summary and the Detailed Minutes captured on Etherpad. Al Morton prepared the meeting report using a post-edited version of the Etherpad minutes. Bhavani Parise, Dean Lee, and Ilya Varlashkin were official Note Takers, and Paul Aitken scribed and monitored the jabber room. SUMMARY: There are two IETF Last Calls in progress, and the protection methodology has significant comments so far. Ron asked that the comments be fixed ASAP, and he will pull it from the IESG agenda for next week to cite the revised draft instead. The standards security paragraphs are working as designed, and Ilya suggested that this might become an XML option that people could simply turn-on and have the text inserted auto-magically. The WGLC on the RFC2544 Applicability draft is complete, and the AD(s) need to evaluate consensus since the chair is a co-author. There will be a WGLC on the IMIX draft, although there was a comment to provide more info about traffic modeling methods. The BGP convergence drafts have been updated immediately after the meeting, and they seek adoption as WG drafts, following presentation of test results using the methods in BMWG and in IDR WG. The SIP Benchmarking work seeks WGLC and Expert Review (already requested from the RAI Directorate). There was a request to check the SIP drafts alignment with the PMOL SIP metrics in RFC 6076, and authors will work this topic. The Content Aware work has made some progress, along with the Security Effectiveness individual draft. There was also a request to have test equipment vendors verify the operation and configuration of their TCP stacks, as this can affect BGP testing that makes use of TCP (message to be sent to the list). AGENDA: 0. Agenda Bashing 1. WG Status and Milestones Published: Benchmarking Link-State IGP Data Plane Route Convergence State 'Testing Eyeball Happiness' as an Informational RFC 1B. IETF Last Call comments on: <draft-ietf-bmwg-protection-meth-09.txt> from Joel M. Halpern and Gregory Mirsky 1C. IETF Last Call on IP Flow Information Accounting and Export Benchmarking Methodology Drafts not presented at this meeting: (none) -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 2. SIP performance benchmarking terms and methods Presenter: Vijay Gurbani Revised Definitions, added baseline test model (tester emulated agent only), Added a description of the testing algorithm for Session Establishment Rate, and revised test procedures to use it. 3. Basic BGP Convergence Benchmarking Methodology status Presenter: Ilya Varlashkin (likely) draft-papneja-bgp-basic-dp-convergence draft-varlashkin-router-conv-bench Status: We will be sharing status updates and edits we did based on the review so far. We will also try to share some feedback that we have received from the community. We would like to continue to request feedback from the group and acceptance as the formal work group item. 4. Benchmarking Methodology for Content-Aware Network Devices Presenter: Mike Hamilton There are some fairly extensive updates in the draft in response to Tom Alexander's comments, as well as Al and Barry Constantine. Since Tom provided so much detail in his comments, authors addressed them one by one as noted on the list. Some comments are not addressed in this version due to time constraints. 5. IMIX Genome Presenter: Al New Run-length coding section 6. RFC 2544 Applicability Statement: Presenter: Al WGLC completed Mar12 with minor comments 7. Milestone Review 8. New Work Proposal Review/Matrix LAST. AOB -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- DETAILED MINUTES: 1. Ron Bonica mentioned that the comments for draft-ietf-bmwg-protection-meth-09.txt need to be addressed 2.Security section need to be added to all drafts - Ilya suggested that XML option to be provided to add Security section 3. BGP Conv benchmarking presentation - Bhavani Parise & Dean Lee Last-minute updated slides, will be published shortly. Background: term doc published in 2005 (4098), started meth on BGP. Two drafts dp-conv..-03 and -router-conv-bench-00 Got feedback from providers. They find it valueable and important. Some test cases were proof-tested - results are good. Newer vers of dp-conv draft posted few days ago, addressed comments from IETF82 and the list. Solicited feedback from IDR WG - presented there during IETF83 (Monday before BMWG). Current dp-conv includes IPv4 and IPv6, want to look at other AFI and e.g. Route-Reflectors. IDR folk notified about their work on L3VPN scaleability. Dean is now presenting, reporting on proof-of-concept test results. RIB-IN conv being presented. Config as per draft, 2 ports on tester emulating S and R1. 1000 prefixes (/32) with unique next-hop. results presented: Average conv 182.5ms, Std.Dev 1.75ms. Next, test 5.7 presented - measure conv time for changing PATH attr change, new attr is better than old. Same conf as prev case. Results: avg 167ms, 2.63. Tried tests with different vendors - results different but consistent within given vendor, good repeatability within vendor. Carrier-class router vs. low-end - big diff as expected. Results can be used to compare platforms. Presented simplified topology will be adopted in new version of the draft. New version NOT published yet, but will be shortly after the meeting. Ron comments: do you have case with widthraw small number of route? Answ: yes Ron: in report have saturated CPU or wait for TCP? Is due to loss or smth else? Answ: observ from outside, not looking into the box (blackbox). Report and config should record CPU and other Ilya: need feedback on testing convergence <1 s for large number of prefixes. Solicitating comments on the list or direct mails to authors. 4.Benchmarking Methodology for Content-Aware Network Devices by Mike Hamilton - gave details about the Malformed Traffic Algorithm - presented list of resolved questions & comments addressed from IETF82 - defined and gave details about Markov code and algorithm Paul comments: slide-5, random& will it clear some bits - Mike replied that it was used to truncate some of the fields Al comments: do not see any MUSTS in the draft only SHOULDs - Mike replied that in the interim he has them as SHOULDs but will finalize and add the MUSTS Al comments: TCP stack considerations, couple of references to congestion window sizes also wanted to take a look at award winning paper from Taipei about DUTs and TCP behavior on opposite sides of the DUT (Al to provide this paper to Mike and Sarah) Aamer Akhter comments: question on scope, should scope be tightened since other references are present regarding WAN optimizations etc 5. IMIX Genome by Al Morton - gave details about the Scope and Spec Aamer Akhter comments - likes the idea, there is not discussion about how to get a good IMIX. Al replied citing CAIDA methodology and said he added some text into the draft regarding how to acquire this 6. RFC2544- draft-ietf-bmwg-2544-as-0 - by Al Morton - WGLC complete, thanks to Bill and Kenneth for comments, no additional comments on revised version. 7.Milestones & New work Proposals - Security Effectiveness - Al: I encourage people to read it, draft has been enhanced and improved 8. SIP Performance Benchmarking - Al Morton presenting for Vijay & Carol - Please to see the test case description is shorter now - Require expert review, approved by Ron, Al will seek RAI Directorate review - Aamer- should align and coordinate with RFC6076 (agreed) - Al - slight inconsitency between draft scope and current BMWG charter item on SIP 9. All other business - open proposals? - iLya - There are some issues with test equipment to verify BGP router convergence. It is not possible to test convergence happening between two ports. The issue is to maintain the same TCP session with a emulated BGP router across two ports. We like to call for other test equipment vendor to look into this test challenge. Al: suggests to post this question to the list, where vendors are present/listening. EOT