Benchmarking Methodology WG (bmwg) Meeting Notes THURSDAY, July 29, 2010 0900-1130 Morning Session I Room: 0.8 Rome CHAIR(s): Al Morton <acmorton@att.com> [Draft August 16, 2010] This report was prepared by Al Morton, based on detailed notes provided by Steve Maxwell as official note taker. Ilya Varlashkin monitored the Jabber room, enabling comments to be displayed and read during the meeting. Mike Hamilton provided a point-to-point Skype call with Ron Bonica, our AD advisor, who was unable to travel at the last minute. Over 20 people attended in person, with two people participating remotely. The meeting required the full 2.5 hour timeslot, and ran-over by a few minutes. This report is divided in two parts: and executive summary with action items, and detailed minutes of the meeting (from Steve Maxwell). Summary of the BMWG Session at IETF-78 -------------------------------------- Expect another short WGLC on the RESET draft, and then publication request shortly after that (September). Some interesting questions on Software resets were raised at the meeting, and these will be worked further on the list. Kris Micheilson attended in-person to summarize the IESG Review of the IGP-Dataplane Convergence drafts. There was quite a bit of working group feedback on ways to address the comments. After the meeting, progress was made with Stewart Bryant on many of the recurring points in his open DISCUSS. There was also some agreement that many issue-areas raised can be addressed by exclusion from the scope. The BGP Convergence work is back on-line, with help from Sure Hares (and her presence will help with other routing-related areas). The IEEE 802.1 liaison reply gives a green-light on Data Center Bridging work, and progress was made on the Content-Aware Device benchmarking. Although the plan is to meet in Beijing, we may request an Interim meeting before or after IETF-79, to progress the work that cannot be presented there. ACTION Items: ------------------ Start another WGLC on the RESET draft when revised to address the post-submission-deadline discussions on the list and at the meeting. Revised versions of IPsec drafts needed (addressing IESG review). Revised versions of IGP-Dataplane drafts needed (addressing IESG review). WG comments on key issues (see the detailed minutes). Complete Re-chartering Discussions in the WG. Detailed Minutes of the BMWG Session at IETF-78 ----------------------------------------------- AGENDA: >>>>>>>Meeting Administration<<<<<<< Notes/Scribe: Steve Maxwell (steve.maxwell@jdsu.com) Question [Rajiv]: He would like to join the Skype call? Answer [AL]: indicated that only a point to point is being used to support the connection to Ron…” Jabber Client moderator: Ilya Varlashkin Ron Bonica attended the meeting via Skype [AL]: Commented about Intellectual Rights slide... 0. Agenda Bashing [AL]: Asked the question: “would it be ok to move the charter discussion to the end of the meeting??? ok?” The audience agreed with this… >>>>>>>1. WG Status and Milestones<<<<<<< http://datatracker.ietf.org/wg/bmwg Approved: http://tools.ietf.org/html/draft-ietf-bmwg-protection-term Recently Expired: draft-ietf-bmwg-ipsec-meth -05 2009-07-28 draft-ietf-bmwg-ipsec-term -12 2009-07-28 >>>>>>>2. SIP Benchmarking Drafts: brief updates and status <<<<<<< http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-term http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-meth Presenter: Al, for the authors Question [AL]: Can we get any reviewers for this draft? [Scribe]: No one responded. Question [ILYA]: Can't find benchmark SIP protocol or device implementation of Trans-coding? Answer [AL]: Indicated that it would be outside of the scope; control plane testing; device implementation of protocol; no tests of the media should be done! There is some in/out for media, but not tested...doesn't remember....transcoding would be beyond the scope of this draft, so “no” >>>>>>>3. RESET benchmarks: Update to RFC 2544<<<<<<< http://tools.ietf.org/html/draft-ietf-bmwg-reset-01 Presenter: Andrew Yourtchenko, for the authors Question [STEVE]: Additional clarity of the use of networking protocols would be helpful. For example, the draft calls out the use of MPLS, but would it be a good idea to specific other networking protocols as well? [AL] responded with "clarity is always good!" [AL]: We will start another working group last call for this draft since we have more comments Question [ILYA] asked the question about when we perform a test to include when a device requires an upgrade of the system software; how to handle this in the draft; do we have two different cases here? [AL]: mention that perhaps we should describe different details for this [ILYA]: "This is my real world issue and I need to measure devices for upgrades!" [AL]: indicated that this might be beyond the scope of this draft..but it is a great question..... [STEVE]: if we run the test before the software upgrade and then again, the tests are meaningless because we are now doing a comparison between different things! So, the best thing would be to perform the upgrade, and then run the test. However, if you want to measure the amount of time it takes to perform an upgrade, then that is a really different test all together. Also, if the issue is that you want to establish a baseline of performance across system releases, then care should be taken because the configurations might be slightly different due to the introduction of software features or bugs. >>>>>>>4. Benchmarking Link-State IGP Data Plane Route Convergence<<<<<<< Need to address results of IESG Review http://tools.ietf.org/html/draft-ietf-bmwg-igp-dataplane-conv-term-21 http://tools.ietf.org/html/draft-ietf-bmwg-igp-dataplane-conv-meth-21 Presenter: Kris Michielson [AL]: indicated that we are working to resolves these issues mentioned on the slides.... Answer [KRIS]: responded with that he agreed, but noted that not a lot of interest was seen. Question [SUE] asked about slide #8, why put broadcast out of scope? She noted that networks have high percent of this traffic and is a reality moment! Dave Ward made this comment before.. ”I’m channeling Dave here”, replied SUE, she also said:..."you should take it seriously and if you need help, please let me know... I'm glad to help with the wording...." [SUE]: Asked to please explain slide #7, why you want to count the duplicates as loss/or out of order..it would be important to tell the difference; ethernet vs point-to-point for example. Indicated that she would be glad to help here. Also, she said “don’t' ignore this....fix it... these are real problems... I misunderstood your comments...about blending these together.... My apology for missing these comments..." [KRIS]: indicated that he was NOT going to combine the duplicates [AL]: Indicated that on Slide 6: from a test method point of view; focus on packet loss... and indicated that more work needs to be done. He said “For example, typically a link is unavailable, to reroute the traffic and when we have loss, look at the full convergence process. But, microloops or software bugs ([SUE ]comment), how can we combine these impairments in the methodology to say convergence is complete??? Thus, we need to measure these events...ie. out of order packet, say T=3 seconds, this event started the re-convergence... is this a reasonable way to go??? or should we just report loss based convergence?” [SUE] said: “so, I recommend content: how does this happen? What software/hardware commodity products...if you are have commodity hardware it can happen based on hardware issues based on many things, hiding it is a problem. This means we ask the question what is this? what happen here?, so I'm always operational we shouldn't hid it... and from a coder point of view; I don't know what operators think about this? I'm giving you a data point here.." [KRIS]: I think it depend on the application...the kind of threshold... but I want to know if we have duplicate or out of order... [SUE]: said “what are you considering?” [KRIS]: said “It depends on what you want...set a threshold for the packets and then report...” [SUE]: she indicated that that they are aligned on the approach...then said “so, since you are talking black box...if we are doing benchmark, we need to fix this...to learn the characteristics” [SUE]: indicated that the threshold might be possible, and said “ i'm a router and testing... reading a report on convergences...we might need a configuration piece... this is what you want to report? and am I clear about this?” [KRIS]: indicated that you can use the forward delay threshold... [SUE]: said “precisely! Truth in advertizing...this is necessary!” [ANDREW]: mentioned that during measurement of convergences, it doesn't cover all the aspects...such as jitter. [KRIS]: said “so, are you suggesting that another threshold be set? for Jitter?” [ANDREW]: indicated that we can talk off line about this… [KRIS]: mentioned that we can set for jitter, we can consider another thresholds too. [AL] : indicated that the problem is that the tester may not know the acceptance value for the device...it would be one of 10 or 100 devices in the network. So, this might be difficult. [ILYA]: mentioned that in the current state, the draft of convergence of the protocol can be an issue and asked if it should it be atomic or be split? Do we have more than one characteristic here? [AL] : Asked if it is possible to measure RIB to FIB transfer (supposing not)? [AL/ILYA/KRIS] Discussed the issue of getting the convergent values and doing comparative testing and perhaps this would be difficult to do. [AL] Indicated that “Yet, that is a reasonable note to all... we might to see the convergence...informative drafts...and could show the time to take the data plane without getting to specific...” [AL]: indicated that he was happy with the discussion and thanked the group. >>>>>>>5. Methodology for benchmarking MPLS protection mechanisms<<<<<<< Need to consider IGP feedback and revise/submit http://tools.ietf.org/html/draft-ietf-bmwg-protection-meth No presentation - discussion to follow IGP-dataplane item. [AL]: indicated that some of the impairment is covered in other drafts; the IGP-dataplane feedback should be fully addressed, and possibly loss-only methods might be deprecated. [AL]: asked Rajiv if he had any question and Rajiv didn’t respond via jabber and after a moment, assume “No”. >>>>>>>6. Basic BGP Convergence Benchmarking Methodology status<<<<<<< Related Draft: http://www.ndzh.com/ietf/bmwg/draft-papneja-bgp-basic-dp-convergence-003.pdf Presenter: Sue Hares [Al] "Ancient" work item, with renewed interest. [AL/ SUE]: Discussed the issue of ISG member work and how this related to BGP. Sue is of the opinion that the churn is a big problem and using/defining the right test is important. Sue also mentioned that working with the vendors is a very big deal too. One challenge is the difference between traffic interface/vs routing interface. [ILYA]: Asked “I’m trying to understand why” [SUE]: replied “because it exists!” [SUE/ILYA]: Had a detailed exchange about the need for using double box configurations when testing and having different hardware setups. [SUE:] ask if Rajiv agrees about what she said... and then asked “Did I miss anything?” From jabber, Rajiv indicated that all was good. Question [ILYA]: “in the 5.3 draft, will it include keep alives or loss per route address?” [SUE:] responded “good question: two answers (1) we don't know... if you are looking at a blackbox, we can have an IGP failure... so, at a certain level, we don't know (2) we should note everything that happens when we do this test.” [ILYA]: then indicated that we need to find out how they ran the test... [SUE]: she indicated that “Yes, we have to record... using the same reporting and scenarios...we need to report everything..I 100 % agree...” [AL]: said “thanks for bringing this work back to life!” [SUE:] said “we need to make sure Rajiv hears that too!!!” [Jabber Rajv]: typed “Thanks Sue!” >>>>>>>7. Re-chartering: Proposed Charter text and Milestones<<<<<<< New Work Item Proposals (status: some of these topics will not be presented): [AL]: Slide #9 New work proposal summary; we are in good shape! >>>>>>>8. Benchmarking Methodology for Content-Aware Network Devices<<<<<<< GOALS: To discuss the updated draft and discuss future of draft Draft file name, or preferably the complete URL: http://tools.ietf.org/id/draft-hamilton-bmwg-ca-bench-meth-04.txt Presenter: Mike Hamilton Question [ILYA ]: said “you want to compare specific networks? What about A or B vendors? Should we have some profile for this using IMIX?” [MIKE]: then indicated that he understood...and then also mentioned that he didn’t care about getting similar data from each vendor using RFC 2544; because that did really help him. [ILYA]: made the comment that we need some guidelines for something similar to IMIX so that people can measure their own network...how can we correlate when doing lab tests?... since they will need the traffic patterns/data. [MIKE]: mentioned that the draft contains reference traffic information that can help when performing the tests. [AL]: indicated that this information could be in the appendix section of this draft. [MIKE ]: mentioned the he also agreed. [MIKE/ILYA]: Back and forth on discussion on different approach when using the IMIX. [AL]: said “if we standardize on a mix of test frame sizes, we WON"T call it IMIX!!” [ILYA]: made the comment about trying to understand your test (slides #10).... traffic pattern…. [MIKE]: mentioned that it describes real traffic, real URLs, and attachments, etc. in this case of this slide: not specified) [ILYA]: asked “you want to measure DUT performance or efficiency of load balancing?” [MIKE]: mentioned that he tried to make the point that if a vendor’s data sheet says, 150,000 connections per second, is this true? He wanted to know how they got that number. [ILYA]: mentioned that for load balancing, IMIX might be good for certain cases. [MIKE]: mentioned that he spent time testing different types: cache, etc and that this is a good discussion, but then added “we should take this offline….” [ANDREW]: made the comment that load balancing may not be good traffic profile... [MIKE]: responded that indicated that perhaps these devices are not the best... [ANDREW]: they replied “and those constrains depend on mode..right?” [MIKE]: said “Yes, absolutely! Not talking only of load balancing...” [AL]: interjected “when you say random..don't you mean pseudo random?” [MIKE]: responded “Yes! my company does this” Then mentioned that when you use seed 10 for example, it would provide same result consistently. [AL]: mentioned that it would be better if two guys across the country can run the same test setup. [MIKE]: said “yes, but we need more work...and I could use some help too...” [AL]: made the comment “in general, I think you made some good progress... but some definitions still need some work...but I'll add some more comments on this offline..” [MIKE]: said “let me know...” [AL]: said “we want to be exhaustive in specifying the traffic here... let's touch on these later...” >>>>>>>9. Benchmarks For Data Center Bridging Devices<<<<<<< changes in the new draft, expressions of interest Liaison Reply: https://datatracker.ietf.org/documents/LIAISON/file1083.pdf Presenter: Al Related Draft: http://tools.ietf.org/html/draft-player-dcb-benchmarking-02 [AL]: he mentioned the issue of having to deal with larger frame sizes.. such as with jumbo frames. Question: [DAN ROMASCANU]: “it this true; are we sending these larger framc sizes...?” [STEVE]: “Yes, but for FCoE there is different ethertype, so I don’t think this will be a problem” [DAN]; He agreed and didn’t think we would have a problem with the conflict with the 802.x group. [STEVE}: asked “would it be helpful to update the draft to include language that make it clear that if a vendor claims support for a certain number of queues, the when executing the DUT tests, that it is required to have all the queues as port of the test?” The primary reason is that these queues are vital to the execution of the flow control mechanism with PAUSE frames. >>>>>>>10. IP Flow Information Accounting and Export Benchmarking Methodology<<<<<<< scope of new work paragraph discussed on list updated it for all the comments received + some other changes, hopefully toward better clarity http://tools.ietf.org/html/draft-novak-bmwg-ipflow-meth-05 Open call for comments >>>>>>>11. Power usage while Networking draft-manral-bmwg-power-usage-01.txt Open call for comments 12. AOB (there may be some time for additional items) >>>>>>>Re-charter Discussion now! <<<<<<< [RON]: asked “did you want to talk about the motivation for the re-charter?” [AL]: “yes” [RON]: Explained the reason for the motivation ...”when we do control plane work, we get into trouble with the IESG review process! I'm seeing a pattern... with the IPSEC drafts for example, etc! The forwarding plane goes through easy..!, but the control plane work gets held up.” they have issues with the terminology drafts , he indicated. Then he mentioned that perhaps we could ask another WG group to do the documents for us… [RON’s Skype Connection Dropped] [RON]: “The problem is that this is like the fox guarding the hen house!!!” He indicated that If we ask them to handle the docs for our WG. [RON’s Skype Connection Dropped Again] [RON]: Repeated the phrase “The problem is that this is like the fox guarding the hen house!!!.”..and then he mentioned that perhaps we can’t trust the other WG to do the documentation work. [AL]:” thanks Ron...” [AL]: “I don't think we can turn over the writing of these docs, because they won't do it...” [RON]: “I agree...” [AL]: indicated that Ron and he tried to get back together to review the chapter text again...which makes the joint ownership for the docs... Thus, [AL ]: asked if “we have a partnership for this effort? “ then indicated that perhaps individual to give us reviews and work out the details... for example, content aware,” we need a combo of groups to help with this effort...” “Ron has made some good suggestions to help us...” [DAN]: mentioned that speaking as a contributor he asked the question “does this working group test vendor equipment and is this historical…?” He was concerned about changing the approach because relying on other groups could be problematic for this WG. He explained that the primary problem is one of priority between WGs. [AL]: “thanks!” [ILYA]: ”The 4th paragraph of the Charter, the sentence is confusing” Should say” It is an implementation of technology” [DAN]: confirmed that it is a ‘technology implementation’ description. [AL]: replied “Good point, the main point is the delivery scope: Lab only... I'll move words around and have better description...” [AL]: “we might need to pare the list of new projects down... need milestones and agree on the text..... any final comments?...thanks for participating today!” EOT