Benchmarking Methodology WG (bmwg) Thursday, July 13, 2006 1300-1500 Afternoon Session I (Room 519A, Channel 7 on Audio Stream) CHAIR(s): Al Morton <acmorton@att.com> The report of the BMWG session at IETF-66 is divided in two main sections: - Summary of the meeting and Action Items - Detailed Minutes of the meeting (typical of a Jabber Log) This Report was prepared by Al Morton, based on Tom Alexander's notes/minutes complied as official note-taker. 30 people signed the blue sheet, and about 5 participated remotely. Session Audio: http://limestone.uoregon.edu/ftp/pub/videolab/media/ietf66/ Jabber Log: http://www.ietf.org/meetings/ietf-logs/bmwg/2006-07-13.html Slides: https://datatracker.ietf.org/public/meeting_materials.cgi?meeting_num=66 BMWG Session Summary and Action Items ------------------------------------- The working group successfully completed several milestones in the past few months, and saw two drafts reach IESG Review. Of these, the Traffic Control Terminology Draft (dsmterm) was approved, and the Resource Reservation-capable Router Terminology (benchres) has an outstanding DISCUSS. There will soon be WGLCs an the Hash and Stuffing Draft, and the Accelerated Stress Benchmarking Drafts following the meeting. We were successful in getting people to sign-up and review most of the active working group items, but readership coming into the meeting was light. However, there were several constructive comments prompted by the presentations on the various work items. The volunteers and their Review Areas are listed below: Hash and Stuffing: Diego Dugatkin and Jerry Perser IPsec: Jason Guy and Scott Poretsky Accelerated Stress: Scott Bradner, Jay Karthik, Chip Popoviciu, and Jerry Perser Protection: (All the people who committed to review on the list!) There was a suggestion to prepare a Supplementary Web Page for BMWG, where the Active Review Template, non-charter Internet-Drafts, and other resources could be listed. This page will be noted on the BMWG Charter page at ietf.org. BMWG adopted new work on Protection Benchmarking, and will look for additional mailing list interest in adding IPv6 benchmarking to our charter with a work proposal following the meeting. There was also considerable interest in a SIP benchmarking Proposal, but the Scope needs to be defined much more clearly before we can consider it again. There is also a constraint to avoid overlap with the SIPPING WG, where a different (but somewhat coordinated) proposal is under consideration. Other new work proposals on LDP convergence time and MPLS forwarding benchmarks were presented in an effort to raise awareness of these work areas and create interest/readership. Regarding the remaining new work proposal, we will communicate with 802.11 T to gauge their interest in the WLAN switch benchmarking (discussed briefly). ACTION Items ------------ Initiate WGLC on Hash and Stuffing Draft - DONE Initiate WGLC on Accelerated Stress Benchmark Drafts - DONE Resolve DISCUSS on the Resource Reservation Terms - Add Protection Benchmark Milestones to the Charter - Communicate WLAN Switch Benchmark Proposal to 802.11 Liaison - DONE "Standard" Security Considerations Section for BMWG Drafts - Prepare a Supplementary Web Page to appear with the Charter - DONE (see http://home.comcast.net/~acmacm/BMWG/ ) ========================================================================== BMWG Minutes - Thursday July 13 2006 ------------------------------------ Meeting started at 1305 hours EDT. Tom Alexander was note-taker. About 40 people attended. Al verified that the people on Jabber could hear him. There were some difficulties with background noise indicated. Al reviewed the agenda and requested any bashing. He mentioned the intellectual property rights portion of the registration packet (yellow sheet). A new work item was IPv6 benchmarking, presented early due to time constraints. Al requested people to review the I-Ds in progress and clear them out so that we could get to the new work items. Al offered to do a tutorial on the XMLmind XML Editor in the interest of getting the drafts into better shape. He noted that some of the current I-Ds were poorly formatted, and showed an example without naming anyone. 1. Working Group Status (Chair) ------------------------------------------ Al reviewed the activity to date, including the drafts in review at the IESG. There was a DISCUSS against the Reservation-capable router benchmarking draft, put there by the AD. The DSMterm draft also had a DISCUSS but was eventually approved. The IGP drafts were at the AD review stage. Al said that we need a standard security section that we could place into all I-Ds, based on recent comments in IESG review. We should emphasize that our work is lab-only. There were two drafts recently last called: hash and stuffing, and IPsec. The milestone to move hash and stuffing to AD Review is in July. Al asked for at least one volunteer to read the hash & stuffing draft and give us a good strong review. Scott Poretsky signed up to read it (later de-volunteered). Diego Dugatin also volunteered to read it. Scott P. said that that draft had direct impact on test equipment vendors who do packet generation. Al noted that it had impact on IPv6 work. Jerry Perser also volunteered (via jabber) to read the draft. The IPsec methodology document needs readers. Jason Guy volunteered to read the IPsec terminology and methodology as well. Scott Poretsky volunteered to read the IPsec documents, and since there were already 2 others on the hash & stuffing draft he would not volunteer for that one. Al reviewed the expired I-Ds. Scott P. noted that the strategy on the accelerated stress benchmarking was to do it after the other terminology and methodology documents were done. Al agreed. 2. New work on IPv6 Benchmarking Methodology - (Popoviciu) --------------- Related Draft: http://tools.ietf.org/wg/bmwg/draft-popoviciu-bmwg-ipv6benchmarking-01.txt Chip Popoviciu presented on IPv6 benchmarking methodology. He noted that the WG feedback had been integrated in the current version of the I-D, and discussed the disposition of each individual feedback item. He reported that there was a lot of interest in this document and in this work; he mentioned active conversations one-on-one with other members of v6ops. In closing, he noted that the issue of having guidelines for IPv6 benchmarking was significant and he would like to propose the adoption of this work as a BMWG item. Al thanked Chip for his presentation and asked for comments or questions. He called for a show of hands on who has read the draft; about 3-4 people raised their hands. He said that getting 3 people to read it at the individual stage was a good start and the work was in good shape; also there was an impetus from the industry to take this up. Scott Bradner asked the question in reverse: how can you not take it on? He said that we did it for v4, we need to do it for v6, and BMWG should take it on. Al asked for a show of hands to support taking it up, and another show of hands for people who objected; about 15 hands went up in support, and none were raised for objections. Al said that there was impetus from all corners to take it up. Al asked Chip about the IANA portion of the draft; he said that he met with the IANA folks during their office hours, and they had suggested BMWG take a look at RFC 4380, and ask IAB to designate an address space. The IANA folks said that they usually assign /48 to networks today. Al wanted feedback on the size of the address space needed. Scott Bradner noted that we should request a space; he also clarified that it was the IANA not the IAB that does the assignments. Scott B. also said that BMWG should look at the intent and what we are trying to achieve; the original request in RFC 2544 was to provide enough space to support the size of the test network he wanted to setup. In this case, we might want to ask for a smaller space than /48, which is 16,000 nets, which seemed a bit heavy. Al suggested that the authors should propose a rationale. Chip noted that this was one of the topics that they were considering as well. Al said that he'd read through the I-D and there were a couple of editorials that should be corrected, and gave Chip a marked-up copy. There were no other comments on IPv6 benchmarking. 3. Techniques for Benchmarking Core Router Accelerated Stress Testing. (Poretsky) ------------------------------------------------------------ http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-term-09.txt http://www.ietf.cnri.reston.va.us/internet-drafts/draft-ietf-bmwg-acc-bench-meth-05.txt Scott Poretsky then presented on core router accelerated stress benchmarking. Scott Poretsky noted first that this work has evolved and it applies to any router, not just core routers. The current status is that the terminology is at 09 and methodology is at 05. He said they had passed a sort of litmus test on the mailing list; a reviewer had complained that the document does not have a pass/fail criteria, did not provide any qualification requirements and so on. But these are areas we must avoid in BMWG. Therefore, he said that this document clearly had fixed most of the issues of the previous drafts. He had also used the "idnits" tool and the methodology had zero nits. As all the issues raised by the group had been addressed, the document was ready for last call in the opinions of the authors. Scott Bradner said that he had not read the document, but the mental image that was brought to mind by accelerated stress was what you do to Twinkies in the lab. A new title might be in order. Scott B. promised to read it, but said it feels a little funny. Al had noted that the WG had struggled with the notion of stress tests in the past. Al also seized upon the interest, and Scott Bradner promised to read the document. Scott Bradner further said that he should establish a stable level for the aggregate of the test protocols, and then individually do a performance test like a throughput test, keeping in mind the old test dictum that you never change two things at once. Al asked for a few volunteers to fill out our active review template. Scott Bradner suggested that Al could put up a BMWG web page containing a list of I-Ds that are being tossed about but aren't working group items yet. Al said that if authors included "bmwg" somewhere in the filename, then the IETF tools would automatically grab it and link it from the BMWG Tools page. Al called for further volunteers. Jay Karthik and Chip Popoviciu both volunteered to read the document. Jerry Perser said that if you put a link to it on the BMWG webpage, he would volunteer to read the draft as well (these drafts have links on the BMWG Charter Page and BMWG Tools Page). 4. Methodology for Benchmarking Network-layer Traffic Control Mechanisms (Poretsky) ---------------------------------------------------------- http://tools.ietf.org/wg/bmwg/draft-ietf-bmwg-dsmmeth/draft-ietf-bmwg-dsmmeth-02.txt Scott Poretsky presented on benchmarking traffic layer control mechanisms. The terminology had already gone to IESG, this is the methodology. An interesting item that came out of the IESG review was classification expanded to include anything in the header, not just DiffServ. The difference between versions 02 and 01 of the methodology was to expand the test cases to the logical interfaces as well as the physical interfaces. He reviewed the methodology and the next steps. Scott closed by saying that the document was not yet ready for WG last call. Al asked if anyone had read the draft. Aamer said that he had read the documents but the comments were in his inbox; he would send them out. Thomas Eriksson said that he hadn't read the document, but asked if the document played with schedulers - are there different types of schedulers in the document? Scott said that the configuration of the QoS mechanisms to get any kinds of benchmarks was device-dependent. Thomas asked if the results were comparable across different devices; Scott urged him to read the document. Thomas then said that he MIGHT read the document. 5. Milestone Status (Chair) --------------------------------------------- Al then reviewed the milestone status. There were three things listed as done, so some space had been freed up. However, some short-term milestones were coming up: the stress methodologies, the hash & stuffing I-D, the IPsec work. Thomas Eriksson asked if there was anyone working on BGP. Al said that they hadn't been active recently, and asked if he was willing to contribute to that. Thomas said that he was interested in having it, but declined to volunteer. Marianne Lepp said that she was one of the authors of the BGP convergence terminology; it was virtually done, but not quite that virtually done to submit it; there were some people interested, and it would be nice to finish it up, and all agreed. ******* New Work Proposals ********* 6. Sub-IP Protection Mechanisms - (Karthik) -------------------------- Related Drafts: draft-papneja-mpls-protection-meth-merge-00.txt draft-poretsky-protection-term-02.txt Jay Karthik presented on MPLS protection mechanisms. He started with the history of the various drafts. The drafts have been merged, posted on the list and received a few comments on the merged drafts. He said that this should be ready to become an official work item. Jay covered the highlights of the merged draft, and the comments received against it. Current status was that they were waiting to hear from the WG leadership on accepting this work item. Al said that the WG leadership has gotten together, and believed that because of the strong support from the WG, we could take this up as a work item. Now people who signed up to review this draft would have to submit their detailed reviews and progress the work. He congratulated the whole author team in making this milestone. Al said that he had some technical comments. First he was struggling with the term called "failover event" which was also synonymously referred to as "recovery" in the document. Scott Poretsky clarified that there were two different events, a failover event and a recovery event; the latter was the restoral from a failure. Scott said that people seemed to like "recovery event" in preference to "restoration event". Al also said that he had a bunch of detailed comments in the terminology, and that we definitely needed a scope section in the methodology. Jay took the provision of a scope as an action item. Al observed that readership was a little light on drafts this time. Dave Kessons (AD) said that if the WG wasn't reading the documents, then the AD would not read them either, or submit to the IESG. He encouraged authors to solicit reviewers. Scott Poretsky said that the reviewers need not even be in the WG; he in fact encouraged authors to get non-WG people to review the draft. Al said that obtaining cross-area review is a recent initiative from the IESG. Scott Poretsky also suggested that new authors should start reviewing other work items, because that also provides valuable background for their own work proposals. Al went over slide 7 of his presentation, which covered the new work evaluation matrix based on several different criteria. He said that the sub-IP protection mechanisms had met all the criteria and it should be clear why we adopted it. Jerry Perser said (on Jabber) that he was looking for a general protection terminology/methodology to implement for a new technology. How do we apply MPLS or fast reroute to this new type of network. Al said that maybe there could be a general protection methodology that could be applied in these cases. 7. LDP Convergence Benchmarking (Old Work Proposal) - (Eriksson) -------- Related Draft: draft-eriksson-ldp-convergence-03.txt Thomas Eriksson then presented on LDP data plane convergence. Al described an earlier discussion with Thomas, where they covered his comment on restoral for the MPLS protection mechanism work. specifically for the make-before-break situation where there was a possibility of duplicated or reordered packets. It might not be applicable for this draft. Al asked the group if there were any questions on Thomas's presentation. There were none. He then asked if any had read the draft. One person had read it (Scott P); Scott had volunteered to be a reviewer. Al then gave a general action item to the group, to read the draft that Thomas had prepared and see if this was a work item that we should take up. A short poll of operators interest was taken, with inconclusive results among those in the room. Jerry Perser made some comments regarding lossless transmission, reordering (being an issue for TCP), and that duplicate packets are filtered by most protocols. Al responded duplicates and reordering are impairments that can be tracked as part of the recovery operation. Al also reminded Thomas to look at some terminology edits. 8. MPLS Benchmarking Methodology (Akhter) ------------------------------- Related Draft: http://tools.ietf.org/wg/bmwg/draft-akhter-bmwg-mpls-meth-00.txt Aamer Akhter presented on MPLS benchmarking methodology. He described the draft goals and the current state of the draft. He then requested BMWG to send in comments, and raised several issues. One issue was IMIX, which raised a lot of controversy. Another was negative test cases: was this out of scope of the WG? He wanted to force the issue of delay cases. Al asked if people had read the draft. Scotts Poretsky and Bradner and others had read the draft. Scott Poretsky noted that the reason that there were no comments was that 0.0 was just a table of contents, and 0.1 was just recently posted. He also asked about the terminology; Aamer said that the terminology and methodology were bundled into one document. Scott Poretsky said that this was redundant to all of the work items around MPLS, and further that this document was 8 years too late. Scott Poretsky said that one of his litmus tests was that BMWG should take up work that was generating excitement in other groups. Aamer responded by saying that he had reviewed the other documents and he did not believe that there was any overlap; Al backed this up, and encouraged people to read the 01 version. Aamer further said that MPLS is no longer sexy but a lot of people would say that about testing in general. Rajiv Sahi agreed with Scott Poretsky that this should have come out 8 years ago, but he wasn't convinced that it downplays the need for a methodology document that helped people in the MPLS industry deploy MPLS technologies. He further said that he had compared the documents with other MPLS work and he did not think there was any overlap. Scott Poretsky said that until we have a clear scope and a work item proposal we can't make any decision, and encouraged Aamer to come up with a clear scope. 9. SIP Performance Benchmarking (Poretsky) Related Draft: http://www.ietf.org/internet-drafts/draft-poretsky-sip-bench-term-01.txt Scott Poretsky presented on SIP networking devices. He covered the motivation, and said that service providers and vendors in the industry are completely confused by who really does what. He then covered the relevance to BMWG as indicated by e-mails. There was some discussion of device performance (which Scott said was the domain of BMWG) and end-to-end application performance (which Scott said was the domain of SIPPING). Henning Schulzrinne said that the division made some intuitive sense, but he was worried about it being as crisp in practice as on a PowerPoint slide. It's really not distinguishable that we can measure the number of calls/second that we can support, whether it consists of a proxy server or a whole farm of proxy servers. He asked if Scott could be a little bit more precise about the difference between device performance and network performance. He also said that there might be other division that could lead to less inter-working-group diplomacy. Scott Poretsky covered the scope; he said that this was the hardest thing that could be defined. He talked about various components of the scope, and said that there could be separate methodology documents covering each component. Scott then discussed various issues, like whether a SUT was required for benchmarking the media, and also whether there was an issue with NAT. The next steps was to determine whether this work was of interest to BMWG, and also whether they had any input for the scope of the work. He suggested that comments via the mailing list would be good. Scott Bradner said that he could have thought of more possibilities to add to the next steps slide, but it would take a large amount of time. He would, however, suggest that it was in order for this group to take a look at parts of the work, not necessarily for a SIP-in-a-box, but from the standpoint that this group was the right one for such work. Henning said that you have a narrowly defined functionality for a device - in one case you can determine that this piece of SW is faster than that piece of SW, and another case being this piece of SW is not working well and we need to find out. These are two different things. Further, a SIP system has a wide range of what it does, which doesn't apply to other systems. Henning gave three examples of customers for the performance work: Marketing/System Sizing, Debugging Engineer, and Customer of an SLA. Unless we are clear as to what customer group we are serving, we aren't going to be serving any of them. Vijay Gurbani added that we tried to put some thought into this. He had seen vendors reporting numbers, and we needed to think a little bit more. Al said that we needed a scope that was not too big and didn't conflict with SIPPING, and produced useful results. 10. Extending the current methodologies to cover wireless LAN switches and wireless LAN meshes (Perser/Alexander) ----------------------------------- Tom Alexander briefly presented on behalf of Jerry Perser on the WLAN switch benchmark proposal. He said that Jerry would present formally at the next meeting. As per a previous discussion with Al a request would be made to the IEEE 802.11T group as to determine whether they considered this item within their scope and would take it up. Al said that he would have a hallway discussion with Dorothy Stanley and request that this question be covered at the 802.11 meeting next week (this was accomplished by e-mail, instead). With that, Al closed the meeting. Meeting ended at 1505 hours EDT.