Benchmarking Methodology WG (bmwg) Thursday, Dec 6, 2007 Morning Session I 9:00 - 11:30 Oak Room CHAIR(s): Al Morton <acmorton@att.com> DRAFT as of 12/21/2007 This report was prepared by Al Morton, based on notes provided by Tom Alexander as official notetaker. This report is divided in two parts: and executive summary with action items, and detailed minutes of the meeting. Summary of the BMWG Session at IETF-70 -------------------------------------- BMWG had 26 people joining the session locally and 2 participating remotely. The IGP-dataplane convergence Design team has revised all the drafts, and responded to almost all of the DISCUSSes. There are about 3 items left to address, including capturing the variation in convergence time over prefixes (min, max, ave). Unfortunately, the revised milestones did not anticipate the additional effort required on the IGP dataplane drafts, and BMWG is about to fall behind on the Accelerated Stress and IPsec items. The Accelerated Stress Benchmarking Drafts require revision before another WGLC, and Merike Kaeo will use a different "trust model" with her IPsec draft co-authors and revise the drafts again, based on extensive comments. Merike will also examine the IKEv1 issues raised at the meeting. There were many recent e-mail comments on the Sub-IP Protection Benchmarking terminology, and these will be addressed shortly. Work Proposals on LDP convergence time, Multicast VPN, MPLS Forwarding, and SIP device benchmarking were also presented. LDP convergence time work is dependent on both the IGP convergence work (this was accepted) AND the MPLS Forwarding work for it's RFC2544 extensions to measure Throughput level of offered load. The Multicast L3VPN spec that the benchmarking draft serves has not reached WGLC, and although there is support for the work it seems wise to wait another meeting cycle before considering this further. The authors had addressed the WG comments and revised the scope to make most of the benchmarks general and applicable to any M-L3VPN arch. The MPLS Forwarding Benchmarking draft has been substantially revised to clarify the motivation and reduce the number of test cases to a minimum. There was good support for this draft and good readership (it is fundamental proposal, like the IPv6 work). The SIP Device Benchmarking drafts were augmented, not too many had read it in the room - most SIP experts were elsewhere, so we need to examine support for this on the list. Action Items ------------ 1) Al to send a link to Sancho on the BMWG reflector. DONE 2) Tom Alexander, Silvija Dry and Matthew Thomas volunteered to read the protection mechanism documents. 3) Yaron Sheffer to type up a list of issues with the IPsec documents and post on the list. 4) Al to investigate the issue of IKEv1 vs IKEv2 and report. DONE (Merike's mail to list) 5) Jay to ask the WG via the list as to whether LDP scalability should be included in the draft. 6) WG to help sort out the name "partial convergence" and converge on a consensus solution (on the list). Detailed Minutes of the Meeting ------------------------------- Al Morton opened the session at 9.00 AM in the Oak conference room. Tom Alexander was note taker. Gunter volunteered to watch the Jabber session. About 25 people attended. About 4 or 5 people indicated that they were attending BMWG for the first time. 0. Agenda Bashing Al reviewed the agenda. He asked everyone to read the yellow sheet discussing IPR, and also to sign in on the blue sheets. Some updates to the presentation titles (mainly names of presenters) were made. He then asked for agenda bashing. Yaron Sheffer requested a slot for firewall performance. Scott Poretsky said that he wasn't prepared to address the accelerated stress work today. Jay Karthik noted that either he himself or Aamer Akhter would address MPLS. Al also said that he would review his BMWG 101 slides presented the routing area, if there was time. With that, the agenda bashing was completed. 1. Working Group Status (Chair) Al then reviewed the activity since the last meeting. He said that there were 4 drafts that had gone through IESG review and were now back in the WG. Two last calls were completed, one on the accelerated benchmarking drafts, where Al had some comments outstanding. In addition, there were 5 drafts being reviewed, including the IPsec work. Yaron volunteered to provide comments on the IPsec drafts in the future. Scott noted that DSM methodology (DiffServ devices) was now expired, and there had been various authors along the way who came and went; he called for volunteers to help complete this one. Al discussed the work proposals pending, and gave the status on the various I-Ds that were outstanding (but not yet BMWG work items). He then noted that there had been no new RFCs issued to date since the last meeting. Jay Karthik noted that he had volunteered along with Rajiv Papneja to get some BGP work done along with Sue Hares. They had missed the deadline, but would continue to work along with Sue to get something in place before the next meeting. 2. IGP Data plane convergence benchmark I-Ds - Publication Requested - IESG Review Having completed the status update, Al moved to the presentations. He started with the IGP Dataplane Convergence work by Scott Poretsky. Scott presented on the three drafts dealing with IGP data plane convergence benchmarking. He noted that the Routing AD, David Ward, had posted a number of DISCUSS items, and so the group had formed a design team to address them. He believed that most of the Discusses (including those from BMWG ADs) had now been addressed, and so he would present the changes. The first change addressed a comment from Dan Romascanu, and responded to the comment by inserting the term "Link State" into the titles. This was acceptable to Dan. Al mentioned that he had re-read the original work proposal message from 2003, and it confirmed that the scope was limited to Link State IGPs at that time. Scott then moved on to the comments from the Routing ADs, going through the Considerations, Terminology and Methodology documents in turn. A new benchmark metric had been added, indicating the first prefix convergence time. He also noted that the diagram indicated a time axis that increased right to left to match what commercial test equipment indicated, and said that a sentence had been added to that effect. There was some discussion between Scott and Al on this topic. Todd Law from Agilent noted that the test equipment from his company showed the time axis going from left to right. Scott professed surprise at this. The units of time had been clarified in the revised document, allowing for microsecond convergence times. Silvija Dry noted that in lots of other drafts we have similar types of issues around convergence instants or convergence points, and wondered if we should have a general document that describes the convergence points. Scott responded that he was hoping that the future documents would simply leverage this document. Al noted that preparing general terms was hard to do, because BMWG typically works on one technology at a time. The term "Restoration Convergence Time" had been changed to "Reversion Convergence Time" to address yet another Discuss. Some terms had also been clarified in the discussion sections in the terminology document. Scott covered the changes to the methodology document as well, noting that two more test cases had been added to cover local administrative shutdown as well as remote interface failure. Aamer asked whether the local administrative shutdown addressed bringing the link down or bringing the protocol down. Scott clarified that it addressed bringing the link down. Aamer said that this was to address graceful restart, and Scott responded saying that the document specifically did not cover graceful restart. Scott highlighted the additional items in the reporting format, saying that this created more work for the test engineer but were essential to be reported. He also noted that the recommendation for packet sampling interval was lowered to 10 msec rather than 100 msec, and asked for feedback. Todd said that this was a contentious issue in Agilent and he would have to take it offline and get back to Scott on whether this could be addressed. There was some discussion on this topic, with the note that Kris had asked for the number to be increased to 100 msec based on test equipment capabilities. Rajiv Asati noted that the lower value was better because it allowed the equipment vendor to improve the granularity of results. Scott agreed. Gunter brought up a Jabber note from Kris Michielson regarding the sampling interval, saying that the sampling had to be done slower than the time it takes to send a packet for each route affected by the convergence event. Finally, Scott reported that the design team had agreed (along with the Routing AD) that NSF, graceful restart and RIP were out of scope. Aamer raised the point that NSF from Cisco's perspective was graceful restart, but Scott rebutted that Cisco's website differed on this topic. Aamer also brought up NSR. Scott mentioned that there were a couple of Discuss items not yet covered in the document: partial convergence and partial convergence time, dealing with convergence to one or a group of route entries, as well as convergence due to link insertion and route withdrawal. Al, Scott and Silvija discussed the title "partial convergence"; during the discussion it came out that this really meant something like "specific route entry convergence". Scott noted that Cisco and the rest of the industry used "partial convergence" in this context. We could use "Partial Route Convergence" if people liked it, but the general issue was with the word "partial". Todd noted that this was a classic design tradeoff between granularity and scale, and the first question was how many routes would be covered under these per-flow measurements. Scott responded that this was dependent on the capabilities of the equipment. Rajiv Asati noted that partial route convergence was used by Cisco in a different context, i.e., SPF, and not in terms of routing table updates. Aamer mentioned that we should not even use the term routing table, because it was the forwarding table that was being updated. Matthew Thomas said that partial convergence is very important these days. Jay Karthik asked if this could be called "incomplete" or "fractional" convergence. Scott proposed that we should take this on the mailing list so that everybody could add their input. Kris suggested the term "per-flow measurement". Finally, Scott urged everyone within the WG to read and understand these drafts before they went back to IESG. Al echoed this. With that, Scott closed the presentation. Al thanked Scott for the presentation, and then turned the floor over back to him for a discussion on Accelerated Stress testing. 3. Techniques for Benchmarking Router Accelerated Stress Testing. Scott gave a short introduction to this work for those who were new to the group. He said that we had done that work item, and were proceeding towards WGLC when it was decided to separate out the methodologies specific to each technology, and keep a single guideline methodology document that was common to all the methodologies. For now, most of the work was focusing on the common guideline document. Scott noted that Al had sent in some excellent comments on this document and he would be addressing these and sending the document out for WGLC. Scott also asked the group to review the document extensively because this was one of the more complex documents being worked on. Jay asked whether the document could include some BFD configuration into the accelerated stress methodology. Scott noted that BFD had come a long way and it would be good to include this. With that, Scott ended his presentation. Al thanked him for the presentation and turned the floor back over to him for a presentation on sub-IP layer protection mechanisms. 4. Sub-IP Protection Mechanisms Scott discussed the scope of the work item, noting that this covered benchmarking at the IP layer of the performance of the sub-IP protection mechanisms. He gave examples of the different protection mechanisms, such as APS and GMPLS-FRR. After this introduction, he went on to talk about the terminology changes from -02 to -03, such as the definitions of "tunnel" and "path"; two new benchmarks on failover and reversion packet loss; providing the sequence of events to describe how the process of failover and restoration would work; and clarification of restoration and reversion. Al suggested replacing the word "act" with "aspect" in the definitions of restoration and reversion. Some equations were also amended in the terminology. Al said that we needed to figure out which ones we really needed and which ones we did not, especially when looking at the equations, which were not really clear. Scott asked for input from the WG. Scott also covered some input received from Al on clarifications in the terminology: the input proposed simplifying some terms, and clarifying the discussion for Failure Detection. There was discussion between Scott and Al on this topic. There was also some input on the terminology from Adrian Farrell of the CCAMP WG, sent in on the reflector yesterday, to align the terms in the document with RFC 4427. and several ITU-T Recommendations. Scott noted that the author team had started work on doing this, but there were some contradictions - for example, the RFC 4427 definition of impairment assumed packet loss, but our benchmark recognized that packet loss could be zero. Al noted that Jerry Perser had presented on this topic at IETF in Vienna and the WG had thus already comprehended this point in our discussions. Al showed the group a quick way to reference the ITU-T documents that Adrian had referred to, using the SANCHO search engine. As an example, Al entered the term "restoration" into the Sancho website maintained by ITU-T, and came up with the definition and also the references to the ITU-T documents. Scott asked Al to send out the link on the mailing list. Matthew Thomas asked if Provider Backbone Transport (PBT) would be included in the protection mechanism documents. Al said that nobody had volunteered to add it, but it would be good to address that kind of stuff. Matthew said that he might be able to find someone who could help out in this regard. Scott encouraged people to come up with such additions at this time. Silvija asked if we should be reviewing the document that would be posted in January or the existing document; Scott responded by saying that people should review the existing document and changes could then be incorporated into the version to be posted in January. Scott thanked various people for their help, including Agilent who had actually taken the methodology document and run it in their lab, which was really good input at this stage. He closed by saying that the revised terminology and methodology would be submitted by the end of January. 5. Milestone Status and New Proposal Summary (Chair) Al thanked Scott for his presentation and proceeded to review the milestones and status. He noted that the IPsec and accelerated test documents were in red status, and asked for volunteers to read them. Scott Poretsky re-volunteered to read the IPsec documents, and noted that the comments he had put in last time hadn't been addressed yet. Al said that IKEv2 was specifically not included in the document. There was considerable discussion between Al, Scott, Aamer, Yaron, and Dan Romascanu on the topic of IKEv1 and IKEv2 relative to this document. Aamer pointed out that IKEv1 had been obsoleted by IETF, but deployments used IKEv1. Yaron said that there was a general need of an update of this document because it had a number of issues. Al asked if Yaron could type up a set of such issues and post it on the mailing list. Al noted that Sue Hares had re-emerged and we might make some progress on BGP convergence. However, Al would have to sort out the network traffic control and router accelerated test methodology documents, which were held up in the review process. Al then covered the work proposal summary. He said that the SIP, multicast VPN, LDP convergence, and MPLS would be covered today. He asked if anyone had read the memos on WLAN switching; Tom Alexander responded that Scott Poretsky, Curtis Villamizar, and Al Morton had read the memos, and Scott Bradner had promised to read them. Al then encouraged people to review and comment on the existing items in the pipeline and get them out of the way so that we could move on to the new proposals. 6. Benchmarking LDP Convergence in the Dataplane Al then invited Jay Karthik to speak on LDP Data Plane Convergence. Jay thanked the group for letting him speak, noting that the last time he wanted to present this he got voted off. Jay then covered the motivation for this work, indicating that LDP was the most widely used MPLS label distribution protocol as per a couple of industry surveys (RFCs 5037 and 5038) and operators wanted tight convergence numbers for LDP with verifiable traffic loss times. Jay noted that the authors had not received many review comments to date and asked for input from the group on the I-Ds. Jay briefly reviewed the work and progress to date on these document, and said that as far as next steps he invited people to review and comment on the work and also would be asking BMWG to take this up as a work item. Silvija commented that she thought that it was important work, and the authors should keep multicast LDP in mind when doing the work. Jay agreed. Scott said that as soon as the IGP convergence is cleared off the agenda, he would fully support this. Al said that we needed the MPLS forwarding draft to be completed (so that throughput and latency measurements could be defined for MPLS) before the LDP drafts could be covered. There was discussion on this topic between Al, Scott, Jay and Silvija. Jay wanted to know whether LDP scalability was of interest to be included in this work item. Al suggested posting this question on the list. Al called for volunteers for reviewing the LDP convergence drafts. Sylvia and Aamer volunteered. Todd volunteered to get someone in Agilent to look at the document as well. 7. Multicast VPN Scalability Benchmarking Al then thanked Jay for his presentation and then turned the floor over to Silvija Dry for the discussion on Multicast VPN scalability benchmarking. She noted that this was a joint proposal from people at Cisco, NTT and AT&T. Sylvia said that most of the presentation would discuss the open issues and also the modified scope proposal. The two open issues from IETF 68 were the scope and the document timing. With regard to timing, Sylvija said that MVPN is the most widely deployed technology today, and scalability was one of the most discussed aspects. There were numerous downside consequences to delaying the development of a common methodology, including delaying the scaling up of existing deployments. Al remarked that there was no slide saying what the *advantages* of delaying were, and asked if there were any such advantages. Silvija said that in her opinion there were no advantages to delaying. Al said that he had attended the L3 VPN WG and had observed that the specification was in a state of flux at the moment, so there may be some advantages to waiting for the specification to settle. Silvija responded that to address this issue the MVPN draft had two pieces: a generic component that was independent of the specific deployed protocol in the service provider core, and a specific component that was dependent on the profile (combination of options) that had been deployed. Jay Karthik said that the delay wasn't necessarily required because the VPN group discussions didn't bear on what was being standardized as metrics. Aamer agreed with this sentiment, and also noted that for a "push button" test methodology to occur the BMWG had to give input to the test equipment vendors. Yuji Kamite commented that they had experience with implementing the work described in the document and it would be of great use to service providers right away. Silvija then briefly described the metrics in the document. Al asked if there were any key metrics that could be identified; Silvija pointed out five parameters that service providers would really like to know about. Silvija then discussed why the work addressed the industry needs and how the document longevity could be ensured. Al asked how many people had read the document (5 people) and how many would support the document as being a BMWG work item (6 people). Instead of counting objections, there was more discussion. Scott asked whether there was a chance that this wouldn't become an RFC based on progress in the multicast VPN WG, expressing the concern that if developments in that WG came out awry then this could wind up not becoming an RFC. Silvija noted that the work in the L3 VPN group was at 95%, and remarked that regardless of activities in the WG, this would be of wide applicability because there were deployments. Scott asked how many deployments; Silvija replied that there were at least 50. Dan asked: "95% of what"? Silvija responded that the multicast VPN WG felt the document was 95% done. A member of the audience (Yigun) said that the authors felt that it was 95% done but the WG chair and AD wanted more review. Dan remarked that the fact that the document had not even entered WGLC and, putting on his AD hat, we should give priority to proposals that referenced documents that are closer to the RFC status. However, this is not to say that the people in the WG cannot review and comment on the document, but he would recommend delaying further consideration of this by a meeting cycle. Rajiv Asati agreed with Dan, but also felt that the document was written in a relatively protocol independent manner, and as the technology was widely deployed there would be considerable interest in seeing the benchmarking work go forward. Al cut the mike line off at that point because we had run out of time. He thanked Silvija for the presentation and also for seriously addressing the comments from the Prague meeting, and then turned the floor over to Aamer for the MPLS benchmarking methodology. 8. MPLS Forwarding Benchmarking Aamer Akhter noted that this was the second time this was being presented, though it was the third and fourth revisions of the drafts. He covered the need for the draft as well as the goals of the drafts. He then went on to discuss the changes from the previous versions, mainly dealing with clarifications, simplifications, and alignments. The document had been condensed from 40 pages to 20 pages. In terms of current state, the document has received considerable support from the list and the document is nearing completion. An -04 version was being worked on. Scott pointed out that with respect to the DoS attack on MPLS forwarding, that was already covered by the accelerated stress documents. Scott also remarked that he would strongly support this as a work item for the BMWG, because it was a natural follow-on from IPv4 and IPv6, the next item would be MPLS. Al asked how many people had read the draft (7 people) and how many people would support this as a work item (7 people). There were no objections to making this a work item. Al then asked the people who hadn't read the draft as to whether it should be a work item; another 5-6 people raised their hands. Al noted that there was a matrix of work going forward on MPLS, even though there wasn't a coherent plan of attack. Jay said that Samir Vapiwala has read the document and supports it as well. Al then thanked Aamer for the presentation, and moved on the SIP performance benchmarking. 9. SIP Performance Benchmarking Scott presented on SIP performance benchmarking, starting out by asking who uses VoIP. Al said that he didn't, whereupon Scott declared that he was an old man. Scott then introduced his coauthors and went on to discuss the motivation for he work. After this, the benchmarks (terminology and methodology) were discussed. After this, Scott went over the revision changes from the previous version, and then talked about the next steps. He said that when it was posted, the SIPPING WG wanted it; in the Montreal meeting, Henning and Scott Bradner attended the meeting, and Scott Bradner was able to convince the SIPPING group to pass this on to BMWG. Scott also remarked that Henning had set up a lab at Columbia to implement these metrics. Saverio Niccolini made a correction to the draft revision number - 05 was not expired. He also wanted to know how the difference between BMWG and SIPPING was to be clarified: Scott said that BMWG had the concept of a DUT and SIPPING was end-to-end. Al also clarified that there would be a consistent terminology between BMWG and SIPPING. Al asked the people who had read it to raise their hands: 3 people did so. Al suggested to Scott that he raise more interest in SIP benchmarking within the WG on the reflector. Todd noted that IP performance metrics were about in-service measurements and BMWG was about lab tests. He wanted to know where he should do an application layer (L4-7) benchmarking submission; Al noted that BMWG was the place, but we do cross-area review with other WGs. Jay said that Aamer and himself would throw a question on the mailing list regarding whether we should wait until BFD becomes an RFC or not, even though BFD was in WGLC. Yaron Sheffer noted that a colleague was looking into extending RFC 3511 with new work with more recent firewall features, and was looking for volunteers to help out. Yaron also noted that his colleague was planning to publish a draft and then look for interest.