Benchmarking Methodology WG (bmwg) Monday, March 10, 2008 13:00 - 15:00 Afternoon Session I Salon J CHAIR(s): Al Morton <acmorton@att.com> For Slides, see: https://datatracker.ietf.org/meeting/71/materials.html This report was prepared by Al Morton, based on notes provided by Tom Alexander as official note taker. This report is divided in two parts: and executive summary with action items, and detailed minutes of the meeting. Summary of the BMWG Session at IETF-71 -------------------------------------- BMWG met with 28 people present and 8 participating remotely. WG Status: the IPv6 Draft has been approved, and the Approval announcement will be sent shortly. The IGP Dataplane convergence drafts have been revised, and after a minor revision suggested at the meeting to elevate the status of Route-Specific Convergence time, will be put forward for WGLC. There has been excellent commentary on all working group items recently, and significant progress will be evident in their next revisions. We need to revise the milestones for the Accelerated Stress, IPsec and Protection Switch Benchmarking work items. The BMWG is working on re-chartering to add new work on MPLS Forwarding and SIP device Benchmarking. There was good feedback on the SIP work at this meeting - the Benchmark names need to explicitly indicate that these are SIP sessions, and not media sessions. There was also excellent feedback from the SIPPING WG at their session the same evening, and there must be more opportunities for the SIPPING Community to provide feedback before WGLC. Also, the work proposals on LDP Convergence time and WLAN Switch Benchmarking were discussed briefly, with an appeal for more readership and comments. Feedback on LDP indicates that they should take-up scalability and convergence time, as these are related topics. ------------ ACTION ITEMS ------------ IPsec I-D announcements should be posted to the IPsec mailing list. The Standard Security Paragraphs to be sent formally to the Security Directorate for review. The remaining IGP-Dataplane Convergence issue is the route withdrawal test case, it was proposed to be removed by someone in the design team because it was more "white box" and implementation specific. Accelerated Stress memos need update to address current comments, then WGLC For Protection Switch Benchmarking, the term "unimpaired packet" was identified as needing a definition in the terminology. Also, need to reconcile the definitions of "PLR" and Head-end. Post the LDP-convergence drafts to the MPLS WG so that we wouldn't repeat Scott's experience with the IGP drafts (authors are asked to do this when they are ready, since this isn't a bmwg work item yet). Also, consider the LDP-IGP sync issue (has not been fully addressed in the protocol dev wgs). Regarding the non-SIP specific metrics in the SIP draft, a possible resolution may be to be to revise the names of the benchmarks to make them SIP-specific (especially if they are anchored on SIP protocol events). Detailed Minutes of the Meeting ------------------------------- Al Morton opened the session at 1.03 PM Eastern time. He asked everyone to sign up on the blue sheets, and went through some other introductory stuff. Tom Alexander was note taker. The significance of the yellow sheet (IPR rules) was discussed, as BMWG was meeting early in the week. He introduced Ron Bonica and Dan Romascanu, the Operations and Management ADs (Ron is AD advisor for bmwg). There were approximately 25 people attending the session at that point. 0. Agenda Bashing (Chair) Al then discussed the agenda. He noted that one of the proposed work items on MPLS performance benchmarking would not be presented today, but the rest would be presented. He then asked for any agenda bashing; there was none. Al noted that he would probably cover the LDP proposal when Rajiv Asati was available to join the meeting (he was needed in two other sessions, simultaneously). 1. Working Group Status (Chair) The WG status was covered next. Al mentioned that there was some discussion about the IPv6 draft, regarding a comment that Tom A. had raised. Tom said that as per the reflector discussion all of his comments had been satisfactorily addressed and he did not see any need for changes to the draft. Ron noted that the draft was being sent to the RFC editor. Al reviewed the status of the IGP dataplane convergence drafts. Scott Poretsky noted that an excellent review of the work on Accelerated Stress had been provided on the reflector this morning and the comments would be incorporated before going to IESG review. Al then briefly reviewed the work proposals, of which there were a fair number. He also talked about the expired I-Ds. Scott noted that he had had three different coauthors on the dsmmeth draft, and that it seemed like the I-D was like a Bermuda Triangle for coauthors, and he was basically waiting for a coauthor before proceeding with the draft. There was also some discussion about RFC 2889 Appendix errata; Al said that there was some discussion but that petered out, and he invited people to continue and complete this work. Al reviewed the updates to the IPsec drafts. Yaron Sheffer noted that he had made his comments on the reflector, and suggested that the I-D announcement should be posted to the IPsec mailing list as well. Al noted this as an action item (AI). A standard paragraph for introduction and security (to be incorporated into all BMWG documents) was discussed next. Al said that these standard paragraphs have helped us defuse concerns about security issues. Dan R. wanted to know if the text was something that had been worked out with the security Area. Al responded that we had received a lot of comments from people unfamiliar with the scope of BMWG, and we had started putting these paragraphs into the drafts going to IESG review, and they seemed to answer a lot of questions. He said that the Security Directorate has not reviewed the paragraphs as Director, but had seen it piece by piece. Dan suggested that these paragraphs be sent formally to the Security Directorate for review. Al agreed and noted this as an action item (AI). There was a technical hitch with Jabber at this point, and Al took a short break to recover it. Topics/Drafts not covered by presentations below: IPv6 Benchmarking Methodology Approved http://www.ietf.org/internet-drafts/draft-ietf-bmwg-ipv6-meth-05.txt IPsec Benchmarking Drafts http://tools.ietf.org/html/draft-ietf-bmwg-ipsec-term-10 http://tools.ietf.org/html/draft-ietf-bmwg-ipsec-meth-03 2. IGP Data plane convergence benchmark I-Ds - Publication Requested - IESG Review Revised I-Ds: http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-term-15.txt http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-meth-15.txt http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-app-15.txt Presenter: Scott Poretsky Al then turned the floor over to Scott Poretsky for the IGP Dataplane Convergence presentation. Scott noted that we were at -15 on the revision level, having covered cross-area reviews, two WGLCs, completed AD Reviews, and closed out two sets of DISCUSS items from IESG (IETFs 69 and 70). Scott introduced the terminology changes in the latest update. Al commented (as participant) that what is practical now - in terms of test equipment - has gone a long way from where it was when the work was started; the equipment can measure thousands and thousands of flows, and thus it is possible to measure route-specific convergence time all along the recovery transition. This was what was underlying Dave's comments: he wanted the first converging route, the last converging route, and the median in between. There was extended discussion between Al and Scott on this topic. After this, Scott introduced the methodology changes, particularly in the reporting format. Al noted that the methodology now measures all the necessary information on Route-Specific Convergence Time to make it a benchmark, and there was sufficient rationale to elevate this method equal to the Rate-Derived Convergence Time in the terminology draft. There was a jabber comment from Ilya Varlashkin regarding iSPF and SPF: he proposed leaving SPF, and having no iSPF. (Ilya later posted this comment to the list) Finally, Scott covered the outstanding issues, of which there were two. He said that based on the discussion at this meeting, the route-specific convergence time would be covered with a sentence or two being added. The other issue is the route withdrawal test case, which was proposed to be removed by someone in the design team because it was more "white box" and implementation specific. Ilya noted that there will probably be different times of intra and inter-area routes which would make things more "white box". (Ilya later posted this comment to the list) Al said that we would have a very short last call, probably after IETF 71 - we could finish this before March is over. Scott covered the next steps for the document. Al noted that we now have much better documents as a result of the reviews and updates. The WG also applauded Scott for his hard work on these drafts. There was no further discussion on the IGP dataplane documents, so Al moved on. 3. Benchmarking LDP Convergence in the Dataplane Related Drafts: https://datatracker.ietf.org/drafts/draft-eriksson-ldp-convergence-term/ https://datatracker.ietf.org/drafts/draft-karthik-bmwg-ldp-convergence-meth/ Presenter: Rajiv Asati Al invited Rajiv Asati to present on LDP Data Plane Convergence benchmarking. There was a short break while Al repaired another technical hitch with Jabber. Kannan Varadhan then volunteered to watch the Jabber. Rajiv noted that his name was not on the draft, but he had agreed to present it so that people could have a messenger to shoot at. He discussed the motivation of the draft, basically to build upon the terminology and methodology of the IGP convergence documents. He then covered the current status of the drafts, and noted that the LGP protocol authors have reviewed them as well. He asked how many people had read the drafts; Scott raised his hand, at which point Rajiv said that 100 people had read the draft, because Scott counted for a hundred. He then talked about the next steps, and asked people to read and review the draft. Al asked for a show of hands on who were attending BMWG for the first time. About eleven people raised their hands, probably more than half the room. Al then pointed out that the best way to join the BMWG was to read the drafts, and encouraged them to do so. Ron Bonica then recommended that Al poll the room for LDP-savvy people, and also to post the draft to the MPLS WG so that we wouldn't repeat Scott's experience with the IGP drafts. Al then asked the room for LDP-savvy people; nobody raised their hands, but Al said that he saw someone hiding and would approach him later for a read. (Yakov Stein was hiding, but he later read the drafts anyway and said they were reasonable.) Rajiv noted that the synchronization problem related to LDP and IGP hasn't been addressed within the MPLS workgroup, and ducked a suggestion that he could volunteer to address this issue. 4. Techniques for Benchmarking Router Accelerated Stress Testing. Discussion GOALS: Get Last Call comments resolved, another WGLC ... Draft URLs: http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-term-13.txt http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-meth-09.txt Presenter: Scott Poretsky Al thanked Rajiv for his presentation and then invited Scott to present on accelerated stress testing. Scott began by noting that this was an extremely difficult work item, and he would appreciate assistance in terms of coauthors as well as reviewers. He then discussed the scope of the work item, noting that the terminology and the current methodology guidelines document were the top-level documents for a large number of technology-specific documents. WGLC had been completed, and the -13/-09 version incorporated the comments. Scott covered some of the key comments. There were some suggestions to incorporate BFD and SIP, but further discussion was needed on these. Al asked if we had SIP stuff was new; Scott noted that it was a recent addition, and explained that there was a definition of the setup for the different phases which now includes SIP. Dan Romascanu wondered whether the inclusion of SIP was based on some statistics, or whether it was because people were interested in it. Scott responded that SIP was incorporated because the WG sent in comments indicating that SIP should not have been omitted. Scott also said that there were a number of planes (e.g., network management plane, security plane, etc.) that included setups for different protocols (e.g., network management protocols). Al noted that some of us were surprised by the inclusion of SIP, and some review was warranted. There was considerable discussion, including Ron (who encouraged Scott to defer SIP, because the SIP sessions would terminate on different kinds of boxes than routers). Ultimately, Scott agreed that BFD would stay, as it was a router protocol, and SIP would come out, as it was not. Scott then covered next steps. Al noted that we should have another WGLC, and also it was time to re-consult other WGs on this topic. With that, Al moved on to sub-IP protection benchmarking, and asked Scott to present. 5. Sub-IP Protection Mechanisms Discussion GOALS: Reach consensus on the various definitions of actions during Protection Switching. Drafts: http://www.ietf.org/internet-drafts/draft-ietf-bmwg-protection-term-04.txt http://www.ietf.org/internet-drafts/draft-ietf-bmwg-protection-meth-03.txt Presenter: Scott Poretsky Scott discussed the protection benchmarking terminology and methodology. He covered the scope of the work item, and remarked that having a common terminology and metrics for the IP layer of the system allowed operators to compare protection mechanisms derived from different physical layer technologies. Ron Bonica commented that at one time the IETF had a sub-IP area, which included MPLS but none of the other items on the list (e.g., APS, VRRP, HA, etc.). Ron also noted that all the items in the list except for HA provided a way to protect a path or a circuit, but HA seemed like a different animal; did it belong in the bucket? There was discussion between Scott, Ron and Al on this. Kannan Varadhan remarked that something different in this is APS, which is a completely L2 function. Scott covered the updates to the document, starting with the terminology. Al (as a participant) said that there really needs to be a section in the document on scope, as well as a section on the general model. Rajiv Asati noted that a definition which was missing in the draft is the PLR; we're making an assumption that the headend is capable of making the switchover, but this is not true in all cases. Scott noted that the PLR is in fact defined in the terminology doc, but it had to be reconciled with the standard use of the term. The updates to the methodology were discussed next. The term "unimpaired packet" was talked about; an action item that resulted was to add a definition of unimpaired packet to the terminology (AI). Al noted that this category of packets could be taken to mean the absence of other specific impairments (which would be listed and referenced). There will be a revision of the methodology document based on the comments received. Rajiv Asati noted that in the figures the headend was the PLR, but this is not the general case. Scott responded that the terminology would address this. Scott then discussed the next steps. Al mentioned that this was a WG draft, and we would have a last call on this soon; he encouraged people to read the draft and get their comments in soon. 6. Milestone Status, Re-Chartering, and New Proposal Summary (Chair) Al thanked Scott for his presentation, then went to the milestone status. He noted that there was a bunch of stuff that didn't get done from Dec 2007. The work proposal summary was then presented. Al noted that in the last few months two of the items on the work proposal had reached a high state of maturity. The SIP draft had garnered much attention, but there was a possibility that the SIP item might be taken up by SIPPING instead of BMWG. There was also lots of interest in the MPLS work, specifically the extension of RFC 2544 metrics to MPLS, and people were doing this ad-hoc now. Al also discussed the multicast VPN and LDP convergence work. Scott noted that the LDP convergence was interesting, but it had two gating items: the IGP convergence and the MPLS forwarding, and we wanted to get them out of the way first. Al responded that the dependency on MPLS depended on the approach taken in the LDP draft. Ron Bonica noted that MPLS forwarding was the low-hanging fruit on the tree, and this should be the first work item. Al then presented a charter text extension, to cover MPLS forwarding and SIP networking. Rajiv Papneja asked (via Jabber) if we should poll the WG for including scaling as part of LDP convergence - we could update the draft accordingly. Rajiv Asati said that the scaling aspect goes hand-in-hand with the convergence aspect and thus he agreed that this should be included. Al said that they have some proposal text prepared and he would like to see that text modified based on the group comments today. Rajiv Papneja agreed to update it. ******* New Work Proposals ********* 7. MPLS Forwarding Benchmarking Related Draft: http://www.ietf.org/internet-drafts/draft-akhter-bmwg-mpls-meth-03.txt Presenter: Rajiv Asati Rajiv then gave a brief verbal review on the MPLS forwarding performance benchmarking. He said that the drafts had been posted, and asked people to review the outstanding drafts and provide feedback. 8. SIP Performance Benchmarking Related Drafts: Draft URLs: http://www.ietf.org/internet-drafts/draft-poretsky-sip-bench-term-04.txt http://www.ietf.org/internet-drafts/draft-poretsky-bmwg-sip-bench-meth-02.txt Presenter: Carol Davids Carol Davids presented on the SIP performance benchmarking drafts. She reviewed the motivation for the drafts, summarizing the problem statement, the goals and the scope. She also discussed the industry collaboration as well as the cross-functional collaboration that was going on in this area, particularly between SIPPING and PMOL WGs, and the SPEC consortium developing test code for SIP benchmarking. She noted that SPEC has expressed interest in this effort as well. Carol put up e-mails that indicated the interest from the SIP community and WG leadership. She presented the benchmarks that were proposed. Dan Romascanu asked how many of these metrics were SIP-specific, and how many were more general session establishment metrics? Carol responded that these were SIP-specific and not general session related. Dan remarked that he had serious reservations about the document, because it seemed to him that 80% of the metrics were not SIP specific and were generally session-specific. Carol said that this was a very well taken point. A possible resolution may be to be to revise the names of the benchmark terms to make them specific to SIP (especially if they are anchored on SIP protocol events). Carol also covered input from BMWG via the e-mail list. The next steps were to incorporate comments into the next revision, and asked Al what happens next. Al said that they were going to work the "where to do the work issue" this week, and would have an extension to the charter to review by the WG ASAP. Scott asked what the issue was: is the issue the fact that SIPPING and BMWG were both addressing this? Al responded that that was one of the issues. Scott asked if they could handle it jointly; Al said that this was unlikely, because there needed to be one owner with this work on the charter, but there can be joint WG last calls, cross-review, etc. 9. Benchmarking Wireless LAN Switches Discussion GOALS: Briefly describe updates and encourage readership/comments Related Drafts: http://tools.ietf.org/wg/bmwg/draft-alexander-bmwg-wlan-switch-term-01.txt http://tools.ietf.org/wg/bmwg/draft-alexander-bmwg-wlan-switch-meth-01.txt Presenter: Tom Alexander Tom gave a brief update on the WLAN drafts that had been posted, noting the changes in the new revision that had been posted and asking people to read them and send in comments. He noted that hospitals were now using WLANs to transport patient monitor information over IP networks, and said that performance measurement that provide incentive to improve the industry could literally save your life in this scenario. With that, Al thanked the audience for their participation, particularly the new members, and then closed the session at 3.02 PM Eastern time. *** EOF ***