Benchmarking Methodology WG (bmwg) Wednesday, July 25, 2007 Afternoon Session I 1300-1500 Room Name: Salon 2 CHAIR(s): Al Morton <acmorton@att.com> This report is divided in two parts: and executive summary with action items, and detailed minutes of the meeting. Summary BMWG had 38 people joining the session locally and 2 participating remotely. We gave resolution of IESG DISCUSSes the highest priority on the agenda. AD Dave Ward, and ISIS & OSPF co-chairs joined the meeting for this discussion, and convinced the group to expand the scope of the IGP-dataplane convergence drafts to cover RIP, "link-up/Good News" and other areas mentioned in the DISCUSSes. A small group of authors, a RIP expert to be named later, Chris Hoppes, Acee Lindem and Dave will work this in the next few weeks. The WG is as much as 6 months behind schedule on many milestones, but the teams are making progress. The Accelerated Stress Benchmarking Drafts require another WGLC, and the IPsec drafts will be reviewed by about 5 volunteer WG members. There was good discussion of the remaining issues on the IPv6 draft and the WGLC will close soon. There were comments on the Sub-IP Protection Benchmarking drafts, and revisions will be needed. Work Proposals on WLAN switch and SIP device benchmarking were also presented. A strong case has been made that the WLAN work is within BMWG's scope, and the WG seeks any alternate opinions. Action Items Jay Karthik has volunteered to restart the BGP convergence methodology, and Rajiv Papneja also volunteered to help out. David Ward will identify a RIP expert to consult on adding RIP to the IGP Dataplane drafts, and a small group of volunteers to help expand the scope and resolve the DISCUSS comments. The IPv6 Author team will look into whether a reference to the Hash and Stuffing RFC 4814 is appropriate in their memo. They will also pursue the frame size issue on the list. WGLC ends on August 10. All authors to check that their drafts print-out properly, with no line wraps. Margins to body text should be 3 spaces, no more. Regarding the IPsec drafts, folks should read and comment on the need for an IKE flood scenario and whether the back-to-back test should be included. Scott Poretsky, Rajiv Papneja, Muninder Sambi, Sanjay Hooda, and Benno Overinder all volunteered to review the IPsec drafts. Jerry Perser also volunteered to help Merike finish the documents. There's lots of work being proposed, so the WG needs to read these drafts too and comment on what is most relevant to do next. Detailed Minutes Al Morton opened the session at 1:05 PM CDT in the Salon 2 room. Tom Alexander was official note taker. Vijay Gurbani agreed to watch the Jabber session. About 40 people attended, including two remote participants. Al requested people to sign up to the reflector and also to fill out the Blue sign-up sheet. He reviewed the agenda and asked if there was any bashing needed. He noted that we would look at our milestones and see how "red" they were. As the agenda was fairly full, there might have to be some triage on the new work presentations. No bashing was required so Al continued with the agenda as shown. Al also mentioned that there was a supplementary BMWG page that could be consulted for new attendees. 1. Working Group Status (Chair) Al reviewed the IESG review and last call status. Gunter said that some key issues with the IPv6 draft would be closed today, along with some work on the accelerated stress benchmarking. Scott noted that Jay Karthik is willing to take on the BGP convergence. Rajiv Papneja also volunteered to help out. Al pointed out that the work proposals outweighed the current work in progress - need more balance here. Two WG drafts have moved through the RFC Editor's queue. New RFCS - RFC 4814 and RFC 4883 have made it through (RFC 4883 just today). (Applause.) Al read aloud the standard paragraph on introduction to BMWG activities plus security. He would like to see the standard paragraph in all the drafts. Scott wanted to know if the paragraphs had been posted to the mailing list. Al replied: lots of times. However, this was the updated version. With that, Al asked Scott Poretsky to discuss the IGP dataplane convergence presentation. 2. IGP Data plane convergence benchmark I-Ds - Publication Requested - IESG Review Scott Poretsky presented on IGP dataplane convergence. There were three drafts - they are now on revision 13 which incorporated comments from second AD review; this revision is currently in IESG review. One Area Director (David Ward) had come back with DISCUSS comments on all three drafts. There were some comments that discussed SCOPE issues, and these would be discussed in the group. There was a significant and active discussion between several members. The first SCOPE comment was about maintaining a narrow focus on OSPF/ISIS versus expanding to RIP/BGP/etc. The IESG wanted to know if the scope could be expanded. David Ward (the originator of the comment) said that BGP wasn't expected to be included, but RIP was close to the scope of the work and could be included. Rajiv Papneja said that we could also look at the applicability of RIP relative to OSPF/ISIS, and he wasn't sure how it could be handled in the same methodology. David thought that the exact same setup could be used, and you could be forced to do a massive cut-and-paste job when expanding to cover this RIP. Al said that the group had decided issue on a need basis - RIP had never come up. There was a Jabber comment, to the effect that RIP is different from ISIS and OSPF as it has TE extensions. David Ward said that TE extensions have not been discussed in the drafts and so it doesn't apply. Ron Bonica said that most of the experiments could be identical for RIP, except for the configuration parameters. One issue is that we don't have a lot of expertise in RIP and if someone from the routing area could look at the documents and see what the delta is, then we could see what could be done. David Ward took on the action item for sending someone from the RIP area to provide a consult on including RIP. (Action Item Note.) The next comment was on dealing with graceful restart (plus non-stop forwarding). David Ward said that the industry had great concern about the amount of time that was lost during graceful restart, and suggested that we could have methods for measuring this in the draft. Scott Poretsky's belief was that he generally agreed that people would like to have those benchmarks, but they should be done in a separate work item (e.g., MPLS fast reroute). David said that this is part of the normal protocol functionality and should be part of these drafts. Rajiv Papneja said that under the sub-IP benchmarking, we already cover similar issues, but we could definitely consider this as a trigger event if that helped. The high availability techniques are an extension to the base draft material. David disagreed. Chris Hoppes said that he had an issue with no note being sent to the ISIS WG at all regarding this draft (though he allowed that it might have gone out on the OSPF list). Al said that it had gone out to all the chairs and that the group had done the necessary information dissemination. Chris said that there might be some additional comments that would come if this were examined by the ISIS group. He continued to say that it is not unexpected to have multiple failures occurring in the network at the same time; if you have a router restarting, for example, what is the impact on convergence during that time? Even if you were to push it into another document, that document would have cross-reference back up to this one. Scott Poretsky asked, in response, if it would be better to have someone in the routing area to see what changes would be needed to incorporate these. David said that he would like really like an opportunity for the ISIS and OSPF community to review these changes and provide input. Scott said that these had been reviewed with Sue Hares et. al., but David said that this didn’t count because Sue was the chair of the BGP group and not ISIS or OSPF. Chris also pointed out a nit with a 100 msec timer that would be caught by the ISIS group - a 201 msec timer would be reported as a 300 msec time which would make a big difference. Al Morton reiterated that a lot of people had looked at the draft. David repeated in response that he would like the ISIS group to be given a chance to take a look as well. Al Morton asked for a show of hands in favor of holding up the work so that a wider review could be had. Six people were in favor and none were opposed. Jay suggested that we could split the draft into two pieces, but this was generally dismissed. Al decided that there was a consensus on holding the draft to get a better one after incorporating some of these comments and getting more review. He then asked Scott to press on with the presentation. Ron Bonica remarked that there was a difference between any old route reconverging vs. the important routes reconverging. It may be impossible to measure that by simply looking at the forwarding plane. David Ward agreed; he commented that it may be important to understand the difference between the first FIB entry converging, vs. the last FIB entry converging, vs. the median or mean convergence. He said that both protocols had policy knobs to allow different FIB entries to be handled differently. Ron asked how one would know which prefixes had converged first. Chris responded saying that operators prioritized the BGP next hop routes to be downloaded to the FIB first, and thus while it may be difficult to measure, they want to know how fast their BGP next hops would be converged relative to all other routes. David gave an example to illustrate, saying that it worked even better with a large number of prefixes vs. a smaller number prefixes. Scott said that now that it had been explained, he could see how to change the document quite easily to accommodate this. Merike asked if it could be done in a couple of weeks so that we could all move forward? She did not want to see it taking a whole year. David said that he would round up a couple of volunteers to take care of these. Al asked David for the names, and David said that he would get them to him. (Action Item.) David said that another thing to note is the differentiation between link-down and link-up testing, which wasn't present in the draft. Scott disagreed with him, and said that there is a term and it is the last step in every methodology. David said that he recollected that the link-up portion was ascribed to the restoration time; Scott clarified that restoration included "putting back what you did", and that would cover the link up. David felt that this was incorrect methodology. Chris echoed this concern - it was not restoration because the 'before' and 'after' views of the network weren’t the same. The issue was what happened when you added a new link. Al said that this is all about terminology; there was discussion on clarifying this term. What is needed here is a test to characterize a new link-up, a.k.a. good news. David Ward asked if, with LDP, BFD and other protocols coming in, could we do this process with the ISIS and OSPF groups as with the IGP convergence? Al Morton said that we have had wide review, and we will try to include these groups even more actively in future. It was agreed by the ISIS/OSPF representatives that the IGP dataplane convergence methodology was much better than previous RFCs on measuring convergence. There was some discussion about the trigger event for the BFD convergence - this is the notification from the forwarding plane to the FIB. Scott clarified that it wouldn't be a benchmark for BFD, but BFD would be a trigger event. A comment from Jabber agreed that test tool vendors do offer tests today based on data plane convergence. Scott Poretsky reviewed some of the terminology revision changes, notably the definition of packet loss, which is modified from frame loss. Also, a sentence has been added to the document relating to the value of "5" for forwarding convergence time, which Scott said came from RFC 2544. Al Morton and Jerry Perser disagreed, saying that there was general guidance; Jerry said that he'd been using 2 seconds to wait for residual frames. David Ward said that 11 seconds was what was used by the ISIS people. (Scott eventually produced his reference, it is the waiting time between tests in RFC 2544.) With that, the presentation ended. Al Morton thanked the ISIS/OSPF people for their participation. He then turned to the accelerated stress work, remarking wryly that the WG was already under accelerated stress. (Laughter.) Scott Poretsky volunteered to speak on this presentation. 3. Techniques for Benchmarking Router Accelerated Stress Testing. Scott Poretsky said that extensive reviews and comments had been received and several revisions had been made (term-12, meth-08). There was a lot of discussion on the mailing list about BFD and what to do about it. Scott noted that the scope of the work item was to define a terminology for accelerated stress, as well as a set of methodology guidelines for accelerated stress benchmarking. Thus BFD could be addressed just as with EBGP instabilities or operational stress, as a separate methodology document. There was no dissent from the group. Jay Karthik said that he agreed but this did not prevent the WG from having a parallel BFD work item. Al Morton said that the work proposal would be treated separately. Scott Poretsky noted that he had received some comments from WGLC, and they had been addressed, so was the document ready to move to IESG? Al Morton polled the commenters. At the end of the poll, Al said that we would have one more WGLC and then we can be done with it. With that the presentation ended. 4. IPv6 Benchmarking Methodology Gunter van de Velde presented on IPv6 benchmarking, and said that the presentation was completely prepared by Chip Popoviciu who couldn't be here today. He gave an overview of the document, saying that it complements RFC 2544. There was a dearth of IPv6 methodology out there which prompted this work. There was a WGLC with reviews from three "official" reviewers (Scott Bradner, Bill Cerveny, Rajiv Asati) and the comments were favorable. Question from Jabber: IPv6 has a lot of different addressing mechanisms, how is this covered? Al Morton felt that this might be addressed by the hash and stuffing RFC, and suggested that a reference be made to this. Gunter said that he would look into whether the Hash and Stuffing RFC had something relevant to reference. (Action Item.) Additional comments and recommendations were also received. These were on throughput (and definition thereof), jumbo frames, back-to-back tests. Gunter van de Velde said that the back-to-back test was not very predictable and not widely used as a result, so it was removed. It was also clarified that the throughput measurement was done with zero-drop rates. All of these were addressed as summarized in the posted PDF document. There was one open item, on SONET minimum frame size and signature fields. The signature was about 20 bytes, but with a minimum frame size the signature cannot be 20 bytes any more. The question was how to select a minimum frame size to include the signature field. For example in SONET the min frame size was 47 bytes and this would not be large enough to include a signature field. Scott Poretsky remarked on the absence of odd-numbered frame sizes in the list, which would produce different performance; and also the incorporation of frame sizes related to page sizes - i.e., if you go one page up or down then you see different performance. Gunter van de Velde said that he saw what he was referring to, but he wasn't sure whether it was practical. Scott clarified that a difference in performance was actually measured with IPv4, so he recommended that we don't repeat the same mistake. Bill Cerveny suggested mentioning the issue, and specifying just one frame size that was out of the ordinary. Gunter van de Velde asked what was the extraordinary value? Scott wanted to know what was the logic behind 1280 bytes. Jerry Perser pointed out that there was no logic. Al Morton said that there should be a standardized list, which is necessary for repeatability in benchmarking, but there could then be other frame sizes that you can also publish. He asked the group what was wrong with the list - are there not enough odd values? One proposal was 73 bytes. Scott Poretsky said that 71 was interesting because that was the size of a VoIP packet. Al Morton said that this current set of mostly even sizes was worrisome, and we should do further investigation and publish a final set of packet sizes. Merike Kaeo said that we should have a separate document that published frame sizes, because she had the same issue in her document. Al brought up the infamous IMIX controversy, and said that there was apparent need for such a document in the future, as a plug-in to other bmwg RFCs. Gunter van de Velde noted that there was already a vendor supporting this work and it already works. He also showed the signature field used by the vendor. he concluded that the next steps were to get all the feedback we could before the WGLC cutoff of 10 August, and to reach agreement on the minimum frame size issue. Al Morton encouraged everyone to post their comments on frame size to the reflector, and also noted that this was a good draft and it should be done soon. Al said that all the comments against the draft had been captured in a PDF file which was a thing of beauty, and if all author teams did this it would be great. Scott Poretsky wanted to know whether the whole draft would go to the IPv6 group for them to comment on. Gunter van de Velde clarified that they had gone around to all these people and they were aware that they were doing this work. With that, the presentation ended. 5. Sub-IP Protection Mechanisms Scott Poretsky presented on sub-IP layer protection benchmarking. The terminology and methodology are at rev 2. The plan is for one terminology document and several methodology documents. A number of changes were made from rev 01 to 02, to accommodate the many comments that had come in. A timestamp based method (TBM) was added in addition to the time-based loss method and packet-based loss method (TBLM and PBLM). More terms were added as well. The methodology was updated to include TBM. Scott Poretsky thanked the reviewers and also noted that Agilent had taken the document and used it in their lab, with excellent results. He asked if there were any more comments and is it ready for WG LC. Al Morton said that he had a ton of comments, including: - the page length is highly variable and the formatting needs to be fixed - The path definition must include links (it currently only includes nodes) - other major comments, most of which will be handled outside the meeting Al Morton said that there was still some more work to do on the terminology, and we all had to learn the science of definition writing. For example the most commonly used term in the draft is "failover" whereas the whole draft was about "protection". Scott Poretsky clarified that there was much discussion in the WG and "failover" was the best term to use to define failover. Al noted that he didn't recall the "much discussion" part but that the term protection goes all the way back to the pre-Bellheads. Scott asked if he was happy with the definition of failover, but didn't like the word? Al said that this was possible - he was always unhappy with the word "failover". Al Morton asked who was the lead author on the methodology draft. Rajiv Papneja confessed to being the culprit. Al then noted that the margins were all screwed up and the formatting really had to be fixed. Al then said that the XML ones worked out fine but the other ones were looking like crap, so we had things to fix. Rajiv protested that the nit checker didn't complain. Al concluded by saying that this document was not ready for WGLC, and recommended that the authors fix it. 6. Terms and Methods for Benchmarking IPsec Devices Meriko Kaeo presented on IPsec benchmarking. She said that some coauthors had dropped out and there was a bit of a hiatus. Most of the comments had been incorporated, and there were only a couple of things left to do, so it would not take much work to get them completed. The terms are at 09 and meth at 02 currently; the terminology doc has been updated with latest references and limited DoS testing was added. Frame sizes were still being cleaned up, and the question of back-to-back frame testing was still present. Scott Poretsky said that we would need to add one more DoS test to the methodology - IKE flood, which was happening in the real world. Scott gave an example from a carrier, which was getting flooded with IKE packets at 100 or 200 packets/second. Jerry Perser said that at the last two companies he had worked for had implemented a back-to-back test and no customer had asked for it. Meriko Kaeo said that she would ask the reflector if anyone was interested, if not she would take it out. There were some methodology changes as well - references, security context, back-to-back frames, DoS resiliency, etc. Meriko Kaeo appealed to the group at large for help on completing the drafts, and said that people were wondering why the drafts were still not done. Al Morton said that comments could be had from the security directors. Meriko suggested that after the next set of comments were incorporated we move to WGLC. Scott Poretsky asked if it was better to expand the scope to include IKEv2. Meriko said that this brings up the companion doc on IKEv2, which had the significantly different terminology associated with IKEv2. Scott pointed out that we should avoid the experience on graceful restart on IGP convergence. Meriko said that she would prefer to have these documents out there, and also that the IKEv2 stuff was quite different from IKEv1. Meriko asked if there was anyone in the room who felt that IKEv2 should be added; one person raised their hand. Nobody was eager to see IKEv2 stuff incorporated into the doc. Scott Poretsky, Rajiv Papneja, Muninder Sambi, Sanjay Hooda, and Benno Overinder all volunteered to review the IPsec draft and comment on it. Jerry Perser also volunteered to help Merike finish the document. (Action Items.) With that, the presentation ended. 7. Milestone Status and New Proposal Summary (Chair) Al Morton covered the milestone status, and said that everything that we covered today was in the red, so nothing new would be added to the charter until this work was completed. He also covered the work proposal summary slide. He then did some triage on the new work proposals and asked which proposals the group would like to hear. MPLS was requested. Al also said that there was interest in the reflector on the WLAN work so he wanted to do them, and also the SIP benchmarking. Ultimately a straw poll was taken of the group's interest in hearing each proposal in turn: SIP - 7 people wanted to hear it WLAN - 8 people wanted to hear it RSVP-TE - 6 people wanted to hear it Al Morton indicated that we would hear the WLAN presentation and also the SIP presentation, and asked if there were objections. The room was silent. 8. WLAN Switch Benchmarking Jerry Perser presented on the WLAN terminology and methodology proposals. He made a plea for comments on the posted documents. He noted that there was some reflector comment, but not enough, and encouraged people to read the drafts. Jerry Perser said that WLANs were slightly different from other technologies because they have more things to solve, and so additional methodologies had to be introduced. Jerry went over the methodology and said that he highly discouraged open-air testing. He also noted that there was a big push in the enterprise to go wireless due to reduced costs and desire for mobility. In response to a question from the group, he stated that there was no significant RF involved in the testing. Jerry said that he was using these documents in public tests and would like to see some review and comments, especially on what metrics we were missing that would better represent real world issues. Jay Karthik asked how we ensured that we did not get interference from running networks. Jerry Perser noted that the document would ensure that there was no interference by using chambers. There was some discussion on whether we would specify properties of the chambers. Muninder noted that the document explained how to avoid interference, and also said that this document was requested by the equipment vendors and enterprise customers. Tom Alexander further pointed out that the document specified two setups, one using the APs with RF cabling, and the other dispensing with APs altogether and connecting directly to the Ethernet ports on the WLAN controller. It was not necessary to assume that RF would be involved or to deal with RF issues if this wasn't wanted. Al Morton then thanked Jerry Perser for the presentation, and brought on the SIP performance proposals. 9. SIP Performance Benchmarking Carol Davids presented on SIP performance benchmarking. A rev 03 terminology and rev 01 methodology had been posted. She said that SIP allows for a wide range of configuration and operational conditions and the documents tried to capture that. The goals of the document were also stated; for example a lot of people were coming up with SIP boxes but there was no way to compare their performance. A list of benchmarks and terminology was presented, plus the key revisions since the last presentation (made at IETF-67). A significant revision to the documents was that, because SIP boxes perform as multimedia systems, a three-dimensional vector had to be introduced to describe a SIP session (i.e., signaling, media and media control). The methodology then incorporated the terminology. The complementary nature of the drafts to the proposed work in SIPPING (or OPS-APM) was discussed. The APM work would be discussed in a BoF this afternoon, and related to end-to-end performance metrics. The BMWG metrics would be used to assess budgets for individual components or SUTs that then could be applied against the end-to-end SIPPING metrics. Carol Davids asked if it could be considered for a BMWG work item. Al noted that the draft had improved substantially since the early days, and felt that the authors had really pounded away and made it a solid work item. However, we had to finish up the existing charter items first before taking on any new work items. Al Morton then closed the presentations, as there was no more time. He apologized to the MPLS and LDP teams for not including their presentations, and hoped that a future session would allow time for them to present. Jerry Perser made the observation (relative to the SIP proposal) that he had never considered a server to be an internetworking device, and asked if this was within the BMWG charter. Al Morton said that it was lab testing and so he felt it would fit. Dan Romascanu said that the Internet is built not only in internetworking devices but also hosts and servers; with his AD hat on, this question should not be asked now (i.e., the decision to adopt was not to being discussed). Carol Davids also said that the Internet is being used as the basis for all kinds of applications, and we can make the argument that it's here, and it belongs (at least in terms of lab testing). With that, Al Morton closed the meeting at 3.00 PM CDT. He thanked all the participants and invited everyone to join the APM BOF session.