Benchmarking Methodology WG (bmwg) Monday, July 28, 2008, 1520-1720 Afternoon Session II Room: Conservatory CHAIR(s): Al Morton <acmorton@att.com> For Slides, see: https://datatracker.ietf.org/meeting/72/materials.html This report was prepared by Al Morton, based on detailed notes provided by Benoit Claise as official note taker. Mike Hamilton monitored the Jabber room, enabling comments to be displayed and read during the meeting. This report is divided in two parts: and executive summary with action items, and detailed minutes of the meeting (from Benoit). Summary of the BMWG Session at IETF-72 -------------------------------------- BMWG met with 22 people present and 10 participating on jabber/remotely. WG Status: The IPv6 memo has been published as RFC 5180. The BMWG successfully rechartered to explicitly begin developing benchmarks for MPLS packet forwarding, and signaling plane scale and service rates for SIP network devices. There is a strong draft ready for the MPLS Forwarding work, and reasonable support was expressed for this work at the meeting. The SIP terms and methodology drafts need to be updated before the working group will consider them as candidate WG drafts. A few minor edits are needed on the new charter/milestones. The next drafts to enter WG Last Call will be the IPsec Benchmarking Terms and Methodology. The meeting supported the mailing list consensus to use the current methodologies and provide incentive to the test equipment industry to augment their capabilities. The drafts were also presented at the new IPsecme WG, with agreement and few comments. The Sub-IP Protection Switching Benchmark work is proceeding, and there was a discussion of the issues with the different Methods of Measurement with good feedback for the authors. There was a new work proposal on IP flow export benchmarking, and good feedback for the author on issues to address. The IGP Dataplane Discusses will be resolved ASAP. There was a proposal investigate a common terminology for all convergence time drafts (not affecting the IGP dataplane memos), and working group will look at this when available. A new proposal on Benchmarking Devices with IP Flow Exporting capabilities was reviewed, and detailed feedback was given to the author. ------------ ACTION ITEMS ------------ IPsec I-D announcements should be posted to the new *IPsecme* mailing list. The "standard" Security paragraphs should be posted on the OPS-AREA page. The IGP Dataplane convergence drafts need to be revised, and will be put forward for WGLC. Accelerated Stress memos need update to address current comments, then WGLC For Protection Switch Benchmarking, the drafts need revision to address comments on the list and at the latest meeting. The 3 different methods to assess Failover time were compared and discussed, and perhaps this aspect of the draft can be simplified. Post the LDP-convergence drafts to the MPLS WG so that we wouldn't repeat Scott's experience with the IGP drafts (authors are asked to do this when they are ready, since this isn't a bmwg work item yet). Also, consider the LDP-IGP sync issue (has not been fully addressed in the protocol dev wgs). The SIP drafts need revision to be brought out of expired status. Detailed Minutes of the Meeting ------------------------------- 1. Working Group Status (Chair) Topics/Drafts not covered by presentations below: RFC 5180, IPv6 Benchmarking Methodology Completed Editing SIP Performance Benchmarking (revised drafts planned) Approved Charter Text Extension: * MPLS Forwarding: Develop specific methods to characterize the latency and forwarding performance of MPLS devices, extending the fundamental recommendations of RFC 1242 and RFC 2544 to this networking technology. * SIP Networking Devices: Develop new terminology and methods to characterize the key performance aspects of network devices using SIP, including the signaling plane scale and service rates while considering load conditions on both the signaling and media planes. This work will be harmonized with related SIP performance metric definitions prepared by the PMOL working group. Activity Status • AD/IESG Review • <draft-ietf-bmwg-igp-dataplane-conv-term-15.txt> Rev I-Ds • <draft-ietf-bmwg-igp-dataplane-conv-meth-15.txt> Needed • <draft-ietf-bmwg-igp-dataplane-conv-app-15.txt> • WG Last Call • draft-ietf-bmwg-ipsec-meth-03.txt Rev I-Ds Needed • draft-ietf-bmwg-ipsec-term-10.txt Then another WGLC • <draft-ietf-bmwg-acc-bench-term-13.txt> Rev I-Ds Needed • <draft-ietf-bmwg-acc-bench-meth-09.txt> Then another WGLC • I-Ds • draft-ietf-bmwg-protection-term-04.txt Discussion Today, then • draft-ietf-bmwg-protection-meth-03.txt Rev I-Ds Needed, WGLC • Current ACTION ITEMS See Slide 5 of the Agenda/Status/Milestone slides for the existing list of action items, one item was revised as below, and all post-meeting action items are listed above. ACTION: Security paragraph posted on OPS area Standard “Paragraph” (intro/security): Al stressed that the charter speaks about laboratory experiment only ACTION: Al to add draft-novak-bmwg-ipflow-meth-00.txt in his list of proposed new WG activity (DONE). For BMWG newcomers, supplementary BMWG Page at http://home.comcast.net/~acmacm/BMWG/ 2. IPsec Benchmarking Drafts Many comments resolved on the list. - WG Last Call SOON URLs: (updates needed before WGLC) http://tools.ietf.org/html/draft-ietf-bmwg-ipsec-term-10 http://tools.ietf.org/html/draft-ietf-bmwg-ipsec-meth-03 Merike Kaeo: Methodology draft needs an update. Will be posted by the end of the week Al: look at the idnits, there are some obsolete references identified. 3. MPLS Forwarding Benchmarking, draft-akhter-bmwg-mpls-meth-04.txt (Rajiv Asati) Discussion GOALS: We would like to lead the discussion on benchmarking the MPLS forwarding methodology. Take the document to the WG LastCall. Abstract The purpose of this draft is to describe a methodology specific to the benchmarking of MPLS forwarding devices. The scope of this benchmarking will be limited to various types of packet-forwarding and delay measurements. It builds upon the tenets set forth in RFC2544 [RFC2544], RFC1242 [RFC1242] and other IETF Benchmarking Methodology Working Group (BMWG) efforts. This document seeks to extend these efforts to the MPLS paradigm. - 4th version of the document • Address an acute and growing need for recommendations on evaluating MPLS forwarding: Focuses on the forwarding plane. Rajiv and Aamer believe this draft is nearly ready for WGLC. Jay question: “Hi Rajiv, I have read the previous version of this draft and I support this work. I had a question - Would "packet replication" be in scope of this work in future ? Example - in the point-to-multipoint scenario, an LSR needs to replicate a packet and it might be highly desirable to benchmark forwarding in the "replication"/"branch node" scenarios.” Rajiv: Will be valid in the future work on multicast, but not this draft Al: TTL or Hop Count (if IPV6). What about the TTL decrement? Benoit: the “reporting format” is not a format, but a list of variables Who read the draft: 5 Who, amongst the readers, supports it: 5 Al: there is support again at this meeting to take-up this draft as a work item to satisfy the new milestone 4. Sub-IP Protection Mechanisms, draft-ietf-bmwg-protection-term-04.txt and draft-ietf-bmwg-protection-meth-03.txt Discussion GOALS: Collect comments on simplifications coming in the next revision. Note: no authors were present Sub-IP Layer Protection Mechanism Performance Benchmarking Abstract This document provides common terminology and metrics for benchmarking the performance of sub-IP layer protection mechanisms. The performance benchmarks are measured at the IP-Layer, so avoid dependence on specific sub-IP protection mechanisms. The benchmarks and terminology can be applied in methodology documents for different sub-IP layer protection mechanisms such as Automatic Protection Switching (APS), Virtual Router Redundancy Protocol (VRRP), Stateful High Availability (HA), and Multi-Protocol Label Switching Fast Reroute (MPLS-FRR). Abstract This draft describes the methodology for benchmarking MPLS Protection mechanisms for link and node protection as defined in [MPLS-FRR-EXT]. The benchmarking and terminology [TERM-ID] are to be used for benchmarking MPLS based protection mechanisms [MPLS-FRR-EXT]. This document provides test methodologies and test-bed setup for measuring failover times while considering all dependencies that might impact faster recovery of real time services riding on MPLS based primary tunnel. The terms used in the procedures included in this document are defined in [TERM-ID]. Current Status: - Under re-review by all the co-authors Incorporating comments from the group Soliciting more comments on the existing documents No new updates for the IETF-72 since none of the co-author could make it Next versions of the Sub-IP Protection Meth and Term to include the following changes: + Formatting + Editorial changes + Splitting the Introduction in the terminology document into two sections: Introduction and Overview + Optimizing the usage and reference to figures/test setups + Recent comments from Bill Cerveny (Thanks to him for his thorough review and comments of the terminology doc) Next Steps: - Incorporate any new comments from meeting and mailing list Version 4 of methodology document will be submitted soon Version 5 of the terminology document will be submitted soon Seek direction from the working group on firming up the progress of terminology document as lot of time and reviews have already been performed Al presented the Sub-IP Protection Methods of Measurement: Considerations Three Methods for Failover & Reversion Time Measurement: • Packet Loss-Based Method (3.6.2) • Time-Based Loss Method (3.6.1) • Timestamp-Based Method (3.6.3) Jeff: would like to have a generic terminology document for convergence time Benchmarks. Rajiv: second that Jay Karthik: Like Al mentioned IGP work is near completion and Sub-IP can certainly reuse or refer to terminologies defined in IGP IDs. As a co-author of LDP IDs wanted to add that, same holds good for LDP work as well. Can we not look at the IGP Terminology as the "generic terminology ID" that we all need ? Al: would probably need some improvements for that Jeff: terminology section of the IGP drafts was not exhaustive enough Nick: (regarding the Timestamp-Based Method, TBM) out of order impairment measurement. Many tools support that, but order can change do to normal parallel processing. AL: So what we need is the out of order WITHIN a flow. If out of order across different flows, then it doesn't affect the customer. So who cares ;-) - need to consider this in the metrics. Jay: We request folks to kindly review the Sub-IP ID excluding the formatting issues that the authors are already aware of and are working on fixing them. 5. Milestone Status, Re-Chartering (Chair) ******* New Work Proposals ********* 6. Proposed Work IP Flow Information Accounting and Export Benchmarking Methodology, draft-novak-bmwg-ipflow-meth-00.txt, Jan Novak Discussion GOALS: introduce the IP flow monitoring performance work, and suggest it as a WG doc. Abstract This document provides methodology and framework for quantifying performance implications of enabling selective monitoring of IP flows on a network device and export of this information to a collector as specified in [RFC5101]. Motivation: Numerous customer requests over a period of last 3-4 years and wide confusion about what to measure and how to measure it. Scope of this work: Performance implications of IP flow monitoring and flow information export on network devices: 1) CPU utilization 2) RFC2544 throughput with IP flow monitoring Does not cover: IP Flow monitoring accuracy Performance of flow export collectors Probes and other non-forwarding monitoring devices Discussed a couple of metrics: Cache States Maintenance, Cache States Update, Flow Expiration Rate, Flow Export Rate, Throughput (based on RFC2544) Dan Romascanu: what is specific to IPFIX, compared to CPU/forwarding device under load/etc…? Jan: doesn't use Ilya: in the CPU section, you mention baselining. Baselining should be done for multiple initial load levels. Otherwise, the results may be not indicative because the same amount of additional traffic may have different impact depending on initial (CPU) load level. Jeff: section where input/output. The picture doesn't match on the text Jeff: multiple modules where CPU should be measured. Not described in the test cases. Al: there's no existing standard definition for Utilization. This document hangs a lot of benchmarks on the Utilization measurement, and today it is supplied by the DUT itself - this is a key issue. Nick: difficulties about the CPU measurement because depends on the OS, and prioritized processes can change the CPU load immediately. Benoit: attention to increasing IP addresses in generated traffic. Maybe a warning Who read the draft: 5 or 6 Al: this memo is tackling an important problem, more people should read it. 7. Proposed Work, but no presentations or updates this time. – LDP Convergence (Asati) – Wireless LAN Switches Alexander) – Multicast VPN Scalability (Dry) MPLS-TE (Vapiwala) Bi-Directional Forwarding Detection (Karthik) 8. New Proposal Summary Matrix (see the Agenda, Status, and Milestones slides) *** EOF ***