APM BOF Agenda Application Performance Metrics BOF Wednesday, June 25 Afternoon Session II BOF Chairs: Alan Clark Al Morton Agenda: 1. Introduction: Problem Statement and Goals 2. Examination of a perceived gap in IETF coverage: Existing Drafts: (< 2 minute, 1 slide overview of each) - draft-malas-performance-metrics-07.txt - draft-venna-ippm-app-loss-metrics-00.txt - draft-ietf-avt-rtcpxr-video-01.txt - draft-ietf-avt-rtcpxr-audio-00.txt - draft-ietf-avt-rtcphr-01.txt - draft-xie-ccamp-lsp-dppm-01.txt - draft-kikuchi-passive-measure-00.txt IDR Stability Performance (draft?) 3. Comparison of Possible Solutions: + APM Directorate (with working process) + APM WG (with charter and working process) 4. Discussion of Pros and Cons 5. Assess consensus on the following questions: 1. Should IETF commit its resources to the formal development of application performance metrics in one or more key areas? 2. If yes, is there a preferred approach to manage metric development: + Protocol Development WGs with advice from an APM Directorate + APM WG with participation by experts from the relevant Protocol WG. + Both an APM Directorate and an APM WG are needed + Neither an APM Directorate or an APM WG are needed 3. If the APM Directorate approach is desirable, is the proposed method of working acceptable, or are modifications needed? 4. If the APM WG approach is desirable, is the proposed charter acceptable, or are modifications needed? ========================================================================== Approved BOF Proposal Follows: APM BOF Application Performance Metrics BOF Chairs: Alan Clark [alan@telchemy.com] Al Morton [acmorton@att.com] Mailing Lists: BOF discussion on Area Directorate: Operations and Management Area Ultimate Area Directorate: TBD BOF Problem Statement: There have always been some uncertainties about the performance and suitability of new technologies and applications for their intended audience, and the Internet is no exception. Most uncertainties are effectively addressed through quantified assessment of key performance indicators. Standardized performance metrics add the desirable features of consistent implementation, interpretation, and comparison. Although the IETF has two Working Groups dedicated to the development of performance metrics, they each have strict limitations in their charters: - The Benchmarking Methodology WG has addressed a range of networking technologies and protocols in their long history (such as IEEE 802.3, ATM, Frame Relay, and Routing Protocols), but the charter strictly limits their performance characterizations to the laboratory environment. - The IP Performance Metrics WG has the mandate to develop metrics applicable to live IP networks, but it is specifically prohibited from developing metrics that characterize traffic (such as a VoIP stream). The IETF does have current and completed activities related to the reporting of application performance metrics (e.g. RAQMON) and is also actively involved in the development of reliable transport protocols which would affect the relationship between IP performance and application performance. APM proposals that are beyond the scope of bmwg and ippm have begun to appear. Some recent drafts include: http://www.ietf.org/internet-drafts/draft-malas-performance-metrics-06.txt http://tools.ietf.org/html/draft-venna-ippm-app-loss-metrics-00 Thus there is a gap in the currently chartered coverage of IETF WGs: development of performance metrics for application-layer protocols that can be used to characterize traffic on live networks. There is a need for an application performance framework and application performance metrics that enable IP based applications to be characterized (and hence managed) when operating over UDP, TCP, FEC and the other robust transport protocols under development within IETF. However, the work undertaken must be strictly evaluated on an item by item basis, to avoid overlap with standards organizations already developing application performance metrics in this general area, ITU-T SG 12 for example. This is the problem to be solved. Potential Solutions: Two possible approaches to solving this problem have been suggested. + APM Directorate This Directorate would advise working groups initiating APM work (e.g. AVT for Voice and IP video quality metrics, sipping for SIP performance metrics, idr for BGP metrics) and provide expert reviews. Members of the directorate could review protocols for manageability from a performance perspective. The Directorate members would be drawn from experts currently involved in ippm, bmwg, ipfix, and other working groups. The APM directorate would be available to consult in all IETF areas, like other directorates. The members of the Directorate would create a BCP document that would provide guidance to the reviewers and reference for editors of IETF APM metrics documents. + APM WG This WG would be responsible for the development of RFCs that characterize the performance of applications, especially those utilizing IETF protocols. The WG would be responsible for advancing the RFCs along the standards track, according to the process defined in draft-bradner-metricstest-01.txt . The WG would partner with a specific protocol development WG whenever possible, to take advantage of willing experts and ensure that the metrics developed are both appropriate and relevant. Last, the WG would create a BCP document that would provide guidance for development of IETF APM metrics documents, including the requirement to include relevant protocol experts in the development. The BCP may also define the process to adopt new work in the APM WG, where a unique feature could be that proposals would be vetted in the relevant protocol WG before reaching the APM WG. The APM WG would avoid overlap with existing SDOs, and seek IESG approval for each new Metric RFC before the WG spends any resources on it. (more is needed for a complete charter)