Benchmarking Methodology WG (bmwg) - Minutes
 
MONDAY, July 27, 2009
1300-1500  Afternoon Session I
Room 300    
 
CHAIR(s): Al Morton <acmorton@att.com>
 

For Slides, see: https://datatracker.ietf.org/meeting/75/materials.html

These minutes are divided in two sections – Summary/Action Items and Detailed Notes.

The BMWG met with 24 people in attendance, and at least 2 participating remotely via Jabber.

This meeting report was prepared by Al Morton, based on notes provided by Benoit Claise, as official note taker.  Several people monitored the jabber room, so that comments could be read and displayed.

SUMMARY
 

The IGP-Dataplane Convergence Benchmarking Drafts have been substantially revised, and are reaching the end of a very productive WGLC. Another short WGLC is planned to review the final changes before re-requesting publication.

The IPsec drafts needed some final, non-technical editing, and this was accomplished during IETF week so the Publication Request has been made (WGLC in April).

Wide review of the Sub-IP Protection Drafts has resulted in a few comments that can be addressed during the WGLC in progress till July 31 (this is review beyond BMWG, for example the Metro Ethernet Forum has considered our terminology draft in their work on network resilience metrics).

The MPLS-Forwarding (Update of RFC 2544) has completed WGLC and AD-Review, and the remaining point to resolve appears to be the Device Reset Testing. A related proposal to update the RFC 2544 Reset Benchmark was received with interest, as this is still an important device aspect but today’s devices have many options in this area.

The SIP device Benchmarking drafts were not updated, but several revisions are in-progress. The Chair and a member of the draft authors compared the metrics with the SIP metrics draft in PMOL, and found no term confusion or overlap. Several comments were raised at the meeting, and the authors may add some test cases, with November as the new target for AD Review.

There were two very solid work proposal discussed. The Proposal on Flow Monitoring Device Benchmarks was very well received (read by 6 or 7, and 4 or 5 of those agreed to work this going forward).  The Benchmarks are now clearly defined, and a sample set of tests on one device showed their value quite clearly. There was a possible issue identified in that this would be a multi-dimensional benchmark, but this should be sorted-out during development.

There was also good support again for the Content-Aware Device Benchmarking proposal, and some good suggestions for further development. 3 or 4 had read this draft, and there will be further discussion on the list.

It may be beneficial to begin to look at the charter changes that would help the adoption of these two items.

Work proposals that have not seen interest in some time will be deleted after this meeting (LDP is on hold).

BMWG may have an Interim meeting by Electronic Conferencing to advance its work program.
 
 
 
DETAILED NOTES
 
 
AGENDA:  
 
0.  Agenda bashing (if we need to shuffle a few items)
    See https://datatracker.ietf.org/meeting/75/materials.html
    for Agenda updates and Slides
 
    Check the BMWG mail archive for comments:
    http://www.ietf.org/mail-archive/web/bmwg/current/maillist.html
 
New work proposed by Ron Bonica during Bashing: Router time reset benchmarking work
 
1.  Working Group Status (Chair)
WG Drafts not covered by presentations below:
               AD/IESG Review

                            <draft-ietf-bmwg-igp-dataplane-conv-term-18.txt>

    Status: A lot of substantial changes done by new co-author, Kris Michielsen

    In figure 1, a timestamp will be added, for the offered load

    Ilya: draft say -> “it MAY establish an adjacency” in section 3.1

           Why not say: it SHOULD?MUST establish adjacencies

           If MAY, we’re missing the convergence aspect

    Ron: fundamental point -> what this draft doesn’t say is how the DUT relays the information

    Al: all the control plane aspects are in the OSPF control plane benchmarking

    Kris Michielsen: “it propagates the LSP the Next-Best Egress as well”

    Al: not sure how it answers the initial comment. Let’s follow up on the mailing list

    Kris: the reason was not to have an unaffected topology on the ingress interface

                            <draft-ietf-bmwg-igp-dataplane-conv-meth-18.txt>

    Status: A lot of substantial changes done by Kris

    In figure 1: differentiate between “Ingress Interface” and “Preferred Egress Interface”

                            <draft-ietf-bmwg-igp-dataplane-conv-app-17.txt>

                            draft-ietf-bmwg-mpls-forwarding-meth-04.txt

                            Ron:

1. Strong linkage with the IP Forwarding Benchmarking (for example, same sections)

2. Doc. Says: you should be running an IGP. Right, but IGP is not too chatty. But maybe we want static. Won’t push for it

3. IF the router crashes or power link is removed, how long does it need for the router to transmit packets again. Depends on many things: interfaces number, line card number, how long is the config, etc… maybe it’s a work item by itself because many variables not related to MPLS

                            Al: this test is called RESET in section 26.6 from RFC2544. The main question here: Is this a fair test?

                            Ilya: this kind of test requires his own document

                            Randy Bush: interested by the RESET time

                            Al: open call for working on this document, with an update to RFC2544.

                            Al: People interested? Cesar Olvera is interested.

                            Al: follow up on the mailing list, along with the interest level

               Recent WG Last Call

                            draft-ietf-bmwg-protection-term-06.txt 

    Status: In-progress till July 31

    The two protection drafts are now getting a wider review than before

    So new readers are giving feedback

    Al: We looked at section 1.2. We might revise the general model for MPLS Fast Reroute as an example

                            draft-ietf-bmwg-protection-meth-05.txt 

    Status: In-progress till July 31

                            draft-ietf-bmwg-ipsec-meth-04.txt      Rev I-Ds Needed

    Merike: no more technical comments have been made

    Must do a new version to fix editorial comments and nits, like the missing security section

    Regarding IKEv2 (even if products still ship with IKEv1), would like to write a new document but need a co-author!

                            draft-ietf-bmwg-ipsec-term-11.txt      (Doc Shep comments)

               WG Last Call

                            <draft-ietf-bmwg-acc-bench-term-13.txt> Rev I-Ds Needed

                            <draft-ietf-bmwg-acc-bench-meth-09.txt> Then another WGLC

    Status: the two -acc- drafts are kind of dying …

               MORE Chartered I-Ds
                      draft-ietf-bmwg-sip-bench-term-00.txt 
                            Status: no yet WG last-call, hoping for revision          
                      draft-ietf-bmwg-sip-bench-meth-00.txt 
                            Status: no yet WG last-call, hoping for revision          
               Work Proposals.
                      draft-karthik-bmwg-ldp-convergence-meth-02.txt 
                            Status: Need IGP prog.
                      draft-eriksson-ldp-convergence-05.txt    
                            Status: Need IGP prog.
                      draft-novak-bmwg-ipflow-meth-03.txt           
                            Status: Revised
                      draft-hamilton-bmwg-ca-bench-meth-01.txt   
                            Status: Revised
               Expected I-Ds
                      <draft-ietf-bmwg-bgpcmeth-00.txt> 
               Expired BMWG I-Ds
    <draft-ietf-bmwg-acc-bench-meth-ebgp-00.txt>, 
    <draft-ietf-bmwg-acc-bench-meth-opsec-00.txt>,
    <draft-ietf-bmwg-benchres-method-00.txt> Pending term 
    <draft-ietf-bmwg-dsmmeth-02.txt>
               RFC Editor Queue
                       
               New RFC:
                       
               Charter Update
                       none
               Supplementary BMWG Page
                      See   http://home.comcast.net/~acmacm/BMWG/
 
 
2.  SIP Performance Benchmarking, Vijay Gurbani 
Discussion GOALS: 
Introduce new WG item at -00.  Request input/comments.
 
Related Drafts:
http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-term-00
 
Abstract
 
   This document provides a terminology for benchmarking SIP performance
   in networking devices.  Terms are included for test components, test
   setup parameters, and performance benchmark metrics for black-box
   benchmarking of SIP networking devices.  The performance benchmark
   metrics are obtained for the SIP control plane and media plane.  The
   terms are intended for use in a companion methodology document for
   complete performance characterization of a device in a variety of
   conditions making it possible to compare performance of different
   devices.  It is critical to provide test setup parameters and a
   methodology document for SIP performance benchmarking because SIP
   allows a wide range of configuration and operational conditions that
   can influence performance benchmark measurements.  It is necessary to
   have terminology and methodology standards to ensure that reported
   benchmarks have consistent definition and were obtained following the
   same procedures.  Benchmarks can be applied to compare performance of
   a variety of SIP networking devices.
 
http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-meth-00
Abstract
 
   This document describes the methodology for benchmarking Session
   Initiation Protocol (SIP) performance as described in SIP
   benchmarking terminology document.  The methodology and terminology
   are to be used for benchmarking signaling plane performance with
   varying signaling and media load.  Both scale and establishment rate
   are measured by signaling plane performance.  The SIP Devices to be
   benchmarked may be a single device under test (DUT) or a system under
   test (SUT).  Benchmarks can be obtained and compared for different
   types of devices such as SIP Proxy Server, SBC, P-CSCF, and Server
   paired with a Firewall/NAT device.
 

Was supposed to go the WG last June this June, but some difficulties

See the slides, “required document updates”

Assess the alignment with PMOL end-to-end SIP benchmarking work

               We set out to determine if the measurements in the BMWG metrics draft can be used to predict measurements made using the metrics defined in the PMOL end-to-end performance draft.

               We conclude that the answer is “No.” 



This was also the chairman's conclusion, see: 

http://home.comcast.net/~acmacm/BMWG/SIP%20Metrics%20Comparison.html
Read the slides
Hadriel: question about the metrics. Is Re-registeration is considered?
Vijay: we could consider adding registration refresher -> what is the impact? 
Hadriel: Tests with media are important, and media as recommended to send …
            Would like to have a second test for that …
Vijay: not too late to change. we should consider a separate metric for media test
 
3.   Milestone Status and New Proposal Summary (Chair)
Current Milestones
               Sep 2008          IPsec Device Benchmarking Terminology to IESG Review
               Sep 2008          IPsec Device Benchmarking Methodology to IESG Review
 
               Dec 2008          Net Traffic Control Benchmarking Methodology to AD Review.
               Dec 2008          Router Accelerated Test Terminology to IESG Review
               Dec 2008          Router Accelerated Test Methodology to IESG Review
 
               Feb 2009          Terminology For Protection Benchmarking to AD Review
               Feb 2009          Methodology For Protection Benchmarking to AD Review
 
               Apr 2009          Methodology for MPLS Forwarding to AD Review
               Jun 2009           Terminology for SIP Device Benchmarking to IESG Review
               Jun 2009           Methodology for SIP Device Benchmarking to IESG Review
 
               Dec 2009          Router Accelerated Test Method for EBGP to IESG Review
               Dec 2009          Router Accelerated Test Method for Op Sec to IESG Review
               Jul 2010            Basic BGP Convergence Benchmarking Meth. to AD Review.
 
Work Proposal Summary Matrix
Work Area >
Criteria \/
IPFIX
Content-Aware 
(was Firewall 
extension)
LDP Converg
Mcast VPN 
Scalability
WLAN 
switch
Proposal
Y
Y
Y
Y
Y
In Scope of 
Charter? (acm)
Y
Y
Y
Y
No Overlap 
w/802.11T
Draft(s)
1+
1
1+1
1+
1+1
Sig. Support 
at meetings
Significant 
questions
Some 
comments
 
Yes, but also 
some objections
 
Sig. Support 
on List
Some 
comments
Questions to 
clarify
Many 2/2008 
comments
Many 12/2007 
comments
 
Dependencies
 
 
IGP prog
L3vpn spec
 
 
Al: if nobody picks up the grey part -> they will disappear next time
*******   New Work Proposals  *********
 
 
4.  IP Flow Monitoring Benchmarking Methodology
Discussion GOALS: Clarify questions from last meeting - What is the
benchmark?
Draft file name, or preferably the complete URL:
http://tools.ietf.org/id/draft-novak-bmwg-ipflow-meth-03.txt
Presenter: Jan Novak
Abstract
   This document provides methodology and framework for quantifying 
   performance of selective monitoring of IP flows on a network device 
   and export of this information to a collector. It is based on the
   Architecture for IP Flow Information Export [RFC5470].
 
Document Goal

               Defines metrics to characterise Flow Monitoring performance based on:

   - Architecture Model for IP Flow Information Export: RFC5470

 

               Defines conditions to perform Network Devices benchmarking in the presence of Flow Monitoring

                    IPv4 Benchmarking Methodology: RFC2544

                    IPv6 Benchmarking Methodology: RFC5180

Two different building blocks:

            Forwarding Plane -> RFC2544 Throughput

            Flow Monitoring Plane -> Flow Monitoring Throughput =

How many Flow Records the DUT can handle per second without dropping any Flow information

Jan recognizes the name could be confusing

So we need two metrics, as proven in the next table

Packets                Flow Expiration        RFC2544
per flow                        Rate                Throughput
    1                        65 000 FR/s     65 000 pps
    2                        43 500 FR/s     87 000 pps
    3                        33 600 FR/s   101 000 pps
 
Two extremes:
Each packet creates a flow record -> maximum impact of the Flow Monitoring
No Flow monitoring -> forward traffic at 10 times the rate faster
 
Al: This was a software based platform. If tested with hardware, completely different
Jan: Also tested on hardware platform -> no much impact on the forwarding plane performance
Ilya: slide 5, collector will be inband most of the time.
      So the export will impact the forwarding plane ever more
      Valuable to have such a test 
      Impact: different scheduling on the packet

Jan: from the performance point of view, it won’t change anything if you have enough export bandwidth

Ilya: can we just say: available bandwidth = full output bandwidth – the export bandwidth ??? there may be cases where available bandwidth is NOT= full output bandwidth - export bandwidth (due to impact on packet scheduling).

Brian Tramell: Collection link: UDP versus TCP versus SCTP comparison would be interesting.

Brian: same set of tasks for dedicated probes. For example, optical splitter to a probe. The same methodologies can be applied to stand-alone devices with small modifications.

Jan: test methodology makes the assumption that RFC5470 is used.

Brian: forwarding plane is not part of RFC5470

Benoit: This is a multidimensional Benchmark (throughput, Flow Export Throughput)
        other dimensions
è                     sampling (1/100, 1/500)
è                     flow keys
è                     cache size, expiration condition, timeout
è                     traffic generator
o packet size
o number of packet per flow 

Difficult to compare benchmark report: understand the interest from the audience. Will it be usefull? (this is a point for further discussion)

Ilya: might want to have realistic traffic. So different profiles could help

Joel Jaeggli: Repeatability is key

Al: it’s difficult to conclude something out of the results with "real" traffic, except for this specific profile. I’m cautiously optimistic about this work, as it helps to answer the perpetual question: what is the impact if I turn on NetFlow (and features similar to NetFlow).

Al: current limitation -> it’s limited to router only.

Al: 6 or 7 have read the draft
Al: from the ones who read it, would would support it
            Elisa Boschi, Brian Trammel, Ilya, Sarah Banks (Cisco) 
 
5.  Content-Aware Device testing methodology
Discussion GOALS: To discuss the updated draft and discuss future of draft
Draft file name, or preferably the complete URL: 
<draft-hamilton-bmwg-ca-bench-meth-01>
Presenter: Mike Hamilton
Post Mortem on Last meeting:
l            Biggest Thing:  Did not separate out the terminology draft.  Time constraints.
l            Traffic content for D/DoS
-                    Could have included the latest generation of D/DoS attacks
l            Need better clarification on the reporting/recording procedure and requirements

Ilya: apart from DOS attack, a category of device have a kind of bypass. When the device comes up, there is some time needed (for example, a established connection). Might be worth looking at this kind of device.

Al: how many read it -> 3 persons. Sarah Banks will read it tonight

Al: if accepted, from the ones who read it.
    Sarah Banks, yes if it’s not in the context of the firewall.
Al: let’s see the interest from the mailing list…
 
 
Miscellaneous

Potentially no face-to-face meeting in Japan at the next IETF, due to scheduling conflicts for the Chair and AD.

An alternative, an electronic meeting in the mean time.