Benchmarking Methodology Working S. Poretsky Group Allot Communications Internet-Draft V. Gurbani Expires: January 13, 2011 Bell Laboratories, Alcatel-Lucent C. Davids Illinois Institute of Technology July 12, 2010 Methodology for Benchmarking SIP Networking Devices draft-ietf-bmwg-sip-bench-meth-02 Abstract This document describes the methodology for benchmarking Session Initiation Protocol (SIP) performance as described in SIP benchmarking terminology document. The methodology and terminology are to be used for benchmarking signaling plane performance with varying signaling and media load. Both scale and establishment rate are measured by signaling plane performance. The SIP Devices to be benchmarked may be a single device under test (DUT) or a system under test (SUT). Benchmarks can be obtained and compared for different types of devices such as SIP Proxy Server, SBC, and server paired with a media relay or Firewall/NAT device. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on January 13, 2011. Poretsky, et al. Expires January 13, 2011 [Page 1] Internet-Draft SIP Benchmarking Methodology July 2010 Copyright Notice Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the BSD License. Poretsky, et al. Expires January 13, 2011 [Page 2] Internet-Draft SIP Benchmarking Methodology July 2010 Table of Contents 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. Test Topologies . . . . . . . . . . . . . . . . . . . . . . . 5 4. Test Considerations . . . . . . . . . . . . . . . . . . . . . 6 4.1. Selection of SIP Transport Protocol . . . . . . . . . . . 6 4.2. Signaling Server . . . . . . . . . . . . . . . . . . . . . 6 4.3. Associated Media . . . . . . . . . . . . . . . . . . . . . 7 4.4. Selection of Associated Media Protocol . . . . . . . . . . 7 4.5. Number of Associated Media Streams per SIP Session . . . . 7 4.6. Session Duration . . . . . . . . . . . . . . . . . . . . . 7 4.7. Attempted Sessions per Second . . . . . . . . . . . . . . 7 4.8. Stress Testing . . . . . . . . . . . . . . . . . . . . . . 8 5. Reporting Format . . . . . . . . . . . . . . . . . . . . . . . 8 5.1. Test Setup Report . . . . . . . . . . . . . . . . . . . . 8 5.2. Device Benchmarks for IS . . . . . . . . . . . . . . . . . 9 5.3. Device Benchmarks for NS . . . . . . . . . . . . . . . . . 9 6. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 9 6.1. Baseline Session Establishment Rate . . . . . . . . . . . 9 6.2. Session Establishment Rate . . . . . . . . . . . . . . . . 10 6.3. Session Establishment Rate with Media . . . . . . . . . . 10 6.4. Session Establishment Rate with Loop Detection Enabled . . 11 6.5. Session Establishment Rate with Forking . . . . . . . . . 12 6.6. Session Establishment Rate with Forking and Loop Detection . . . . . . . . . . . . . . . . . . . . . . . . 12 6.7. Session Establishment Rate with TLS Encrypted SIP . . . . 13 6.8. Session Establishment Rate with IPsec Encrypted SIP . . . 13 6.9. Session Establishment Rate with SIP Flooding . . . . . . . 14 6.10. Maximum Registration Rate . . . . . . . . . . . . . . . . 14 6.11. Maximum Re-Registration Rate . . . . . . . . . . . . . . . 15 6.12. Maximum IM Rate . . . . . . . . . . . . . . . . . . . . . 16 6.13. Session Capacity without Media . . . . . . . . . . . . . . 16 6.14. Session Capacity with Media . . . . . . . . . . . . . . . 17 6.15. Session Capacity with Media and a Media Relay/NAT and/or Firewall . . . . . . . . . . . . . . . . . . . . . 17 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 8. Security Considerations . . . . . . . . . . . . . . . . . . . 18 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 18 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 18 10.1. Normative References . . . . . . . . . . . . . . . . . . . 18 10.2. Informative References . . . . . . . . . . . . . . . . . . 19 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 19 Poretsky, et al. Expires January 13, 2011 [Page 3] Internet-Draft SIP Benchmarking Methodology July 2010 1. Terminology In this document, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as described in BCP 14, conforming to [RFC2119] and indicate requirement levels for compliant implementations. Terms specific to SIP [RFC3261] performance benchmarking are defined in [I-D.sip-bench-term]. RFC 2119 defines the use of these key words to help make the intent of standards track documents as clear as possible. While this document uses these keywords, this document is not a standards track document. The term Throughput is defined in [RFC2544]. 2. Introduction This document describes the methodology for benchmarking Session Initiation Protocol (SIP) performance as described in Terminology document [I-D.sip-bench-term]. The methodology and terminology are to be used for benchmarking signaling plane performance with varying signaling and media load. Both scale and establishment rate are measured by signaling plane performance. The SIP Devices to be benchmarked may be a single device under test (DUT) or a system under test (SUT). The DUT is a SIP Server, which may be any [RFC3261] conforming device. The SUT can be any device or group of devices containing RFC 3261 conforming functionality along with Firewall and/or NAT functionality. This enables benchmarks to be obtained and compared for different types of devices such as SIP Proxy Server, SBC, SIP proxy server paired with a media relay or Firewall/NAT device. SIP Associated Media benchmarks can also be made when testing SUTs. The test cases covered in this methodology document provide benchmarks metrics of Registration Rate, SIP Session Establishment Rate, Session Capacity, IM Rate, and Presence Rate. These can be benchmarked with or without associated Media. Some cases are also included to cover Forking, Loop detecion, Encrypted SIP, and SIP Flooding. The test topologies that can be used are described in the Test Setup section. Topologies are provided for benchmarking of a DUT or SUT. Benchmarking with Associated Media can be performed when using a SUT. SIP permits a wide range of configuration options that are also explained in the Test Setup section. Benchmark metrics could Poretsky, et al. Expires January 13, 2011 [Page 4] Internet-Draft SIP Benchmarking Methodology July 2010 possibly be impacted by Associated Media. The selected values for Session Duration and Media Streams Per Session enable benchmark metrics to be benchmarked without Associated Media. Session Setup Rate could possibly be impacted by the selected value for Maximum Sessions Attempted. The benchmark for Session Establishment Rate is measured with a fixed value for maximum Session Attempts. 3. Test Topologies Figures 1 through 3 below provide various topologies to perform the SIP Performance Benchmarking. These figures show the Device Under Test (DUT) to be a single server or a System Under Test (SUT). Test Topology options to include benchmarking with Associated Media require use of a SUT and are shown in Figures 3. DUT --------- --------- | | | | |signal-| |ing | | | |server | SIP | | | |<------------->| EA | | | | | | | | | | | | | --------- --------- Figure 1: Basic SIP Test Topology SUT ------------------------ --------- --------- --------- | | |media- | | | |signal-| | relay | | | |ing | SIP | OR | SIP | | |server |<---------------------->| EA | | | |fire | | | | | |wall OR| | | | | | NAT | | | --------- --------- --------- Figure 2: SIP Test Topology with Firewall Poretsky, et al. Expires January 13, 2011 [Page 5] Internet-Draft SIP Benchmarking Methodology July 2010 SUT ------------------------ --------- --------- --------- | | | media-| | | |signal-| | relay| | | |ing | SIP | OR | SIP | | |server |<---------------------->| EA | | | |fire | | | | | |wall OR| Media | | | | ---| NAT |---------| | --------- | --------- --------- | Media ^ -------------------------| Figure 3: SIP Test Topology with Media through Firewall --------- | | |------------->| | ^ | | | SIP | | |<-------------| EA | | | | | | | --------- Figure 4: Baseline Test Topology 4. Test Considerations 4.1. Selection of SIP Transport Protocol Test cases may be performed with any transport protocol supported by SIP. This includes, but is not limited to, SIP TCP, SIP UDP, and TLS. The protocol used for the SIP transport protocol must be reported with benchmarking results. 4.2. Signaling Server Poretsky, et al. Expires January 13, 2011 [Page 6] Internet-Draft SIP Benchmarking Methodology July 2010 The Server is a SIP-speaking device that complies with RFC 3261. The purpose of this document is to benchmark SIP performance, not conformance. Conformance to [RFC3261] is assumed for all tests. The Server may be the DUT or a component of a SUT that includes Firewall and/or NAT functionality. The components of the SUT may be a single physical device or separate devices. 4.3. Associated Media Some tests may require associated media to be present for each SIP session. The Server is not involved in the forwarding of media. Associated Media can be benchmarked only with a SUT in which the media traverses a Media Relay, Firewall, NAT, or Firewall/NAT device. The test topologies to be used when benchmarking SUT performance for Associated Media are shown in Figures 4 and 5, in which the SIP signaling is bidirectional and the Associated Media is unidirectional. 4.4. Selection of Associated Media Protocol The test cases specified in this document provide SIP performance independent of the protocol used for the media stream. Any media protocol supported by SIP may be used. This includes, but is not limited to, RTP, RTSP, and SRTP. The protocol used for Associated Media MUST be reported with benchmarking results. 4.5. Number of Associated Media Streams per SIP Session Benchmarking results may vary with the number of media streams per SIP session. When benchmarking a SUT for voice, a single media stream is used. When benchmarking a SUT for voice and video, two media streams are used. The number of Associated Media Streams MUST be reported with benchmarking results. 4.6. Session Duration SUT performance benchmarks may vary with the duration of SIP sessions. Session Duration MUST be reported with benchmarking results. A Session Duration of zero seconds indicates transmission of a BYE immediately following successful SIP establishment indicate by receipt of a 200 OK. An infinite Session Duration indicates that a BYE is never transmitted. 4.7. Attempted Sessions per Second Poretsky, et al. Expires January 13, 2011 [Page 7] Internet-Draft SIP Benchmarking Methodology July 2010 DUT and SUT performance benchmarks may vary with the the rate of attempted sessions offered by the Tester. Attempted Sessions per Second MUST be reported with benchmarking results. 4.8. Stress Testing Discussion: The purpose of this document is to benchmark SIP performance, not system stability under stressful conditions such as a high rate of Attempted Sessions per Second. 5. Reporting Format 5.1. Test Setup Report SIP Transport Protocol = ___________________________ (valid values: TCP|UDP|TLS|SCTP|specify-other) Session Attempt Rate = _____________________________ (session attempts/sec) IS Media Attempt Rate = ____________________________ (IS media attempts/sec) Total Sessions Attempted = _________________________ (total sessions to be created over duration of test) Media Streams Per Session = _______________________ (number of streams per session) Associated Media Protocol = _______________________ (RTP|RTSP|specify-other) Media Packet Size = _______________________________ (bytes) Media Offered Load = ______________________________ (packets per second) Media Session Hold Time = _________________________ (seconds) Establishment Threshold time = ____________________ (seconds) Loop Detecting Option = ___________________________ (on|off) Forking Option Number of endpoints request sent to = ___________ (1, means forking is not enabled) Type of forking = _______________________________ (serial|parallel) Note: Total Sessions Attempted is used in the calculation of the Session Establishment Performance ([I-D.sip-bench-term], Section 3.4.5). It is the number of session attempts ([I-D.sip-bench-term], Poretsky, et al. Expires January 13, 2011 [Page 8] Internet-Draft SIP Benchmarking Methodology July 2010 Section 3.1.6) that will be made over the duration of the test. 5.2. Device Benchmarks for IS Registration Rate = _______________________________ (registrations per second) Re-registration Rate = ____________________________ (registrations per second) Session Capacity = _________________________________ (sessions) Session Overload Capacity = ________________________ (sessions) Session Establishment Rate = ______________________ (sessions per second) Session Establishment Performance = _______________ (total established sessions/total sessions attempted)(no units) Session Attempt Delay = ___________________________ (seconds) 5.3. Device Benchmarks for NS IM Rate = _______________________________ (IM messages per second) 6. Test Cases 6.1. Baseline Session Establishment Rate Objective: To benchmark the Session Establishment Rate of the DUT/SUT with zero failures. Procedure: 1. Configure the test bed in the test topology shown in Figure 4. 2. Configure Tester with a Session Attempt Rate = 100 SPS, maximum Session Attempts = 100,000 and Media Streams Per Session=0. 3. Start Tester to initiate SIP Session establishment with the DUT. 4. Measure Session Attempt Failures and total Established Sessions at the Tester. 5. If a Session Attempt Failure is recorded then reduce the Session Attempt Rate configured on the Tester by 50%. 6. If no Session Attempt Failure is recorded then increase the Session Attempt Rate configured on the Tester by 50%. Poretsky, et al. Expires January 13, 2011 [Page 9] Internet-Draft SIP Benchmarking Methodology July 2010 7. Repeat steps 3 through 6 until the Session Establishment Rate is obtained and recorded. Expected Results: This is the scenario to obtain the maximum Session Establishment Rate of the test bed when no DUT?SUT is present. The results of this test might be used to normalize test results performed on different test beds or simply to better understand the impact of the DUT/SUT on the test bed in question. 6.2. Session Establishment Rate Objective: To benchmark the Session Establishment Rate of the DUT/SUT with zero failures. Procedure: 1. Configure the DUT in the test topology shown in Figure 1 or SUT as shown in Figures 2 or 3. 2. Configure Tester with a Session Attempt Rate = 100 SPS, maximum Session Attempts = 100,000 and Media Streams Per Session=0. 3. Start Tester to initiate SIP Session establishment with the DUT. 4. Measure Session Attempt Failures and total Established Sessions at the Tester. 5. If a Session Attempt Failure is recorded then reduce the Session Attempt Rate configured on the Tester by 50%. 6. If no Session Attempt Failure is recorded then increase the Session Attempt Rate configured on the Tester by 50%. 7. Repeat steps 3 through 6 until the Session Establishment Rate is obtained and recorded. Expected Results: This is the scenario to obtain the maximum Session Establishment Rate of the DUT/SUT. 6.3. Session Establishment Rate with Media Objective: To benchmark the Session Establishment Rate of the SUT with zero failures when Associated Media is included in the benchmark test. Procedure: 1. Configure the SUT in the test topology shown in Figure 2 or 3. 2. Configure Tester for a Session Attempt Rate = 100 SPS, maximum Session Attempts = 100,000 and Media Streams Per Session = 1. The rate of offered load for each media stream SHOULD be (eq 1) Offered Load per Media Stream = Throughput / maximum sessions attempted, where Throughput is defined in [RFC2544]. 3. Start Tester to initiate SIP Session establishment with the SUT and transmit media through the SUT to a destination other than the server. Poretsky, et al. Expires January 13, 2011 [Page 10] Internet-Draft SIP Benchmarking Methodology July 2010 4. At the Tester measure Session Attempt Failures, total Established Sessions, and Packet Loss [RFC2544] of the media. 5. If a Session Attempt Failure or Packet Loss is recorded then reduce the Session Attempt Rate configured on the Tester by 50%. 6. If no Session Attempt Failure or Packet Loss is recorded then increase the Session Attempt Rate configured on the Tester by 50%. 7. Repeat steps 3 through 6 until the Session Establishment Rate is obtained and recorded. 8. Repeat steps 1 through 7 for multimedia in which Media Streams Per Session = 2. Expected Results: Session Establishment Rate results obtained with Associated Media with any number of media streams per SIP session are expected to be identical to the Session Establishment Rate results obtained without media in the case where the server is running on a platform separate from the platform on which the Media Relay, NAT or Firewall is running. Session Establishment Rate results obtained with Associated Media may be lower than those obtained without media in the case where the server and the NAT, Firewall or Media Relay are running on the same platform. 6.4. Session Establishment Rate with Loop Detection Enabled Objective: To benchmark the Session Establishment Rate of the DUT/SUT with zero failures when the Loop Detection option is enabled. Procedure: 1. Configure the DUT in the test topology shown in Figure 1 or SUT as shown in Figures 2 or 3. 2. Configure Tester for a Session Attempt Rate = 100 SPS, maximum Session Attempts = 100,000 and Media Streams Per Session=0. 3. Turn on the Loop Detection option in the DUT or SUT. 4. Start Tester to initiate SIP Session establishment with the DUT. 5. Measure Session Attempt Failures and total Established Sessions at the Tester. 6. If a Session Attempt Failure is recorded then reduce the Session Attempt Rate configured on the Tester by 50%. 7. If no Session Attempt Failure is recorded then increase the Session Attempt Rate configured on the Tester by 50%. 8. Repeat steps 4 through 7 until the Session Establishment Rate is obtained and recorded. Expected Results: Session Establishment Rate results obtained with Loop Detection may be lower than those obtained without Loop Detection enabled. Poretsky, et al. Expires January 13, 2011 [Page 11] Internet-Draft SIP Benchmarking Methodology July 2010 6.5. Session Establishment Rate with Forking Objective: To benchmark the Session Establishment Rate of the DUT/SUT with zero failures when the Forking Option is enabled. Procedure: 1. Configure the DUT in the test topology shown in Figure 1 or SUT as shown in Figures 2 or 3. 2. Configure Tester for a Session Attempt Rate = 100 SPS, maximum Session Attempts = 100,000 and Media Streams Per Session=0. 3. Set the number of endpoints that will receive the forked invitation to a value of 2 or more (subsequent tests may increase this value at the discretion of the tester.) 4. Start Tester to initiate SIP Session establishment with the DUT. 5. Measure Session Attempt Failures and total Established Sessions at the Tester. 6. If a Session Attempt Failure is recorded then reduce the Session Attempt Rate configured on the Tester by 50%. 7. If no Session Attempt Failure is recorded then increase the Session Attempt Rate configured on the Tester by 50%. 8. Repeat steps 4 through 7 until the Session Establishment Rate is obtained and recorded. Expected Results: Session Establishment Rate results obtained with Forking may be lower than those obtained without Forking enabled. 6.6. Session Establishment Rate with Forking and Loop Detection Objective: To benchmark the Session Establishment Rate of the DUT/SUT with zero failures when both the Forking and Loop Detection Options are enabled. Procedure: 1. Configure the DUT in the test topology shown in Figure 1 or SUT as shown in Figures 2 or 3. 2. Configure Tester for a Session Attempt Rate = 100 SPS, maximum Session Attempts = 100,000 and Media Streams Per Session=0. 3. Start Tester to initiate SIP Session establishment with the DUT. 4. Enable the Loop Detection Options on the DUT. 5. Set the number of endpoints that will receive the forked invitation to a value of 2 or more (subsequent tests may increase this value at the discretion of the tester.) 6. Measure Session Attempt Failures and total Established Sessions at the Tester. 7. If a Session Attempt Failure is recorded then reduce the Session Attempt Rate configured on the Tester by 50%. Poretsky, et al. Expires January 13, 2011 [Page 12] Internet-Draft SIP Benchmarking Methodology July 2010 8. If no Session Attempt Failure is recorded then increase the Session Attempt Rate configured on the Tester by 50%. 9. Repeat steps 4 through 7 until the Session Establishment Rate is obtained and recorded. Expected Results: Session Establishment Rate results obtained with Forking and Loop Detection may be lower than those obtained with only Forking or Loop Detection enabled. 6.7. Session Establishment Rate with TLS Encrypted SIP Objective: To benchmark the Session Establishment Rate of the DUT/SUT with zero failures when using TLS encrypted SIP. Procedure: 1. Configure the DUT in the test topology shown in Figure 1 or SUT as shown in Figures 2 or 3. 2. Configure Tester for SIP TCP, enable TLS, Session Attempt Rate = 100 SPS, maximum Session Attempts = 100,000 and Media Streams Per Session = 0. 3. Start Tester to initiate SIP Session establishment with the DUT. 4. Measure Session Attempt Failures and total Established Sessions at the Tester. 5. If a Session Attempt Failure is recorded then reduce the Session Attempt Rate configured on the Tester by 50%. 6. If no Session Attempt Failure is recorded then increase the Session Attempt Rate configured on the Tester by 50%. 7. Repeat steps 3 through 6 until the Session Establishment Rate is obtained and recorded. Expected Results: Session Establishment Rate results obtained with TLS Encrypted SIP may be lower than those obtained with plaintext SIP. 6.8. Session Establishment Rate with IPsec Encrypted SIP Objective: To benchmark the Session Establishment Rate of the DUT/SUT with zero failures when using IPsec Encryoted SIP. Procedure: 1. Configure the DUT in the test topology shown in Figure 1 or SUT as shown in Figures 2 or 3. 2. Configure Tester for SIP TCP, enable IPSec, Session Attempt Rate = 100 SPS, maximum Session Attempts = 100,000 and Media Streams Per Session = 0. 3. Start Tester to initiate SIP Session establishment with the DUT. Poretsky, et al. Expires January 13, 2011 [Page 13] Internet-Draft SIP Benchmarking Methodology July 2010 4. Measure Session Attempt Failures and total Established Sessions at the Tester. 5. If a Session Attempt Failure is recorded then reduce the Session Attempt Rate configured on the Tester by 50%. 6. If no Session Attempt Failure is recorded then increase the Session Attempt Rate configured on the Tester by 50%. 7. Repeat steps 3 through 6 until the Session Establishment Rate is obtained and recorded. Expected Results: Session Establishment Rate results obtained with IPSec Encrypted SIP may be lower than those obtained with plaintext SIP. 6.9. Session Establishment Rate with SIP Flooding Objective: To benchmark the Session Establishment Rate of the SUT with zero failures when SIP Flooding is occurring. Procedure: 1. Configure the DUT in the test topology shown in Figure 1 or the SUT as shown in Figure 2. 2. Configure Tester for SIP UDP with an Session Attempt Rate = 100 SPS, maximum Session Attempts = 100,000, Associated Media Streams Per Session = 0, and SIP INVITE Message Flood = 500 per second. 3. Start Tester to initiate SIP Session establishment with the SUT and SIP Flood targetted at the Server. 4. At the Tester measure Session Attempt Failures, total Established Sessions, and Packet Loss [RFC2544] of the media. 5. If a Session Attempt Failure or Packet Loss is recorded then reduce the Session Attempt Rate configured on the Tester by 50%. 6. If no Session Attempt Failure or Packet Loss is recorded then increase the Session Attempt Rate configured on the Tester by 50%. 7. Repeat steps 3 through 6 until the Session Establishment Rate is obtained and recorded. 8. Repeat steps 1 through 7 with SIP INVITE Message Flood = 1000 per second. Expected Results: Session Establishment Rate results obtained with SIP Flooding may be degraded. 6.10. Maximum Registration Rate Objective: Poretsky, et al. Expires January 13, 2011 [Page 14] Internet-Draft SIP Benchmarking Methodology July 2010 To benchmark the maximum registration rate of the DUT/SUT with zero failures. Procedure: 1. Configure the DUT in the test topology shown in Figure 1 or SUT as shown in Figures 2 or 3. 2. Configure Tester with a Registration Rate = 100 SPS and maximum registrations attempted = 100,000. 3. Set the registration timeout value to at least 3600 seconds. 4. At the Tester measure failed registration attempts, total registrations and packet loss. 5. If a Failed Registration Attempt or Packet Loss is recorded then reduce the Attempted Registration Rate configured on the Tester by 50%. 6. If no Failed Registration or Packet Loss is recorded then increase the Attempted Registration Rate configured on the Tester by 50%. 7. Repeat steps 5 and 6 until the all registrations have succeeded. This number is obtained and recorded. Expected Results: 6.11. Maximum Re-Registration Rate Objective: To benchmark the maximum re-registration rate of the DUT/SUT with zero failures. Procedure: 1. Configure the DUT in the test topology shown in Figure 1 or SUT as shown in Figures 2 or 3. 2. Execute test detailed in Section 6.10 to register the endpoints with the registrar. The rest of the steps below MUST be performed at least 5 minutes after, but no more than 15 minutes after the test performed in Section 6.10. 3. Configure Tester for an attempted Registration Rate = 100 SPS and maximum registrations attempted = 100,000. 4. Configure Tester to re-register the same address-of-records that were registered in Section 6.10. 5. At the Tester measure failed registration attempts, total registrations and packet loss. 6. If a Failed Registration Attempt or Packet Loss is recorded then reduce the Attempted Registration Rate configured on the Tester by 50%. 7. If no Failed Registration or Packet Loss is recorded then increase the Attempted Registration Rate configured on the Tester by 50%. 8. Repeat steps 6 and 7 until the all re-registrations have succeeded. This number is obtained and recorded. Poretsky, et al. Expires January 13, 2011 [Page 15] Internet-Draft SIP Benchmarking Methodology July 2010 Expected Results: The rate should be at least equal to but not more than the result of Section 6.10. 6.12. Maximum IM Rate Objective: To benchmark the maximum IM rate of the SUT with zero failures. Procedure: 1. Configure the DUT in the test topology shown in Figure 1 or SUT as shown in Figures 2 or 3. 2. Configure Tester for an IM Rate = 100 SPS, Maximum IM Attempted = 100,000. 3. At the Tester measure Failed IM Attempts, Total IM and Packet Loss. 4. If a Failed IM Attempt or Packet Loss is recorded then reduce the Attempted IM Rate configured on the Tester by 50%. 5. If no Failed IM or Packet Loss is recorded then increase the Attempted IM Rate configured on the Tester by 50%. 6. Repeat steps 3 through 6 until the Maximum IM Rate is obtained and recorded. Expected Results: 6.13. Session Capacity without Media Objective: To benchmark the Session Capacity of the SUT without Associated Media. Procedure: 1. Configure the DUT in the test topology shown in Figure 1 or SUT as shown in Figures 2 or 3. 2. Configure Tester for a Session Attempt Rate = Session Establishment Rate, maximum Session Attempts = 10,000 and Media Streams Per Session = 0. 3. Start Tester to initiate SIP Session establishment with the DUT. 4. Measure Session Attempt Failures, total Established Sessions, and Packet Loss [RFC2544] at the Tester. 5. If a Session Attempt Failure or Packet Loss is recorded then reduce the maximum Session Attempts configured on the Tester by 5,000. 6. If no Session Attempt Failure or Packet Loss is recorded then increase the maximum Session Attempts configured on the Tester by 10,000. 7. Repeat steps 3 through 6 until the Session Capacity is obtained and recorded. 8. Repeat steps 1 through 7 for multimedia in which media streams per session = 2. Poretsky, et al. Expires January 13, 2011 [Page 16] Internet-Draft SIP Benchmarking Methodology July 2010 Expected Results: This is the scenario to obtain the maximum Session Capacity of the DUT/SUT. 6.14. Session Capacity with Media Objective: To benchmark the session capacity of the DUT/SUT with Associated Media. Procedure: 1. Configure the DUT in the test topology shown in Figure 1 or SUT as shown in Figures 2 or 3. 2. Configure Tester for a Session Attempt Rate = 100 SPS, Session Duration = 30 sec, maximum Session Attempts = 100,000 and Media Streams Per Session = 1. NOTE: The total offered load to the DUT/SUT SHOULD be equal to the Throughput of the DUT/SUT as defined in [RFC2544]. The offered load to the DUT/SUT for each media stream SHOULD be equal to Throughput/Maximum Session Attemps. 3. Start Tester to initiate SIP Session establishment with the SUT and transmit media through the SUT to a destination other than the server. 4. Measure Session Attempt Failures and total Established Sessions at the Tester. 5. If a Session Attempt Failure is recorded then reduce the maximum Session Attempts configured on the Tester by 5,000. 6. If no Session Attempt Failure is recorded then increase the maximum Session Attempts configured on the Tester by 10,000. 7. Repeat steps 3 through 6 until the Session Capacity is obtained and recorded. Expected Results: Session Capacity results obtained with Associated Media with any number of media streams per SIP session will be identical to the Session Capacity results obtained without media. 6.15. Session Capacity with Media and a Media Relay/NAT and/or Firewall Objective: To benchmark the Session Establishment Rate of the SUT with Associated Media. Procedure: 1. Configure the SUT in the test topology shown in Figure 3. 2. Configure Tester for a Session Attempt Rate = 100 SPS, Session Duration = 30 sec, maximum Session Attempts = 100,000 and Media Streams Per Session = 1. NOTE: The offered load for each media stream SHOULD be as in Equation 1. Poretsky, et al. Expires January 13, 2011 [Page 17] Internet-Draft SIP Benchmarking Methodology July 2010 3. Start Tester to initiate SIP Session establishment with the SUT and transmit media through the SUT to a destination other than the server. 4. Measure Session Attempt Failures and total Established Sessions at the Tester. 5. If a Session Attempt Failure is recorded then reduce the maximum Session Attempts configured on the Tester by 5,000. 6. If no Session Attempt Failure is recorded then increase the maximum Session Attempts configured on the Tester by 10,000. 7. Repeat steps 3 through 6 until the Session Capacity is obtained and recorded. Expected Results: Session Capacity results obtained with Associated Media with any number of media streams per SIP session may be lower than the Session Capacity without Media result if the Media Relay, NAT or Firewall is sharing a platform with the server. 7. IANA Considerations This document does not requires any IANA considerations. 8. Security Considerations Documents of this type do not directly affect the security of Internet or corporate networks as long as benchmarking is not performed on devices or systems connected to production networks. Security threats and how to counter these in SIP and the media layer is discussed in RFC3261, RFC3550, and RFC3711 and various other drafts. This document attempts to formalize a set of common methodology for benchmarking performance of SIP devices in a lab environment. 9. Acknowledgments The authors would like to thank Keith Drage and Daryl Malas for their contributions to this document. 10. References 10.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Poretsky, et al. Expires January 13, 2011 [Page 18] Internet-Draft SIP Benchmarking Methodology July 2010 Network Interconnect Devices", RFC 2544, March 1999. [I-D.sip-bench-term] Poretsky, S., Gurbani, V., and C. Davids, "SIP Performance Benchmarking Terminology", draft-ietf-bmwg-sip-bench-term-02 (work in progress), July 2010. 10.2. Informative References [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, "SIP: Session Initiation Protocol", RFC 3261, June 2002. Authors' Addresses Scott Poretsky Allot Communications 300 TradeCenter, Suite 4680 Woburn, MA 08101 USA Phone: +1 508 309 2179 Email: sporetsky@allot.com Vijay K. Gurbani Bell Laboratories, Alcatel-Lucent 1960 Lucent Lane Rm 9C-533 Naperville, IL 60566 USA Phone: +1 630 224 0216 Email: vkg@bell-labs.com Carol Davids Illinois Institute of Technology 201 East Loop Road Wheaton, IL 60187 USA Phone: +1 630 682 6024 Email: davids@iit.edu Poretsky, et al. Expires January 13, 2011 [Page 19]