idnits 2.17.1 draft-ietf-bmwg-sip-bench-meth-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** The document seems to lack a License Notice according IETF Trust Provisions of 28 Dec 2009, Section 6.b.i or Provisions of 12 Sep 2009 Section 6.b -- however, there's a paragraph with a matching beginning. Boilerplate error? (You're using the IETF Trust Provisions' Section 6.b License Notice from 12 Feb 2009 rather than one of the newer Notices. See https://trustee.ietf.org/license-info/.) Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (March 4, 2009) is 5526 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Downref: Normative reference to an Informational RFC: RFC 2544 == Outdated reference: A later version (-12) exists of draft-ietf-bmwg-sip-bench-term-00 ** Downref: Normative reference to an Informational draft: draft-ietf-bmwg-sip-bench-term (ref. 'I-D.sip-bench-term') Summary: 3 errors (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Methodology Working S. Poretsky 3 Group Allot Communications 4 Internet-Draft V. Gurbani 5 Expires: September 5, 2009 Bell Laboratories, Alcatel-Lucent 6 C. Davids 7 Illinois Institute of Technology 8 March 4, 2009 10 Methodology for Benchmarking SIP Networking Devices 11 draft-ietf-bmwg-sip-bench-meth-00 13 Status of this Memo 15 This Internet-Draft is submitted to IETF in full conformance with the 16 provisions of BCP 78 and BCP 79. 18 Internet-Drafts are working documents of the Internet Engineering 19 Task Force (IETF), its areas, and its working groups. Note that 20 other groups may also distribute working documents as Internet- 21 Drafts. 23 Internet-Drafts are draft documents valid for a maximum of six months 24 and may be updated, replaced, or obsoleted by other documents at any 25 time. It is inappropriate to use Internet-Drafts as reference 26 material or to cite them other than as "work in progress." 28 The list of current Internet-Drafts can be accessed at 29 http://www.ietf.org/ietf/1id-abstracts.txt. 31 The list of Internet-Draft Shadow Directories can be accessed at 32 http://www.ietf.org/shadow.html. 34 This Internet-Draft will expire on September 5, 2009. 36 Copyright Notice 38 Copyright (c) 2009 IETF Trust and the persons identified as the 39 document authors. All rights reserved. 41 This document is subject to BCP 78 and the IETF Trust's Legal 42 Provisions Relating to IETF Documents in effect on the date of 43 publication of this document (http://trustee.ietf.org/license-info). 44 Please review these documents carefully, as they describe your rights 45 and restrictions with respect to this document. 47 Abstract 49 This document describes the methodology for benchmarking Session 50 Initiation Protocol (SIP) performance as described in SIP 51 benchmarking terminology document. The methodology and terminology 52 are to be used for benchmarking signaling plane performance with 53 varying signaling and media load. Both scale and establishment rate 54 are measured by signaling plane performance. The SIP Devices to be 55 benchmarked may be a single device under test (DUT) or a system under 56 test (SUT). Benchmarks can be obtained and compared for different 57 types of devices such as SIP Proxy Server, SBC, P-CSCF, and Server 58 paired with a Firewall/NAT device. 60 Table of Contents 62 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 63 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 64 3. Test Topologies . . . . . . . . . . . . . . . . . . . . . . . 5 65 4. Test Considerations . . . . . . . . . . . . . . . . . . . . . 7 66 4.1. Selection of SIP Transport Protocol . . . . . . . . . . . 7 67 4.2. Server . . . . . . . . . . . . . . . . . . . . . . . . . . 7 68 4.3. Associated Media . . . . . . . . . . . . . . . . . . . . . 7 69 4.4. Selection of Associated Media Protocol . . . . . . . . . . 8 70 4.5. Number of Associated Media Streams per SIP Session . . . . 8 71 4.6. Session Duration . . . . . . . . . . . . . . . . . . . . . 8 72 4.7. Attempted Sessions per Second . . . . . . . . . . . . . . 8 73 4.8. Stress Testing . . . . . . . . . . . . . . . . . . . . . . 8 74 5. Reporting Format . . . . . . . . . . . . . . . . . . . . . . . 9 75 5.1. Test setup Report . . . . . . . . . . . . . . . . . . . . 9 76 5.2. Device Benchmarks for IS . . . . . . . . . . . . . . . . . 9 77 5.3. Device Benchmarks for NS . . . . . . . . . . . . . . . . . 9 78 6. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 9 79 6.1. Session Establisment Rate . . . . . . . . . . . . . . . . 9 80 6.2. Session Establishment Rate with Media . . . . . . . . . . 10 81 6.3. Session Establishment Rate with Loop Detection Enabled . . 11 82 6.4. Session Establishment Rate with Forking . . . . . . . . . 11 83 6.5. Session Establishment Rate with Forking and Loop 84 Detection . . . . . . . . . . . . . . . . . . . . . . . . 12 85 6.6. Session Establishment Rate with TLS Encrypted SIP . . . . 12 86 6.7. Session Establishment Rate with IPsec Encrypted SIP . . . 13 87 6.8. Session Establishment Rate with SIP Flooding . . . . . . . 13 88 6.9. Maximum Registration Rate . . . . . . . . . . . . . . . . 14 89 6.10. Maximum IM Rate . . . . . . . . . . . . . . . . . . . . . 14 90 6.11. Session Capacity . . . . . . . . . . . . . . . . . . . . . 15 91 6.12. Session Capacity with Media . . . . . . . . . . . . . . . 15 92 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 16 93 8. Security Considerations . . . . . . . . . . . . . . . . . . . 16 94 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 16 95 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 17 96 10.1. Normative References . . . . . . . . . . . . . . . . . . . 17 97 10.2. Informative References . . . . . . . . . . . . . . . . . . 17 98 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 17 100 1. Terminology 102 In this document, the key words "MUST", "MUST NOT", "REQUIRED", 103 "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT 104 RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as 105 described in BCP 14, conforming to [RFC2119] and indicate requirement 106 levels for compliant implementations. 108 Terms specific to SIP [RFC3261] performance benchmarking are defined 109 in [I-D.sip-bench-term]. 111 RFC 2119 defines the use of these key words to help make the intent 112 of standards track documents as clear as possible. While this 113 document uses these keywords, this document is not a standards track 114 document. The term Throughput is defined in [RFC2544]. 116 2. Introduction 118 This document describes the methodology for benchmarking Session 119 Initiation Protocol (SIP) performance as described in Terminology 120 document [I-D.sip-bench-term]. The methodology and terminology are 121 to be used for benchmarking signaling plane performance with varying 122 signaling and media load. Both scale and establishment rate are 123 measured by signaling plane performance. 125 The SIP Devices to be benchmarked may be a single device under test 126 (DUT) or a system under test (SUT). The DUT is a SIP Server, which 127 may be any [RFC3261] conforming device. The SUT can be any device or 128 group of devices containing RFC 3261 conforming functionality along 129 with Firewall and/or NAT functionality. This enables benchmarks to 130 be obtained and compared for different types of devices such as SIP 131 Proxy Server, SBC, P-CSCF, Proxy Server paired with a Firewall/NAT 132 device, and P-CSCF paired with a Firewall/NAT device. SIP Associated 133 Media benchmarks can also be made when testing SUTs. 135 The test cases covered in this methodology document provide 136 benchmarks metrics of Registration Rate, SIP Session Establishment 137 Rate, Session Capacity, IM Rate, and Presence Rate. These can be 138 benchmarked with or without associated Media. Some cases are also 139 included to cover Forking, Loop detecion, Encrypted SIP, and SIP 140 Flooding. The test topologies that can be used are described in the 141 Test Setup section. Topologies are provided for benchmarking of a 142 DUT or SUT. Benchmarking with Associated Media can be performed when 143 using a SUT. 145 SIP permits a wide range of configuration options that are also 146 explained in the Test Setup section. Benchmark metrics could 147 possibly be impacted by Associated Media. The selected values for 148 Session Duration and Media Streams Per Session enable benchmark 149 metrics to be benchmarked without Associated Media. Session Setup 150 Rate could possibly be impacted by the selected value for Maximum 151 Sessions Attempted. The benchmark for Session Establishment Rate is 152 measured with a fixed value for maximum Session Attempts. 154 3. Test Topologies 156 Figures 1 through 5 below provide various topologies to perform the 157 SIP Performance Benchmarking. These figures show the Device Under 158 Test (DUT) to be a single server or a System Under Test (SUT). Test 159 Topology options to include benchmarking with Associated Media 160 require use of a SUT and are shown in Figures 4 and 5. 162 DUT 163 --------- --------- 164 | | | | 165 | | | | 166 | | SIP | | 167 |Server |<------------->| Tester| 168 | | | | 169 | | | | 170 | | | | 171 --------- --------- 173 Figure 1: Basic SIP Test Topology 175 SUT 176 ------------------------ 177 --------- --------- --------- 178 | | | | | | 179 | | | | | | 180 | | SIP |Fire- | SIP | | 181 | Server|<---------------------->| Tester| 182 | | |Wall | | | 183 | | | | | | 184 | | | | | | 185 --------- --------- --------- 187 Figure 2: SIP Test Topology with Firewall 188 SUT 189 ------------------------ 190 --------- --------- --------- 191 | | | | | | 192 | | | | | | 193 | | SIP | NAT | SIP | | 194 | Server|<---------------------->| Tester| 195 | | | | | | 196 | | | | | | 197 | | | | | | 198 --------- --------- --------- 200 Figure 3: SIP Test Topology with NAT Device 202 SUT 203 ------------------------ 204 --------- --------- --------- 205 | | | | | | 206 | | | | | | 207 | | SIP |Fire- | SIP | | 208 | Server|<---------------------->| Tester| 209 | | |Wall | | | 210 | | | | Media | | 211 | | ---| |---------| | 212 --------- | --------- --------- 213 | Media ^ 214 -------------------------| 216 Figure 4: SIP Test Topology with Media through Firewall 217 SUT 218 ------------------------ 219 --------- --------- --------- 220 | | | | | | 221 | | | | | | 222 | | SIP | NAT | SIP | | 223 | Server|<---------------------->| Tester| 224 | | | | | | 225 | | | | Media | | 226 | | ---| |---------| | 227 --------- | --------- --------- 228 | Media ^ 229 -------------------------| 231 Figure 5: SIP Test Topology with Media through NAT Device 233 4. Test Considerations 235 4.1. Selection of SIP Transport Protocol 237 Discussion: 238 Test cases may be performed with any transport protocol supported 239 by SIP. This includes, but is not limited to, SIP TCP, SIP UDP, 240 and TLS. The protocol used for the SIP transport protocol must be 241 reported with benchmarking results. 243 4.2. Server 245 Discussion: 246 The Server is a SIP-speaking device that complies with RFC 3261. 247 The purpose of this document is to benchmark SIP performance, not 248 conformance. Conformance to [RFC3261] is assumed for all tests. 249 The Server may be the DUT or a component of a SUT that includes 250 Firewall and/or NAT functionality. The components of the SUT may 251 be a single physical device or separate devices. 253 4.3. Associated Media 255 Discussion: 257 Some tests may require associated media to be present for each SIP 258 session. The Server is not involved in the forwarding of media. 259 Associated Media can be benchmarked only with a SUT in which the 260 media traverses a Firewall, NAT, or Firewall NAT device. The test 261 topologies to be used when benchmarking SUT performance for 262 Associated Media are shown in Figures 4 and 5, in which the SIP 263 signaling is bidirectional and the Associated Media is 264 unidirectional. 266 4.4. Selection of Associated Media Protocol 268 Discussion: 269 The test cases specified in this document provide SIP performance 270 independent of the protocol used for the media stream. Any media 271 protocol supported by SIP may be used. This includes, but is not 272 limited to, RTP, RTSP, and SRTP. The protocol used for Associated 273 Media must be reported with benchmarking results. 275 4.5. Number of Associated Media Streams per SIP Session 277 Discussion: 278 Benchmarking results may vary with the number of media streams per 279 SIP session. When benchmarking a SUT for voice, a single media 280 stream is used. When benchmarking a SUT for voice and video, two 281 media streams are used. The number of Associated Media Streams 282 must be reported with benchmarking results. 284 4.6. Session Duration 286 Discussion: 287 SUT performance benchmarks may vary with the duration of SIP 288 sessions. Session Duration must be reported with benchmarking 289 results. A Session Duration of zero seconds indicates 290 transmission of a BYE immediately following successful SIP 291 establishment indicate by receipt of a 200 OK. An infinite 292 Session Duration indicates that a BYE is never transmitted. 294 4.7. Attempted Sessions per Second 296 Discussion: 297 DUT and SUT performance benchmarks may vary with the the rate of 298 attempted sessions offered by the Tester. Attempted Sessions per 299 Second must be reported with benchmarking results. 301 4.8. Stress Testing 302 Discussion: 303 The purpose of this document is to benchmark SIP performance, not 304 system stability under stressful conditions such as a high rate of 305 Attempted Sessions per Second. 307 5. Reporting Format 309 5.1. Test setup Report 311 SIP Transport Protocol = ___________________________ 312 Session Attempt Rate = _____________________________ 313 IS Media Attempt Rate = ____________________________ 314 Total Sessions Attempted = _________________________ 315 Media Streams Per Session = _______________________ 316 Associated Media Protocol = _______________________ 317 Media Packet Size = _______________________________ 318 Media Offered Load = ______________________________ 319 Media Session Hold Time = _________________________ 320 Establishment Threshold Time = ____________________ 321 Loop Detecting Option = ___________________________ 322 Forking Option = __________________________________ 324 5.2. Device Benchmarks for IS 326 Registration Rate = _______________________________ 327 Session Capacity = _________________________________ 328 Session Overload Capacity = ________________________ 329 Session Establishment Rate = ______________________ 330 Session Establishment Performance = ______________ 331 Session Attempt Delay = ___________________________ 332 Session Disconnect Delay = ________________________ 334 5.3. Device Benchmarks for NS 336 IM Rate = _______________________________ 338 6. Test Cases 340 6.1. Session Establisment Rate 341 Objective: 342 To benchmark the Session Establishment Rate of the DUT/SUT with 343 zero failures. 344 Procedure: 345 1. Configure the DUT in the test topology shown in Figure 1 or 346 SUT as shown in Figures 2 or 3. 347 2. Configure Tester for SIP UDP with an Session Attempt Rate = 348 100 SPS, maximum Session Attempts = 100,000 and Media Streams 349 Per Session=0. 350 3. Start Tester to initiate SIP Session establishment with the 351 DUT. 352 4. Measure Session Attempt Failures and total Established 353 Sessions at the Tester. 354 5. If a Session Attempt Failure is recorded then reduce the 355 Session Attempt Rate configured on the Tester by 50%. 356 6. If no Session Attempt Failure is recorded then increase the 357 Session Attempt Rate configured on the Tester by 50%. 358 7. Repeat steps 3 through 6 until the Session Establishment Rate 359 is obtained and recorded. 360 Expected Results: 362 6.2. Session Establishment Rate with Media 364 Objective: 365 To benchmark the Session Establishment Rate of the SUT with zero 366 failures when Associated Media is included in the benchmark test. 367 Procedure: 368 1. Configure the SUT in the test topology shown in Figure 4 or 5. 369 2. Configure Tester for SIP UDP with an Session Attempt Rate = 370 100 SPS, maximum Session Attempts = 100,000 and Media Streams 371 Per Session = 1. The rate of offered load for each media 372 stream SHOULD be (eq 1) Offered Load per Media Stream = 373 Throughput / maximum sessions attempted, where Throughput is 374 defined in [RFC2544]. 375 3. Start Tester to initiate SIP Session establishment with the 376 SUT and transmit media through the SUT to a destination other 377 than the server. 378 4. At the Tester measure Session Attempt Failures, total 379 Established Sessions, and Packet Loss [RFC2544] of the media. 380 5. If a Session Attempt Failure or Packet Loss is recorded then 381 reduce the Session Attempt Rate configured on the Tester by 382 50%. 383 6. If no Session Attempt Failure or Packet Loss is recorded then 384 increase the Session Attempt Rate configured on the Tester by 385 50%. 386 7. Repeat steps 3 through 6 until the Session Establishment Rate 387 is obtained and recorded. 389 8. Repeat steps 1 through 7 for multimedia in which Media Streams 390 Per Session = 2. 391 Expected Results: 392 Session Establishment Rate results obtained with Associated Media 393 with any number of media streams per SIP session will be identical 394 to the Session Establishment Rate results obtained without media. 396 6.3. Session Establishment Rate with Loop Detection Enabled 398 Objective: 399 To benchmark the Session Establishment Rate of the DUT/SUT with 400 zero failures when the Loop Detection option is enabled. 401 Procedure: 402 1. Configure the DUT in the test topology shown in Figure 1 or 403 SUT as shown in Figures 2 or 3. 404 2. Configure Tester for SIP UDP with an Session Attempt Rate = 405 100 SPS, maximum Session Attempts = 100,000 and Media Streams 406 Per Session=0. 407 3. Turn on the Loop Detection option in the DUT or SUT. 408 4. Start Tester to initiate SIP Session establishment with the 409 DUT. 410 5. Measure Session Attempt Failures and total Established 411 Sessions at the Tester. 412 6. If a Session Attempt Failure is recorded then reduce the 413 Session Attempt Rate configured on the Tester by 50%. 414 7. If no Session Attempt Failure is recorded then increase the 415 Session Attempt Rate configured on the Tester by 50%. 416 8. Repeat steps 4 through 7 until the Session Establishment Rate 417 is obtained and recorded. 418 Expected Results: 420 6.4. Session Establishment Rate with Forking 422 Objective: 423 To benchmark the Session Establishment Rate of the DUT/SUT with 424 zero failures when the Forking Option is enabled. 425 Procedure: 426 1. Configure the DUT in the test topology shown in Figure 1 or 427 SUT as shown in Figures 2 or 3. 428 2. Configure Tester for SIP UDP with an Session Attempt Rate = 429 100 SPS, maximum Session Attempts = 100,000 and Media Streams 430 Per Session=0. 431 3. Turn on the Forking Option in the DUT or SUT. 432 4. Start Tester to initiate SIP Session establishment with the 433 DUT. 434 5. Measure Session Attempt Failures and total Established 435 Sessions at the Tester. 437 6. If a Session Attempt Failure is recorded then reduce the 438 Session Attempt Rate configured on the Tester by 50%. 439 7. If no Session Attempt Failure is recorded then increase the 440 Session Attempt Rate configured on the Tester by 50%. 441 8. Repeat steps 4 through 7 until the Session Establishment Rate 442 is obtained and recorded. 443 Expected Results: 445 6.5. Session Establishment Rate with Forking and Loop Detection 447 Objective: 448 To benchmark the Session Establishment Rate of the DUT/SUT with 449 zero failures when both the Forking and Loop Detection Options are 450 enabled. 451 Procedure: 452 1. Configure the DUT in the test topology shown in Figure 1 or 453 SUT as shown in Figures 2 or 3. 454 2. Configure Tester for SIP UDP with an Session Attempt Rate = 455 100 SPS, maximum Session Attempts = 100,000 and Media Streams 456 Per Session=0. 457 3. Start Tester to initiate SIP Session establishment with the 458 DUT. 459 4. Enable both the Forking and Loop Detection Options on the DUT. 460 5. Measure Session Attempt Failures and total Established 461 Sessions at the Tester. 462 6. If a Session Attempt Failure is recorded then reduce the 463 Session Attempt Rate configured on the Tester by 50%. 464 7. If no Session Attempt Failure is recorded then increase the 465 Session Attempt Rate configured on the Tester by 50%. 466 8. Repeat steps 4 through 7 until the Session Establishment Rate 467 is obtained and recorded. 468 Expected Results: 470 6.6. Session Establishment Rate with TLS Encrypted SIP 472 Objective: 473 To benchmark the Session Establishment Rate of the DUT/SUT with 474 zero failures when using TLS encrypted SIP. 475 Procedure: 476 1. Configure the DUT in the test topology shown in Figure 1 or 477 SUT as shown in Figures 2 or 3. 478 2. Configure Tester for SIP TCP, enable TLS, Session Attempt Rate 479 = 100 SPS, maximum Session Attempts = 100,000 and Media 480 Streams Per Session = 0. 481 3. Start Tester to initiate SIP Session establishment with the 482 DUT. 484 4. Measure Session Attempt Failures and total Established 485 Sessions at the Tester. 486 5. If a Session Attempt Failure is recorded then reduce the 487 Session Attempt Rate configured on the Tester by 50%. 488 6. If no Session Attempt Failure is recorded then increase the 489 Session Attempt Rate configured on the Tester by 50%. 490 7. Repeat steps 3 through 6 until the Session Establishment Rate 491 is obtained and recorded. 492 Expected Results: 494 6.7. Session Establishment Rate with IPsec Encrypted SIP 496 Objective: 497 To benchmark the Session Establishment Rate of the DUT/SUT with 498 zero failures when using IPsec Encryoted SIP. 499 Procedure: 500 1. Configure the DUT in the test topology shown in Figure 1 or 501 SUT as shown in Figures 2 or 3. 502 2. Configure Tester for SIP TCP, enable IPSec, Session Attempt 503 Rate = 100 SPS, maximum Session Attempts = 100,000 and Media 504 Streams Per Session = 0. 505 3. Start Tester to initiate SIP Session establishment with the 506 DUT. 507 4. Measure Session Attempt Failures and total Established 508 Sessions at the Tester. 509 5. If a Session Attempt Failure is recorded then reduce the 510 Session Attempt Rate configured on the Tester by 50%. 511 6. If no Session Attempt Failure is recorded then increase the 512 Session Attempt Rate configured on the Tester by 50%. 513 7. Repeat steps 3 through 6 until the Session Establishment Rate 514 is obtained and recorded. 515 Expected Results: 517 6.8. Session Establishment Rate with SIP Flooding 519 Objective: 520 To benchmark the Session Establishment Rate of the SUT with zero 521 failures when SIP Flooding is occurring. 522 Procedure: 523 1. Configure the DUT in the test topology shown in Figure 1 or 524 the SUT as shown in Figure 2. 525 2. Configure Tester for SIP UDP with an Session Attempt Rate = 526 100 SPS, maximum Session Attempts = 100,000, Associated Media 527 Streams Per Session = 0, and SIP INVITE Message Flood = 500 528 per second. 529 3. Start Tester to initiate SIP Session establishment with the 530 SUT and SIP Flood targetted at the Server. 532 4. At the Tester measure Session Attempt Failures, total 533 Established Sessions, and Packet Loss [RFC2544] of the media. 534 5. If a Session Attempt Failure or Packet Loss is recorded then 535 reduce the Session Attempt Rate configured on the Tester by 536 50%. 537 6. If no Session Attempt Failure or Packet Loss is recorded then 538 increase the Session Attempt Rate configured on the Tester by 539 50%. 540 7. Repeat steps 3 through 6 until the Session Establishment Rate 541 is obtained and recorded. 542 8. Repeat steps 1 through 7 with SIP INVITE Message Flood = 1000 543 per second. 544 Expected Results: Session Establishment Rate results obtained with 545 SIP Flooding may be degraded. 547 6.9. Maximum Registration Rate 549 Objective: 550 To benchmark the maximum registration rate of the SUT with zero 551 failures. 552 Procedure: 553 1. Configure the DUT in the test topology shown in Figure 1 or 554 SUT as shown in Figures 2 or 3. 555 2. Configure Tester for SIP UDP with an attempted Registration 556 Rate = 100 SPS and maximum registrations attempted = 100,000. 557 3. At the Tester measure failed registration attempts, total 558 registrations and packet loss. 559 4. If a Failed Registration Attempt or Packet Loss is recorded 560 then reduce the Attempted Registration Rate configured on the 561 Tester by 50%. 562 5. If no Failed Registration or Packet Loss is recorded then 563 increase the Attempted Registration Rate configured on the 564 Tester by 50%. 565 6. Repeat steps 3 through 6 until the Session Establishment Rate 566 is obtained and recorded. 567 Expected Results: 569 6.10. Maximum IM Rate 571 Objective: 572 To benchmark the maximum IM rate of the SUT with zero failures. 573 Procedure: 574 1. Configure the DUT in the test topology shown in Figure 1 or 575 SUT as shown in Figures 2 or 3. 576 2. Configure Tester for SIP UDP with an Attempted IM Rate = 100 577 SPS, Maximum IM Attempted = 100,000. 579 3. At the Tester measure Failed IM Attempts, Total IM and Packet 580 Loss. 581 4. If a Failed IM Attempt or Packet Loss is recorded then reduce 582 the Attempted IM Rate configured on the Tester by 50%. 583 5. If no Failed IM or Packet Loss is recorded then increase the 584 Attempted IM Rate configured on the Tester by 50%. 585 6. Repeat steps 3 through 6 until the Session Establishment Rate 586 is obtained and recorded. 587 Expected Results: 589 6.11. Session Capacity 591 Objective: 592 To benchmark the Session Capacity of the SUT with Associated 593 Media. 594 Procedure: 595 1. Configure the DUT in the test topology shown in Figure 1 or 596 SUT as shown in Figures 2 or 3. 597 2. Configure Tester for SIP UDP with an Session Attempt Rate = 598 Session Establishment Rate, maximum Session Attempts = 10,000 599 and Media Streams Per Session = 0. 600 3. Start Tester to initiate SIP Session establishment with the 601 DUT. 602 4. Measure Session Attempt Failures, total Established Sessions, 603 and Packet Loss [RFC2544] at the Tester. 604 5. If a Session Attempt Failure or Packet Loss is recorded then 605 reduce the maximum Session Attempts configured on the Tester 606 by 5,000. 607 6. If no Session Attempt Failure or Packet Loss is recorded then 608 increase the maximum Session Attempts configured on the Tester 609 by 10,000. 610 7. Repeat steps 3 through 6 until the Session Capacity is 611 obtained and recorded. 612 8. Repeat steps 1 through 7 for multimedia in which media streams 613 per session = 2. 614 Expected Results: 616 6.12. Session Capacity with Media 618 Objective: 619 To benchmark the Session Establishment Rate of the DUT/SUT with 620 Associated Media. 621 Procedure: 622 1. Configure the DUT in the test topology shown in Figure 1 or 623 SUT as shown in Figures 2 or 3. 624 2. Configure Tester for SIP UDP with a Session Attempt Rate = 100 625 SPS, Session Duration = 30 sec, maximum Session Attempts = 626 100,000 and Media Streams Per Session = 1. The rate of 627 offered load for each media stream SHOULD be (eq 1) Offered 628 Load per Media Stream = Throughput / maximum Session Attempts, 629 where Throughput is defined in [RFC2544]. 630 3. Start Tester to initiate SIP Session establishment with the 631 SUT and transmit media through the SUT to a destination other 632 than the server. 633 4. Measure Session Attempt Failures and total Established 634 Sessions at the Tester. 635 5. If a Session Attempt Failure is recorded then reduce the 636 maximum Session Attempts configured on the Tester by 5,000. 637 6. If no Session Attempt Failure is recorded then increase the 638 maximum Session Attempts configured on the Tester by 10,000. 639 7. Repeat steps 3 through 6 until the Session Capacity is 640 obtained and recorded. 641 Expected Results: Session establishment rate results obtained with 642 Associated Media with any number of media streams per SIP session 643 will be identical to the Session Capacity results obtained without 644 media. 646 7. IANA Considerations 648 This document does not requires any IANA considerations. 650 8. Security Considerations 652 Documents of this type do not directly affect the security of 653 Internet or corporate networks as long as benchmarking is not 654 performed on devices or systems connected to production networks. 655 Security threats and how to counter these in SIP and the media layer 656 is discussed in RFC3261, RFC3550, and RFC3711 and various other 657 drafts. This document attempts to formalize a set of common 658 methodology for benchmarking performance of SIP devices in a lab 659 environment. 661 9. Acknowledgments 663 The authors would like to thank Keith Drage and Daryl Malas for their 664 contributions to this document. 666 10. References 667 10.1. Normative References 669 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 670 Requirement Levels", BCP 14, RFC 2119, March 1997. 672 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 673 Network Interconnect Devices", RFC 2544, March 1999. 675 [I-D.sip-bench-term] 676 Poretsky, S., Gurbani, V., and C. Davids, "SIP Performance 677 Benchmarking Terminology", 678 draft-ietf-bmwg-sip-bench-term-00 (work in progress), 679 March 2009. 681 10.2. Informative References 683 [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, 684 A., Peterson, J., Sparks, R., Handley, M., and E. 685 Schooler, "SIP: Session Initiation Protocol", RFC 3261, 686 June 2002. 688 Authors' Addresses 690 Scott Poretsky 691 Allot Communications 692 67 South Bedford Street, Suite 400 693 Burlington, MA 08103 694 USA 696 Phone: +1 508 309 2179 697 Email: sporetsky@allot.com 699 Vijay K. Gurbani 700 Bell Laboratories, Alcatel-Lucent 701 1960 Lucent Lane 702 Rm 9C-533 703 Naperville, IL 60566 704 USA 706 Phone: +1 630 224 0216 707 Email: vkg@alcatel-lucent.com 708 Carol Davids 709 Illinois Institute of Technology 710 201 East Loop Road 711 Wheaton, IL 60187 712 USA 714 Email: davids@iit.edu