idnits 2.17.1 draft-ietf-bmwg-sip-bench-meth-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (March 12, 2012) is 4428 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Downref: Normative reference to an Informational RFC: RFC 2544 == Outdated reference: A later version (-12) exists of draft-ietf-bmwg-sip-bench-term-04 ** Downref: Normative reference to an Informational draft: draft-ietf-bmwg-sip-bench-term (ref. 'I-D.sip-bench-term') Summary: 2 errors (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Methodology Working C. Davids 3 Group Illinois Institute of Technology 4 Internet-Draft V. Gurbani 5 Expires: September 13, 2012 Bell Laboratories, Alcatel-Lucent 6 S. Poretsky 7 Allot Communications 8 March 12, 2012 10 Methodology for Benchmarking SIP Networking Devices 11 draft-ietf-bmwg-sip-bench-meth-04 13 Abstract 15 This document describes the methodology for benchmarking Session 16 Initiation Protocol (SIP) performance as described in SIP 17 benchmarking terminology document. The methodology and terminology 18 are to be used for benchmarking signaling plane performance with 19 varying signaling and media load. Both scale and establishment rate 20 are measured by signaling plane performance. The SIP Devices to be 21 benchmarked may be a single device under test (DUT) or a system under 22 test (SUT). Benchmarks can be obtained and compared for different 23 types of devices such as SIP Proxy Server, SBC, and server paired 24 with a media relay or Firewall/NAT device. 26 Status of this Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at http://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on September 13, 2012. 43 Copyright Notice 45 Copyright (c) 2012 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (http://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 61 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 62 3. Benchmarking Topologies . . . . . . . . . . . . . . . . . . . 5 63 4. Benchmarking Considerations . . . . . . . . . . . . . . . . . 5 64 4.1. Selection of SIP Transport Protocol . . . . . . . . . . . 5 65 4.2. Signaling Server . . . . . . . . . . . . . . . . . . . . . 5 66 4.3. Associated Media . . . . . . . . . . . . . . . . . . . . . 5 67 4.4. Selection of Associated Media Protocol . . . . . . . . . . 6 68 4.5. Number of Associated Media Streams per SIP Session . . . . 6 69 4.6. Session Duration . . . . . . . . . . . . . . . . . . . . . 6 70 4.7. Attempted Sessions per Second . . . . . . . . . . . . . . 6 71 4.8. Stress Testing . . . . . . . . . . . . . . . . . . . . . . 6 72 4.9. Benchmarking algorithm . . . . . . . . . . . . . . . . . . 6 73 5. Reporting Format . . . . . . . . . . . . . . . . . . . . . . . 9 74 5.1. Test Setup Report . . . . . . . . . . . . . . . . . . . . 9 75 5.2. Device Benchmarks for IS . . . . . . . . . . . . . . . . . 9 76 5.3. Device Benchmarks for NS . . . . . . . . . . . . . . . . . 10 77 6. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 10 78 6.1. Baseline Session Establishment Rate of the test bed . . . 10 79 6.2. Session Establishment Rate without media . . . . . . . . . 11 80 6.3. Session Establishment Rate with Media on DUT/SUT . . . . . 11 81 6.4. Session Establishment Rate with Media not on DUT/SUT . . . 12 82 6.5. Session Establishment Rate with Loop Detection Enabled . . 12 83 6.6. Session Establishment Rate with Forking . . . . . . . . . 13 84 6.7. Session Establishment Rate with Forking and Loop 85 Detection . . . . . . . . . . . . . . . . . . . . . . . . 14 86 6.8. Session Establishment Rate with TLS Encrypted SIP . . . . 14 87 6.9. Session Establishment Rate with IPsec Encrypted SIP . . . 15 88 6.10. Session Establishment Rate with SIP Flooding . . . . . . . 15 89 6.11. Maximum Registration Rate . . . . . . . . . . . . . . . . 16 90 6.12. Maximum Re-Registration Rate . . . . . . . . . . . . . . . 16 91 6.13. Maximum IM Rate . . . . . . . . . . . . . . . . . . . . . 17 92 6.14. Session Capacity without Media . . . . . . . . . . . . . . 17 93 6.15. Session Capacity with Media . . . . . . . . . . . . . . . 18 94 6.16. Session Capacity with Media and a Media Relay/NAT 95 and/or Firewall . . . . . . . . . . . . . . . . . . . . . 18 96 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 97 8. Security Considerations . . . . . . . . . . . . . . . . . . . 19 98 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 19 99 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 19 100 10.1. Normative References . . . . . . . . . . . . . . . . . . . 19 101 10.2. Informative References . . . . . . . . . . . . . . . . . . 20 102 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20 104 1. Terminology 106 In this document, the key words "MUST", "MUST NOT", "REQUIRED", 107 "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT 108 RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as 109 described in BCP 14, conforming to [RFC2119] and indicate requirement 110 levels for compliant implementations. 112 Terms specific to SIP [RFC3261] performance benchmarking are defined 113 in [I-D.sip-bench-term]. 115 RFC 2119 defines the use of these key words to help make the intent 116 of standards track documents as clear as possible. While this 117 document uses these keywords, this document is not a standards track 118 document. The term Throughput is defined in [RFC2544]. 120 2. Introduction 122 This document describes the methodology for benchmarking Session 123 Initiation Protocol (SIP) performance as described in Terminology 124 document [I-D.sip-bench-term]. The methodology and terminology are 125 to be used for benchmarking signaling plane performance with varying 126 signaling and media load. Both scale and establishment rate are 127 measured by signaling plane performance. 129 The SIP Devices to be benchmarked may be a single device under test 130 (DUT) or a system under test (SUT). The DUT is a SIP Server, which 131 may be any [RFC3261] conforming device. The SUT can be any device or 132 group of devices containing RFC 3261 conforming functionality along 133 with Firewall and/or NAT functionality. This enables benchmarks to 134 be obtained and compared for different types of devices such as SIP 135 Proxy Server, SBC, SIP proxy server paired with a media relay or 136 Firewall/NAT device. SIP Associated Media benchmarks can also be 137 made when testing SUTs. 139 The test cases covered in this methodology document provide 140 benchmarks metrics of Registration Rate, SIP Session Establishment 141 Rate, Session Capacity, IM Rate, and Presence Rate. These can be 142 benchmarked with or without associated Media. Some cases are also 143 included to cover Forking, Loop detecion, Encrypted SIP, and SIP 144 Flooding. The test topologies that can be used are described in the 145 Test Setup section. Topologies are provided for benchmarking of a 146 DUT or SUT. Benchmarking with Associated Media can be performed when 147 using a SUT. 149 SIP permits a wide range of configuration options that are also 150 explained in the Test Setup section. Benchmark metrics could 151 possibly be impacted by Associated Media. The selected values for 152 Session Duration and Media Streams Per Session enable benchmark 153 metrics to be benchmarked without Associated Media. Session Setup 154 Rate could possibly be impacted by the selected value for Maximum 155 Sessions Attempted. The benchmark for Session Establishment Rate is 156 measured with a fixed value for maximum Session Attempts. 158 Finally, the overall value of these tests is to serve as a comparison 159 function between multiple SIP implementations. One way to use these 160 tests is to derive benchmarks with SIP devices from Vendor-A, derive 161 a new set of benchmarks with similar SIP devices from Vendor-B and 162 perform a comparison on the results of Vendor-A and Vendor-B. This 163 document does not make any claims on the interpretation of such 164 results. 166 3. Benchmarking Topologies 168 Familiarity with the benchmarking models in Section 2.2 of 169 [I-D.sip-bench-term] is assumed. Figures 1 through 10 in 170 [I-D.sip-bench-term] contain the canonical topologies that can be 171 used to perform the benchmarking tests listed in this document. 173 4. Benchmarking Considerations 175 4.1. Selection of SIP Transport Protocol 177 Test cases may be performed with any transport protocol supported by 178 SIP. This includes, but is not limited to, SIP TCP, SIP UDP, and 179 TLS. The protocol used for the SIP transport protocol must be 180 reported with benchmarking results. 182 4.2. Signaling Server 184 The Server is a SIP-speaking device that complies with RFC 3261. The 185 purpose of this document is to benchmark SIP performance, not 186 conformance. Conformance to [RFC3261] is assumed for all tests. The 187 Server may be the DUT or a component of a SUT that includes Firewall 188 and/or NAT functionality. The components of the SUT may be a single 189 physical device or separate devices. 191 4.3. Associated Media 193 Some tests may require associated media to be present for each SIP 194 session. The Server is not involved in the forwarding of media. 195 Associated Media can be benchmarked only with a SUT in which the 196 media traverses a Media Relay, Firewall, NAT, or Firewall/NAT device. 198 The test topologies to be used when benchmarking SUT performance for 199 Associated Media are shown in Figures 4 and 5, in which the SIP 200 signaling is bidirectional and the Associated Media is 201 unidirectional. 203 4.4. Selection of Associated Media Protocol 205 The test cases specified in this document provide SIP performance 206 independent of the protocol used for the media stream. Any media 207 protocol supported by SIP may be used. This includes, but is not 208 limited to, RTP, RTSP, and SRTP. The protocol used for Associated 209 Media MUST be reported with benchmarking results. 211 4.5. Number of Associated Media Streams per SIP Session 213 Benchmarking results may vary with the number of media streams per 214 SIP session. When benchmarking a SUT for voice, a single media 215 stream is used. When benchmarking a SUT for voice and video, two 216 media streams are used. The number of Associated Media Streams MUST 217 be reported with benchmarking results. 219 4.6. Session Duration 221 SUT performance benchmarks may vary with the duration of SIP 222 sessions. Session Duration MUST be reported with benchmarking 223 results. A Session Duration of zero seconds indicates transmission 224 of a BYE immediately following successful SIP establishment indicate 225 by receipt of a 200 OK. An infinite Session Duration indicates that 226 a BYE is never transmitted. 228 4.7. Attempted Sessions per Second 230 DUT and SUT performance benchmarks may vary with the the rate of 231 attempted sessions offered by the Tester. Attempted Sessions per 232 Second MUST be reported with benchmarking results. 234 4.8. Stress Testing 236 The purpose of this document is to benchmark SIP performance; this 237 document does not benchmark stability of SIP systems under stressful 238 conditions such as a high rate of Attempted Sessions per Second. 240 4.9. Benchmarking algorithm 242 In order to benchmark the test cases uniformly in Section 6, the 243 algorithm described in this section should be used. Both, a prosaic 244 description of the algorithm and a pseudo-code description are 245 provided. 247 The goal is to find the largest value of a SIP session-request-rate, 248 measured in sessions-per-second, which the DUT/SUT can process with 249 zero errors. To discover that number, an iterative process (defined 250 below) is used to find a candidate for this rate. Once the candidate 251 rate has been found, the DUT/SUT is subjected to an offered load 252 whose arrival rate is set to that of the candidate rate. This test 253 is run for an extended period of time, which is referred to as 254 infinity, and which is, itself, a parameter of the test labeled T in 255 the pseudo-code). This latter phase of testing is called the steady- 256 state phase. If errors are encountered during this steady-state 257 phase, then the candidate rate is reduced by a defined percent, also 258 a parameter of test, and the steady-state phase is entered again 259 until a final (new) steady-state rate is achieved. 261 The iterative process itself is defined as follows: a starting rate 262 of 100 sessions per second (sps) is selected. The test is executed 263 for the time period identified by t in the pseudo-code below. If no 264 failures occur, the rate is increased to 150 sps and again tested for 265 time period t. The attempt rate is continuously ramped up until a 266 failure is encountered before the end of the test time t. Then an 267 attempt rate is calculated that is higher than the last successful 268 attempt rate by a quantity equal to half the difference between the 269 rate at which failures occurred and the last successful rate. If 270 this new attempt rate also results in errors, a new attempt rate is 271 tried that is higher than the last successful attempt rate by a 272 quantity equal to half the difference between the rate at which 273 failures occurred and the last successful rate. Continuing in this 274 way, an attempt rate without errors is found. The operator can 275 specify margin of error using the parameter G, measured in units of 276 sessions per second. 278 The pseudo-code corresponding to the description above follows. 280 ; ---- Parameters of test, adjust as needed 281 t := 5000 ; local maximum; used to figure out largest 282 ; value 283 T := 50000 ; global maximum; once largest value has been 284 ; figured out, pump this many requests before calling 285 ; the test a success 286 m := {...} ; other attributes that affect testing, such 287 ; as media streams, etc. 288 s := 100 ; Initial session attempt rate (in sessions/sec) 289 G := 5 ; granularity of results - the margin of error in sps 290 C := 0.05 ; caliberation amount: How much to back down if we 291 ; have found candidate s but cannot send at rate s for 292 ; time T without failures 294 ; ---- End of parameters of test 295 ; ---- Initialization of flags, candidate values and upper bounds 297 f := false ; indicates that you had a success after the upper limit 298 F := false ; indicates that test is done 299 c := 0 ; indicates that we have found an upper limit 301 proc main 302 find_largest_value ; First, figure out the largest value. 304 ; Now that the largest value (saved in s) has been figured out, 305 ; use it for sending out s requests/s and send out T requests. 307 do { 308 send_traffic(s, m, T) ; send_traffic not shown 309 if (all requests succeeded) { 310 F := true ; test is done 311 } else if (one or more requests fail) { 312 s := s - (C * s) ; Reduce s by calibration amount 313 steady_state 314 } 315 } while (F == false) 316 end proc 318 proc find_largest_value 319 ; Iterative process to figure out the largest value we can 320 ; handle with no failures 321 do { 322 send_traffic(s, m, t) ; Send s request/sec with m 323 ; characteristics until t requests have 324 ; been sent 325 if (all requests succeeded) { 326 s' := s ; save candidate value of metric 328 if ( c == 0 ) { 329 s := s + (0.5 * s) 331 }else if ((c == 1) && (s''-s')) > 2*G ) { 332 s := s + ( 0.5 * (s'' - s ); 334 }else if ((c == 1) && ((s''-s') <= 2*G ) { 335 f := true; 337 } 338 else if (one or more requests fail) { 339 c := 1 ; we have found an upper bound for the metric 340 s'' := s ; save new upper bound 341 s := s - (0.5 * (s - s')) 343 } 344 } while (f == false) 345 end proc 347 5. Reporting Format 349 5.1. Test Setup Report 351 SIP Transport Protocol = ___________________________ 352 (valid values: TCP|UDP|TLS|SCTP|specify-other) 353 Session Attempt Rate = _____________________________ 354 (session attempts/sec) 355 IS Media Attempt Rate = ____________________________ 356 (IS media attempts/sec) 357 Total Sessions Attempted = _________________________ 358 (total sessions to be created over duration of test) 359 Media Streams Per Session = _______________________ 360 (number of streams per session) 361 Associated Media Protocol = _______________________ 362 (RTP|RTSP|specify-other) 363 Media Packet Size = _______________________________ 364 (bytes) 365 Media Offered Load = ______________________________ 366 (packets per second) 367 Media Session Hold Time = _________________________ 368 (seconds) 369 Establishment Threshold time = ____________________ 370 (seconds) 371 Loop Detecting Option = ___________________________ 372 (on|off) 373 Forking Option 374 Number of endpoints request sent to = ___________ 375 (1, means forking is not enabled) 376 Type of forking = _______________________________ 377 (serial|parallel) 379 Note: Total Sessions Attempted is used in the calculation of the 380 Session Establishment Performance ([I-D.sip-bench-term], Section 381 3.4.5). It is the number of session attempts ([I-D.sip-bench-term], 382 Section 3.1.6) that will be made over the duration of the test. 384 5.2. Device Benchmarks for IS 385 Registration Rate = _______________________________ 386 (registrations per second) 387 Re-registration Rate = ____________________________ 388 (registrations per second) 389 Session Capacity = _________________________________ 390 (sessions) 391 Session Overload Capacity = ________________________ 392 (sessions) 393 Session Establishment Rate = ______________________ 394 (sessions per second) 395 Session Establishment Performance = _______________ 396 (total established sessions/total sessions attempted)(no units) 397 Session Attempt Delay = ___________________________ 398 (seconds) 400 5.3. Device Benchmarks for NS 402 IM Rate = _______________________________ (IM messages per second) 404 6. Test Cases 406 6.1. Baseline Session Establishment Rate of the test bed 408 Objective: 409 To benchmark the Session Establishment Rate of the Emulated Agent 410 (EA) with zero failures. 412 Procedure: 413 1. Configure the DUT in the test topology shown in Figure 1 in 414 [I-D.sip-bench-term]. 415 2. Set media streams per session to 0. 416 3. Execute benchmarking algorithm as defined in Section 4.9 to 417 get the baseline session establishment rate. This rate MUST 418 be recorded using any pertinent parameters as shown in the 419 reporting format of Section 5.1. 421 Expected Results: This is the scenario to obtain the maximum Session 422 Establishment Rate of the EA and the test bed when no DUT/SUT is 423 present. The results of this test might be used to normalize test 424 results performed on different test beds or simply to better 425 understand the impact of the DUT/SUT on the test bed in question. 427 6.2. Session Establishment Rate without media 429 Objective: 430 To benchmark the Session Establishment Rate of the DUT/SUT with no 431 associated media and zero failures. 433 Procedure: 434 1. If the DUT/SUT is being benchmarked as a user agent client or 435 a user agent server, configure the DUT in the test topology 436 shown in Figure 1 or Figure 2 in [I-D.sip-bench-term]. 437 Alternatively, if the DUT is being benchmarked as a proxy or a 438 B2BUA, configure the DUT in the test topology shown in Figure 439 5 in [I-D.sip-bench-term]. 440 2. Configure a SUT according to the test topology shown in Figure 441 7 in [I-D.sip-bench-term]. 442 3. Set media streams per session to 0. 443 4. Execute benchmarking algorithm as defined in Section 4.9 to 444 get the session establishment rate. This rate MUST be 445 recorded using any pertinent parameters as shown in the 446 reporting format of Section 5.1. 448 Expected Results: This is the scenario to obtain the maximum Session 449 Establishment Rate of the DUT/SUT. 451 6.3. Session Establishment Rate with Media on DUT/SUT 453 Objective: 454 To benchmark the Session Establishment Rate of the DUT/SUT with 455 zero failures when Associated Media is included in the benchmark 456 test and the media is running through the DUT/SUT. 458 Procedure: 459 1. If the DUT is being benchmarked as a user agent client or a 460 user agent server, configure the DUT in the test topology 461 shown in Figure 3 or Figure 4 of [I-D.sip-bench-term]. 462 Alternatively, if the DUT is being benchmarked as a B2BUA, 463 configure the DUT in the test topology shown in Figure 6 in 464 [I-D.sip-bench-term]. 465 2. Configure a SUT according to the test topology shown in Figure 466 9 in [I-D.sip-bench-term]. 467 3. Set media streams per session to 1. 468 4. Execute benchmarking algorithm as defined in Section 4.9 to 469 get the session establishment rate with media. This rate MUST 470 be recorded using any pertinent parameters as shown in the 471 reporting format of Section 5.1. 473 Expected Results: Session Establishment Rate results obtained with 474 Associated Media with any number of media streams per SIP session 475 are expected to be identical to the Session Establishment Rate 476 results obtained without media in the case where the server is 477 running on a platform separate from the platform on which the 478 Media Relay, NAT or Firewall is running. Session Establishment 479 Rate results obtained with Associated Media may be lower than 480 those obtained without media in the case where the server and the 481 NAT, Firewall or Media Relay are running on the same platform. 483 6.4. Session Establishment Rate with Media not on DUT/SUT 485 Objective: 486 To benchmark the Session Establishment Rate of the DUT/SUT with 487 zero failures when Associated Media is included in the benchmark 488 test but the media is not running through the DUT/SUT. 490 Procedure: 491 1. If the DUT is being benchmarked as proxy or B2BUA, configure 492 the DUT in the test topology shown in Figure 7 in 493 [I-D.sip-bench-term]. 494 2. Configure a SUT according to the test topology shown in Figure 495 8 in [I-D.sip-bench-term]. 496 3. Set media streams per session to 1. 497 4. Execute benchmarking algorithm as defined in Section 4.9 to 498 get the session establishment rate with media. This rate MUST 499 be recorded using any pertinent parameters as shown in the 500 reporting format of Section 5.1. 502 Expected Results: Session Establishment Rate results obtained with 503 Associated Media with any number of media streams per SIP session 504 are expected to be identical to the Session Establishment Rate 505 results obtained without media in the case where the server is 506 running on a platform separate from the platform on which the 507 Media Relay, NAT or Firewall is running. Session Establishment 508 Rate results obtained with Associated Media may be lower than 509 those obtained without media in the case where the server and the 510 NAT, Firewall or Media Relay are running on the same platform. 512 6.5. Session Establishment Rate with Loop Detection Enabled 514 Objective: 515 To benchmark the Session Establishment Rate of the DUT/SUT with 516 zero failures when the Loop Detection option is enabled and no 517 media streams are present. 519 Procedure: 520 1. If the DUT is being benchmarked as a proxy or B2BUA, and loop 521 detection is supported in the DUT, then configure the DUT in 522 the test topology shown in Figure 5 in [I-D.sip-bench-term]. 523 If the DUT does not support loop detection, then this step can 524 be skipped. 525 2. Configure a SUT according to the test topology shown in Figure 526 8 of [I-D.sip-bench-term]. 527 3. Set media streams per session to 0. 528 4. Turn on the Loop Detection option in the DUT or SUT. 529 5. Execute benchmarking algorithm as defined in Section 4.9 to 530 get the session establishment rate with loop detection 531 enabled. This rate MUST be recorded using any pertinent 532 parameters as shown in the reporting format of Section 5.1. 534 Expected Results: Session Establishment Rate results obtained with 535 Loop Detection may be lower than those obtained without Loop 536 Detection enabled. 538 6.6. Session Establishment Rate with Forking 540 Objective: 541 To benchmark the Session Establishment Rate of the DUT/SUT with 542 zero failures when the Forking Option is enabled. 544 Procedure: 545 1. If the DUT is being benchmarked as a proxy or B2BUA, and 546 forking is supported in the DUT, then configure the DUT in the 547 test topology shown in Figure 5 in [I-D.sip-bench-term]. If 548 the DUT does not support forking, then this step can be 549 skipped. 550 2. Configure a SUT according to the test topology shown in Figure 551 8 of [I-D.sip-bench-term]. 552 3. Set media streams per session to 0. 553 4. Set the number of endpoints that will receive the forked 554 invitation to a value of 2 or more (subsequent tests may 555 increase this value at the discretion of the tester.) 556 5. Execute benchmarking algorithm as defined in Section 4.9 to 557 get the session establishment rate with forking. This rate 558 MUST be recorded using any pertinent parameters as shown in 559 the reporting format of Section 5.1. 561 Expected Results: Session Establishment Rate results obtained with 562 Forking may be lower than those obtained without Forking enabled. 564 6.7. Session Establishment Rate with Forking and Loop Detection 566 Objective: 567 To benchmark the Session Establishment Rate of the DUT/SUT with 568 zero failures when both the Forking and Loop Detection Options are 569 enabled. 571 Procedure: 572 1. If the DUT is being benchmarked as a proxy or B2BUA, then 573 configure the DUT in the test topology shown in Figure 5 in 574 [I-D.sip-bench-term]. 575 2. Configure a SUT according to the test topology shown in Figure 576 8 of [I-D.sip-bench-term]. 577 3. Set media streams per session to 0. 578 4. Enable the Loop Detection Options on the DUT. 579 5. Set the number of endpoints that will receive the forked 580 invitation to a value of 2 or more (subsequent tests may 581 increase this value at the discretion of the tester.) 582 6. Execute benchmarking algorithm as defined in Section 4.9 to 583 get the session establishment rate with forking and loop 584 detection. This rate MUST be recorded using any pertinent 585 parameters as shown in the reporting format of Section 5.1. 587 Expected Results: Session Establishment Rate results obtained with 588 Forking and Loop Detection may be lower than those obtained with 589 only Forking or Loop Detection enabled. 591 6.8. Session Establishment Rate with TLS Encrypted SIP 593 Objective: 594 To benchmark the Session Establishment Rate of the DUT/SUT with 595 zero failures when using TLS encrypted SIP. 597 Procedure: 598 1. If the DUT is being benchmarked as a proxy or B2BUA, then 599 configure the DUT in the test topology shown in Figure 5 in 600 [I-D.sip-bench-term]. 601 2. Configure a SUT according to the test topology shown in Figure 602 8 of [I-D.sip-bench-term]. 603 3. Set media streams per session to 0. 604 4. Configure Tester to enable TLS over the transport being 605 benchmarked. Make a note the transport when compiling 606 results. May need to run for each transport of interest. 607 5. Execute benchmarking algorithm as defined in Section 4.9 to 608 get the session establishment rate with encryption. This rate 609 MUST be recorded using any pertinent parameters as shown in 610 the reporting format of Section 5.1. 612 Expected Results: Session Establishment Rate results obtained with 613 TLS Encrypted SIP may be lower than those obtained with plaintext 614 SIP. 616 6.9. Session Establishment Rate with IPsec Encrypted SIP 618 Objective: 619 To benchmark the Session Establishment Rate of the DUT/SUT with 620 zero failures when using IPsec Encryoted SIP. 622 Procedure: 623 1. If the DUT is being benchmarked as a proxy or B2BUA, then 624 configure the DUT in the test topology shown in Figure 5 in 625 [I-D.sip-bench-term]. 626 2. Configure a SUT according to the test topology shown in Figure 627 8 of [I-D.sip-bench-term]. 628 3. Set media streams per session to 0. 629 4. Configure Tester for IPSec. 630 5. Execute benchmarking algorithm as defined in Section 4.9 to 631 get the session establishment rate with encryption. This rate 632 MUST be recorded using any pertinent parameters as shown in 633 the reporting format of Section 5.1. 635 Expected Results: Session Establishment Rate results obtained with 636 IPSec Encrypted SIP may be lower than those obtained with 637 plaintext SIP. 639 6.10. Session Establishment Rate with SIP Flooding 641 Objective: 642 To benchmark the Session Establishment Rate of the SUT with zero 643 failures when SIP Flooding is occurring. 645 Procedure: 646 1. If the DUT is being benchmarked as a proxy or B2BUA, then 647 configure the DUT in the test topology shown in Figure 5 in 648 [I-D.sip-bench-term]. 649 2. Configure a SUT according to the test topology shown in Figure 650 8 of [I-D.sip-bench-term]. 651 3. Set media streams per session to 0. 652 4. Set s = 500 (c.f. Section 4.9). 653 5. Execute benchmarking algorithm as defined in Section 4.9 to 654 get the session establishment rate with flooding. This rate 655 MUST be recorded using any pertinent parameters as shown in 656 the reporting format of Section 5.1. 658 Expected Results: Session Establishment Rate results obtained with 659 SIP Flooding may be degraded. 661 6.11. Maximum Registration Rate 663 Objective: 664 To benchmark the maximum registration rate of the DUT/SUT with 665 zero failures. 667 Procedure: 668 1. If the DUT is being benchmarked as a proxy or B2BUA, then 669 configure the DUT in the test topology shown in Figure 5 in 670 [I-D.sip-bench-term]. 671 2. Configure a SUT according to the test topology shown in Figure 672 8 of [I-D.sip-bench-term]. 673 3. Set media streams per session to 0. 674 4. Set the registration timeout value to at least 3600 seconds. 675 5. Execute benchmarking algorithm as defined in Section 4.9 to 676 get the maximum registration rate. This rate MUST be recorded 677 using any pertinent parameters as shown in the reporting 678 format of Section 5.1. 679 Expected Results: 681 6.12. Maximum Re-Registration Rate 683 Objective: 684 To benchmark the maximum re-registration rate of the DUT/SUT with 685 zero failures. 687 Procedure: 688 1. If the DUT is being benchmarked as a proxy or B2BUA, then 689 configure the DUT in the test topology shown in Figure 5 in 690 [I-D.sip-bench-term]. 691 2. Configure a SUT according to the test topology shown in Figure 692 8 of [I-D.sip-bench-term]. 693 3. First, execute test detailed in Section 6.11 to register the 694 endpoints with the registrar. 695 4. After at least 5 mintes of Step 2, but no more than 10 minutes 696 after Step 2 has been performed, execute test detailed in 697 Section 6.11 again (this will count as a re-registration). 698 5. Execute benchmarking algorithm as defined in Section 4.9 to 699 get the maximum re-registration rate. This rate MUST be 700 recorded using any pertinent parameters as shown in the 701 reporting format of Section 5.1. 703 Expected Results: The rate should be at least equal to but not more 704 than the result of Section 6.11. 706 6.13. Maximum IM Rate 708 Objective: 709 To benchmark the maximum IM rate of the SUT with zero failures. 711 Procedure: 712 1. If the DUT/SUT is being benchmarked as a user agent client or 713 a user agent server, configure the DUT in the test topology 714 shown in Figure 1 or Figure 2 in [I-D.sip-bench-term]. 715 Alternatively, if the DUT is being benchmarked as a proxy or a 716 B2BUA, configure the DUT in the test topology shown in Figure 717 5 in [I-D.sip-bench-term]. 718 2. Configure a SUT according to the test topology shown in Figure 719 5 in [I-D.sip-bench-term]. 720 3. Execute benchmarking algorithm as defined in Section 4.9 to 721 get the maximum IM rate. This rate MUST be recorded using any 722 pertinent parameters as shown in the reporting format of 723 Section 5.1. 725 Expected Results: 727 6.14. Session Capacity without Media 729 Objective: 730 To benchmark the Session Capacity of the SUT without Associated 731 Media. 732 Procedure: 733 1. If the DUT/SUT is being benchmarked as a user agent client or 734 a user agent server, configure the DUT in the test topology 735 shown in Figure 1 or Figure 2 in [I-D.sip-bench-term]. 736 Alternatively, if the DUT is being benchmarked as a proxy or a 737 B2BUA, configure the DUT in the test topology shown in Figure 738 5 in [I-D.sip-bench-term]. 739 2. Configure a SUT according to the test topology shown in Figure 740 7 in [I-D.sip-bench-term]. 741 3. Set the media treams per session to be 0. 742 4. Set the Session Duration to be a value greater than T. 743 5. Execute benchmarking algorithm as defined in Section 4.9 to 744 get the baseline session establishment rate. This rate MUST 745 be recorded using any pertinent parameters as shown in the 746 reporting format of Section 5.1. 747 6. The Session Capacity is the product of T and the Session 748 Establishment Rate. 750 Expected Results: The maximum rate at which the DUT/SUT can handle 751 session establishment requests with no media for an infinitely 752 long period with no errors. This is the SIP "throughput" of the 753 system with no media. 755 6.15. Session Capacity with Media 757 Objective: 758 To benchmark the session capacity of the DUT/SUT with Associated 759 Media. 760 Procedure: 761 1. Configure the DUT in the test topology shown in Figure 3 or 762 Figure 4 of [I-D.sip-bench-term] depending on whether the DUT 763 is being benchmarked as a user agent client or user agent 764 server. Alternatively, configure the DUT in the test topology 765 shown in Figure 6 or Figure 7 in [I-D.sip-bench-term] 766 depending on whether the DUT is being benchmarked as a B2BUA 767 or as a proxy. If a SUT is being benchmarked, configure the 768 SUT as shown in Figure 9 of [I-D.sip-bench-term]. 769 2. Set the media streams per session to 1. 770 3. Set the Session Duration to be a value greater than T. 771 4. Execute benchmarking algorithm as defined in Section 4.9 to 772 get the baseline session establishment rate. This rate MUST 773 be recorded using any pertinent parameters as shown in the 774 reporting format of Section 5.1. 775 5. The Session Capacity is the product of T and the Session 776 Establishment Rate. 777 Expected Results: Session Capacity results obtained with Associated 778 Media with any number of media streams per SIP session will be 779 identical to the Session Capacity results obtained without media. 781 6.16. Session Capacity with Media and a Media Relay/NAT and/or Firewall 783 Objective: 784 To benchmark the Session Establishment Rate of the SUT with 785 Associated Media. 786 Procedure: 787 1. Configure the SUT as shown in Figure 7 or Figure 10 in 788 [I-D.sip-bench-term]. 789 2. Set media streams per session to 1. 790 3. Execute benchmarking algorithm as defined in Section 4.9 to 791 get the session establishment rate with media. This rate MUST 792 be recorded using any pertinent parameters as shown in the 793 reporting format of Section 5.1. 795 Expected Results: Session Capacity results obtained with Associated 796 Media with any number of media streams per SIP session may be 797 lower than the Session Capacity without Media result if the Media 798 Relay, NAT or Firewall is sharing a platform with the server. 800 7. IANA Considerations 802 This document does not requires any IANA considerations. 804 8. Security Considerations 806 Documents of this type do not directly affect the security of 807 Internet or corporate networks as long as benchmarking is not 808 performed on devices or systems connected to production networks. 809 Security threats and how to counter these in SIP and the media layer 810 is discussed in RFC3261, RFC3550, and RFC3711 and various other 811 drafts. This document attempts to formalize a set of common 812 methodology for benchmarking performance of SIP devices in a lab 813 environment. 815 9. Acknowledgments 817 The authors would like to thank Keith Drage and Daryl Malas for their 818 contributions to this document. 820 10. References 822 10.1. Normative References 824 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 825 Requirement Levels", BCP 14, RFC 2119, March 1997. 827 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 828 Network Interconnect Devices", RFC 2544, March 1999. 830 [I-D.sip-bench-term] 831 Davids, C., Gurbani, V., and S. Poretsky, "SIP Performance 832 Benchmarking Terminology", 833 draft-ietf-bmwg-sip-bench-term-04 (work in progress), 834 March 2012. 836 10.2. Informative References 838 [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, 839 A., Peterson, J., Sparks, R., Handley, M., and E. 840 Schooler, "SIP: Session Initiation Protocol", RFC 3261, 841 June 2002. 843 Authors' Addresses 845 Carol Davids 846 Illinois Institute of Technology 847 201 East Loop Road 848 Wheaton, IL 60187 849 USA 851 Phone: +1 630 682 6024 852 Email: davids@iit.edu 854 Vijay K. Gurbani 855 Bell Laboratories, Alcatel-Lucent 856 1960 Lucent Lane 857 Rm 9C-533 858 Naperville, IL 60566 859 USA 861 Phone: +1 630 224 0216 862 Email: vkg@bell-labs.com 864 Scott Poretsky 865 Allot Communications 866 300 TradeCenter, Suite 4680 867 Woburn, MA 08101 868 USA 870 Phone: +1 508 309 2179 871 Email: sporetsky@allot.com