idnits 2.17.1 draft-ietf-bmwg-sip-bench-meth-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (November 8, 2012) is 4186 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Downref: Normative reference to an Informational RFC: RFC 2544 == Outdated reference: A later version (-12) exists of draft-ietf-bmwg-sip-bench-term-04 ** Downref: Normative reference to an Informational draft: draft-ietf-bmwg-sip-bench-term (ref. 'I-D.sip-bench-term') Summary: 2 errors (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Methodology Working C. Davids 3 Group Illinois Institute of Technology 4 Internet-Draft V. Gurbani 5 Expires: May 12, 2013 Bell Laboratories, Alcatel-Lucent 6 S. Poretsky 7 Allot Communications 8 November 8, 2012 10 Methodology for Benchmarking SIP Networking Devices 11 draft-ietf-bmwg-sip-bench-meth-06 13 Abstract 15 This document describes the methodology for benchmarking Session 16 Initiation Protocol (SIP) performance as described in SIP 17 benchmarking terminology document. The methodology and terminology 18 are to be used for benchmarking signaling plane performance with 19 varying signaling and media load. Both scale and establishment rate 20 are measured by signaling plane performance. The SIP Devices to be 21 benchmarked may be a single device under test (DUT) or a system under 22 test (SUT). Benchmarks can be obtained and compared for different 23 types of devices such as SIP Proxy Server, SBC, and server paired 24 with a media relay or Firewall/NAT device. 26 Status of this Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at http://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on May 12, 2013. 43 Copyright Notice 45 Copyright (c) 2012 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (http://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 61 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 62 3. Benchmarking Topologies . . . . . . . . . . . . . . . . . . . 5 63 4. Test Setup Parameters . . . . . . . . . . . . . . . . . . . . 5 64 4.1. Selection of SIP Transport Protocol . . . . . . . . . . . 5 65 4.2. Signaling Server . . . . . . . . . . . . . . . . . . . . . 5 66 4.3. Associated Media . . . . . . . . . . . . . . . . . . . . . 5 67 4.4. Selection of Associated Media Protocol . . . . . . . . . . 6 68 4.5. Number of Associated Media Streams per SIP Session . . . . 6 69 4.6. Session Duration . . . . . . . . . . . . . . . . . . . . . 6 70 4.7. Attempted Sessions per Second . . . . . . . . . . . . . . 6 71 4.8. Stress Testing . . . . . . . . . . . . . . . . . . . . . . 6 72 4.9. Benchmarking algorithm . . . . . . . . . . . . . . . . . . 6 73 5. Reporting Format . . . . . . . . . . . . . . . . . . . . . . . 9 74 5.1. Test Setup Report . . . . . . . . . . . . . . . . . . . . 9 75 5.2. Device Benchmarks for IS . . . . . . . . . . . . . . . . . 10 76 5.3. Device Benchmarks for NS . . . . . . . . . . . . . . . . . 10 77 6. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 10 78 6.1. Baseline Session Establishment Rate of the test bed . . . 10 79 6.2. Session Establishment Rate without media . . . . . . . . . 11 80 6.3. Session Establishment Rate with Media on DUT/SUT . . . . . 11 81 6.4. Session Establishment Rate with Media not on DUT/SUT . . . 12 82 6.5. Session Establishment Rate with Loop Detection Enabled . . 13 83 6.6. Session Establishment Rate with Forking . . . . . . . . . 13 84 6.7. Session Establishment Rate with Forking and Loop 85 Detection . . . . . . . . . . . . . . . . . . . . . . . . 14 86 6.8. Session Establishment Rate with TLS Encrypted SIP . . . . 14 87 6.9. Session Establishment Rate with IPsec Encrypted SIP . . . 15 88 6.10. Session Establishment Rate with SIP Flooding . . . . . . . 15 89 6.11. Maximum Registration Rate . . . . . . . . . . . . . . . . 16 90 6.12. Maximum Re-Registration Rate . . . . . . . . . . . . . . . 16 91 6.13. Maximum IM Rate . . . . . . . . . . . . . . . . . . . . . 17 92 6.14. Session Capacity without Media . . . . . . . . . . . . . . 17 93 6.15. Session Capacity with Media . . . . . . . . . . . . . . . 18 94 6.16. Session Capacity with Media and a Media Relay/NAT 95 and/or Firewall . . . . . . . . . . . . . . . . . . . . . 18 96 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 97 8. Security Considerations . . . . . . . . . . . . . . . . . . . 19 98 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 19 99 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 19 100 10.1. Normative References . . . . . . . . . . . . . . . . . . . 19 101 10.2. Informative References . . . . . . . . . . . . . . . . . . 20 102 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20 104 1. Terminology 106 In this document, the key words "MUST", "MUST NOT", "REQUIRED", 107 "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT 108 RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as 109 described in BCP 14, conforming to [RFC2119] and indicate requirement 110 levels for compliant implementations. 112 Terms specific to SIP [RFC3261] performance benchmarking are defined 113 in [I-D.sip-bench-term]. 115 RFC 2119 defines the use of these key words to help make the intent 116 of standards track documents as clear as possible. While this 117 document uses these keywords, this document is not a standards track 118 document. The term Throughput is defined in [RFC2544]. 120 2. Introduction 122 This document describes the methodology for benchmarking Session 123 Initiation Protocol (SIP) performance as described in Terminology 124 document [I-D.sip-bench-term]. The methodology and terminology are 125 to be used for benchmarking signaling plane performance with varying 126 signaling and media load. Both scale and establishment rate are 127 measured by signaling plane performance. 129 The SIP Devices to be benchmarked may be a single device under test 130 (DUT) or a system under test (SUT). The DUT is a SIP Server, which 131 may be any [RFC3261] conforming device. The SUT can be any device or 132 group of devices containing RFC 3261 conforming functionality along 133 with Firewall and/or NAT functionality. This enables benchmarks to 134 be obtained and compared for different types of devices such as SIP 135 Proxy Server, SBC, SIP proxy server paired with a media relay or 136 Firewall/NAT device. SIP Associated Media benchmarks can also be 137 made when testing SUTs. 139 The test cases covered in this methodology document provide 140 benchmarks metrics of Registration Rate, SIP Session Establishment 141 Rate, Session Capacity, and IM Rate. These can be benchmarked with 142 or without associated Media. Some cases are also included to cover 143 Forking, Loop detection, Encrypted SIP, and SIP Flooding. The test 144 topologies that can be used are described in the Test Setup section. 145 Topologies are provided for benchmarking of a DUT or SUT. 146 Benchmarking with Associated Media can be performed when using a SUT. 148 SIP permits a wide range of configuration options that are also 149 explained in the Test Setup section. Benchmark metrics could 150 possibly be impacted by Associated Media. The selected values for 151 Session Duration and Media Streams Per Session enable benchmark 152 metrics to be benchmarked without Associated Media. Session Setup 153 Rate could possibly be impacted by the selected value for Maximum 154 Sessions Attempted. The benchmark for Session Establishment Rate is 155 measured with a fixed value for maximum Session Attempts. 157 Finally, the overall value of these tests is to serve as a comparison 158 function between multiple SIP implementations. One way to use these 159 tests is to derive benchmarks with SIP devices from Vendor-A, derive 160 a new set of benchmarks with similar SIP devices from Vendor-B and 161 perform a comparison on the results of Vendor-A and Vendor-B. This 162 document does not make any claims on the interpretation of such 163 results. 165 3. Benchmarking Topologies 167 Familiarity with the benchmarking models in Section 2.2 of 168 [I-D.sip-bench-term] is assumed. Figures 1 through 10 in 169 [I-D.sip-bench-term] contain the canonical topologies that can be 170 used to perform the benchmarking tests listed in this document. 172 4. Test Setup Parameters 174 4.1. Selection of SIP Transport Protocol 176 Test cases may be performed with any transport protocol supported by 177 SIP. This includes, but is not limited to, SIP TCP, SIP UDP, and 178 TLS. The protocol used for the SIP transport protocol must be 179 reported with benchmarking results. 181 4.2. Signaling Server 183 The Signaling Server is defined in the companion terminology 184 document, ([I-D.sip-bench-term], Section 3.2.2) It is a SIP-speaking 185 device that complies with RFC 3261. Conformance to [RFC3261] is 186 assumed for all tests. The Signaling Server may be the DUT or a 187 component of a SUT. The Signaling Server may include Firewall and/or 188 NAT functionality. The components of the SUT may be a single 189 physical device or separate devices. 191 4.3. Associated Media 193 Some tests require Associated Media to be present for each SIP 194 session. The test topologies to be used when benchmarking SUT 195 performance for Associated Media are shown in [I-D.sip-bench-term], 196 Figures 4 and 5. 198 4.4. Selection of Associated Media Protocol 200 The test cases specified in this document provide SIP performance 201 independent of the protocol used for the media stream. Any media 202 protocol supported by SIP may be used. This includes, but is not 203 limited to, RTP, RTSP, and SRTP. The protocol used for Associated 204 Media MUST be reported with benchmarking results. 206 4.5. Number of Associated Media Streams per SIP Session 208 Benchmarking results may vary with the number of media streams per 209 SIP session. When benchmarking a SUT for voice, a single media 210 stream is used. When benchmarking a SUT for voice and video, two 211 media streams are used. The number of Associated Media Streams MUST 212 be reported with benchmarking results. 214 4.6. Session Duration 216 SUT performance benchmarks may vary with the duration of SIP 217 sessions. Session Duration MUST be reported with benchmarking 218 results. A Session Duration of zero seconds indicates transmission 219 of a BYE immediately following successful SIP establishment indicate 220 by receipt of a 200 OK. An infinite Session Duration indicates that 221 a BYE is never transmitted. 223 4.7. Attempted Sessions per Second 225 DUT and SUT performance benchmarks may vary with the the rate of 226 attempted sessions offered by the Tester. Attempted Sessions per 227 Second MUST be reported with benchmarking results. 229 4.8. Stress Testing 231 The purpose of this document is to benchmark SIP performance; this 232 document does not benchmark stability of SIP systems under stressful 233 conditions such as a high rate of Attempted Sessions per Second. 235 4.9. Benchmarking algorithm 237 In order to benchmark the test cases uniformly in Section 6, the 238 algorithm described in this section should be used. Both, a prosaic 239 description of the algorithm and a pseudo-code description are 240 provided. 242 The goal is to find the largest value of a SIP session-request-rate, 243 measured in sessions-per-second, which the DUT/SUT can process with 244 zero errors. To discover that number, an iterative process (defined 245 below) is used to find a candidate for this rate. Once the candidate 246 rate has been found, the DUT/SUT is subjected to an offered load 247 whose arrival rate is set to that of the candidate rate. This test 248 is run for an extended period of time, which is referred to as 249 infinity, and which is, itself, a parameter of the test labeled T in 250 the pseudo-code. This latter phase of testing is called the steady- 251 state phase. If errors are encountered during this steady-state 252 phase, then the candidate rate is reduced by a defined percent, also 253 a parameter of test, and the steady-state phase is entered again 254 until a final (new) steady-state rate is achieved. 256 The iterative process itself is defined as follows: a starting rate 257 of 100 sessions per second (sps) is selected. The test is executed 258 for the time period identified by t in the pseudo-code below. If no 259 failures occur, the rate is increased to 150 sps and again tested for 260 time period t. The attempt rate is continuously ramped up until a 261 failure is encountered before the end of the test time t. Then an 262 attempt rate is calculated that is higher than the last successful 263 attempt rate by a quantity equal to half the difference between the 264 rate at which failures occurred and the last successful rate. If 265 this new attempt rate also results in errors, a new attempt rate is 266 tried that is higher than the last successful attempt rate by a 267 quantity equal to half the difference between the rate at which 268 failures occurred and the last successful rate. Continuing in this 269 way, an attempt rate without errors is found. The operator can 270 specify margin of error using the parameter G, measured in units of 271 sessions per second. 273 The pseudo-code corresponding to the description above follows. 275 ; ---- Parameters of test, adjust as needed 276 t := 5000 ; local maximum; used to figure out largest 277 ; value 278 T := 50000 ; global maximum; once largest value has been 279 ; figured out, pump this many requests before calling 280 ; the test a success 281 m := {...} ; other attributes that affect testing, such 282 ; as media streams, etc. 283 s := 100 ; Initial session attempt rate (in sessions/sec) 284 G := 5 ; granularity of results - the margin of error in sps 285 C := 0.05 ; caliberation amount: How much to back down if we 286 ; have found candidate s but cannot send at rate s for 287 ; time T without failures 289 ; ---- End of parameters of test 290 ; ---- Initialization of flags, candidate values and upper bounds 292 f := false ; indicates that you had a success after the upper limit 293 F := false ; indicates that test is done 294 c := 0 ; indicates that we have found an upper limit 296 proc main 297 find_largest_value ; First, figure out the largest value. 299 ; Now that the largest value (saved in s) has been figured out, 300 ; use it for sending out s requests/s and send out T requests. 302 do { 303 send_traffic(s, m, T) ; send_traffic not shown 304 if (all requests succeeded) { 305 F := true ; test is done 306 } else if (one or more requests fail) { 307 s := s - (C * s) ; Reduce s by calibration amount 308 steady_state 309 } 310 } while (F == false) 311 end proc 313 proc find_largest_value 314 ; Iterative process to figure out the largest value we can 315 ; handle with no failures 316 do { 317 send_traffic(s, m, t) ; Send s request/sec with m 318 ; characteristics until t requests have 319 ; been sent 320 if (all requests succeeded) { 321 s' := s ; save candidate value of metric 323 if ( c == 0 ) { 324 s := s + (0.5 * s) 326 }else if ((c == 1) && (s''-s')) > 2*G ) { 327 s := s + ( 0.5 * (s'' - s ); 329 }else if ((c == 1) && ((s''-s') <= 2*G ) { 330 f := true; 332 } 333 else if (one or more requests fail) { 334 c := 1 ; we have found an upper bound for the metric 335 s'' := s ; save new upper bound 336 s := s - (0.5 * (s - s')) 337 } 338 } while (f == false) 339 end proc 341 5. Reporting Format 343 5.1. Test Setup Report 345 SIP Transport Protocol = ___________________________ 346 (valid values: TCP|UDP|TLS|SCTP|specify-other) 347 Session Attempt Rate = _____________________________ 348 (session attempts/sec) 349 IS Media Attempt Rate = ____________________________ 350 (IS media attempts/sec) 351 Total Sessions Attempted = _________________________ 352 (total sessions to be created over duration of test) 353 Media Streams Per Session = _______________________ 354 (number of streams per session) 355 Associated Media Protocol = _______________________ 356 (RTP|RTSP|specify-other) 357 Media Packet Size = _______________________________ 358 (bytes) 359 Media Offered Load = ______________________________ 360 (packets per second) 361 Media Session Hold Time = _________________________ 362 (seconds) 363 Establishment Threshold time = ____________________ 364 (seconds) 365 Loop Detecting Option = ___________________________ 366 (on|off) 367 Forking Option 368 Number of endpoints request sent to = ___________ 369 (1, means forking is not enabled) 370 Type of forking = _______________________________ 371 (serial|parallel) 372 Authentication option = ___________________________________ 373 (on|off; if on, please see Notes 2 and 3 below). 375 Note 1: Total Sessions Attempted is used in the calculation of the 376 Session Establishment Performance ([I-D.sip-bench-term], Section 377 3.4.5). It is the number of session attempts ([I-D.sip-bench-term], 378 Section 3.1.6) that will be made over the duration of the test. 380 Note 2: When the Authentication Option is "on" the test tool must be 381 set to ignore 401 and 407 failure responses in any test described as 382 a "test to failure." If this is not done, all such tests will yield 383 trivial benchmarks, as all attempt rates will lead to a failure after 384 the first attempt. 386 Note 3: When the Authentication Option is "on" the DUT/SUT uses two 387 transactions instead of one when it is establishing a session or 388 accomplishing a registration. The first transaction ends with the 389 401 or 407. The second ends with the 200 OK or another failure 390 message. The Test Organization interested in knowing how many times 391 the EA was intended to send a REGISTER as distinct from how many 392 times the EA wound up actually sending a REGISTER may wish to record 393 the following data as well: Number of responses of the following 394 type: 401: _____________ (if authentication turned on; N/A otherwise) 395 407: _____________ (if authentication turned on; N/A otherwise) 397 5.2. Device Benchmarks for IS 399 Registration Rate = _______________________________ 400 (registrations per second) 401 Re-registration Rate = ____________________________ 402 (registrations per second) 403 Session Capacity = _________________________________ 404 (sessions) 405 Session Overload Capacity = ________________________ 406 (sessions) 407 Session Establishment Rate = ______________________ 408 (sessions per second) 409 Session Establishment Performance = _______________ 410 (total established sessions/total sessions attempted)(no units) 411 Session Attempt Delay = ___________________________ 412 (seconds) 414 5.3. Device Benchmarks for NS 416 IM Rate = _______________________________ (IM messages per second) 418 6. Test Cases 420 6.1. Baseline Session Establishment Rate of the test bed 422 Objective: 423 To benchmark the Session Establishment Rate of the Emulated Agent 424 (EA) with zero failures. 426 Procedure: 427 1. Configure the DUT in the test topology shown in Figure 1 in 428 [I-D.sip-bench-term]. 430 2. Set media streams per session to 0. 431 3. Execute benchmarking algorithm as defined in Section 4.9 to 432 get the baseline session establishment rate. This rate MUST 433 be recorded using any pertinent parameters as shown in the 434 reporting format of Section 5.1. 436 Expected Results: This is the scenario to obtain the maximum Session 437 Establishment Rate of the EA and the test bed when no DUT/SUT is 438 present. The results of this test might be used to normalize test 439 results performed on different test beds or simply to better 440 understand the impact of the DUT/SUT on the test bed in question. 442 6.2. Session Establishment Rate without media 444 Objective: 445 To benchmark the Session Establishment Rate of the DUT/SUT with no 446 associated media and zero failures. 448 Procedure: 449 1. If the DUT/SUT is being benchmarked as a user agent client or 450 a user agent server, configure the DUT in the test topology 451 shown in Figure 1 or Figure 2 in [I-D.sip-bench-term]. 452 Alternatively, if the DUT is being benchmarked as a proxy or a 453 B2BUA, configure the DUT in the test topology shown in Figure 454 5 in [I-D.sip-bench-term]. 455 2. Configure a SUT according to the test topology shown in Figure 456 7 in [I-D.sip-bench-term]. 457 3. Set media streams per session to 0. 458 4. Execute benchmarking algorithm as defined in Section 4.9 to 459 get the session establishment rate. This rate MUST be 460 recorded using any pertinent parameters as shown in the 461 reporting format of Section 5.1. 463 Expected Results: This is the scenario to obtain the maximum Session 464 Establishment Rate of the DUT/SUT. 466 6.3. Session Establishment Rate with Media on DUT/SUT 468 Objective: 469 To benchmark the Session Establishment Rate of the DUT/SUT with 470 zero failures when Associated Media is included in the benchmark 471 test and the media is running through the DUT/SUT. 473 Procedure: 474 1. If the DUT is being benchmarked as a user agent client or a 475 user agent server, configure the DUT in the test topology 476 shown in Figure 3 or Figure 4 of [I-D.sip-bench-term]. 477 Alternatively, if the DUT is being benchmarked as a B2BUA, 478 configure the DUT in the test topology shown in Figure 6 in 479 [I-D.sip-bench-term]. 480 2. Configure a SUT according to the test topology shown in Figure 481 9 in [I-D.sip-bench-term]. 482 3. Set media streams per session to 1. 483 4. Execute benchmarking algorithm as defined in Section 4.9 to 484 get the session establishment rate with media. This rate MUST 485 be recorded using any pertinent parameters as shown in the 486 reporting format of Section 5.1. 488 Expected Results: Session Establishment Rate results obtained with 489 Associated Media with any number of media streams per SIP session 490 are expected to be identical to the Session Establishment Rate 491 results obtained without media in the case where the server is 492 running on a platform separate from the platform on which the 493 Media Relay, NAT or Firewall is running. Session Establishment 494 Rate results obtained with Associated Media may be lower than 495 those obtained without media in the case where the server and the 496 NAT, Firewall or Media Relay are running on the same platform. 498 6.4. Session Establishment Rate with Media not on DUT/SUT 500 Objective: 501 To benchmark the Session Establishment Rate of the DUT/SUT with 502 zero failures when Associated Media is included in the benchmark 503 test but the media is not running through the DUT/SUT. 505 Procedure: 506 1. If the DUT is being benchmarked as proxy or B2BUA, configure 507 the DUT in the test topology shown in Figure 7 in 508 [I-D.sip-bench-term]. 509 2. Configure a SUT according to the test topology shown in Figure 510 8 in [I-D.sip-bench-term]. 511 3. Set media streams per session to 1. 512 4. Execute benchmarking algorithm as defined in Section 4.9 to 513 get the session establishment rate with media. This rate MUST 514 be recorded using any pertinent parameters as shown in the 515 reporting format of Section 5.1. 517 Expected Results: Session Establishment Rate results obtained with 518 Associated Media with any number of media streams per SIP session 519 are expected to be identical to the Session Establishment Rate 520 results obtained without media in the case where the server is 521 running on a platform separate from the platform on which the 522 Media Relay, NAT or Firewall is running. Session Establishment 523 Rate results obtained with Associated Media may be lower than 524 those obtained without media in the case where the server and the 525 NAT, Firewall or Media Relay are running on the same platform. 527 6.5. Session Establishment Rate with Loop Detection Enabled 529 Objective: 530 To benchmark the Session Establishment Rate of the DUT/SUT with 531 zero failures when the Loop Detection option is enabled and no 532 media streams are present. 534 Procedure: 535 1. If the DUT is being benchmarked as a proxy or B2BUA, and loop 536 detection is supported in the DUT, then configure the DUT in 537 the test topology shown in Figure 5 in [I-D.sip-bench-term]. 538 If the DUT does not support loop detection, then this step can 539 be skipped. 540 2. Configure a SUT according to the test topology shown in Figure 541 8 of [I-D.sip-bench-term]. 542 3. Set media streams per session to 0. 543 4. Turn on the Loop Detection option in the DUT or SUT. 544 5. Execute benchmarking algorithm as defined in Section 4.9 to 545 get the session establishment rate with loop detection 546 enabled. This rate MUST be recorded using any pertinent 547 parameters as shown in the reporting format of Section 5.1. 549 Expected Results: Session Establishment Rate results obtained with 550 Loop Detection may be lower than those obtained without Loop 551 Detection enabled. 553 6.6. Session Establishment Rate with Forking 555 Objective: 556 To benchmark the Session Establishment Rate of the DUT/SUT with 557 zero failures when the Forking Option is enabled. 559 Procedure: 560 1. If the DUT is being benchmarked as a proxy or B2BUA, and 561 forking is supported in the DUT, then configure the DUT in the 562 test topology shown in Figure 5 in [I-D.sip-bench-term]. If 563 the DUT does not support forking, then this step can be 564 skipped. 565 2. Configure a SUT according to the test topology shown in Figure 566 8 of [I-D.sip-bench-term]. 567 3. Set media streams per session to 0. 568 4. Set the number of endpoints that will receive the forked 569 invitation to a value of 2 or more (subsequent tests may 570 increase this value at the discretion of the tester.) 571 5. Execute benchmarking algorithm as defined in Section 4.9 to 572 get the session establishment rate with forking. This rate 573 MUST be recorded using any pertinent parameters as shown in 574 the reporting format of Section 5.1. 576 Expected Results: Session Establishment Rate results obtained with 577 Forking may be lower than those obtained without Forking enabled. 579 6.7. Session Establishment Rate with Forking and Loop Detection 581 Objective: 582 To benchmark the Session Establishment Rate of the DUT/SUT with 583 zero failures when both the Forking and Loop Detection Options are 584 enabled. 586 Procedure: 587 1. If the DUT is being benchmarked as a proxy or B2BUA, then 588 configure the DUT in the test topology shown in Figure 5 in 589 [I-D.sip-bench-term]. 590 2. Configure a SUT according to the test topology shown in Figure 591 8 of [I-D.sip-bench-term]. 592 3. Set media streams per session to 0. 593 4. Enable the Loop Detection Options on the DUT. 594 5. Set the number of endpoints that will receive the forked 595 invitation to a value of 2 or more (subsequent tests may 596 increase this value at the discretion of the tester.) 597 6. Execute benchmarking algorithm as defined in Section 4.9 to 598 get the session establishment rate with forking and loop 599 detection. This rate MUST be recorded using any pertinent 600 parameters as shown in the reporting format of Section 5.1. 602 Expected Results: Session Establishment Rate results obtained with 603 Forking and Loop Detection may be lower than those obtained with 604 only Forking or Loop Detection enabled. 606 6.8. Session Establishment Rate with TLS Encrypted SIP 608 Objective: 609 To benchmark the Session Establishment Rate of the DUT/SUT with 610 zero failures when using TLS encrypted SIP. 612 Procedure: 613 1. If the DUT is being benchmarked as a proxy or B2BUA, then 614 configure the DUT in the test topology shown in Figure 5 in 615 [I-D.sip-bench-term]. 616 2. Configure a SUT according to the test topology shown in Figure 617 8 of [I-D.sip-bench-term]. 618 3. Set media streams per session to 0. 619 4. Configure Tester to enable TLS over the transport being 620 benchmarked. Make a note the transport when compiling 621 results. May need to run for each transport of interest. 623 5. Execute benchmarking algorithm as defined in Section 4.9 to 624 get the session establishment rate with encryption. This rate 625 MUST be recorded using any pertinent parameters as shown in 626 the reporting format of Section 5.1. 628 Expected Results: Session Establishment Rate results obtained with 629 TLS Encrypted SIP may be lower than those obtained with plaintext 630 SIP. 632 6.9. Session Establishment Rate with IPsec Encrypted SIP 634 Objective: 635 To benchmark the Session Establishment Rate of the DUT/SUT with 636 zero failures when using IPsec Encryoted SIP. 638 Procedure: 639 1. If the DUT is being benchmarked as a proxy or B2BUA, then 640 configure the DUT in the test topology shown in Figure 5 in 641 [I-D.sip-bench-term]. 642 2. Configure a SUT according to the test topology shown in Figure 643 8 of [I-D.sip-bench-term]. 644 3. Set media streams per session to 0. 645 4. Configure Tester for IPSec. 646 5. Execute benchmarking algorithm as defined in Section 4.9 to 647 get the session establishment rate with encryption. This rate 648 MUST be recorded using any pertinent parameters as shown in 649 the reporting format of Section 5.1. 651 Expected Results: Session Establishment Rate results obtained with 652 IPSec Encrypted SIP may be lower than those obtained with 653 plaintext SIP. 655 6.10. Session Establishment Rate with SIP Flooding 657 Objective: 658 To benchmark the Session Establishment Rate of the SUT with zero 659 failures when SIP Flooding is occurring. 661 Procedure: 662 1. If the DUT is being benchmarked as a proxy or B2BUA, then 663 configure the DUT in the test topology shown in Figure 5 in 664 [I-D.sip-bench-term]. 665 2. Configure a SUT according to the test topology shown in Figure 666 8 of [I-D.sip-bench-term]. 667 3. Set media streams per session to 0. 668 4. Set s = 500 (c.f. Section 4.9). 670 5. Execute benchmarking algorithm as defined in Section 4.9 to 671 get the session establishment rate with flooding. This rate 672 MUST be recorded using any pertinent parameters as shown in 673 the reporting format of Section 5.1. 675 Expected Results: Session Establishment Rate results obtained with 676 SIP Flooding may be degraded. 678 6.11. Maximum Registration Rate 680 Objective: 681 To benchmark the maximum registration rate of the DUT/SUT with 682 zero failures. 684 Procedure: 685 1. If the DUT is being benchmarked as a proxy or B2BUA, then 686 configure the DUT in the test topology shown in Figure 5 in 687 [I-D.sip-bench-term]. 688 2. Configure a SUT according to the test topology shown in Figure 689 8 of [I-D.sip-bench-term]. 690 3. Set media streams per session to 0. 691 4. Set the registration timeout value to at least 3600 seconds. 692 5. Execute benchmarking algorithm as defined in Section 4.9 to 693 get the maximum registration rate. This rate MUST be recorded 694 using any pertinent parameters as shown in the reporting 695 format of Section 5.1. 696 Expected Results: 698 6.12. Maximum Re-Registration Rate 700 Objective: 701 To benchmark the maximum re-registration rate of the DUT/SUT with 702 zero failures. 704 Procedure: 705 1. If the DUT is being benchmarked as a proxy or B2BUA, then 706 configure the DUT in the test topology shown in Figure 5 in 707 [I-D.sip-bench-term]. 708 2. Configure a SUT according to the test topology shown in Figure 709 8 of [I-D.sip-bench-term]. 710 3. First, execute test detailed in Section 6.11 to register the 711 endpoints with the registrar. 712 4. After at least 5 mintes of Step 2, but no more than 10 minutes 713 after Step 2 has been performed, execute test detailed in 714 Section 6.11 again (this will count as a re-registration). 715 5. Execute benchmarking algorithm as defined in Section 4.9 to 716 get the maximum re-registration rate. This rate MUST be 717 recorded using any pertinent parameters as shown in the 718 reporting format of Section 5.1. 719 Expected Results: The rate should be at least equal to but not more 720 than the result of Section 6.11. 722 6.13. Maximum IM Rate 724 Objective: 725 To benchmark the maximum IM rate of the SUT with zero failures. 727 Procedure: 728 1. If the DUT/SUT is being benchmarked as a user agent client or 729 a user agent server, configure the DUT in the test topology 730 shown in Figure 1 or Figure 2 in [I-D.sip-bench-term]. 731 Alternatively, if the DUT is being benchmarked as a proxy or a 732 B2BUA, configure the DUT in the test topology shown in Figure 733 5 in [I-D.sip-bench-term]. 734 2. Configure a SUT according to the test topology shown in Figure 735 5 in [I-D.sip-bench-term]. 736 3. Execute benchmarking algorithm as defined in Section 4.9 to 737 get the maximum IM rate. This rate MUST be recorded using any 738 pertinent parameters as shown in the reporting format of 739 Section 5.1. 741 Expected Results: 743 6.14. Session Capacity without Media 745 Objective: 746 To benchmark the Session Capacity of the SUT without Associated 747 Media. 748 Procedure: 749 1. If the DUT/SUT is being benchmarked as a user agent client or 750 a user agent server, configure the DUT in the test topology 751 shown in Figure 1 or Figure 2 in [I-D.sip-bench-term]. 752 Alternatively, if the DUT is being benchmarked as a proxy or a 753 B2BUA, configure the DUT in the test topology shown in Figure 754 5 in [I-D.sip-bench-term]. 755 2. Configure a SUT according to the test topology shown in Figure 756 7 in [I-D.sip-bench-term]. 757 3. Set the media treams per session to be 0. 758 4. Set the Session Duration to be a value greater than T. 759 5. Execute benchmarking algorithm as defined in Section 4.9 to 760 get the baseline session establishment rate. This rate MUST 761 be recorded using any pertinent parameters as shown in the 762 reporting format of Section 5.1. 763 6. The Session Capacity is the product of T and the Session 764 Establishment Rate. 766 Expected Results: The maximum rate at which the DUT/SUT can handle 767 session establishment requests with no media for an infinitely 768 long period with no errors. This is the SIP "throughput" of the 769 system with no media. 771 6.15. Session Capacity with Media 773 Objective: 774 To benchmark the session capacity of the DUT/SUT with Associated 775 Media. 776 Procedure: 777 1. Configure the DUT in the test topology shown in Figure 3 or 778 Figure 4 of [I-D.sip-bench-term] depending on whether the DUT 779 is being benchmarked as a user agent client or user agent 780 server. Alternatively, configure the DUT in the test topology 781 shown in Figure 6 or Figure 7 in [I-D.sip-bench-term] 782 depending on whether the DUT is being benchmarked as a B2BUA 783 or as a proxy. If a SUT is being benchmarked, configure the 784 SUT as shown in Figure 9 of [I-D.sip-bench-term]. 785 2. Set the media streams per session to 1. 786 3. Set the Session Duration to be a value greater than T. 787 4. Execute benchmarking algorithm as defined in Section 4.9 to 788 get the baseline session establishment rate. This rate MUST 789 be recorded using any pertinent parameters as shown in the 790 reporting format of Section 5.1. 791 5. The Session Capacity is the product of T and the Session 792 Establishment Rate. 793 Expected Results: Session Capacity results obtained with Associated 794 Media with any number of media streams per SIP session will be 795 identical to the Session Capacity results obtained without media. 797 6.16. Session Capacity with Media and a Media Relay/NAT and/or Firewall 799 Objective: 800 To benchmark the Session Establishment Rate of the SUT with 801 Associated Media. 802 Procedure: 803 1. Configure the SUT as shown in Figure 7 or Figure 10 in 804 [I-D.sip-bench-term]. 805 2. Set media streams per session to 1. 806 3. Execute benchmarking algorithm as defined in Section 4.9 to 807 get the session establishment rate with media. This rate MUST 808 be recorded using any pertinent parameters as shown in the 809 reporting format of Section 5.1. 811 Expected Results: Session Capacity results obtained with Associated 812 Media with any number of media streams per SIP session may be 813 lower than the Session Capacity without Media result if the Media 814 Relay, NAT or Firewall is sharing a platform with the server. 816 7. IANA Considerations 818 This document does not requires any IANA considerations. 820 8. Security Considerations 822 Documents of this type do not directly affect the security of 823 Internet or corporate networks as long as benchmarking is not 824 performed on devices or systems connected to production networks. 825 Security threats and how to counter these in SIP and the media layer 826 is discussed in RFC3261, RFC3550, and RFC3711 and various other 827 drafts. This document attempts to formalize a set of common 828 methodology for benchmarking performance of SIP devices in a lab 829 environment. 831 9. Acknowledgments 833 The authors would like to thank Keith Drage and Daryl Malas for their 834 contributions to this document. Dale Worley provided an extensive 835 review that lead to improvements in the documents. 837 10. References 839 10.1. Normative References 841 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 842 Requirement Levels", BCP 14, RFC 2119, March 1997. 844 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 845 Network Interconnect Devices", RFC 2544, March 1999. 847 [I-D.sip-bench-term] 848 Davids, C., Gurbani, V., and S. Poretsky, "SIP Performance 849 Benchmarking Terminology", 850 draft-ietf-bmwg-sip-bench-term-04 (work in progress), 851 March 2012. 853 10.2. Informative References 855 [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, 856 A., Peterson, J., Sparks, R., Handley, M., and E. 857 Schooler, "SIP: Session Initiation Protocol", RFC 3261, 858 June 2002. 860 Authors' Addresses 862 Carol Davids 863 Illinois Institute of Technology 864 201 East Loop Road 865 Wheaton, IL 60187 866 USA 868 Phone: +1 630 682 6024 869 Email: davids@iit.edu 871 Vijay K. Gurbani 872 Bell Laboratories, Alcatel-Lucent 873 1960 Lucent Lane 874 Rm 9C-533 875 Naperville, IL 60566 876 USA 878 Phone: +1 630 224 0216 879 Email: vkg@bell-labs.com 881 Scott Poretsky 882 Allot Communications 883 300 TradeCenter, Suite 4680 884 Woburn, MA 08101 885 USA 887 Phone: +1 508 309 2179 888 Email: sporetsky@allot.com