idnits 2.17.1 draft-ietf-bmwg-sip-bench-meth-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (January 8, 2013) is 4127 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-12) exists of draft-ietf-bmwg-sip-bench-term-08 Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Methodology Working C. Davids 3 Group Illinois Institute of Technology 4 Internet-Draft V. Gurbani 5 Intended status: Informational Bell Laboratories, Alcatel-Lucent 6 Expires: July 12, 2013 S. Poretsky 7 Allot Communications 8 January 8, 2013 10 Methodology for Benchmarking SIP Networking Devices 11 draft-ietf-bmwg-sip-bench-meth-08 13 Abstract 15 This document describes the methodology for benchmarking Session 16 Initiation Protocol (SIP) performance as described in SIP 17 benchmarking terminology document. The methodology and terminology 18 are to be used for benchmarking signaling plane performance with 19 varying signaling and media load. Both scale and establishment rate 20 are measured by signaling plane performance. The SIP Devices to be 21 benchmarked may be a single device under test (DUT) or a system under 22 test (SUT). Benchmarks can be obtained and compared for different 23 types of devices such as SIP Proxy Server, SBC, and server paired 24 with a media relay or Firewall/NAT device. 26 Status of this Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at http://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on July 12, 2013. 43 Copyright Notice 45 Copyright (c) 2013 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (http://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 61 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 62 3. Benchmarking Topologies . . . . . . . . . . . . . . . . . . . 5 63 4. Test Setup Parameters . . . . . . . . . . . . . . . . . . . . 5 64 4.1. Selection of SIP Transport Protocol . . . . . . . . . . . 5 65 4.2. Signaling Server . . . . . . . . . . . . . . . . . . . . . 5 66 4.3. Associated Media . . . . . . . . . . . . . . . . . . . . . 5 67 4.4. Selection of Associated Media Protocol . . . . . . . . . . 6 68 4.5. Number of Associated Media Streams per SIP Session . . . . 6 69 4.6. Session Duration . . . . . . . . . . . . . . . . . . . . . 6 70 4.7. Attempted Sessions per Second . . . . . . . . . . . . . . 6 71 4.8. Stress Testing . . . . . . . . . . . . . . . . . . . . . . 6 72 4.9. Benchmarking algorithm . . . . . . . . . . . . . . . . . . 6 73 5. Reporting Format . . . . . . . . . . . . . . . . . . . . . . . 9 74 5.1. Test Setup Report . . . . . . . . . . . . . . . . . . . . 9 75 5.2. Device Benchmarks for IS . . . . . . . . . . . . . . . . . 10 76 5.3. Device Benchmarks for NS . . . . . . . . . . . . . . . . . 10 77 6. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 10 78 6.1. Baseline Session Establishment Rate of the test bed . . . 10 79 6.2. Session Establishment Rate without media . . . . . . . . . 11 80 6.3. Session Establishment Rate with Media not on DUT/SUT . . . 11 81 6.4. Session Establishment Rate with Media on DUT/SUT . . . . . 12 82 6.5. Session Establishment Rate with Loop Detection Enabled . . 13 83 6.6. Session Establishment Rate with Forking . . . . . . . . . 13 84 6.7. Session Establishment Rate with Forking and Loop 85 Detection . . . . . . . . . . . . . . . . . . . . . . . . 14 86 6.8. Session Establishment Rate with TLS Encrypted SIP . . . . 14 87 6.9. Session Establishment Rate with IPsec Encrypted SIP . . . 15 88 6.10. Session Establishment Rate with SIP Flooding . . . . . . . 16 89 6.11. Maximum Registration Rate . . . . . . . . . . . . . . . . 16 90 6.12. Maximum Re-Registration Rate . . . . . . . . . . . . . . . 16 91 6.13. Maximum IM Rate . . . . . . . . . . . . . . . . . . . . . 17 92 6.14. Session Capacity without Media . . . . . . . . . . . . . . 17 93 6.15. Session Capacity with Media . . . . . . . . . . . . . . . 18 94 6.16. Session Capacity with Media and a Media Relay/NAT 95 and/or Firewall . . . . . . . . . . . . . . . . . . . . . 19 96 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 97 8. Security Considerations . . . . . . . . . . . . . . . . . . . 19 98 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 19 99 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 20 100 10.1. Normative References . . . . . . . . . . . . . . . . . . . 20 101 10.2. Informative References . . . . . . . . . . . . . . . . . . 20 102 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20 104 1. Terminology 106 In this document, the key words "MUST", "MUST NOT", "REQUIRED", 107 "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT 108 RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as 109 described in BCP 14, conforming to [RFC2119] and indicate requirement 110 levels for compliant implementations. 112 Terms specific to SIP [RFC3261] performance benchmarking are defined 113 in [I-D.sip-bench-term]. 115 RFC 2119 defines the use of these key words to help make the intent 116 of standards track documents as clear as possible. While this 117 document uses these keywords, this document is not a standards track 118 document. The term Throughput is defined in [RFC2544]. 120 2. Introduction 122 This document describes the methodology for benchmarking Session 123 Initiation Protocol (SIP) performance as described in Terminology 124 document [I-D.sip-bench-term]. The methodology and terminology are 125 to be used for benchmarking signaling plane performance with varying 126 signaling and media load. Both scale and establishment rate are 127 measured by signaling plane performance. 129 The SIP Devices to be benchmarked may be a single device under test 130 (DUT) or a system under test (SUT). The DUT is a SIP Server, which 131 may be any [RFC3261] conforming device. The SUT can be any device or 132 group of devices containing RFC 3261 conforming functionality along 133 with Firewall and/or NAT functionality. This enables benchmarks to 134 be obtained and compared for different types of devices such as SIP 135 Proxy Server, SBC, SIP proxy server paired with a media relay or 136 Firewall/NAT device. SIP Associated Media benchmarks can also be 137 made when testing SUTs. 139 The test cases provide benchmarks metrics of Registration Rate, SIP 140 Session Establishment Rate, Session Capacity, and IM Rate. These can 141 be benchmarked with or without associated Media. Some cases are also 142 included to cover Forking, Loop detection, Encrypted SIP, and SIP 143 Flooding. The test topologies that can be used are described in the 144 Test Setup section. Topologies are provided for benchmarking of a 145 DUT or SUT. Benchmarking with Associated Media can be performed when 146 using a SUT. 148 SIP permits a wide range of configuration options that are explained 149 in Section 4 and Section 2 of [I-D.sip-bench-term]. Benchmark 150 metrics could possibly be impacted by Associated Media. The selected 151 values for Session Duration and Media Streams per Session enable 152 benchmark metrics to be benchmarked without Associated Media. 153 Session Setup Rate could possibly be impacted by the selected value 154 for Maximum Sessions Attempted. The benchmark for Session 155 Establishment Rate is measured with a fixed value for maximum Session 156 Attempts. 158 Finally, the overall value of these tests is to serve as a comparison 159 function between multiple SIP implementations. One way to use these 160 tests is to derive benchmarks with SIP devices from Vendor-A, derive 161 a new set of benchmarks with similar SIP devices from Vendor-B and 162 perform a comparison on the results of Vendor-A and Vendor-B. This 163 document does not make any claims on the interpretation of such 164 results. 166 3. Benchmarking Topologies 168 Familiarity with the benchmarking models in Section 2.2 of 169 [I-D.sip-bench-term] is assumed. Figures 1 through 10 in 170 [I-D.sip-bench-term] contain the canonical topologies that can be 171 used to perform the benchmarking tests listed in this document. 173 4. Test Setup Parameters 175 4.1. Selection of SIP Transport Protocol 177 Test cases may be performed with any transport protocol supported by 178 SIP. This includes, but is not limited to, SIP TCP, SIP UDP, and 179 TLS. The protocol used for the SIP transport protocol must be 180 reported with benchmarking results. 182 4.2. Signaling Server 184 The Signaling Server is defined in the companion terminology 185 document, ([I-D.sip-bench-term], Section 3.2.2) It is a SIP-speaking 186 device that complies with RFC 3261. Conformance to [RFC3261] is 187 assumed for all tests. The Signaling Server may be the DUT or a 188 component of a SUT. The Signaling Server may include Firewall and/or 189 NAT functionality. The components of the SUT may be a single 190 physical device or separate devices. 192 4.3. Associated Media 194 Some tests require Associated Media to be present for each SIP 195 session. The test topologies to be used when benchmarking SUT 196 performance for Associated Media are shown in [I-D.sip-bench-term], 197 Figures 4 and 5. 199 4.4. Selection of Associated Media Protocol 201 The test cases specified in this document provide SIP performance 202 independent of the protocol used for the media stream. Any media 203 protocol supported by SIP may be used. This includes, but is not 204 limited to, RTP, RTSP, and SRTP. The protocol used for Associated 205 Media MUST be reported with benchmarking results. 207 4.5. Number of Associated Media Streams per SIP Session 209 Benchmarking results may vary with the number of media streams per 210 SIP session. When benchmarking a SUT for voice, a single media 211 stream is used. When benchmarking a SUT for voice and video, two 212 media streams are used. The number of Associated Media Streams MUST 213 be reported with benchmarking results. 215 4.6. Session Duration 217 SUT performance benchmarks may vary with the duration of SIP 218 sessions. Session Duration MUST be reported with benchmarking 219 results. A Session Duration of zero seconds indicates transmission 220 of a BYE immediately following successful SIP establishment indicate 221 by receipt of a 200 OK. An infinite Session Duration indicates that 222 a BYE is never transmitted. 224 4.7. Attempted Sessions per Second 226 DUT and SUT performance benchmarks may vary with the the rate of 227 attempted sessions offered by the Tester. Attempted Sessions per 228 Second MUST be reported with benchmarking results. 230 4.8. Stress Testing 232 The purpose of this document is to benchmark SIP performance; this 233 document does not benchmark stability of SIP systems under stressful 234 conditions such as a high rate of Attempted Sessions per Second. 236 4.9. Benchmarking algorithm 238 In order to benchmark the test cases uniformly in Section 6, the 239 algorithm described in this section should be used. Both, a prosaic 240 description of the algorithm and a pseudo-code description are 241 provided. 243 The goal is to find the largest value of a SIP session-request-rate, 244 measured in sessions-per-second, which the DUT/SUT can process with 245 zero errors. To discover that number, an iterative process (defined 246 below) is used to find a candidate for this rate. Once the candidate 247 rate has been found, the DUT/SUT is subjected to an offered load 248 whose arrival rate is set to that of the candidate rate. This test 249 is run for an extended period of time, which is referred to as 250 infinity, and which is, itself, a parameter of the test labeled T in 251 the pseudo-code. This latter phase of testing is called the steady- 252 state phase. If errors are encountered during this steady-state 253 phase, then the candidate rate is reduced by a defined percent, also 254 a parameter of test, and the steady-state phase is entered again 255 until a final (new) steady-state rate is achieved. 257 The iterative process itself is defined as follows: a starting rate 258 of 100 sessions per second (sps) is selected. The test is executed 259 for the time period identified by t in the pseudo-code below. If no 260 failures occur, the rate is increased to 150 sps and again tested for 261 time period t. The attempt rate is continuously ramped up until a 262 failure is encountered before the end of the test time t. Then an 263 attempt rate is calculated that is higher than the last successful 264 attempt rate by a quantity equal to half the difference between the 265 rate at which failures occurred and the last successful rate. If 266 this new attempt rate also results in errors, a new attempt rate is 267 tried that is higher than the last successful attempt rate by a 268 quantity equal to half the difference between the rate at which 269 failures occurred and the last successful rate. Continuing in this 270 way, an attempt rate without errors is found. The operator can 271 specify margin of error using the parameter G, measured in units of 272 sessions per second. 274 The pseudo-code corresponding to the description above follows. 276 ; ---- Parameters of test, adjust as needed 277 t := 5000 ; local maximum; used to figure out largest 278 ; value 279 T := 50000 ; global maximum; once largest value has been 280 ; figured out, pump this many requests before calling 281 ; the test a success 282 m := {...} ; other attributes that affect testing, such 283 ; as media streams, etc. 284 s := 100 ; Initial session attempt rate (in sessions/sec) 285 G := 5 ; granularity of results - the margin of error in sps 286 C := 0.05 ; calibration amount: How much to back down if we 287 ; have found candidate s but cannot send at rate s for 288 ; time T without failures 290 ; ---- End of parameters of test 291 ; ---- Initialization of flags, candidate values and upper bounds 292 f := false ; indicates that you had a success after the upper limit 293 F := false ; indicates that test is done 294 c := 0 ; indicates that we have found an upper limit 296 proc main 297 find_largest_value ; First, figure out the largest value. 299 ; Now that the largest value (saved in s) has been figured out, 300 ; use it for sending out s requests/s and send out T requests. 302 do { 303 send_traffic(s, m, T) ; send_traffic not shown 304 if (all requests succeeded) { 305 F := true ; test is done 306 } else if (one or more requests fail) { 307 s := s - (C * s) ; Reduce s by calibration amount 308 steady_state 309 } 310 } while (F == false) 311 end proc 313 proc find_largest_value 314 ; Iterative process to figure out the largest value we can 315 ; handle with no failures 316 do { 317 send_traffic(s, m, t) ; Send s request/sec with m 318 ; characteristics until t requests have 319 ; been sent 320 if (all requests succeeded) { 321 s' := s ; save candidate value of metric 323 if ( c == 0 ) { 324 s := s + (0.5 * s) 326 }else if ((c == 1) && (s''-s')) > 2*G ) { 327 s := s + ( 0.5 * (s'' - s ); 329 }else if ((c == 1) && ((s''-s') <= 2*G ) { 330 f := true; 332 } 333 else if (one or more requests fail) { 334 c := 1 ; we have found an upper bound for the metric 335 s'' := s ; save new upper bound 336 s := s - (0.5 * (s - s')) 337 } 338 } while (f == false) 339 end proc 341 5. Reporting Format 343 5.1. Test Setup Report 345 SIP Transport Protocol = ___________________________ 346 (valid values: TCP|UDP|TLS|SCTP|specify-other) 347 Session Attempt Rate = _____________________________ 348 (session attempts/sec) 349 IS Media Attempt Rate = ____________________________ 350 (IS media attempts/sec) 351 Total Sessions Attempted = _________________________ 352 (total sessions to be created over duration of test) 353 Media Streams Per Session = _______________________ 354 (number of streams per session) 355 Associated Media Protocol = _______________________ 356 (RTP|RTSP|specify-other) 357 Media Packet Size = _______________________________ 358 (bytes) 359 Media Offered Load = ______________________________ 360 (packets per second) 361 Media Session Hold Time = _________________________ 362 (seconds) 363 Establishment Threshold time = ____________________ 364 (seconds) 365 Loop Detecting Option = ___________________________ 366 (on|off) 367 Forking Option 368 Number of endpoints request sent to = ___________ 369 (1, means forking is not enabled) 370 Type of forking = _______________________________ 371 (serial|parallel) 372 Authentication option = ___________________________________ 373 (on|off; if on, please see Notes 2 and 3 below). 375 Note 1: Total Sessions Attempted is used in the calculation of the 376 Session Establishment Performance ([I-D.sip-bench-term], Section 377 3.4.5). It is the number of session attempts ([I-D.sip-bench-term], 378 Section 3.1.6) that will be made over the duration of the test. 380 Note 2: When the Authentication Option is "on" the test tool must be 381 set to ignore 401 and 407 failure responses in any test described as 382 a "test to failure." If this is not done, all such tests will yield 383 trivial benchmarks, as all attempt rates will lead to a failure after 384 the first attempt. 386 Note 3: When the Authentication Option is "on" the DUT/SUT uses two 387 transactions instead of one when it is establishing a session or 388 accomplishing a registration. The first transaction ends with the 389 401 or 407. The second ends with the 200 OK or another failure 390 message. The Test Organization interested in knowing how many times 391 the EA was intended to send a REGISTER as distinct from how many 392 times the EA wound up actually sending a REGISTER may wish to record 393 the following data as well: 395 Number of responses of the following type: 396 401: _____________ (if authentication turned on; N/A 397 otherwise) 398 407: _____________ (if authentication turned on; N/A 399 otherwise) 401 5.2. Device Benchmarks for IS 403 Registration Rate = _______________________________ 404 (registrations per second) 405 Re-registration Rate = ____________________________ 406 (registrations per second) 407 Session Capacity = _________________________________ 408 (sessions) 409 Session Overload Capacity = ________________________ 410 (sessions) 411 Session Establishment Rate = ______________________ 412 (sessions per second) 413 Session Establishment Performance = _______________ 414 (total established sessions/total sessions attempted)(no units) 415 Session Attempt Delay = ___________________________ 416 (seconds) 418 5.3. Device Benchmarks for NS 420 IM Rate = _______________________________ (IM messages per second) 422 6. Test Cases 424 6.1. Baseline Session Establishment Rate of the test bed 425 Objective: 426 To benchmark the Session Establishment Rate of the Emulated Agent 427 (EA) with zero failures. 429 Procedure: 430 1. Configure the DUT in the test topology shown in Figure 1 in 431 [I-D.sip-bench-term]. 432 2. Set media streams per session to 0. 433 3. Execute benchmarking algorithm as defined in Section 4.9 to 434 get the baseline session establishment rate. This rate MUST 435 be recorded using any pertinent parameters as shown in the 436 reporting format of Section 5.1. 438 Expected Results: This is the scenario to obtain the maximum Session 439 Establishment Rate of the EA and the test bed when no DUT/SUT is 440 present. The results of this test might be used to normalize test 441 results performed on different test beds or simply to better 442 understand the impact of the DUT/SUT on the test bed in question. 444 6.2. Session Establishment Rate without media 446 Objective: 447 To benchmark the Session Establishment Rate of the DUT/SUT with no 448 associated media and zero failures. 450 Procedure: 451 1. If the DUT/SUT is being benchmarked as a user agent client or 452 a user agent server, configure the DUT in the test topology 453 shown in Figure 1 or Figure 2 in [I-D.sip-bench-term]. 454 Alternatively, if the DUT is being benchmarked as a proxy or a 455 B2BUA, configure the DUT in the test topology shown in Figure 456 5 in [I-D.sip-bench-term]. 457 2. Configure a SUT according to the test topology shown in Figure 458 7 in [I-D.sip-bench-term]. 459 3. Set media streams per session to 0. 460 4. Execute benchmarking algorithm as defined in Section 4.9 to 461 get the session establishment rate. This rate MUST be 462 recorded using any pertinent parameters as shown in the 463 reporting format of Section 5.1. 465 Expected Results: This is the scenario to obtain the maximum Session 466 Establishment Rate of the DUT/SUT. 468 6.3. Session Establishment Rate with Media not on DUT/SUT 469 Objective: 470 To benchmark the Session Establishment Rate of the DUT/SUT with 471 zero failures when Associated Media is included in the benchmark 472 test but the media is not running through the DUT/SUT. 474 Procedure: 475 1. If the DUT is being benchmarked as proxy or B2BUA, configure 476 the DUT in the test topology shown in Figure 7 in 477 [I-D.sip-bench-term]. 478 2. Configure a SUT according to the test topology shown in Figure 479 8 in [I-D.sip-bench-term]. 480 3. Set media streams per session to 1. 481 4. Execute benchmarking algorithm as defined in Section 4.9 to 482 get the session establishment rate with media. This rate MUST 483 be recorded using any pertinent parameters as shown in the 484 reporting format of Section 5.1. 486 Expected Results: Session Establishment Rate results obtained with 487 Associated Media with any number of media streams per SIP session 488 are expected to be identical to the Session Establishment Rate 489 results obtained without media in the case where the server is 490 running on a platform separate from the platform on which the 491 Media Relay, NAT or Firewall is running. 493 6.4. Session Establishment Rate with Media on DUT/SUT 495 Objective: 496 To benchmark the Session Establishment Rate of the DUT/SUT with 497 zero failures when Associated Media is included in the benchmark 498 test and the media is running through the DUT/SUT. 500 Procedure: 501 1. If the DUT is being benchmarked as a user agent client or a 502 user agent server, configure the DUT in the test topology 503 shown in Figure 3 or Figure 4 of [I-D.sip-bench-term]. 504 Alternatively, if the DUT is being benchmarked as a B2BUA, 505 configure the DUT in the test topology shown in Figure 6 in 506 [I-D.sip-bench-term]. 507 2. Configure a SUT according to the test topology shown in Figure 508 9 in [I-D.sip-bench-term]. 509 3. Set media streams per session to 1. 510 4. Execute benchmarking algorithm as defined in Section 4.9 to 511 get the session establishment rate with media. This rate MUST 512 be recorded using any pertinent parameters as shown in the 513 reporting format of Section 5.1. 515 Expected Results: Session Establishment Rate results obtained with 516 Associated Media may be lower than those obtained without media in 517 the case where the server and the NAT, Firewall or Media Relay are 518 running on the same platform. 520 6.5. Session Establishment Rate with Loop Detection Enabled 522 Objective: 523 To benchmark the Session Establishment Rate of the DUT/SUT with 524 zero failures when the Loop Detection option is enabled and no 525 media streams are present. 527 Procedure: 528 1. If the DUT is being benchmarked as a proxy or B2BUA, and loop 529 detection is supported in the DUT, then configure the DUT in 530 the test topology shown in Figure 5 in [I-D.sip-bench-term]. 531 If the DUT does not support loop detection, then this step can 532 be skipped. 533 2. Configure a SUT according to the test topology shown in Figure 534 8 of [I-D.sip-bench-term]. 535 3. Set media streams per session to 0. 536 4. Turn on the Loop Detection option in the DUT or SUT. 537 5. Execute benchmarking algorithm as defined in Section 4.9 to 538 get the session establishment rate with loop detection 539 enabled. This rate MUST be recorded using any pertinent 540 parameters as shown in the reporting format of Section 5.1. 542 Expected Results: Session Establishment Rate results obtained with 543 Loop Detection may be lower than those obtained without Loop 544 Detection enabled. 546 6.6. Session Establishment Rate with Forking 548 Objective: 549 To benchmark the Session Establishment Rate of the DUT/SUT with 550 zero failures when the Forking Option is enabled. 552 Procedure: 553 1. If the DUT is being benchmarked as a proxy or B2BUA, and 554 forking is supported in the DUT, then configure the DUT in the 555 test topology shown in Figure 5 in [I-D.sip-bench-term]. If 556 the DUT does not support forking, then this step can be 557 skipped. 558 2. Configure a SUT according to the test topology shown in Figure 559 8 of [I-D.sip-bench-term]. 561 3. Set media streams per session to 0. 562 4. Set the number of endpoints that will receive the forked 563 invitation to a value of 2 or more (subsequent tests may 564 increase this value at the discretion of the tester.) 565 5. Execute benchmarking algorithm as defined in Section 4.9 to 566 get the session establishment rate with forking. This rate 567 MUST be recorded using any pertinent parameters as shown in 568 the reporting format of Section 5.1. 570 Expected Results: Session Establishment Rate results obtained with 571 Forking may be lower than those obtained without Forking enabled. 573 6.7. Session Establishment Rate with Forking and Loop Detection 575 Objective: 576 To benchmark the Session Establishment Rate of the DUT/SUT with 577 zero failures when both the Forking and Loop Detection Options are 578 enabled. 580 Procedure: 581 1. If the DUT is being benchmarked as a proxy or B2BUA, then 582 configure the DUT in the test topology shown in Figure 5 in 583 [I-D.sip-bench-term]. 584 2. Configure a SUT according to the test topology shown in Figure 585 8 of [I-D.sip-bench-term]. 586 3. Set media streams per session to 0. 587 4. Enable the Loop Detection Options on the DUT. 588 5. Set the number of endpoints that will receive the forked 589 invitation to a value of 2 or more (subsequent tests may 590 increase this value at the discretion of the tester.) 591 6. Execute benchmarking algorithm as defined in Section 4.9 to 592 get the session establishment rate with forking and loop 593 detection. This rate MUST be recorded using any pertinent 594 parameters as shown in the reporting format of Section 5.1. 596 Expected Results: Session Establishment Rate results obtained with 597 Forking and Loop Detection may be lower than those obtained with 598 only Forking or Loop Detection enabled. 600 6.8. Session Establishment Rate with TLS Encrypted SIP 602 Objective: 603 To benchmark the Session Establishment Rate of the DUT/SUT with 604 zero failures when using TLS encrypted SIP signaling. 606 Procedure: 607 1. If the DUT is being benchmarked as a proxy or B2BUA, then 608 configure the DUT in the test topology shown in Figure 5 in 609 [I-D.sip-bench-term]. 610 2. Configure a SUT according to the test topology shown in Figure 611 8 of [I-D.sip-bench-term]. 612 3. Set media streams per session to 0 (media is not used in this 613 test). 614 4. Configure Tester to enable TLS over the transport being 615 benchmarked. Make a note the transport when compiling 616 results. May need to run for each transport of interest. 617 5. Execute benchmarking algorithm as defined in Section 4.9 to 618 get the session establishment rate with encryption. This rate 619 MUST be recorded using any pertinent parameters as shown in 620 the reporting format of Section 5.1. 622 Expected Results: Session Establishment Rate results obtained with 623 TLS Encrypted SIP may be lower than those obtained with plaintext 624 SIP. 626 6.9. Session Establishment Rate with IPsec Encrypted SIP 628 Objective: 629 To benchmark the Session Establishment Rate of the DUT/SUT with 630 zero failures when using IPsec Encrypted SIP signaling. 632 Procedure: 633 1. If the DUT is being benchmarked as a proxy or B2BUA, then 634 configure the DUT in the test topology shown in Figure 5 in 635 [I-D.sip-bench-term]. 636 2. Configure a SUT according to the test topology shown in Figure 637 8 of [I-D.sip-bench-term]. 638 3. Set media streams per session to 0 (media is not used in this 639 test). 640 4. Configure Tester for IPSec. 641 5. Execute benchmarking algorithm as defined in Section 4.9 to 642 get the session establishment rate with encryption. This rate 643 MUST be recorded using any pertinent parameters as shown in 644 the reporting format of Section 5.1. 646 Expected Results: Session Establishment Rate results obtained with 647 IPSec Encrypted SIP may be lower than those obtained with 648 plaintext SIP. 650 6.10. Session Establishment Rate with SIP Flooding 652 Objective: 653 To benchmark the Session Establishment Rate of the SUT with zero 654 failures when SIP Flooding is occurring. 656 Procedure: 657 1. If the DUT is being benchmarked as a proxy or B2BUA, then 658 configure the DUT in the test topology shown in Figure 5 in 659 [I-D.sip-bench-term]. 660 2. Configure a SUT according to the test topology shown in Figure 661 8 of [I-D.sip-bench-term]. 662 3. Set media streams per session to 0. 663 4. Set s to a high value (e.g., 500) (c.f. Section 4.9). 664 5. Execute benchmarking algorithm as defined in Section 4.9 to 665 get the session establishment rate with flooding. This rate 666 MUST be recorded using any pertinent parameters as shown in 667 the reporting format of Section 5.1. 669 Expected Results: Session Establishment Rate results obtained with 670 SIP Flooding may be degraded. 672 6.11. Maximum Registration Rate 674 Objective: 675 To benchmark the maximum registration rate of the DUT/SUT with 676 zero failures. 678 Procedure: 679 1. If the DUT is being benchmarked as a proxy or B2BUA, then 680 configure the DUT in the test topology shown in Figure 5 in 681 [I-D.sip-bench-term]. 682 2. Configure a SUT according to the test topology shown in Figure 683 8 of [I-D.sip-bench-term]. 684 3. Set media streams per session to 0. 685 4. Set the registration timeout value to at least 3600 seconds. 686 5. Execute benchmarking algorithm as defined in Section 4.9 to 687 get the maximum registration rate. This rate MUST be recorded 688 using any pertinent parameters as shown in the reporting 689 format of Section 5.1. 690 Expected Results: 692 6.12. Maximum Re-Registration Rate 693 Objective: 694 To benchmark the maximum re-registration rate of the DUT/SUT with 695 zero failures. 697 Procedure: 698 1. If the DUT is being benchmarked as a proxy or B2BUA, then 699 configure the DUT in the test topology shown in Figure 5 in 700 [I-D.sip-bench-term]. 701 2. Configure a SUT according to the test topology shown in Figure 702 8 of [I-D.sip-bench-term]. 703 3. First, execute test detailed in Section 6.11 to register the 704 endpoints with the registrar. 705 4. After at least 5 minutes of Step 2, but no more than 10 706 minutes after Step 2 has been performed, execute test detailed 707 in Section 6.11 again (this will count as a re-registration). 708 5. Execute benchmarking algorithm as defined in Section 4.9 to 709 get the maximum re-registration rate. This rate MUST be 710 recorded using any pertinent parameters as shown in the 711 reporting format of Section 5.1. 712 Expected Results: The rate should be at least equal to but not more 713 than the result of Section 6.11. 715 6.13. Maximum IM Rate 717 Objective: 718 To benchmark the maximum IM rate of the SUT with zero failures. 720 Procedure: 721 1. If the DUT/SUT is being benchmarked as a user agent client or 722 a user agent server, configure the DUT in the test topology 723 shown in Figure 1 or Figure 2 in [I-D.sip-bench-term]. 724 Alternatively, if the DUT is being benchmarked as a proxy or a 725 B2BUA, configure the DUT in the test topology shown in Figure 726 5 in [I-D.sip-bench-term]. 727 2. Configure a SUT according to the test topology shown in Figure 728 5 in [I-D.sip-bench-term]. 729 3. Execute benchmarking algorithm as defined in Section 4.9 to 730 get the maximum IM rate. This rate MUST be recorded using any 731 pertinent parameters as shown in the reporting format of 732 Section 5.1. 734 Expected Results: 736 6.14. Session Capacity without Media 737 Objective: 738 To benchmark the Session Capacity of the SUT without Associated 739 Media. 740 Procedure: 741 1. If the DUT/SUT is being benchmarked as a user agent client or 742 a user agent server, configure the DUT in the test topology 743 shown in Figure 1 or Figure 2 in [I-D.sip-bench-term]. 744 Alternatively, if the DUT is being benchmarked as a proxy or a 745 B2BUA, configure the DUT in the test topology shown in Figure 746 5 in [I-D.sip-bench-term]. 747 2. Configure a SUT according to the test topology shown in Figure 748 7 in [I-D.sip-bench-term]. 749 3. Set the media treams per session to be 0. 750 4. Set the Session Duration to be a value greater than T. 751 5. Execute benchmarking algorithm as defined in Section 4.9 to 752 get the baseline session establishment rate. This rate MUST 753 be recorded using any pertinent parameters as shown in the 754 reporting format of Section 5.1. 755 6. The Session Capacity is the product of T and the Session 756 Establishment Rate. 757 Expected Results: The maximum rate at which the DUT/SUT can handle 758 session establishment requests with no media for an infinitely 759 long period with no errors. This is the SIP "throughput" of the 760 system with no media. 762 6.15. Session Capacity with Media 764 Objective: 765 To benchmark the session capacity of the DUT/SUT with Associated 766 Media. 767 Procedure: 768 1. Configure the DUT in the test topology shown in Figure 3 or 769 Figure 4 of [I-D.sip-bench-term] depending on whether the DUT 770 is being benchmarked as a user agent client or user agent 771 server. Alternatively, configure the DUT in the test topology 772 shown in Figure 6 or Figure 7 in [I-D.sip-bench-term] 773 depending on whether the DUT is being benchmarked as a B2BUA 774 or as a proxy. If a SUT is being benchmarked, configure the 775 SUT as shown in Figure 9 of [I-D.sip-bench-term]. 776 2. Set the media streams per session to 1. 777 3. Set the Session Duration to be a value greater than T. 778 4. Execute benchmarking algorithm as defined in Section 4.9 to 779 get the baseline session establishment rate. This rate MUST 780 be recorded using any pertinent parameters as shown in the 781 reporting format of Section 5.1. 782 5. The Session Capacity is the product of T and the Session 783 Establishment Rate. 785 Expected Results: Session Capacity results obtained with Associated 786 Media with any number of media streams per SIP session will be 787 less than the Session Capacity results obtained without media. 789 6.16. Session Capacity with Media and a Media Relay/NAT and/or Firewall 791 Objective: 792 To benchmark the Session Establishment Rate of the SUT with 793 Associated Media. 794 Procedure: 795 1. Configure the SUT as shown in Figure 7 or Figure 10 in 796 [I-D.sip-bench-term]. 797 2. Set media streams per session to 1. 798 3. Execute benchmarking algorithm as defined in Section 4.9 to 799 get the session establishment rate with media. This rate MUST 800 be recorded using any pertinent parameters as shown in the 801 reporting format of Section 5.1. 803 Expected Results: Session Capacity results obtained with Associated 804 Media with any number of media streams per SIP session may be 805 lower than the Session Capacity without Media result if the Media 806 Relay, NAT or Firewall is sharing a platform with the server. 808 7. IANA Considerations 810 This document does not requires any IANA considerations. 812 8. Security Considerations 814 Documents of this type do not directly affect the security of 815 Internet or corporate networks as long as benchmarking is not 816 performed on devices or systems connected to production networks. 817 Security threats and how to counter these in SIP and the media layer 818 is discussed in RFC3261, RFC3550, and RFC3711 and various other 819 drafts. This document attempts to formalize a set of common 820 methodology for benchmarking performance of SIP devices in a lab 821 environment. 823 9. Acknowledgments 825 The authors would like to thank Keith Drage and Daryl Malas for their 826 contributions to this document. Dale Worley provided an extensive 827 review that lead to improvements in the documents. We are grateful 828 to Barry Constantine for providing valuable comments during the 829 document's WGLC. 831 10. References 833 10.1. Normative References 835 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 836 Requirement Levels", BCP 14, RFC 2119, March 1997. 838 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 839 Network Interconnect Devices", RFC 2544, March 1999. 841 [I-D.sip-bench-term] 842 Davids, C., Gurbani, V., and S. Poretsky, "SIP Performance 843 Benchmarking Terminology", 844 draft-ietf-bmwg-sip-bench-term-08 (work in progress), 845 January 2013. 847 10.2. Informative References 849 [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, 850 A., Peterson, J., Sparks, R., Handley, M., and E. 851 Schooler, "SIP: Session Initiation Protocol", RFC 3261, 852 June 2002. 854 Authors' Addresses 856 Carol Davids 857 Illinois Institute of Technology 858 201 East Loop Road 859 Wheaton, IL 60187 860 USA 862 Phone: +1 630 682 6024 863 Email: davids@iit.edu 865 Vijay K. Gurbani 866 Bell Laboratories, Alcatel-Lucent 867 1960 Lucent Lane 868 Rm 9C-533 869 Naperville, IL 60566 870 USA 872 Phone: +1 630 224 0216 873 Email: vkg@bell-labs.com 874 Scott Poretsky 875 Allot Communications 876 300 TradeCenter, Suite 4680 877 Woburn, MA 08101 878 USA 880 Phone: +1 508 309 2179 881 Email: sporetsky@allot.com