idnits 2.17.1 draft-ietf-bmwg-sip-bench-meth-12.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (November 12, 2014) is 3424 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '1' on line 782 Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Methodology Working Group C. Davids 3 Internet-Draft Illinois Institute of Technology 4 Intended status: Informational V. Gurbani 5 Expires: May 16, 2015 Bell Laboratories, 6 Alcatel-Lucent 7 S. Poretsky 8 Allot Communications 9 November 12, 2014 11 Methodology for Benchmarking Session Initiation Protocol (SIP) Devices: 12 Basic session setup and registration 13 draft-ietf-bmwg-sip-bench-meth-12 15 Abstract 17 This document provides a methodology for benchmarking the Session 18 Initiation Protocol (SIP) performance of devices. Terminology 19 related to benchmarking SIP devices is described in the companion 20 terminology document. Using these two documents, benchmarks can be 21 obtained and compared for different types of devices such as SIP 22 Proxy Servers, Registrars and Session Border Controllers. The term 23 "performance" in this context means the capacity of the device-under- 24 test (DUT) to process SIP messages. Media streams are used only to 25 study how they impact the signaling behavior. The intent of the two 26 documents is to provide a normalized set of tests that will enable an 27 objective comparison of the capacity of SIP devices. Test setup 28 parameters and a methodology are necessary because SIP allows a wide 29 range of configuration and operational conditions that can influence 30 performance benchmark measurements. 32 Status of this Memo 34 This Internet-Draft is submitted in full conformance with the 35 provisions of BCP 78 and BCP 79. 37 Internet-Drafts are working documents of the Internet Engineering 38 Task Force (IETF). Note that other groups may also distribute 39 working documents as Internet-Drafts. The list of current Internet- 40 Drafts is at http://datatracker.ietf.org/drafts/current/. 42 Internet-Drafts are draft documents valid for a maximum of six months 43 and may be updated, replaced, or obsoleted by other documents at any 44 time. It is inappropriate to use Internet-Drafts as reference 45 material or to cite them other than as "work in progress." 47 This Internet-Draft will expire on May 16, 2015. 49 Copyright Notice 51 Copyright (c) 2014 IETF Trust and the persons identified as the 52 document authors. All rights reserved. 54 This document is subject to BCP 78 and the IETF Trust's Legal 55 Provisions Relating to IETF Documents 56 (http://trustee.ietf.org/license-info) in effect on the date of 57 publication of this document. Please review these documents 58 carefully, as they describe your rights and restrictions with respect 59 to this document. Code Components extracted from this document must 60 include Simplified BSD License text as described in Section 4.e of 61 the Trust Legal Provisions and are provided without warranty as 62 described in the Simplified BSD License. 64 Table of Contents 66 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 67 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 68 3. Benchmarking Topologies . . . . . . . . . . . . . . . . . . . 5 69 4. Test Setup Parameters . . . . . . . . . . . . . . . . . . . . 7 70 4.1. Selection of SIP Transport Protocol . . . . . . . . . . . 7 71 4.2. Connection-oriented Transport Management . . . . . . . . . 7 72 4.3. Signaling Server . . . . . . . . . . . . . . . . . . . . . 8 73 4.4. Associated Media . . . . . . . . . . . . . . . . . . . . . 8 74 4.5. Selection of Associated Media Protocol . . . . . . . . . . 8 75 4.6. Number of Associated Media Streams per SIP Session . . . . 8 76 4.7. Codec Type . . . . . . . . . . . . . . . . . . . . . . . . 8 77 4.8. Session Duration . . . . . . . . . . . . . . . . . . . . . 8 78 4.9. Attempted Sessions per Second (sps) . . . . . . . . . . . 9 79 4.10. Benchmarking algorithm . . . . . . . . . . . . . . . . . . 9 80 5. Reporting Format . . . . . . . . . . . . . . . . . . . . . . . 11 81 5.1. Test Setup Report . . . . . . . . . . . . . . . . . . . . 11 82 5.2. Device Benchmarks for session setup . . . . . . . . . . . 12 83 5.3. Device Benchmarks for registrations . . . . . . . . . . . 12 84 6. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 13 85 6.1. Baseline Session Establishment Rate of the test bed . . . 13 86 6.2. Session Establishment Rate without media . . . . . . . . . 13 87 6.3. Session Establishment Rate with Media not on DUT . . . . . 13 88 6.4. Session Establishment Rate with Media on DUT . . . . . . . 14 89 6.5. Session Establishment Rate with TLS Encrypted SIP . . . . 14 90 6.6. Session Establishment Rate with IPsec Encrypted SIP . . . 15 91 6.7. Registration Rate . . . . . . . . . . . . . . . . . . . . 15 92 6.8. Re-Registration Rate . . . . . . . . . . . . . . . . . . . 16 93 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 16 94 8. Security Considerations . . . . . . . . . . . . . . . . . . . 16 95 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 17 96 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 17 97 10.1. Normative References . . . . . . . . . . . . . . . . . . . 17 98 10.2. Informative References . . . . . . . . . . . . . . . . . . 17 99 Appendix A. R Code Component to simulate benchmarking 100 algorithm . . . . . . . . . . . . . . . . . . . . . . 18 101 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20 103 1. Terminology 105 In this document, the key words "MUST", "MUST NOT", "REQUIRED", 106 "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT 107 RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as 108 described in BCP 14, conforming to [RFC2119] and indicate requirement 109 levels for compliant implementations. 111 RFC 2119 defines the use of these key words to help make the intent 112 of standards track documents as clear as possible. While this 113 document uses these keywords, this document is not a standards track 114 document. The term Throughput is defined in [RFC2544]. 116 Terms specific to SIP [RFC3261] performance benchmarking are defined 117 in [I-D.sip-bench-term]. 119 2. Introduction 121 This document describes the methodology for benchmarking Session 122 Initiation Protocol (SIP) performance as described in the Terminology 123 document [I-D.sip-bench-term]. The methodology and terminology are 124 to be used for benchmarking signaling plane performance with varying 125 signaling and media load. Media streams, when used, are used only to 126 study how they impact the signaling behavior. This document 127 concentrates on benchmarking SIP session setup and SIP registrations 128 only. 130 The device-under-test (DUT) is a RFC3261-capable [RFC3261] network 131 intermediary that plays the role of a registrar, redirect server, 132 stateful proxy, a Session Border Controller (SBC) or a B2BUA. This 133 document does not require the intermediary to assume the role of a 134 stateless proxy. Benchmarks can be obtained and compared for 135 different types of devices such as a SIP proxy server, Session Border 136 Controllers (SBC), SIP registrars and a SIP proxy server paired with 137 a media relay. 139 The test cases provide metrics for benchmarking the maximum 'SIP 140 Registration Rate' and maximum 'SIP Session Establishment Rate' that 141 the DUT can sustain over an extended period of time without failures 142 (extended period of time is defined in the algorithm in 143 Section 4.10). Some cases are included to cover encrypted SIP. The 144 test topologies that can be used are described in the Test Setup 145 section. Topologies in which the DUT handles media as well as those 146 in which the DUT does not handle media are both considered. The 147 measurement of the performance characteristics of the media itself is 148 outside the scope of these documents. 150 Benchmark metrics could possibly be impacted by Associated Media. 151 The selected values for Session Duration and Media Streams per 152 Session enable benchmark metrics to be benchmarked without Associated 153 Media. Session Setup Rate could possibly be impacted by the selected 154 value for Maximum Sessions Attempted. The benchmark for Session 155 Establishment Rate is measured with a fixed value for maximum Session 156 Attempts. 158 Finally, the overall value of these tests is to serve as a comparison 159 function between multiple SIP implementations. One way to use these 160 tests is to derive benchmarks with SIP devices from Vendor-A, derive 161 a new set of benchmarks with similar SIP devices from Vendor-B and 162 perform a comparison on the results of Vendor-A and Vendor-B. This 163 document does not make any claims on the interpretation of such 164 results. 166 3. Benchmarking Topologies 168 Test organizations need to be aware that these tests generate large 169 volumes of data and consequently ensure that networking devices like 170 hubs, switches or routers are able to handle the generated volume. 172 The test cases enumerated in Section 6.1 to Section 6.6 operate on 173 two test topologies: one in which the DUT does not process the media 174 (Figure 1) and the other in which it does process media (Figure 2). 175 In both cases, the tester or emulated agent (EA) sends traffic into 176 the DUT and absorbs traffic from the DUT. The diagrams in Figure 1 177 and Figure 2 represent the logical flow of information and do not 178 dictate a particular physical arrangement of the entities. 180 Figure 1 depicts a layout in which the DUT is an intermediary between 181 the two interfaces of the EA. If the test case requires the exchange 182 of media, the media does not flow through the DUT but rather passes 183 directly between the two endpoints. Figure 2 shows the DUT as an 184 intermediary between the two interfaces of the EA. If the test case 185 requires the exchange of media, the media flows through the DUT 186 between the endpoints. 188 +--------+ Session +--------+ Session +--------+ 189 | | Attempt | | Attempt | | 190 | |------------>+ |------------>+ | 191 | | | | | | 192 | | Response | | Response | | 193 | Tester +<------------| DUT +<------------| Tester | 194 | (EA) | | | | (EA) | 195 | | | | | | 196 +--------+ +--------+ +--------+ 197 /|\ /|\ 198 | Media (optional) | 199 +==============================================+ 201 Figure 1: DUT as an intermediary, end-to-end media 203 +--------+ Session +--------+ Session +--------+ 204 | | Attempt | | Attempt | | 205 | |------------>+ |------------>+ | 206 | | | | | | 207 | | Response | | Response | | 208 | Tester +<------------| DUT +<------------| Tester | 209 | (EA) | | | | (EA) | 210 | |<===========>| |<===========>| | 211 +--------+ Media +--------+ Media +--------+ 212 (Optional) (Optional) 214 Figure 2: DUT as an intermediary forwarding media 216 The test cases enumerated in Section 6.7 and Section 6.8 use the 217 topology in Figure 3 below. 219 +--------+ Registration +--------+ 220 | | request | | 221 | |------------->+ | 222 | | | | 223 | | Response | | 224 | Tester +<-------------| DUT | 225 | (EA) | | | 226 | | | | 227 +--------+ +--------+ 229 Figure 3: Registration and Re-registration tests 231 During registration or re-registration, the DUT may involve backend 232 network elements and data stores. These network elements and data 233 stores are not shown in Figure 3, but it is understood that they will 234 impact the time required for the DUT to generate a response. 236 This document explicitly separates a registration test (Section 6.7) 237 from a re-registration test (Section 6.8) because in certain 238 networks, the time to re-register may vary from the time to perform 239 an initial registration due to the backend processing involved. It 240 is expected that the registration tests and the re-registration test 241 will be performed with the same set of backend network elements in 242 order to derive a stable metric. 244 4. Test Setup Parameters 246 4.1. Selection of SIP Transport Protocol 248 Test cases may be performed with any transport protocol supported by 249 SIP. This includes, but is not limited to, TCP, UDP, TLS and 250 websockets. The protocol used for the SIP transport protocol must be 251 reported with benchmarking results. 253 SIP allows a DUT to use different transports for signaling on either 254 side of the connection to the EAs. Therefore, this document assumes 255 that the same transport is used on both sides of the connection; if 256 this is not the case in any of the tests, the transport on each side 257 of the connection MUST be reported in the test reporting template. 259 4.2. Connection-oriented Transport Management 261 SIP allows a device to open one connection and send multiple requests 262 over the same connection (responses are normally received over the 263 same connection that the request was sent out on). The protocol also 264 allows a device to open a new connection for each individual request. 265 A connection management strategy will have an impact on the results 266 obtained from the test cases, especially for connection-oriented 267 transports such as TLS. For such transports, the cryptographic 268 handshake must occur every time a connection is opened. 270 The connection management strategy, i.e., use of one connection to 271 send all requests or closing an existing connection and opening a new 272 connection to send each request, MUST be reported with the 273 benchmarking result. 275 4.3. Signaling Server 277 The Signaling Server is defined in the companion terminology 278 document, ([I-D.sip-bench-term], Section 3.2.2). The Signaling 279 Server is a DUT. 281 4.4. Associated Media 283 Some tests require Associated Media to be present for each SIP 284 session. The test topologies to be used when benchmarking DUT 285 performance for Associated Media are shown in Figure 1 and Figure 2. 287 4.5. Selection of Associated Media Protocol 289 The test cases specified in this document provide SIP performance 290 independent of the protocol used for the media stream. Any media 291 protocol supported by SIP may be used. This includes, but is not 292 limited to, RTP, and SRTP. The protocol used for Associated Media 293 MUST be reported with benchmarking results. 295 4.6. Number of Associated Media Streams per SIP Session 297 Benchmarking results may vary with the number of media streams per 298 SIP session. When benchmarking a DUT for voice, a single media 299 stream is used. When benchmarking a DUT for voice and video, two 300 media streams are used. The number of Associated Media Streams MUST 301 be reported with benchmarking results. 303 4.7. Codec Type 305 The test cases specified in this document provide SIP performance 306 independent of the media stream codec. Any codec supported by the 307 EAs may be used. The codec used for Associated Media MUST be 308 reported with the benchmarking results. 310 4.8. Session Duration 312 The value of the DUT's performance benchmarks may vary with the 313 duration of SIP sessions. Session Duration MUST be reported with 314 benchmarking results. A Session Duration of zero seconds indicates 315 transmission of a BYE immediately following a successful SIP 316 establishment. Setting this parameter to the value '0' indicates 317 that a BYE will be sent by the EA immediately after the EA receives a 318 200 OK to the INVITE. Setting this parameter to a time value greater 319 than the duration of the test indicates that a BYE is never sent. 321 4.9. Attempted Sessions per Second (sps) 323 The value of the DUT's performance benchmarks may vary with the 324 Session Attempt Rate offered by the tester. Session Attempt Rate 325 MUST be reported with the benchmarking results. 327 The test cases enumerated in Section 6.1 to Section 6.6 require that 328 the EA is configured to send the final 2xx-class response as quickly 329 as it can. This document does not require the tester to add any 330 delay between receiving a request and generating a final response. 332 4.10. Benchmarking algorithm 334 In order to benchmark the test cases uniformly in Section 6, the 335 algorithm described in this section should be used. A prosaic 336 description of the algorithm and a pseudo-code description are 337 provided below, and a simulation written in the R statistical 338 language [Rtool] is provided in Appendix A. 340 The goal is to find the largest value, R, a SIP Session Attempt Rate, 341 measured in sessions-per-second (sps), which the DUT can process with 342 zero errors over a defined, extended period. This period is defined 343 as the amount of time needed to attempt N SIP sessions, where N is a 344 parameter of test, at the attempt rate, R. An iterative process is 345 used to find this rate. The algorithm corresponding to this process 346 converges to R. 348 If the DUT vendor provides a value for R, the tester can use this 349 value. In cases where the DUT vendor does not provide a value for R, 350 or where the tester wants to establish the R of a system using local 351 media characteristics, the algorithm should be run by setting "r", 352 the session attempt rate, equal to a value of the tester's choice. 353 For example the tester may initialize "r = 100" to start the 354 algorithm and observe the value at convergence. The algorithm 355 dynamically increases and decreases "r" as it converges to the a 356 maximum sps value for R. The dynamic increase and decrease rate is 357 controlled by the weights "w" and "d", respectively. 359 The pseudo-code corresponding to the description above follows, and a 360 simulation written in the R statistical language is provided in 361 Appendix A. 363 ; ---- Parameters of test, adjust as needed 364 N := 50000 ; Global maximum; once largest session rate has 365 ; been established, send this many requests before 366 ; calling the test a success 367 m := {...} ; Other attributes that affect testing, such 368 ; as media streams, etc. 369 r := 100 ; Initial session attempt rate (in sessions/sec). 370 ; Adjust as needed (for example, if DUT can handle 371 ; thousands of calls in steady state, set to 372 ; appropriate value in the thousands). 373 w := 0.10 ; Traffic increase weight (0 < w <= 1.0) 374 d := max(0.10, w / 2) ; Traffic decrease weight 376 ; ---- End of parameters of test 378 proc find_R 380 R = max_sps(r, m, N) ; Setup r sps, each with m media 381 ; characteristics until N sessions have been attempted. 382 ; Note that if a DUT vendor provides this number, the tester 383 ; can use the number as a Session Attempt Rate, R, instead 384 ; of invoking max_sps() 386 end proc 388 ; Iterative process to figure out the largest number of 389 ; sps that we can achieve in order to setup n sessions. 390 ; This function converges to R, the Session Attempt Rate. 391 proc max_sps(r, m, n) 392 s := 0 ; session setup rate 393 old_r := 0 ; old session setup rate 394 h := 0 ; Return value, R 395 count := 0 397 ; Note that if w is small (say, 0.10) and r is small 398 ; (say, <= 9), the algorithm will not converge since it 399 ; uses floor() to increment r dynamically. It is best 400 ; off to start with the defaults (w = 0.10 and 401 ; r >= 100) 403 while (TRUE) { 404 s := send_traffic(r, m, n) ; Send r sps, with m media 405 ; characteristics until n sessions have been attempted. 406 if (s == n) { 407 if (r > old_r) { 408 old_r = r 409 } 410 else { 411 count = count + 1 412 if (count >= 10) { 413 # We've converged. 414 h := max(r, old_r) 415 break 417 } 418 } 420 r := floor(r + (w * r)) 421 } 422 else { 423 r := floor(r - (d * r)) 424 d := max(0.10, d / 2) 425 w := max(0.10, w / 2) 426 } 428 } 429 return h 430 end proc 432 5. Reporting Format 434 5.1. Test Setup Report 436 SIP Transport Protocol = ___________________________ 437 (valid values: TCP|UDP|TLS|SCTP|websockets|specify-other) 438 (specify if same transport used for connections to the DUT 439 and connections from the DUT. If different transports 440 used on each connection, enumerate the transports used) 442 Connection management strategy for connection oriented 443 transports 444 DUT receives requests on one connection = _______ 445 (Yes or no. If no, DUT accepts a new connection for 446 every incoming request, sends a response on that 447 connection and closes the connection) 448 DUT sends requests on one connection = __________ 449 (yes or no. If no, DUT initiates a new connection to 450 send out each request, gets a response on that 451 connection and closes the connection) 453 Session Attempt Rate _______________________________ 454 (Session attempts/sec) 455 (The initial value for "r" in Benchmarking Algorithm of 456 Section 4.10) 458 Session Duration = _________________________________ 459 (In seconds) 460 Total Sessions Attempted = _________________________ 461 (Total sessions to be created over duration of test) 463 Media Streams Per Session = _______________________ 464 (number of streams per session) 466 Associated Media Protocol = _______________________ 467 (RTP|SRTP|specify-other) 469 Codec = ____________________________________________ 470 (Codec type as identified by the organization that 471 specifies the codec) 473 Media Packet Size (audio only) = __________________ 474 (Number of bytes in an audio packet) 476 Establishment Threshold time = ____________________ 477 (Seconds) 479 TLS ciphersuite used 480 (for tests involving TLS) = ________________________ 481 (E.g., TLS_RSA_WITH_AES_128_CBC_SHA) 483 IPSec profile used 484 (For tests involving IPSEC) = _____________________ 486 5.2. Device Benchmarks for session setup 488 Session Establishment Rate, "R" = __________________ 489 (sessions per second) 490 Is DUT acting as a media relay (yes/no) = _________ 492 5.3. Device Benchmarks for registrations 494 Registration Rate = ____________________________ 495 (registrations per second) 497 Re-registration Rate = ____________________________ 498 (registrations per second) 500 Notes = ____________________________________________ 501 (List any specific backend processing required or 502 other parameters that may impact the rate) 504 6. Test Cases 506 6.1. Baseline Session Establishment Rate of the test bed 508 Objective: 509 To benchmark the Session Establishment Rate of the Emulated Agent 510 (EA) with zero failures. 512 Procedure: 513 1. Configure the DUT in the test topology shown in Figure 1. 514 2. Set media streams per session to 0. 515 3. Execute benchmarking algorithm as defined in Section 4.10 to 516 get the baseline session establishment rate. This rate MUST 517 be recorded using any pertinent parameters as shown in the 518 reporting format of Section 5.1. 520 Expected Results: This is the scenario to obtain the maximum Session 521 Establishment Rate of the EA and the test bed when no DUT is 522 present. The results of this test might be used to normalize test 523 results performed on different test beds or simply to better 524 understand the impact of the DUT on the test bed in question. 526 6.2. Session Establishment Rate without media 528 Objective: 529 To benchmark the Session Establishment Rate of the DUT with no 530 associated media and zero failures. 532 Procedure: 533 1. Configure a DUT according to the test topology shown in 534 Figure 1 or Figure 2. 535 2. Set media streams per session to 0. 536 3. Execute benchmarking algorithm as defined in Section 4.10 to 537 get the session establishment rate. This rate MUST be 538 recorded using any pertinent parameters as shown in the 539 reporting format of Section 5.1. 541 Expected Results: Find the Session Establishment Rate of the DUT 542 when the EA is not sending media streams. 544 6.3. Session Establishment Rate with Media not on DUT 546 Objective: 547 To benchmark the Session Establishment Rate of the DUT with zero 548 failures when Associated Media is included in the benchmark test 549 but the media is not running through the DUT. 551 Procedure: 552 1. Configure a DUT according to the test topology shown in 553 Figure 1. 554 2. Set media streams per session to 1. 555 3. Execute benchmarking algorithm as defined in Section 4.10 to 556 get the session establishment rate with media. This rate MUST 557 be recorded using any pertinent parameters as shown in the 558 reporting format of Section 5.1. 560 Expected Results: Session Establishment Rate results obtained with 561 Associated Media with any number of media streams per SIP session 562 are expected to be identical to the Session Establishment Rate 563 results obtained without media in the case where the DUT is 564 running on a platform separate from the Media Relay. 566 6.4. Session Establishment Rate with Media on DUT 568 Objective: 569 To benchmark the Session Establishment Rate of the DUT with zero 570 failures when Associated Media is included in the benchmark test 571 and the media is running through the DUT. 573 Procedure: 574 1. Configure a DUT according to the test topology shown in 575 Figure 2. 576 2. Set media streams per session to 1. 577 3. Execute benchmarking algorithm as defined in Section 4.10 to 578 get the session establishment rate with media. This rate MUST 579 be recorded using any pertinent parameters as shown in the 580 reporting format of Section 5.1. 582 Expected Results: Session Establishment Rate results obtained with 583 Associated Media may be lower than those obtained without media in 584 the case where the DUT and the Media Relay are running on the same 585 platform. It may be helpful for the tester to be aware of the 586 reasons for this degradation, although these reasons are not 587 parameters of the test. For example, the degree of performance 588 degradation may be due to what the DUT does with the media (e.g., 589 relaying vs. transcoding), the type of media (audio vs. video vs. 590 data), and the codec used for the media. There may also be cases 591 where there is no performance impact, if the DUT has dedicated 592 media-path hardware. 594 6.5. Session Establishment Rate with TLS Encrypted SIP 595 Objective: 596 To benchmark the Session Establishment Rate of the DUT with zero 597 failures when using TLS encrypted SIP signaling. 599 Procedure: 600 1. If the DUT is being benchmarked as a proxy or B2BUA, then 601 configure the DUT in the test topology shown in Figure 1 or 602 Figure 2. 603 2. Configure the tester to enable TLS over the transport being 604 used during benchmarking. Note the ciphersuite being used for 605 TLS and record it in Section 5.1. 606 3. Set media streams per session to 0 (media is not used in this 607 test). 608 4. Execute benchmarking algorithm as defined in Section 4.10 to 609 get the session establishment rate with TLS encryption. 611 Expected Results: Session Establishment Rate results obtained with 612 TLS Encrypted SIP may be lower than those obtained with plaintext 613 SIP. 615 6.6. Session Establishment Rate with IPsec Encrypted SIP 617 Objective: 618 To benchmark the Session Establishment Rate of the DUT with zero 619 failures when using IPsec Encrypted SIP signaling. 621 Procedure: 622 1. Configure a DUT according to the test topology shown in 623 Figure 1 or Figure 2. 624 2. Set media streams per session to 0 (media is not used in this 625 test). 626 3. Configure tester for IPSec. Note the IPSec profile being used 627 for and record it in Section 5.1. 628 4. Execute benchmarking algorithm as defined in Section 4.10 to 629 get the session establishment rate with encryption. 631 Expected Results: Session Establishment Rate results obtained with 632 IPSec Encrypted SIP may be lower than those obtained with 633 plaintext SIP. 635 6.7. Registration Rate 637 Objective: 638 To benchmark the maximum registration rate the DUT can handle over 639 an extended time period with zero failures. 641 Procedure: 642 1. Configure a DUT according to the test topology shown in 643 Figure 3. 644 2. Set the registration timeout value to at least 3600 seconds. 645 3. Each register request MUST be made to a distinct address of 646 record (AoR). Execute benchmarking algorithm as defined in 647 Section 4.10 to get the maximum registration rate. This rate 648 MUST be recorded using any pertinent parameters as shown in 649 the reporting format of Section 5.1. For example, the use of 650 TLS or IPSec during registration must be noted in the 651 reporting format. In the same vein, any specific backend 652 processing (use of databases, authentication servers, etc.) 653 SHOULD be recorded as well. 655 Expected Results: Provides a maximum registration rate. 657 6.8. Re-Registration Rate 659 Objective: 660 To benchmark the re-registration rate of the DUT with zero 661 failures using the same backend processing and parameters used 662 during Section 6.7. 664 Procedure: 665 1. Configure a DUT according to the test topology shown in 666 Figure 3. 667 2. First, execute test detailed in Section 6.7 to register the 668 endpoints with the registrar and obtain the registration rate. 669 3. After at least 5 minutes of Step 2, but no more than 10 670 minutes after Step 2 has been performed, re-register the same 671 AoRs used in Step 3 of Section 6.7. This will count as a re- 672 registration because the SIP AoRs have not yet expired. 674 Expected Results: Note the rate obtained through this test for 675 comparison with the rate obtained in Section 6.7. 677 7. IANA Considerations 679 This document does not requires any IANA considerations. 681 8. Security Considerations 683 Documents of this type do not directly affect the security of 684 Internet or corporate networks as long as benchmarking is not 685 performed on devices or systems connected to production networks. 687 Security threats and how to counter these in SIP and the media layer 688 is discussed in RFC3261, RFC3550, and RFC3711 and various other 689 drafts. This document attempts to formalize a set of common 690 methodology for benchmarking performance of SIP devices in a lab 691 environment. 693 9. Acknowledgments 695 The authors would like to thank Keith Drage and Daryl Malas for their 696 contributions to this document. Dale Worley provided an extensive 697 review that lead to improvements in the documents. We are grateful 698 to Barry Constantine, William Cerveny and Robert Sparks for providing 699 valuable comments during the document's last calls and expert 700 reviews. Al Morton and Sarah Banks have been exemplary working group 701 chairs, we thank them for tracking this work to completion. Tom 702 Taylor provided an in-depth review and subsequent comments on the 703 benchmarking convergence algorithm in Section 4.10. 705 10. References 707 10.1. Normative References 709 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 710 Requirement Levels", BCP 14, RFC 2119, March 1997. 712 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 713 Network Interconnect Devices", RFC 2544, March 1999. 715 [I-D.sip-bench-term] 716 Davids, C., Gurbani, V., and S. Poretsky, "SIP Performance 717 Benchmarking Terminology", 718 draft-ietf-bmwg-sip-bench-term-12 (work in progress), 719 November 2014. 721 10.2. Informative References 723 [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, 724 A., Peterson, J., Sparks, R., Handley, M., and E. 725 Schooler, "SIP: Session Initiation Protocol", RFC 3261, 726 June 2002. 728 [Rtool] R Development Core Team, "R: A language and environment 729 for statistical computing. R Foundation for Statistical 730 Computing, Vienna, Austria. ISBN 3-900051-07-0, URL 731 http://www.R-project.org/", , 2011. 733 Appendix A. R Code Component to simulate benchmarking algorithm 735 # Copyright (c) 2014 IETF Trust and Vijay K. Gurbani. All 736 # rights reserved. 737 # 738 # Redistribution and use in source and binary forms, with 739 # or without modification, are permitted provided that the 740 # following conditions are met: 741 # 742 # * Redistributions of source code must retain the above 743 # copyright notice, this list of conditions and the following 744 # disclaimer. 745 # * Redistributions in binary form must reproduce the above 746 # copyright notice, this list of conditions and the following 747 # disclaimer in the documentation and/or other materials 748 # provided with the distribution. 749 # * Neither the name of Internet Society, IETF or IETF Trust, 750 # nor the names of specific contributors, may be used 751 # to endorse or promote products derived from this software 752 # without specific prior written permission. 753 # 754 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND 755 # CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, 756 # INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 757 # MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 758 # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR 759 # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 760 # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 761 # BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 762 # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 763 # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, 764 # WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 765 # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE 766 # USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY 767 # OF SUCH DAMAGE. 769 w = 0.10 770 d = max(0.10, w / 2) 771 DUT_max_sps = 460 # Change as needed to set the max sps value 772 # for a DUT 774 # Returns R, given r (initial session attempt rate). 775 # E.g., assume that a DUT handles 460 sps in steady state 776 # and you have saved this code in a file simulate.r. Then, 777 # start an R session and do the following: 778 # 779 # > source("simulate.r") 780 # > find_R(100) 781 # ... debug output omitted ... 782 # [1] 458 783 # 784 # Thus, the max sps that the DUT can handle is 458 sps, which is 785 # close to the absolute maximum of 460 sps the DUT is specified to 786 # do. 787 find_R <- function(r) { 788 s = 0 789 old_r = 0 790 h = 0 791 count = 0 793 # Note that if w is small (say, 0.10) and r is small 794 # (say, <= 9), the algorithm will not converge since it 795 # uses floor() to increment r dynamically. It is best 796 # off to start with the defaults (w = 0.10 and 797 # r >= 100) 799 cat("r old_r w d \n") 800 while (TRUE) { 801 cat(r, ' ', old_r, ' ', w, ' ', d, '\n') 802 s = send_traffic(r) 803 if (s == TRUE) { # All sessions succeeded 805 if (r > old_r) { 806 old_r = r 807 } 808 else { 809 count = count + 1 811 if (count >= 10) { 812 # We've converged. 813 h = max(r, old_r) 814 break 815 } 816 } 818 r = floor(r + (w * r)) 819 } 820 else { 821 r = floor(r - (d * r)) 822 d = max(0.10, d / 2) 823 w = max(0.10, w / 2) 824 } 825 } 827 h 829 } 831 send_traffic <- function(r) { 832 n = TRUE 834 if (r > DUT_max_sps) { 835 n = FALSE 836 } 838 n 839 } 841 Authors' Addresses 843 Carol Davids 844 Illinois Institute of Technology 845 201 East Loop Road 846 Wheaton, IL 60187 847 USA 849 Phone: +1 630 682 6024 850 Email: davids@iit.edu 852 Vijay K. Gurbani 853 Bell Laboratories, Alcatel-Lucent 854 1960 Lucent Lane 855 Rm 9C-533 856 Naperville, IL 60566 857 USA 859 Phone: +1 630 224 0216 860 Email: vkg@bell-labs.com 862 Scott Poretsky 863 Allot Communications 864 300 TradeCenter, Suite 4680 865 Woburn, MA 08101 866 USA 868 Phone: +1 508 309 2179 869 Email: sporetsky@allot.com