idnits 2.17.1 draft-ietf-bmwg-sip-bench-term-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 497 has weird spacing: '...UTs and hop- ...' -- The document date (January 6, 2013) is 4128 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-12) exists of draft-ietf-bmwg-sip-bench-meth-06 == Outdated reference: A later version (-15) exists of draft-ietf-soc-overload-control-11 Summary: 0 errors (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Methodology Working Group C. Davids 3 Internet-Draft Illinois Institute of Technology 4 Intended status: Informational V. Gurbani 5 Expires: July 10, 2013 Bell Laboratories, 6 Alcatel-Lucent 7 S. Poretsky 8 Allot Communications 9 January 6, 2013 11 Terminology for Benchmarking Session Initiation Protocol (SIP) 12 Networking Devices 13 draft-ietf-bmwg-sip-bench-term-07 15 Abstract 17 This document provides a terminology for benchmarking the SIP 18 performance of networking devices. The term performance in this 19 context means the capacity of the device- or system-under-test to 20 process SIP messages. Terms are included for test components, test 21 setup parameters, and performance benchmark metrics for black-box 22 benchmarking of SIP networking devices. The performance benchmark 23 metrics are obtained for the SIP signaling plane only. The terms are 24 intended for use in a companion methodology document for 25 characterizing the performance of a SIP networking device under a 26 variety of conditions. The intent of the two documents is to enable 27 a comparison of the capacity of SIP networking devices. Test setup 28 parameters and a methodology document are necessary because SIP 29 allows a wide range of configuration and operational conditions that 30 can influence performance benchmark measurements. A standard 31 terminology and methodology will ensure that benchmarks have 32 consistent definition and were obtained following the same 33 procedures. 35 Status of this Memo 37 This Internet-Draft is submitted in full conformance with the 38 provisions of BCP 78 and BCP 79. 40 Internet-Drafts are working documents of the Internet Engineering 41 Task Force (IETF). Note that other groups may also distribute 42 working documents as Internet-Drafts. The list of current Internet- 43 Drafts is at http://datatracker.ietf.org/drafts/current/. 45 Internet-Drafts are draft documents valid for a maximum of six months 46 and may be updated, replaced, or obsoleted by other documents at any 47 time. It is inappropriate to use Internet-Drafts as reference 48 material or to cite them other than as "work in progress." 49 This Internet-Draft will expire on July 10, 2013. 51 Copyright Notice 53 Copyright (c) 2013 IETF Trust and the persons identified as the 54 document authors. All rights reserved. 56 This document is subject to BCP 78 and the IETF Trust's Legal 57 Provisions Relating to IETF Documents 58 (http://trustee.ietf.org/license-info) in effect on the date of 59 publication of this document. Please review these documents 60 carefully, as they describe your rights and restrictions with respect 61 to this document. Code Components extracted from this document must 62 include Simplified BSD License text as described in Section 4.e of 63 the Trust Legal Provisions and are provided without warranty as 64 described in the Simplified BSD License. 66 Table of Contents 68 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 69 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 6 70 2.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 7 71 2.2. Benchmarking Models . . . . . . . . . . . . . . . . . . . 9 72 3. Term Definitions . . . . . . . . . . . . . . . . . . . . . . . 14 73 3.1. Protocol Components . . . . . . . . . . . . . . . . . . . 14 74 3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 14 75 3.1.2. Signaling Plane . . . . . . . . . . . . . . . . . . . 17 76 3.1.3. Media Plane . . . . . . . . . . . . . . . . . . . . . 18 77 3.1.4. Associated Media . . . . . . . . . . . . . . . . . . . 18 78 3.1.5. Overload . . . . . . . . . . . . . . . . . . . . . . . 19 79 3.1.6. Session Attempt . . . . . . . . . . . . . . . . . . . 20 80 3.1.7. Established Session . . . . . . . . . . . . . . . . . 20 81 3.1.8. Invite-initiated Session (IS) . . . . . . . . . . . . 21 82 3.1.9. Non-INVITE-initiated Session (NS) . . . . . . . . . . 22 83 3.1.10. Session Attempt Failure . . . . . . . . . . . . . . . 22 84 3.1.11. Standing Sessions Count . . . . . . . . . . . . . . . 23 85 3.2. Test Components . . . . . . . . . . . . . . . . . . . . . 23 86 3.2.1. Emulated Agent . . . . . . . . . . . . . . . . . . . . 24 87 3.2.2. Signaling Server . . . . . . . . . . . . . . . . . . . 24 88 3.2.3. SIP-Aware Stateful Firewall . . . . . . . . . . . . . 24 89 3.2.4. SIP Transport Protocol . . . . . . . . . . . . . . . . 25 90 3.3. Test Setup Parameters . . . . . . . . . . . . . . . . . . 26 91 3.3.1. Session Attempt Rate . . . . . . . . . . . . . . . . . 26 92 3.3.2. IS Media Attempt Rate . . . . . . . . . . . . . . . . 26 93 3.3.3. Establishment Threshold Time . . . . . . . . . . . . . 27 94 3.3.4. Session Duration . . . . . . . . . . . . . . . . . . . 27 95 3.3.5. Media Packet Size . . . . . . . . . . . . . . . . . . 28 96 3.3.6. Media Offered Load . . . . . . . . . . . . . . . . . . 28 97 3.3.7. Media Session Hold Time . . . . . . . . . . . . . . . 29 98 3.3.8. Loop Detection Option . . . . . . . . . . . . . . . . 29 99 3.3.9. Forking Option . . . . . . . . . . . . . . . . . . . . 30 100 3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 31 101 3.4.1. Registration Rate . . . . . . . . . . . . . . . . . . 31 102 3.4.2. Session Establishment Rate . . . . . . . . . . . . . . 31 103 3.4.3. Session Capacity . . . . . . . . . . . . . . . . . . . 32 104 3.4.4. Session Overload Capacity . . . . . . . . . . . . . . 33 105 3.4.5. Session Establishment Performance . . . . . . . . . . 33 106 3.4.6. Session Attempt Delay . . . . . . . . . . . . . . . . 34 107 3.4.7. IM Rate . . . . . . . . . . . . . . . . . . . . . . . 34 108 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 35 109 5. Security Considerations . . . . . . . . . . . . . . . . . . . 35 110 6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 36 111 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 36 112 7.1. Normative References . . . . . . . . . . . . . . . . . . . 36 113 7.2. Informational References . . . . . . . . . . . . . . . . . 36 115 Appendix A. White Box Benchmarking Terminology . . . . . . . . . 37 116 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 37 118 1. Terminology 120 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 121 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 122 document are to be interpreted as described in BCP 14, RFC2119 123 [RFC2119]. RFC 2119 defines the use of these key words to help make 124 the intent of standards track documents as clear as possible. While 125 this document uses these keywords, this document is not a standards 126 track document. The term Throughput is defined in RFC2544 [RFC2544]. 128 For the sake of clarity and continuity, this document adopts the 129 template for definitions set out in Section 2 of RFC 1242 [RFC1242]. 131 The terms Device Under Test (DUT) and System Under Test (SUT) are 132 defined in the following BMWG documents: 134 Device Under Test (DUT) (c.f., Section 3.1.1 RFC 2285 [RFC2285]). 135 System Under Test (SUT) (c.f., Section 3.1.2, RFC 2285 [RFC2285]). 137 Many commonly used SIP terms in this document are defined in RFC 3261 138 [RFC3261]. For convenience the most important of these are 139 reproduced below. Use of these terms in this document is consistent 140 with their corresponding definition in [RFC3261]. 141 o Call Stateful: A proxy is call stateful if it retains state for a 142 dialog from the initiating INVITE to the terminating BYE request. 143 A call stateful proxy is always transaction stateful, but the 144 converse is not necessarily true. 145 o Stateful Proxy: A logical entity that maintains the client and 146 server transaction state machines defined by this specification 147 during the processing of a request, also known as a transaction 148 stateful proxy. The behavior of a stateful proxy is further 149 defined in Section 16 of RFC 3261 [RFC3261] . A transaction 150 stateful proxy is not the same as a call stateful proxy. 151 o Stateless Proxy: A logical entity that does not maintain the 152 client or server transaction state machines defined in this 153 specification when it processes requests. A stateless proxy 154 forwards every request it receives downstream and every response 155 it receives upstream. 156 o Back-to-back User Agent: A back-to-back user agent (B2BUA) is a 157 logical entity that receives a request and processes it as a user 158 agent server (UAS). In order to determine how the request should 159 be answered, it acts as a user agent client (UAC) and generates 160 requests. Unlike a proxy server, it maintains dialog state and 161 must participate in all requests sent on the dialogs it has 162 established. Since it is a concatenation of a UAC and a UAS, no 163 explicit definitions are needed for its behavior. 165 o Loop: A request that arrives at a proxy, is forwarded, and later 166 arrives back at the same proxy. When it arrives the second time, 167 its Request-URI is identical to the first time, and other header 168 fields that affect proxy operation are unchanged, so that the 169 proxy will make the same processing decision on the request it 170 made the first time. Looped requests are errors, and the 171 procedures for detecting them and handling them are described by 172 the SIP protocol[RFC3261] and also by RFC 5393 174 2. Introduction 176 Service Providers and IT Organizations deliver Voice Over IP (VoIP) 177 and Multimedia network services based on the IETF Session Initiation 178 Protocol (SIP) [RFC3261]. SIP is a signaling protocol originally 179 intended to be used to dynamically establish, disconnect and modify 180 streams of media between end users. As it has evolved it has been 181 adopted for use in a growing number of services and applications. 182 Many of these result in the creation of a media session, but some do 183 not. Examples of this latter group include text messaging and 184 subscription services. The set of benchmarking terms provided in 185 this document is intended for use with any SIP-enabled device 186 performing SIP functions in the interior of the network, whether or 187 not these result in the creation of media sessions. The performance 188 of end-user devices is outside the scope of this document. 190 A number of networking devices have been developed to support SIP- 191 based VoIP services. These include SIP Servers, Session Border 192 Controllers (SBC), Back-to-back User Agents (B2BUA), and SIP-Aware 193 Stateful Firewalls. These devices contain a mix of voice and IP 194 functions whose performance may be reported using metrics defined by 195 the equipment manufacturer or vendor. The Service Provider or IT 196 Organization seeking to compare the performance of such devices will 197 not be able to do so using these vendor-specific metrics, whose 198 conditions of test and algorithms for collection are often 199 unspecified. SIP functional elements and the devices that include 200 them can be configured many different ways and can be organized into 201 various topologies. These configuration and topological choices 202 impact the value of any chosen signaling benchmark. Unless these 203 conditions-of-test are defined, a true comparison of performance 204 metrics will not be possible. Some SIP-enabled network devices 205 terminate or relay media as well as signaling. The processing of 206 media by the device impacts the signaling performance. As a result, 207 the conditions-of-test must include information as to whether or not 208 the device under test processes media and if the device does process 209 media, a description of the media handled and the manner in which it 210 is handled. This document and its companion methodology document 211 [I-D.ietf-bmwg-sip-bench-meth] provide a set of black-box benchmarks 212 for describing and comparing the performance of devices that 213 incorporate the SIP User Agent Client and Server functions and that 214 operate in the network's core. 216 The definition of SIP performance benchmarks necessarily includes 217 definitions of Test Setup Parameters and a test methodology. These 218 enable the Tester to perform benchmarking tests on different devices 219 and to achieve comparable results. This document provides a common 220 set of definitions for Test Components, Test Setup Parameters, and 221 Benchmarks. All the benchmarks defined are black-box measurements of 222 the SIP signaling plane. The Test Setup Parameters and Benchmarks 223 defined in this document are intended for use with the companion 224 Methodology document. Benchmarks of internal DUT characteristics 225 (also known as white-box benchmarks) such as Session Attempt Arrival 226 Rate, which is measured at the DUT, are described in Appendix A to 227 allow additional characterization of DUT behavior with different 228 distribution models. 230 2.1. Scope 232 The scope of this work item is summarized as follows: 233 o This terminology document describes SIP signaling performance 234 benchmarks for black-box measurements of SIP networking devices. 235 Stress and debug scenarios are not addressed in this work item. 236 o The DUT must be an RFC 3261 capable network equipment. This may 237 be a Registrar, Redirect Server, Stateless Proxy or Stateful 238 Proxy. A DUT MAY also include a B2BUA, SBC functionality. The 239 DUT MAY be a multi-port SIP-to-switched network gateway 240 implemented as a SIP UAC or UAS. 241 o The DUT MAY include an internal SIP Application Level Gateway 242 (ALG), firewall, and/or a Network Address Translator (NAT). This 243 is referred to as the "SIP Aware Stateful Firewall." 244 o The DUT or SUT MUST NOT be end user equipment, such as personal 245 digital assistant, a computer-based client, or a user terminal. 246 o The Tester acts as multiple "Emulated Agents" (EA) that initiate 247 (or respond to) SIP messages as session endpoints and source (or 248 receive) associated media for established connections. 249 o SIP Signaling in presence of Media 250 * The media performance is not benchmarked in this work item. 251 * It is RECOMMENDED that SIP signaling plane benchmarks be 252 performed with media present, but this is optional. 253 * The SIP INVITE requests MUST include the SDP body. 254 * The type of DUT dictates whether the associated media streams 255 traverse the DUT or SUT. Both scenarios are within the scope 256 of this work item. 257 * SIP is frequently used to create media streams; the signaling 258 plane and media plane are treated as orthogonal to each other 259 in this document. While many devices support the creation of 260 media streams, benchmarks that measure the performance of these 261 streams are outside the scope of this document and its 262 companion methodology document [I-D.ietf-bmwg-sip-bench-meth]. 263 Tests may be performed with or without the creation of media 264 streams. The presence or absence of media streams MUST be 265 noted as a condition of the test as the performance of SIP 266 devices may vary accordingly. Even if the media is used during 267 benchmarking, only the SIP performance will be benchmarked, not 268 the media performance or quality. 269 o Both INVITE and non-INVITE scenarios (such as Instant Messages or 270 IM) are addressed in this document. However, benchmarking SIP 271 presence is not a part of this work item. 272 o Different transport mechanisms -- such as UDP, TCP, SCTP, or TLS 273 -- may be used. The specific transport mechanism MUST be noted as 274 a condition of the test as the performance of SIP devices may vary 275 accordingly. 276 o Looping and forking options are also considered since they impact 277 processing at SIP proxies. 278 o REGISTER and INVITE requests may be challenged or remain 279 unchallenged for authentication purpose. Whether or not the 280 REGISTER and INVITE requests are challenged is a condition of test 281 which will be recorded along with other such parameters which may 282 impact the SIP performance of the device or system under test. 283 o Re-INVITE requests are not considered in scope of this work item 284 since the benchmarks for INVITEs are based on the dialog created 285 by the INVITE and not on the transactions that take place within 286 that dialog. 287 o Only session establishment is considered for the performance 288 benchmarks. Session disconnect is not considered in the scope of 289 this work item. This is because our goal is to determine the 290 maximum capacity of the device or system under test, that is the 291 number of simultaneous SIP sessions that the device or system can 292 support. It is true that there are BYE requests being created 293 during the test process. These transactions do contribute to the 294 load on the device or system under test and thus are accounted for 295 in the metric we derive. We do not seek a separate metric for the 296 number of BYE transactions a device or system can support. 297 o SIP Overload [RFC6357] is within the scope of this work item. We 298 test to failure and then can continue to observe and record the 299 behavior of the system after failures are recorded. The cause of 300 failure is not within the scope of this work. We note the failure 301 and may continue to test until a different failure or condition is 302 encountered. Considerations on how to handle overload are 303 deferred to work progressing in the SOC working group 304 [I-D.ietf-soc-overload-control]. Vendors are, of course, free to 305 implement their specific overload control behavior as the expected 306 test outcome if it is different from the IETF recommendations. 307 However, such behavior MUST be documented and interpreted 308 appropriately across multiple vendor implementations. This will 309 make it more meaningful to compare the performance of different 310 SIP overload implementations. 311 o IMS-specific scenarios are not considered, but test cases can be 312 applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT. 314 2.2. Benchmarking Models 316 This section shows ten models to be used when benchmarking SIP 317 performance of a networking device. Figure 1 shows shows the 318 configuration needed to benchmark the tester itself. This model will 319 be used to establish the limitations of the test apparatus. 321 +--------+ Signaling request +--------+ 322 | +----------------------------->| | 323 | Tester | | Tester | 324 | EA | Signaling response | EA | 325 | |<-----------------------------+ | 326 +--------+ +--------+ 327 /|\ /|\ 328 | Media | 329 +=========================================+ 331 Figure 1: Baseline performance of the Emulated Agent without 332 a DUT present 334 Figure 2 shows the DUT playing the role of a user agent client (UAC), 335 initiating requests and absorbing responses. This model can be used 336 to baseline the performance of the DUT acting as an UAC without 337 associated media. 339 +--------+ Signaling request +--------+ 340 | +----------------------------->| | 341 | DUT | | Tester | 342 | | Signaling response | EA | 343 | |<-----------------------------+ | 344 +--------+ +--------+ 346 Figure 2: Baseline performance for DUT acting as a 347 user agent client without associated media 349 Figure 3 shows the DUT plays the role of a user agent server (UAS), 350 absorbing the requests and sending responses. This model can be used 351 as a baseline performance for the DUT acting as a UAS without 352 associated media. 354 +--------+ Signaling request +--------+ 355 | +----------------------------->| | 356 | Tester | | DUT | 357 | EA | Response | | 358 | |<-----------------------------+ | 359 +--------+ +--------+ 361 Figure 3: Baseline performance for DUT acting as a 362 user agent server without associated media 364 Figure 4 shows the DUT plays the role of a user agent client (UAC), 365 initiating requests and absorbing responses. This model can be used 366 as a baseline performance for the DUT acting as a UAC with associated 367 media. 369 +--------+ Signaling request +--------+ 370 | +----------------------------->| | 371 | DUT | | Tester | 372 | | Signaling response | (EA) | 373 | |<-----------------------------+ | 374 | |<============ Media =========>| | 375 +--------+ +--------+ 377 Figure 4: Baseline performance for DUT acting as a 378 user agent client with associated media 380 Figure 5 shows the DUT plays the role of a user agent server (UAS), 381 absorbing the requests and sending responses. This model can be used 382 as a baseline performance for the DUT acting as a UAS with associated 383 media. 385 +--------+ Signaling request +--------+ 386 | +----------------------------->| | 387 | Tester | | DUT | 388 | (EA) | Response | | 389 | |<-----------------------------+ | 390 | |<============ Media =========>| | 391 +--------+ +--------+ 393 Figure 5: Baseline performance for DUT acting as a 394 user agent server with associated media 396 Figure 6 shows that the Tester acts as the initiating and responding 397 EA as the DUT/SUT forwards Session Attempts. 399 +--------+ Session +--------+ Session +--------+ 400 | | Attempt | | Attempt | | 401 | |<------------+ |<------------+ | 402 | | | | | | 403 | | Response | | Response | | 404 | Tester +------------>| DUT +------------>| Tester | 405 | (EA) | | | | (EA) | 406 | | | | | | 407 +--------+ +--------+ +--------+ 409 Figure 6: DUT/SUT performance benchmark for session 410 establishment without media 412 Figure 7 is used when performing those same benchmarks with 413 Associated Media traversing the DUT/SUT. 415 +--------+ Session +--------+ Session +--------+ 416 | | Attempt | | Attempt | | 417 | |<------------+ |<------------+ | 418 | | | | | | 419 | | Response | | Response | | 420 | Tester +------------>| DUT +------------>| Tester | 421 | (EA) | | | | (EA) | 422 | | Media | | Media | | 423 | |<===========>| |<===========>| | 424 +--------+ +--------+ +--------+ 426 Figure 7: DUT/SUT performance benchmark for session 427 establishment with media traversing the DUT 429 Figure 8 is to be used when performing those same benchmarks with 430 Associated Media, but the media does not traverse the DUT/SUT. 431 Again, the benchmarking of the media is not within the scope of this 432 work item. The SIP control signaling is benchmarked in the presence 433 of Associated Media to determine if the SDP body of the signaling and 434 the handling of media impacts the performance of the DUT/SUT. 436 +--------+ Session +--------+ Session +--------+ 437 | | Attempt | | Attempt | | 438 | |<------------+ |<------------+ | 439 | | | | | | 440 | | Response | | Response | | 441 | Tester +------------>| DUT +------------>| Tester | 442 | (EA) | | | | (EA) | 443 | | | | | | 444 +--------+ +--------+ +--------+ 445 /|\ /|\ 446 | Media | 447 +=============================================+ 449 Figure 8: DUT/SUT performance benchmark for session 450 establishment with media external to the DUT 452 Figure 9 is used when performing benchmarks that require one or more 453 intermediaries to be in the signaling path. The intent is to gather 454 benchmarking statistics with a series of DUTs in place. In this 455 topology, the media is delivered end-to-end and does not traverse the 456 DUT. 458 SUT 459 ------------------^^^^^^^^------------- 460 / \ 461 +------+ Session +---+ Session +---+ Session +------+ 462 | | Attempt | | Attempt | | Attempt | | 463 | |<---------+ |<---------+ |<---------+ | 464 | | | | | | | | 465 | | Response | | Response | | Response | | 466 |Tester+--------->|DUT+--------->|DUT|--------->|Tester| 467 | (EA) | | | | | | (EA) | 468 | | | | | | | | 469 +------+ +---+ +---+ +------+ 470 /|\ /|\ 471 | Media | 472 +=============================================+ 474 Figure 9: DUT/SUT performance benchmark for session 475 establishment with multiple DUTs and end-to-end media 477 Figure 10 is used when performing benchmarks that require one or more 478 intermediaries to be in the signaling path. The intent is to gather 479 benchmarking statistics with a series of DUTs in place. In this 480 topology, the media is delivered hop-by-hop through each DUT. 482 SUT 483 -----------------^^^^^^^^------------- 484 / \ 485 +------+ Session +---+ Session +---+ Session +------+ 486 | | Attempt | | Attempt | | Attempt | | 487 | |<---------+ |<---------+ |<---------+ | 488 | | | | | | | | 489 | | Response | | Response | | Response | | 490 |Tester+--------->|DUT+--------->|DUT|--------->|Tester| 491 | (EA) | | | | | | (EA) | 492 | | | | | | | | 493 | |<========>| |<========>| |<========>| | 494 +------+ Media +---+ Media +---+ Media +------+ 496 Figure 10: DUT/SUT performance benchmark for session 497 establishment with multiple DUTs and hop- by-hop 498 media 500 Figure 11 illustrates the SIP signaling for an Established Session. 501 The Tester acts as the EAs and initiates a Session Attempt with the 502 DUT/SUT. When the Emulated Agent (EA) receives a 200 OK from the 503 DUT/SUT that session is considered to be an Established Session. The 504 illustration indicates three states of the session bring created by 505 the EA - Attempting, Established, and Disconnecting. Sessions can be 506 one of two type: Invite-Initiated Session (IS) or Non-Invite 507 Initiated Session (NS). Failure for the DUT/SUT to successfully 508 respond within the Establishment Threshold Time is considered a 509 Session Attempt Failure. SIP Invite messages MUST include the SDP 510 body to specify the Associated Media. Use of Associated Media, to be 511 sourced from the EA, is optional. When Associated Media is used, it 512 may traverse the DUT/SUT depending upon the type of DUT/SUT. The 513 Associated Media is shown in Figure 11 as "Media" connected to media 514 ports M1 and M2 on the EA. After the EA sends a BYE, the session 515 disconnects. Performance test cases for session disconnects are not 516 considered in this work item (the BYE request is shown for 517 completeness.) 518 EA DUT/SUT M1 M2 519 | | | | 520 | INVITE | | | 521 ---------+-------------->| | | 522 | | | | 523 Attempting | | | 524 | 200 OK | | | 525 ---------+<--------------| | | 526 | ACK | | | 527 |-------------->| | | 528 | | | | 529 | | | | 530 | | | Media | 531 Established | |<=====>| 532 | | | | 533 | BYE | | | 534 --------+--------------> | | | 535 | | | | 536 Disconnecting | | | 537 | 200 OK | | | 538 --------|<-------------- | | | 539 | | | | 541 Figure 11: Invite-initiated Session States 543 3. Term Definitions 545 3.1. Protocol Components 547 3.1.1. Session 549 Definition: 550 The combination of signaling and media messages and processes that 551 support a SIP-based service. 553 Discussion: 554 SIP messages are used to create and manage services for end users. 555 Often, these services include the creation of media streams that 556 are defined in the SDP body of a SIP message and carried in RTP 557 protocol data units. However, SIP messages can also be used to 558 create Instant Message services and subscription services, and 559 such services are not associated with media streams. SIP reserves 560 the term "session" to describe services that are analogous to 561 telephone calls on a circuit switched network. SIP reserves the 562 term "dialog" to refer to a signaling-only relationship between 563 User Agent peers. SIP reserves the term "transaction" to refer to 564 the brief communication between a client and a server that lasts 565 only until the final response to the SIP request. None of these 566 terms describes the entity whose performance we want to benchmark. 567 For example, the MESSAGE request does not create a dialog and can 568 be sent either within or outside of a dialog. It is not 569 associated with media, but it resembles a phone call in its 570 dependence on human rather than machine initiated responses. The 571 SUBSCRIBE method does create a dialog between the originating end- 572 user and the subscription service. It, too, is not associated 573 with a media session. 574 In light of the above observations we have extended the term 575 "session" to include SIP-based services that are not initiated by 576 INVITE requests and that do not have associated media. In this 577 extended definition, a session always has a signaling component 578 and may also have a media component. Thus, a session can be 579 defined as signaling-only or a combination of signaling and media. 580 We define the term "Associated Media", see Section 3.1.4, to 581 describe the situation in which media is associated with a SIP 582 dialog. The terminology "Invite-initiated Session" (IS) 583 Section 3.1.8 and "Non-invite-Initiated Session" (NS) 584 Section 3.1.9 are used to distinguish between these two types of 585 session. An Invite-initiated Session is a session as defined in 586 SIP. The performance of a device or system that supports Invite- 587 initiated Sessions that do not create media sessions, "Invite- 588 initiated Sessions without Associated Media", can be measured and 589 is of interest for comparison and as a limiting case. The 590 REGISTER request can be considered to be a "Non-invite-initiated 591 Session without Associated Media." A separate set of benchmarks 592 is provided for REGISTER requests since most implementations of 593 SIP-based services require this request and since a registrar may 594 be a device under test. 596 A Session in the context of this document, can be considered to be 597 a vector with three components: 599 1. A component in the signaling plane (SIP messages), sess.sig; 600 2. A media component in the media plane (RTP and SRTP streams for 601 example), sess.med (which may be null); 602 3. A control component in the media plane (RTCP messages for 603 example), sess.medc (which may be null). 605 An IS is expected to have non-null sess.sig and sess.med 606 components. The use of control protocols in the media component 607 is media dependent, thus the expected presence or absence of 608 sess.medc is media dependent and test-case dependent. An NS is 609 expected to have a non-null sess.sig component, but null sess.med 610 and sess.medc components. 612 Packets in the Signaling Plane and Media Plane will be handled by 613 different processes within the DUT. They will take different 614 paths within a SUT. These different processes and paths may 615 produce variations in performance. The terminology and benchmarks 616 defined in this document and the methodology for their use are 617 designed to enable us to compare performance of the DUT/SUT with 618 reference to the type of SIP-supported application it is handling. 620 Note that one or more sessions can simultaneously exist between 621 any participants. This can be the case, for example, when the EA 622 sets up both an IM and a voice call through the DUT/SUT. These 623 sessions are represented as an array session[x]. 625 Sessions will be represented as a vector array with three 626 components, as follows: 627 session-> 628 session[x].sig, the signaling component 629 session[x].medc[y], the media control component (e.g. RTCP) 630 session[x].med[y], an array of associated media streams (e.g. 631 RTP, SRTP, RTSP, MSRP). This media component may consist of zero 632 or more media streams. 633 Figure 12 models the vectors of the session. 635 Measurement Units: 636 N/A. 638 Issues: 639 None. 641 See Also: 642 Media Plane 643 Signaling Plane 644 Associated Media 645 Invite-initiated Session (IS) 646 Non-invite-initiated Session (NS) 647 |\ 648 | 649 | \ 650 sess.sig| 651 | \ 652 | 653 | \ 654 | o 655 | / 656 | / | 657 | / 658 | / | 659 | / 660 | / | 661 | / 662 | / | sess.medc 663 |/_____________________ 664 / / 665 / | 666 / / 667 sess.med / | 668 /_ _ _ _ _ _ _ _/ 669 / 670 / 671 / 672 / 674 Figure 12: Session components 676 3.1.2. Signaling Plane 678 Definition: 679 The plane in which SIP messages [RFC3261] are exchanged between 680 SIP Agents [RFC3261]. 682 Discussion: 683 SIP messages are used to establish sessions in several ways: 684 directly between two User Agents [RFC3261], through a Proxy Server 685 [RFC3261], or through a series of Proxy Servers. The Session 686 Description Protocol is included in the Signaling Plane. (SDP). 687 The Signaling Plane for a single Session is represented by 688 session.sig. 690 Measurement Units: 691 N/A. 693 Issues: 694 None. 696 See Also: 697 Media Plane 698 EAs 700 3.1.3. Media Plane 702 Definition: 703 The data plane in which one or more media streams and their 704 associated media control protocols are exchanged between User 705 Agents after a media connection has been created by the exchange 706 of signaling messages in the Signaling Plane. 708 Discussion: 709 Media may also be known as the "bearer channel". The Media Plane 710 MUST include the media control protocol, if one is used, and the 711 media stream(s). Examples of media are audio and video. The 712 media streams are described in the SDP of the Signaling Plane. 713 The media for a single Session is represented by session.med. The 714 media control protocol for a single media description is 715 represented by session.medc. 717 Measurement Units: 718 N/A. 720 Issues: 721 None. 723 See Also: 724 Signaling Plane 726 3.1.4. Associated Media 728 Definition: 729 Media that corresponds to an 'm' line in the SDP payload of the 730 Signaling Plane. 732 Discussion: 734 Any media protocol MAY be used. 735 For any session's signaling component, session.sig, there may be 736 zero, one, or multiple associated media streams. When there are 737 multiple media streams, these are represented be a vector array 738 session.med[y]. When there are multiple media streams there will 739 be multiple media control protocol descriptions as well. They are 740 represented by a vector array session.medc[y]. 742 Measurement Units: 743 N/A. 745 Issues: 746 None. 748 3.1.5. Overload 750 Definition: 751 Overload is defined as the state where a SIP server does not have 752 sufficient resources to process all incoming SIP messages 753 [RFC6357]. 755 Discussion: 756 The distinction between an overload condition and other failure 757 scenarios is outside the scope of black box testing and of this 758 document. Under overload conditions, all or a percentage of 759 Session Attempts will fail due to lack of resources. In black box 760 testing the cause of the failure is not explored. The fact that a 761 failure occurred for whatever reason, will trigger the tester to 762 reduce the offered load, as described in the companion methodology 763 document, [I-D.ietf-bmwg-sip-bench-meth]. SIP server resources 764 may include CPU processing capacity, network bandwidth, input/ 765 output queues, or disk resources. Any combination of resources 766 may be fully utilized when a SIP server (the DUT/SUT) is in the 767 overload condition. For proxy-only type of devices, it is 768 expected that the proxy will be driven into overload based on the 769 delivery rate of signaling requests. 770 For UA-type of network devices such as gateways, it is expected 771 that the UA will be driven into overload based on the volume of 772 media streams it is processing. 774 Measurement Units: 775 N/A. 777 Issues: 778 The issue of overload in SIP networks is currently a topic of 779 discussion in the SIPPING WG. The normal response to an overload 780 stimulus -- sending a 503 response -- is considered inadequate and 781 new response codes and behaviors may be specified in the future. 782 From the perspective of this document, all these responses will be 783 considered to be failures. There is thus no dependency between 784 this document and the ongoing work on the treatment of overload 785 failure. 787 3.1.6. Session Attempt 789 Definition: 790 A SIP request sent by the EA that has not received a final 791 response. 793 Discussion: 794 The attempted session may be Invite Initiated or Non-invite 795 Initiated. When counting the number of session attempts we 796 include all INVITEs that are rejected for lack of authentication 797 information. The EA needs to record the total number of session 798 attempts including those attempts that are routinely rejected by a 799 proxy that requires the UA to authenticate itself. The EA is 800 provisioned to deliver a specific number of session attempts per 801 second. But the EA must also count the actual number of session 802 attempts per given tie interval. 804 Measurement Units: 805 N/A. 807 Issues: 808 None. 810 See Also: 811 Session 812 Session Attempt Rate 813 Invite-initiated Session 814 Non-Invite initiated Session 816 3.1.7. Established Session 818 Definition: 819 A SIP session for which the EA acting as the UE/UA has received a 820 200 OK message. 822 Discussion: 823 An Established Session MAY be Invite Initiated or Non-invite 824 Initiated. 826 Measurement Units: 827 N/A. 829 Issues: 830 None. 832 See Also: 833 Invite-initiated Session 834 Session Attempting State 835 Session Disconnecting State 837 3.1.8. Invite-initiated Session (IS) 839 Definition: 840 A Session that is created by an exchange of messages in the 841 Signaling Plane, the first of which is a SIP INVITE request. 843 Discussion: 844 When an IS becomes an Established Session its signaling component 845 is identified by the SIP dialog parameter values, Call-ID, To-tag, 846 and From-tag (RFC3261 [RFC3261]). An IS may have zero, one or 847 multiple Associated Media descriptions in the SDP body. The 848 inclusion of media is test case dependent. An IS is successfully 849 established if the following two conditions are met: 850 1. Sess.sig is established by the end of Establishment Threshold 851 Time (c.f. Section 3.3.3), and 852 2. If a media session is described in the SDP body of the 853 signaling message, then the media session is established by 854 the end of Establishment Threshold Time (c.f. Section 3.3.3). 855 An SBC or B2BUA may receive media from a calling or called 856 party before a signaling dialog is established and certainly 857 before a confirmed dialog is established. The EA can be built 858 in such a way that it does not send early media or it needs to 859 include a parameter that indicates when it will send media. 860 This parameter must be included in the list of test setup 861 parameters in Section 5.1 of [I-D.ietf-bmwg-sip-bench-meth] 863 Measurement Units: 864 N/A. 866 Issues: 867 None. 869 See Also: 870 Session 871 Non-Invite initiated Session 872 Associated Media 874 3.1.9. Non-INVITE-initiated Session (NS) 876 Definition: 877 A session that is created by an exchange of SIP messages in the 878 Signaling Plane the first of which is not a SIP INVITE message. 880 Discussion: 881 An NS is successfully established if the Session Attempt via a 882 non- INVITE request results in the EA receiving a 2xx reply before 883 the expiration of the Establishment Threshold timer (c.f., 884 Section 3.3.3). An example of a NS is a session created by the 885 SUBSCRIBE request. 887 Measurement Units: 888 N/A. 890 Issues: 891 None. 893 See Also: 894 Session 895 Invite-initiated Session 897 3.1.10. Session Attempt Failure 899 Definition: 900 A session attempt that does not result in an Established Session. 902 Discussion: 903 The session attempt failure may be indicated by the following 904 observations at the EA: 905 1. Receipt of a SIP 4xx, 5xx, or 6xx class response to a Session 906 Attempt. 907 2. The lack of any received SIP response to a Session Attempt 908 within the Establishment Threshold Time (c.f. Section 3.3.3). 910 Measurement Units: 911 N/A. 913 Issues: 914 None. 916 See Also: 917 Session Attempt 919 3.1.11. Standing Sessions Count 921 Definition: 922 The number of Sessions currently established on the DUT/SUT at any 923 instant. 925 Discussion: 926 The number of Standing Sessions is influenced by the Session 927 Duration and the Session Attempt Rate. Benchmarks MUST be 928 reported with the maximum and average Standing Sessions for the 929 DUT/SUT for the duration of the test. In order to determine the 930 maximum and average Standing Sessions on the DUT/SUT for the 931 duration of the test it is necessary to make periodic measurements 932 of the number of Standing Sessions on the DUT/SUT. The 933 recommended value for the measurement period is 1 second. Since 934 we cannot directly poll the DUT/SUT, we take the number of 935 standing sessions on the DUT/SUT to be the number of distinct 936 calls as measured by the number of distinct Call-IDs that the EA 937 is processing at the time of measurement. The EA must make that 938 count available for viewing and recording. 940 Measurement Units: 941 Number of sessions 943 Issues: 944 None. 946 See Also: 947 Session Duration 948 Session Attempt Rate 949 Session Attempt Rate 950 Emulated Agent 952 3.2. Test Components 953 3.2.1. Emulated Agent 955 Definition: 956 A device in the test topology that initiates/responds to SIP 957 messages as one or more session endpoints and, wherever 958 applicable, sources/receives Associated Media for Established 959 Sessions. 961 Discussion: 962 The EA functions in the Signaling and Media Planes. The Tester 963 may act as multiple EAs. 965 Measurement Units: 966 N/A 968 Issues: 969 None. 971 See Also: 972 Media Plane 973 Signaling Plane 974 Established Session 975 Associated Media 977 3.2.2. Signaling Server 979 Definition: 980 Device in the test topology that acts to create sessions between 981 EAs. This device is either a DUT or a component of a SUT. 983 Discussion: 984 The DUT MUST be an RFC 3261 capable network equipment such as a 985 Registrar, Redirect Server, User Agent Server, Stateless Proxy, or 986 Stateful Proxy. A DUT MAY also include B2BUA or SBC. 988 Measurement Units: 989 NA 991 Issues: 992 None. 994 See Also: 995 Signaling Plane 997 3.2.3. SIP-Aware Stateful Firewall 998 Definition: 999 Device in the test topology that provides protection against 1000 various types of security threats to which the Signaling and Media 1001 Planes of the EAs and Signaling Server are vulnerable. 1003 Discussion: 1004 Threats may include Denial-of-Service, theft of service and misuse 1005 of service. The SIP-Aware Stateful Firewall MAY be an internal 1006 component or function of the Session Server. The SIP-Aware 1007 Stateful Firewall MAY be a standalone device. If it is a 1008 standalone device it MUST be paired with a Signaling Server. If 1009 it is a standalone device it MUST be benchmarked as part of a SUT. 1010 SIP-Aware Stateful Firewalls MAY include Network Address 1011 Translation (NAT) functionality. Ideally, the inclusion of the 1012 SIP-Aware Stateful Firewall in the SUT does not lower the measured 1013 values of the performance benchmarks. 1015 Measurement Units: 1016 N/A 1018 Issues: 1019 None. 1021 See Also: 1023 3.2.4. SIP Transport Protocol 1025 Definition: 1026 The protocol used for transport of the Signaling Plane messages. 1028 Discussion: 1029 Performance benchmarks may vary for the same SIP networking device 1030 depending upon whether TCP, UDP, TLS, SCTP, or another transport 1031 layer protocol is used. For this reason it MAY be necessary to 1032 measure the SIP Performance Benchmarks using these various 1033 transport protocols. Performance Benchmarks MUST report the SIP 1034 Transport Protocol used to obtain the benchmark results. 1036 Measurement Units: 1037 TCP,UDP, SCTP, TLS over TCP, TLS over UDP, or TLS over SCTP 1039 Issues: 1040 None. 1042 See Also: 1044 3.3. Test Setup Parameters 1046 3.3.1. Session Attempt Rate 1048 Definition: 1049 Configuration of the EA for the number of sessions per second that 1050 the EA attempts to establish using the services of the DUT/SUT. 1052 Discussion: 1053 The Session Attempt Rate is the number of sessions per second that 1054 the EA sends toward the DUT/SUT. Some of the sessions attempted 1055 may not result in a session being established. A session in this 1056 case may be either an IS or an NS. 1058 Measurement Units: 1059 Session attempts per second 1061 Issues: 1062 None. 1064 See Also: 1065 Session 1066 Session Attempt 1068 3.3.2. IS Media Attempt Rate 1070 Definition: 1071 Configuration on the EA for the rate, measured in sessions per 1072 second, at which the EA attempts to establish INVITE-initiated 1073 sessions with Associated Media, using the services of the DUT/SUT. 1075 Discussion: 1076 An IS is not required to include a media description. The IS 1077 Media Attempt Rate defines the number of media sessions we are 1078 trying to create, not the number of media sessions that are 1079 actually created. Some attempts might not result in successful 1080 sessions established on the DUT. 1082 Measurement Units: 1083 session attempts per second (saps) 1085 Issues: 1087 None. 1089 See Also: 1090 IS 1092 3.3.3. Establishment Threshold Time 1094 Definition: 1095 Configuration of the EA for representing the amount of time that 1096 an EA will wait before declaring a Session Attempt Failure. 1098 Discussion: 1099 This time duration is test dependent. 1100 It is RECOMMENDED that the Establishment Threshold Time value be 1101 set to Timer B (for ISs) or Timer F (for NSs) as specified in RFC 1102 3261, Table 4 [RFC3261]. Following the default value of T1 1103 (500ms) specified in the table and a constant multiplier of 64 1104 gives a value of 32 seconds for this timer (i.e., 500ms * 64 = 1105 32s). 1107 Measurement Units: 1108 seconds 1110 Issues: 1111 None. 1113 See Also: 1114 session establishment failure 1116 3.3.4. Session Duration 1118 Definition: 1119 Configuration of the EA that represents the amount of time that 1120 the SIP dialog is intended to exist between the two EAs associated 1121 with the test. 1123 Discussion: 1124 The time at which the BYE is sent will control the Session 1125 Duration 1126 Normally the Session Duration will be the same as the Media 1127 Session Hold Time. However, it is possible that the dialog 1128 established between the two EAs can support different media 1129 sessions at different points in time. Providing both parameters 1130 allows the testing agency to explore this possibility. 1132 Measurement Units: 1133 seconds 1135 Issues: 1136 None. 1138 See Also: 1139 Media Session Hold Time 1141 3.3.5. Media Packet Size 1143 Definition: 1144 Configuration on the EA for a fixed size of packets used for media 1145 streams. 1147 Discussion: 1148 For a single benchmark test, all sessions use the same size packet 1149 for media streams. The size of packets can cause variation in 1150 performance benchmark measurements. 1152 Measurement Units: 1153 bytes 1155 Issues: 1156 None. 1158 See Also: 1160 3.3.6. Media Offered Load 1162 Definition: 1163 Configuration of the EA for the constant rate of Associated Media 1164 traffic offered by the EA to the DUT/SUT for one or more 1165 Established Sessions of type IS. 1167 Discussion: 1168 The Media Offered Load to be used for a test MUST be reported with 1169 three components: 1170 1. per Associated Media stream; 1171 2. per IS; 1172 3. aggregate. 1173 For a single benchmark test, all sessions use the same Media 1174 Offered Load per Media Stream. There may be multiple Associated 1175 Media streams per IS. The aggregate is the sum of all Associated 1176 Media for all IS. 1178 Measurement Units: 1179 packets per second (pps) 1181 Issues: 1182 None. 1184 See Also: 1185 Established Session 1186 Invite Initiated Session 1187 Associated Media 1189 3.3.7. Media Session Hold Time 1191 Definition: 1192 Parameter configured at the EA, that represents the amount of time 1193 that the Associated Media for an Established Session of type IS 1194 will last. 1196 Discussion: 1197 The Associated Media streams may be bi-directional or uni- 1198 directional as indicated in the test methodology. 1199 Normally the Media Session Hold Time will be the same as the 1200 Session Duration. However, it is possible that the dialog 1201 established between the two EAs can support different media 1202 sessions at different points in time. Providing both parameters 1203 allows the testing agency to explore this possibility. 1205 Measurement Units: 1206 seconds 1208 Issues: 1209 None. 1211 See Also: 1212 Associated Media 1213 Established Session 1214 Invite-initiated Session (IS) 1216 3.3.8. Loop Detection Option 1218 Definition: 1219 An option that causes a Proxy to check for loops in the routing of 1220 a SIP request before forwarding the request. 1222 Discussion: 1223 This is an optional process that a SIP proxy may employ; the 1224 process is described under Proxy Behavior in RFC 3261 [RFC3261] in 1225 Section 16.3 Request Validation and that section also contains 1226 suggestions as to how the option could be implemented. Any 1227 procedure to detect loops will use processor cycles and hence 1228 could impact the performance of a proxy. 1230 Measurement Units: 1231 NA 1233 Issues: 1234 None. 1236 See Also: 1238 3.3.9. Forking Option 1240 Definition: 1241 An option that enables a Proxy to fork requests to more than one 1242 destination. 1244 Discussion: 1245 This is an process that a SIP proxy may employ to find the UAS. 1246 The option is described under Proxy Behavior in RFC 3261 in 1247 Section 16.1. A proxy that uses forking must maintain state 1248 information and this will use processor cycles and memory. Thus 1249 the use of this option could impact the performance of a proxy and 1250 different implementations could produce different impacts. 1251 SIP supports serial or parallel forking. When performing a test, 1252 the type of forking mode MUST be indicated. 1254 Measurement Units: 1255 The number of endpoints that will receive the forked invitation. 1256 A value of 1 indicates that the request is destined to only one 1257 endpoint, a value of 2 indicates that the request is forked to two 1258 endpoints, and so on. This is an integer value ranging between 1 1259 and N inclusive, where N is the maximum number of endpoints to 1260 which the invitation is sent. 1261 Type of forking used, namely parallel or serial. 1263 Issues: 1264 None. 1266 See Also: 1268 3.4. Benchmarks 1270 3.4.1. Registration Rate 1272 Definition: 1273 The maximum number of registrations that can be successfully 1274 completed by the DUT/SUT in a given time period without 1275 registration failures in that time period. 1277 Discussion: 1278 This benchmark is obtained with zero failure in which 100% of the 1279 registrations attempted by the EA are successfully completed by 1280 the DUT/SUT. The registration rate provisioned on the Emulated 1281 Agent is raised and lowered as described in the algorithm in the 1282 companion methodology draft [I-D.ietf-bmwg-sip-bench-meth] until a 1283 traffic load consisting of registrations at the given attempt rate 1284 over the sustained period of time identified by T in the algorithm 1285 completes without failure. 1287 Measurement Units: 1288 registrations per second (rps) 1290 Issues: 1291 None. 1293 See Also: 1295 3.4.2. Session Establishment Rate 1297 Definition: 1298 The maximum number of sessions that can be successfully completed 1299 by the DUT/SUT in a given time period without session 1300 establishment failures in that time period. 1302 Discussion: 1303 This benchmark is obtained with zero failure in which 100% of the 1304 sessions attempted by the Emulated Agent are successfully 1305 completed by the DUT/SUT. The session attempt rate provisioned on 1306 the EA is raised and lowered as described in the algorithm in the 1307 accompanying methodology document, until a traffic load at the 1308 given attempt rate over the sustained period of time identified by 1309 T in the algorithm completes without any failed session attempts. 1310 Sessions may be IS or NS or a mix of both and will be defined in 1311 the particular test. 1313 Measurement Units: 1314 sessions per second (sps) 1316 Issues: 1317 None. 1319 See Also: 1320 Invite-initiated Sessions 1321 Non-INVITE initiated Sessions 1322 Session Attempt Rate 1324 3.4.3. Session Capacity 1326 Definition: 1327 The maximum value of Standing Sessions Count achieved by the DUT/ 1328 SUT during a time period T in which the EA is sending session 1329 establishment messages at the Session Establishment Rate. 1331 Discussion: 1332 Sessions may be IS or NS. If they are IS they can be with or 1333 without media. When benchmarking Session Capacity for sessions 1334 with media it is required that these sessions be permanently 1335 established (i.e., they remain active for the duration of the 1336 test.) This can be achieved by causing the EA not to send a BYE 1337 for the duration of the testing. In the signaling plane, this 1338 requirement means that the dialog lasts as long as the test lasts. 1339 When media is present, the Media Session Hold Time MUST be set to 1340 infinity so that sessions remain established for the duration of 1341 the test. If the DUT/SUT is dialog-stateful, then we expect its 1342 performance will be impacted by setting Media Session Hold Time to 1343 infinity, since the DUT/SUT will need to allocate resources to 1344 process and store the state information. The report of the 1345 Session Capacity must include the Session Establishment Rate at 1346 which it was measured. 1348 Measurement Units: 1349 sessions 1351 Issues: 1352 None. 1354 See Also: 1355 Established Session 1356 Session Attempt Rate 1357 Session Attempt Failure 1359 3.4.4. Session Overload Capacity 1361 Definition: 1362 The maximum number of Established Sessions that can exist 1363 simultaneously on the DUT/SUT until it stops responding to Session 1364 Attempts. 1366 Discussion: 1367 Session Overload Capacity is measured after the Session Capacity 1368 is measured. The Session Overload Capacity is greater than or 1369 equal to the Session Capacity. When benchmarking Session Overload 1370 Capacity, continue to offer Session Attempts to the DUT/SUT after 1371 the first Session Attempt Failure occurs and measure Established 1372 Sessions until no there is no SIP message response for the 1373 duration of the Establishment Threshold. Note that the Session 1374 Establishment Performance is expected to decrease after the first 1375 Session Attempt Failure occurs. 1377 Units: 1378 Sessions 1380 Issues: 1381 None. 1383 See Also: 1384 Overload 1385 Session Capacity 1386 Session Attempt Failure 1388 3.4.5. Session Establishment Performance 1390 Definition: 1391 The percent of Session Attempts that become Established Sessions 1392 over the duration of a benchmarking test. 1394 Discussion: 1395 Session Establishment Performance is a benchmark to indicate 1396 session establishment success for the duration of a test. The 1397 duration for measuring this benchmark is to be specified in the 1398 Methodology. The Session Duration SHOULD be configured to 1399 infinity so that sessions remain established for the entire test 1400 duration. 1402 Session Establishment Performance is calculated as shown in the 1403 following equation: 1405 Session Establishment = Total Established Sessions 1406 Performance -------------------------- 1407 Total Session Attempts 1409 Session Establishment Performance may be monitored real-time 1410 during a benchmarking test. However, the reporting benchmark MUST 1411 be based on the total measurements for the test duration. 1413 Measurement Units: 1414 Percent (%) 1416 Issues: 1417 None. 1419 See Also: 1420 Established Session 1421 Session Attempt 1423 3.4.6. Session Attempt Delay 1425 Definition: 1426 The average time measured at the EA for a Session Attempt to 1427 result in an Established Session. 1429 Discussion: 1430 Time is measured from when the EA sends the first INVITE for the 1431 call-ID in the case of an IS. Time is measured from when the EA 1432 sends the first non-INVITE message in the case of an NS. Session 1433 Attempt Delay MUST be measured for every established session to 1434 calculate the average. Session Attempt Delay MUST be measured at 1435 the Session Establishment Rate. 1437 Measurement Units: 1438 Seconds 1440 Issues: 1441 None. 1443 See Also: 1444 Session Establishment Rate 1446 3.4.7. IM Rate 1447 Definition: 1448 Maximum number of IM messages completed by the DUT/SUT. 1450 Discussion: 1451 For a UAS, the definition of success is the receipt of an IM 1452 request and the subsequent sending of a final response. 1453 For a UAC, the definition of success is the sending of an IM 1454 request and the receipt of a final response to it. For a proxy, 1455 the definition of success is as follows: 1456 A. the number of IM requests it receives from the upstream client 1457 MUST be equal to the number of IM requests it sent to the 1458 downstream server; and 1459 B. the number of IM responses it receives from the downstream 1460 server MUST be equal to the number of IM requests sent to the 1461 downstream server; and 1462 C. the number of IM responses it sends to the upstream client 1463 MUST be equal to the number of IM requests it received from 1464 the upstream client. 1466 Measurement Units: 1467 IM messages per second 1469 Issues: 1470 None. 1472 See Also: 1474 4. IANA Considerations 1476 This document requires no IANA considerations. 1478 5. Security Considerations 1480 Documents of this type do not directly affect the security of 1481 Internet or corporate networks as long as benchmarking is not 1482 performed on devices or systems connected to production networks. 1483 Security threats and how to counter these in SIP and the media layer 1484 is discussed in RFC3261 [RFC3261], RFC 3550 [RFC3550], RFC3711 1485 [RFC3711] and various other drafts. This document attempts to 1486 formalize a set of common terminology for benchmarking SIP networks. 1487 Packets with unintended and/or unauthorized DSCP or IP precedence 1488 values may present security issues. Determining the security 1489 consequences of such packets is out of scope for this document. 1491 6. Acknowledgments 1493 The authors would like to thank Keith Drage, Cullen Jennings, Daryl 1494 Malas, Al Morton, and Henning Schulzrinne for invaluable 1495 contributions to this document. Dale Worley provided an extensive 1496 review that lead to improvements in the documents. 1498 7. References 1500 7.1. Normative References 1502 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1503 Requirement Levels", BCP 14, RFC 2119, March 1997. 1505 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 1506 Network Interconnect Devices", RFC 2544, March 1999. 1508 [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, 1509 A., Peterson, J., Sparks, R., Handley, M., and E. 1510 Schooler, "SIP: Session Initiation Protocol", RFC 3261, 1511 June 2002. 1513 [I-D.ietf-bmwg-sip-bench-meth] 1514 Davids, C., Gurbani, V., and S. Poretsky, "Methodology for 1515 Benchmarking SIP Networking Devices", 1516 draft-ietf-bmwg-sip-bench-meth-06 (work in progress), 1517 November 2012. 1519 7.2. Informational References 1521 [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN 1522 Switching Devices", RFC 2285, February 1998. 1524 [RFC1242] Bradner, S., "Benchmarking terminology for network 1525 interconnection devices", RFC 1242, July 1991. 1527 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 1528 Jacobson, "RTP: A Transport Protocol for Real-Time 1529 Applications", STD 64, RFC 3550, July 2003. 1531 [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 1532 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 1533 RFC 3711, March 2004. 1535 [RFC6357] Hilt, V., Noel, E., Shen, C., and A. Abdelal, "Design 1536 Considerations for Session Initiation Protocol (SIP) 1537 Overload Control", RFC 6357, August 2011. 1539 [I-D.ietf-soc-overload-control] 1540 Gurbani, V., Hilt, V., and H. Schulzrinne, "Session 1541 Initiation Protocol (SIP) Overload Control", 1542 draft-ietf-soc-overload-control-11 (work in progress), 1543 November 2012. 1545 Appendix A. White Box Benchmarking Terminology 1547 Session Attempt Arrival Rate 1549 Definition: 1550 The number of Session Attempts received at the DUT/SUT over a 1551 specified time period. 1553 Discussion: 1554 Sessions Attempts are indicated by the arrival of SIP INVITES OR 1555 SUBSCRIBE NOTIFY messages. Session Attempts Arrival Rate 1556 distribution can be any model selected by the user of this 1557 document. It is important when comparing benchmarks of different 1558 devices that same distribution model was used. Common 1559 distributions are expected to be Uniform and Poisson. 1561 Measurement Units: 1562 Session attempts/sec 1564 Issues: 1565 None. 1567 See Also: 1568 Session Attempt 1570 Authors' Addresses 1572 Carol Davids 1573 Illinois Institute of Technology 1574 201 East Loop Road 1575 Wheaton, IL 60187 1576 USA 1578 Phone: +1 630 682 6024 1579 Email: davids@iit.edu 1580 Vijay K. Gurbani 1581 Bell Laboratories, Alcatel-Lucent 1582 1960 Lucent Lane 1583 Rm 9C-533 1584 Naperville, IL 60566 1585 USA 1587 Phone: +1 630 224 0216 1588 Email: vkg@bell-labs.com 1590 Scott Poretsky 1591 Allot Communications 1592 300 TradeCenter, Suite 4680 1593 Woburn, MA 08101 1594 USA 1596 Phone: +1 508 309 2179 1597 Email: sporetsky@allot.com