idnits 2.17.1 draft-ietf-bmwg-sip-bench-term-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 376 has weird spacing: '...r agent clien...' == Line 392 has weird spacing: '...r agent serve...' == Line 496 has weird spacing: '...UTs and hop- ...' -- The document date (January 8, 2013) is 4123 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-12) exists of draft-ietf-bmwg-sip-bench-meth-08 == Outdated reference: A later version (-15) exists of draft-ietf-soc-overload-control-11 Summary: 0 errors (**), 0 flaws (~~), 6 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Methodology Working C. Davids 3 Group Illinois Institute of Technology 4 Internet-Draft V. Gurbani 5 Intended status: Informational Bell Laboratories, Alcatel-Lucent 6 Expires: July 12, 2013 S. Poretsky 7 Allot Communications 8 January 8, 2013 10 Terminology for Benchmarking Session Initiation Protocol (SIP) 11 Networking Devices 12 draft-ietf-bmwg-sip-bench-term-08 14 Abstract 16 This document provides a terminology for benchmarking the SIP 17 performance of networking devices. The term performance in this 18 context means the capacity of the device- or system-under-test to 19 process SIP messages. Terms are included for test components, test 20 setup parameters, and performance benchmark metrics for black-box 21 benchmarking of SIP networking devices. The performance benchmark 22 metrics are obtained for the SIP signaling plane only. The terms are 23 intended for use in a companion methodology document for 24 characterizing the performance of a SIP networking device under a 25 variety of conditions. The intent of the two documents is to enable 26 a comparison of the capacity of SIP networking devices. Test setup 27 parameters and a methodology document are necessary because SIP 28 allows a wide range of configuration and operational conditions that 29 can influence performance benchmark measurements. A standard 30 terminology and methodology will ensure that benchmarks have 31 consistent definition and were obtained following the same 32 procedures. 34 Status of this Memo 36 This Internet-Draft is submitted in full conformance with the 37 provisions of BCP 78 and BCP 79. 39 Internet-Drafts are working documents of the Internet Engineering 40 Task Force (IETF). Note that other groups may also distribute 41 working documents as Internet-Drafts. The list of current Internet- 42 Drafts is at http://datatracker.ietf.org/drafts/current/. 44 Internet-Drafts are draft documents valid for a maximum of six months 45 and may be updated, replaced, or obsoleted by other documents at any 46 time. It is inappropriate to use Internet-Drafts as reference 47 material or to cite them other than as "work in progress." 48 This Internet-Draft will expire on July 12, 2013. 50 Copyright Notice 52 Copyright (c) 2013 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents 57 (http://trustee.ietf.org/license-info) in effect on the date of 58 publication of this document. Please review these documents 59 carefully, as they describe your rights and restrictions with respect 60 to this document. Code Components extracted from this document must 61 include Simplified BSD License text as described in Section 4.e of 62 the Trust Legal Provisions and are provided without warranty as 63 described in the Simplified BSD License. 65 Table of Contents 67 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 68 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 6 69 2.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 7 70 2.2. Benchmarking Models . . . . . . . . . . . . . . . . . . . 9 71 3. Term Definitions . . . . . . . . . . . . . . . . . . . . . . . 14 72 3.1. Protocol Components . . . . . . . . . . . . . . . . . . . 14 73 3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 14 74 3.1.2. Signaling Plane . . . . . . . . . . . . . . . . . . . 17 75 3.1.3. Media Plane . . . . . . . . . . . . . . . . . . . . . 18 76 3.1.4. Associated Media . . . . . . . . . . . . . . . . . . . 18 77 3.1.5. Overload . . . . . . . . . . . . . . . . . . . . . . . 19 78 3.1.6. Session Attempt . . . . . . . . . . . . . . . . . . . 20 79 3.1.7. Established Session . . . . . . . . . . . . . . . . . 20 80 3.1.8. Invite-initiated Session (IS) . . . . . . . . . . . . 21 81 3.1.9. Non-INVITE-initiated Session (NS) . . . . . . . . . . 22 82 3.1.10. Session Attempt Failure . . . . . . . . . . . . . . . 22 83 3.1.11. Standing Sessions Count . . . . . . . . . . . . . . . 23 84 3.2. Test Components . . . . . . . . . . . . . . . . . . . . . 23 85 3.2.1. Emulated Agent . . . . . . . . . . . . . . . . . . . . 24 86 3.2.2. Signaling Server . . . . . . . . . . . . . . . . . . . 24 87 3.2.3. SIP-Aware Stateful Firewall . . . . . . . . . . . . . 24 88 3.2.4. SIP Transport Protocol . . . . . . . . . . . . . . . . 25 89 3.3. Test Setup Parameters . . . . . . . . . . . . . . . . . . 26 90 3.3.1. Session Attempt Rate . . . . . . . . . . . . . . . . . 26 91 3.3.2. IS Media Attempt Rate . . . . . . . . . . . . . . . . 26 92 3.3.3. Establishment Threshold Time . . . . . . . . . . . . . 27 93 3.3.4. Session Duration . . . . . . . . . . . . . . . . . . . 27 94 3.3.5. Media Packet Size . . . . . . . . . . . . . . . . . . 28 95 3.3.6. Media Offered Load . . . . . . . . . . . . . . . . . . 28 96 3.3.7. Media Session Hold Time . . . . . . . . . . . . . . . 29 97 3.3.8. Loop Detection Option . . . . . . . . . . . . . . . . 29 98 3.3.9. Forking Option . . . . . . . . . . . . . . . . . . . . 30 99 3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 31 100 3.4.1. Registration Rate . . . . . . . . . . . . . . . . . . 31 101 3.4.2. Session Establishment Rate . . . . . . . . . . . . . . 31 102 3.4.3. Session Capacity . . . . . . . . . . . . . . . . . . . 32 103 3.4.4. Session Overload Capacity . . . . . . . . . . . . . . 33 104 3.4.5. Session Establishment Performance . . . . . . . . . . 33 105 3.4.6. Session Attempt Delay . . . . . . . . . . . . . . . . 34 106 3.4.7. IM Rate . . . . . . . . . . . . . . . . . . . . . . . 34 107 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 35 108 5. Security Considerations . . . . . . . . . . . . . . . . . . . 35 109 6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 35 110 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 36 111 7.1. Normative References . . . . . . . . . . . . . . . . . . . 36 112 7.2. Informational References . . . . . . . . . . . . . . . . . 36 114 Appendix A. White Box Benchmarking Terminology . . . . . . . . . 37 115 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 37 117 1. Terminology 119 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 120 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 121 document are to be interpreted as described in BCP 14, RFC2119 122 [RFC2119]. RFC 2119 defines the use of these key words to help make 123 the intent of standards track documents as clear as possible. While 124 this document uses these keywords, this document is not a standards 125 track document. The term Throughput is defined in RFC2544 [RFC2544]. 127 For the sake of clarity and continuity, this document adopts the 128 template for definitions set out in Section 2 of RFC 1242 [RFC1242]. 130 The terms Device Under Test (DUT) and System Under Test (SUT) are 131 defined in the following BMWG documents: 133 Device Under Test (DUT) (c.f., Section 3.1.1 RFC 2285 [RFC2285]). 134 System Under Test (SUT) (c.f., Section 3.1.2, RFC 2285 [RFC2285]). 136 Many commonly used SIP terms in this document are defined in RFC 3261 137 [RFC3261]. For convenience the most important of these are 138 reproduced below. Use of these terms in this document is consistent 139 with their corresponding definition in [RFC3261]. 140 o Call Stateful: A proxy is call stateful if it retains state for a 141 dialog from the initiating INVITE to the terminating BYE request. 142 A call stateful proxy is always transaction stateful, but the 143 converse is not necessarily true. 144 o Stateful Proxy: A logical entity that maintains the client and 145 server transaction state machines defined by this specification 146 during the processing of a request, also known as a transaction 147 stateful proxy. The behavior of a stateful proxy is further 148 defined in Section 16 of RFC 3261 [RFC3261] . A transaction 149 stateful proxy is not the same as a call stateful proxy. 150 o Stateless Proxy: A logical entity that does not maintain the 151 client or server transaction state machines defined in this 152 specification when it processes requests. A stateless proxy 153 forwards every request it receives downstream and every response 154 it receives upstream. 155 o Back-to-back User Agent: A back-to-back user agent (B2BUA) is a 156 logical entity that receives a request and processes it as a user 157 agent server (UAS). In order to determine how the request should 158 be answered, it acts as a user agent client (UAC) and generates 159 requests. Unlike a proxy server, it maintains dialog state and 160 must participate in all requests sent on the dialogues it has 161 established. Since it is a concatenation of a UAC and a UAS, no 162 explicit definitions are needed for its behavior. 164 o Loop: A request that arrives at a proxy, is forwarded, and later 165 arrives back at the same proxy. When it arrives the second time, 166 its Request-URI is identical to the first time, and other header 167 fields that affect proxy operation are unchanged, so that the 168 proxy will make the same processing decision on the request it 169 made the first time. Looped requests are errors, and the 170 procedures for detecting them and handling them are described by 171 the SIP protocol[RFC3261] and also by RFC 5393 173 2. Introduction 175 Service Providers and IT Organizations deliver Voice Over IP (VoIP) 176 and Multimedia network services based on the IETF Session Initiation 177 Protocol (SIP) [RFC3261]. SIP is a signaling protocol originally 178 intended to be used to dynamically establish, disconnect and modify 179 streams of media between end users. As it has evolved it has been 180 adopted for use in a growing number of services and applications. 181 Many of these result in the creation of a media session, but some do 182 not. Examples of this latter group include text messaging and 183 subscription services. The set of benchmarking terms provided in 184 this document is intended for use with any SIP-enabled device 185 performing SIP functions in the interior of the network, whether or 186 not these result in the creation of media sessions. The performance 187 of end-user devices is outside the scope of this document. 189 A number of networking devices have been developed to support SIP- 190 based VoIP services. These include SIP Servers, Session Border 191 Controllers (SBC), Back-to-back User Agents (B2BUA), and SIP-Aware 192 Stateful Firewalls. These devices contain a mix of voice and IP 193 functions whose performance may be reported using metrics defined by 194 the equipment manufacturer or vendor. The Service Provider or IT 195 Organization seeking to compare the performance of such devices will 196 not be able to do so using these vendor-specific metrics, whose 197 conditions of test and algorithms for collection are often 198 unspecified. SIP functional elements and the devices that include 199 them can be configured many different ways and can be organized into 200 various topologies. These configuration and topological choices 201 impact the value of any chosen signaling benchmark. Unless these 202 conditions-of-test are defined, a true comparison of performance 203 metrics will not be possible. Some SIP-enabled network devices 204 terminate or relay media as well as signaling. The processing of 205 media by the device impacts the signaling performance. As a result, 206 the conditions-of-test must include information as to whether or not 207 the device under test processes media and if the device does process 208 media, a description of the media handled and the manner in which it 209 is handled. This document and its companion methodology document 210 [I-D.ietf-bmwg-sip-bench-meth] provide a set of black-box benchmarks 211 for describing and comparing the performance of devices that 212 incorporate the SIP User Agent Client and Server functions and that 213 operate in the network's core. 215 The definition of SIP performance benchmarks necessarily includes 216 definitions of Test Setup Parameters and a test methodology. These 217 enable the Tester to perform benchmarking tests on different devices 218 and to achieve comparable results. This document provides a common 219 set of definitions for Test Components, Test Setup Parameters, and 220 Benchmarks. All the benchmarks defined are black-box measurements of 221 the SIP signaling plane. The Test Setup Parameters and Benchmarks 222 defined in this document are intended for use with the companion 223 Methodology document. Benchmarks of internal DUT characteristics 224 (also known as white-box benchmarks) such as Session Attempt Arrival 225 Rate, which is measured at the DUT, are described in Appendix A to 226 allow additional characterization of DUT behavior with different 227 distribution models. 229 2.1. Scope 231 The scope of this work item is summarized as follows: 232 o This terminology document describes SIP signaling performance 233 benchmarks for black-box measurements of SIP networking devices. 234 Stress and debug scenarios are not addressed in this work item. 235 o The DUT must be an RFC 3261 capable network equipment. This may 236 be a Registrar, Redirect Server, Stateless Proxy or Stateful 237 Proxy. A DUT MAY also include a B2BUA, SBC functionality. The 238 DUT MAY be a multi-port SIP-to-switched network gateway 239 implemented as a SIP UAC or UAS. 240 o The DUT MAY include an internal SIP Application Level Gateway 241 (ALG), firewall, and/or a Network Address Translator (NAT). This 242 is referred to as the "SIP Aware Stateful Firewall." 243 o The DUT or SUT MUST NOT be end user equipment, such as personal 244 digital assistant, a computer-based client, or a user terminal. 245 o The Tester acts as multiple "Emulated Agents" (EA) that initiate 246 (or respond to) SIP messages as session endpoints and source (or 247 receive) associated media for established connections. 248 o SIP Signaling in presence of Media 249 * The media performance is not benchmarked in this work item. 250 * It is RECOMMENDED that SIP signaling plane benchmarks be 251 performed with media present, but this is optional. 252 * The SIP INVITE requests MUST include the SDP body. 253 * The type of DUT dictates whether the associated media streams 254 traverse the DUT or SUT. Both scenarios are within the scope 255 of this work item. 256 * SIP is frequently used to create media streams; the signaling 257 plane and media plane are treated as orthogonal to each other 258 in this document. While many devices support the creation of 259 media streams, benchmarks that measure the performance of these 260 streams are outside the scope of this document and its 261 companion methodology document [I-D.ietf-bmwg-sip-bench-meth]. 262 Tests may be performed with or without the creation of media 263 streams. The presence or absence of media streams MUST be 264 noted as a condition of the test as the performance of SIP 265 devices may vary accordingly. Even if the media is used during 266 benchmarking, only the SIP performance will be benchmarked, not 267 the media performance or quality. 268 o Both INVITE and non-INVITE scenarios (such as Instant Messages or 269 IM) are addressed in this document. However, benchmarking SIP 270 presence is not a part of this work item. 271 o Different transport mechanisms -- such as UDP, TCP, SCTP, or TLS 272 -- may be used. The specific transport mechanism MUST be noted as 273 a condition of the test as the performance of SIP devices may vary 274 accordingly. 275 o Looping and forking options are also considered since they impact 276 processing at SIP proxies. 277 o REGISTER and INVITE requests may be challenged or remain 278 unchallenged for authentication purpose. Whether or not the 279 REGISTER and INVITE requests are challenged is a condition of test 280 which will be recorded along with other such parameters which may 281 impact the SIP performance of the device or system under test. 282 o Re-INVITE requests are not considered in scope of this work item 283 since the benchmarks for INVITEs are based on the dialog created 284 by the INVITE and not on the transactions that take place within 285 that dialog. 286 o Only session establishment is considered for the performance 287 benchmarks. Session disconnect is not considered in the scope of 288 this work item. This is because our goal is to determine the 289 maximum capacity of the device or system under test, that is the 290 number of simultaneous SIP sessions that the device or system can 291 support. It is true that there are BYE requests being created 292 during the test process. These transactions do contribute to the 293 load on the device or system under test and thus are accounted for 294 in the metric we derive. We do not seek a separate metric for the 295 number of BYE transactions a device or system can support. 296 o SIP Overload [RFC6357] is within the scope of this work item. We 297 test to failure and then can continue to observe and record the 298 behavior of the system after failures are recorded. The cause of 299 failure is not within the scope of this work. We note the failure 300 and may continue to test until a different failure or condition is 301 encountered. Considerations on how to handle overload are 302 deferred to work progressing in the SOC working group 303 [I-D.ietf-soc-overload-control]. Vendors are, of course, free to 304 implement their specific overload control behavior as the expected 305 test outcome if it is different from the IETF recommendations. 306 However, such behavior MUST be documented and interpreted 307 appropriately across multiple vendor implementations. This will 308 make it more meaningful to compare the performance of different 309 SIP overload implementations. 310 o IMS-specific scenarios are not considered, but test cases can be 311 applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT. 313 2.2. Benchmarking Models 315 This section shows ten models to be used when benchmarking SIP 316 performance of a networking device. Figure 1 shows shows the 317 configuration needed to benchmark the tester itself. This model will 318 be used to establish the limitations of the test apparatus. 320 +--------+ Signaling request +--------+ 321 | +----------------------------->| | 322 | Tester | | Tester | 323 | EA | Signaling response | EA | 324 | |<-----------------------------+ | 325 +--------+ +--------+ 326 /|\ /|\ 327 | Media | 328 +=========================================+ 330 Figure 1: Baseline performance of the Emulated Agent without a DUT 331 present 333 Figure 2 shows the DUT playing the role of a user agent client (UAC), 334 initiating requests and absorbing responses. This model can be used 335 to baseline the performance of the DUT acting as an UAC without 336 associated media. 338 +--------+ Signaling request +--------+ 339 | +----------------------------->| | 340 | DUT | | Tester | 341 | | Signaling response | EA | 342 | |<-----------------------------+ | 343 +--------+ +--------+ 345 Figure 2: Baseline performance for DUT acting as a user agent client 346 without associated media 348 Figure 3 shows the DUT playing the role of a user agent server (UAS), 349 absorbing the requests and sending responses. This model can be used 350 as a baseline performance for the DUT acting as a UAS without 351 associated media. 353 +--------+ Signaling request +--------+ 354 | +----------------------------->| | 355 | Tester | | DUT | 356 | EA | Response | | 357 | |<-----------------------------+ | 358 +--------+ +--------+ 360 Figure 3: Baseline performance for DUT acting as a user agent server 361 without associated media 363 Figure 4 shows the DUT plays the role of a user agent client (UAC), 364 initiating requests and absorbing responses. This model can be used 365 as a baseline performance for the DUT acting as a UAC with associated 366 media. 368 +--------+ Signaling request +--------+ 369 | +----------------------------->| | 370 | DUT | | Tester | 371 | | Signaling response | (EA) | 372 | |<-----------------------------+ | 373 | |<============ Media =========>| | 374 +--------+ +--------+ 376 Figure 4: Baseline performance for DUT acting as a user agent client 377 with associated media 379 Figure 5 shows the DUT plays the role of a user agent server (UAS), 380 absorbing the requests and sending responses. This model can be used 381 as a baseline performance for the DUT acting as a UAS with associated 382 media. 384 +--------+ Signaling request +--------+ 385 | +----------------------------->| | 386 | Tester | | DUT | 387 | (EA) | Response | | 388 | |<-----------------------------+ | 389 | |<============ Media =========>| | 390 +--------+ +--------+ 392 Figure 5: Baseline performance for DUT acting as a user agent server 393 with associated media 395 Figure 6 shows that the Tester acts as the initiating and responding 396 EA as the DUT/SUT forwards Session Attempts. 398 +--------+ Session +--------+ Session +--------+ 399 | | Attempt | | Attempt | | 400 | |<------------+ |<------------+ | 401 | | | | | | 402 | | Response | | Response | | 403 | Tester +------------>| DUT +------------>| Tester | 404 | (EA) | | | | (EA) | 405 | | | | | | 406 +--------+ +--------+ +--------+ 408 Figure 6: DUT/SUT performance benchmark for session establishment 409 without media 411 Figure 7 is used when performing those same benchmarks with 412 Associated Media traversing the DUT/SUT. 414 +--------+ Session +--------+ Session +--------+ 415 | | Attempt | | Attempt | | 416 | |<------------+ |<------------+ | 417 | | | | | | 418 | | Response | | Response | | 419 | Tester +------------>| DUT +------------>| Tester | 420 | (EA) | | | | (EA) | 421 | | Media | | Media | | 422 | |<===========>| |<===========>| | 423 +--------+ +--------+ +--------+ 425 Figure 7: DUT/SUT performance benchmark for session establishment 426 with media traversing the DUT 428 Figure 8 is to be used when performing those same benchmarks with 429 Associated Media, but the media does not traverse the DUT/SUT. 430 Again, the benchmarking of the media is not within the scope of this 431 work item. The SIP control signaling is benchmarked in the presence 432 of Associated Media to determine if the SDP body of the signaling and 433 the handling of media impacts the performance of the DUT/SUT. 435 +--------+ Session +--------+ Session +--------+ 436 | | Attempt | | Attempt | | 437 | |<------------+ |<------------+ | 438 | | | | | | 439 | | Response | | Response | | 440 | Tester +------------>| DUT +------------>| Tester | 441 | (EA) | | | | (EA) | 442 | | | | | | 443 +--------+ +--------+ +--------+ 444 /|\ /|\ 445 | Media | 446 +=============================================+ 448 Figure 8: DUT/SUT performance benchmark for session establishment 449 with media external to the DUT 451 Figure 9 is used when performing benchmarks that require one or more 452 intermediaries to be in the signaling path. The intent is to gather 453 benchmarking statistics with a series of DUTs in place. In this 454 topology, the media is delivered end-to-end and does not traverse the 455 DUT. 457 SUT 458 ------------------^^^^^^^^------------- 459 / \ 460 +------+ Session +---+ Session +---+ Session +------+ 461 | | Attempt | | Attempt | | Attempt | | 462 | |<---------+ |<---------+ |<---------+ | 463 | | | | | | | | 464 | | Response | | Response | | Response | | 465 |Tester+--------->|DUT+--------->|DUT|--------->|Tester| 466 | (EA) | | | | | | (EA) | 467 | | | | | | | | 468 +------+ +---+ +---+ +------+ 469 /|\ /|\ 470 | Media | 471 +=============================================+ 473 Figure 9: DUT/SUT performance benchmark for session establishment 474 with multiple DUTs and end-to-end media 476 Figure 10 is used when performing benchmarks that require one or more 477 intermediaries to be in the signaling path. The intent is to gather 478 benchmarking statistics with a series of DUTs in place. In this 479 topology, the media is delivered hop-by-hop through each DUT. 481 SUT 482 -----------------^^^^^^^^------------- 483 / \ 484 +------+ Session +---+ Session +---+ Session +------+ 485 | | Attempt | | Attempt | | Attempt | | 486 | |<---------+ |<---------+ |<---------+ | 487 | | | | | | | | 488 | | Response | | Response | | Response | | 489 |Tester+--------->|DUT+--------->|DUT|--------->|Tester| 490 | (EA) | | | | | | (EA) | 491 | | | | | | | | 492 | |<========>| |<========>| |<========>| | 493 +------+ Media +---+ Media +---+ Media +------+ 495 Figure 10: DUT/SUT performance benchmark for session establishment 496 with multiple DUTs and hop- by-hop media 498 Figure 11 illustrates the SIP signaling for an Established Session. 499 The Tester acts as the EAs and initiates a Session Attempt with the 500 DUT/SUT. When the EA receives a 200 OK from the DUT/SUT that session 501 is considered to be an Established Session. The illustration 502 indicates three states of the session bring created by the EA - (1) 503 Attempting, (2) Established, and (3) Disconnecting. Sessions can be 504 one of two type: Invite-Initiated Session (IS) or Non-Invite 505 Initiated Session (NS). Failure for the DUT/SUT to successfully 506 respond within the Establishment Threshold Time is considered a 507 Session Attempt Failure. SIP Invite messages MUST include the SDP 508 body to specify the Associated Media. Use of Associated Media, to be 509 sourced from the EA, is optional. When Associated Media is used, it 510 may traverse the DUT/SUT depending upon the type of DUT/SUT. The 511 Associated Media is shown in Figure 11 as "Media" connected to media 512 ports M1 and M2 on the EA. After the EA sends a BYE, the session 513 disconnects. Performance test cases for session disconnects are not 514 considered in this work item (the BYE request is shown for 515 completeness.) 516 EA DUT/SUT M1 M2 517 | | | | 518 | INVITE | | | 519 ---------+-------------->| | | 520 | | | | 521 Attempting | | | 522 | 200 OK | | | 523 ---------+<--------------| | | 524 | ACK | | | 525 |-------------->| | | 526 | | | | 527 | | | | 528 | | | Media | 529 Established | |<=====>| 530 | | | | 531 | BYE | | | 532 --------+--------------> | | | 533 | | | | 534 Disconnecting | | | 535 | 200 OK | | | 536 --------|<-------------- | | | 537 | | | | 539 Figure 11: Invite-initiated Session States 541 3. Term Definitions 543 3.1. Protocol Components 545 3.1.1. Session 547 Definition: 548 The combination of signaling and media messages and processes that 549 support a SIP-based service. 551 Discussion: 552 SIP messages are used to create and manage services for end users. 553 Often, these services include the creation of media streams that 554 are defined in the SDP body of a SIP message and carried in RTP 555 protocol data units. However, SIP messages can also be used to 556 create Instant Message services and subscription services, and 557 such services are not associated with media streams. SIP reserves 558 the term "session" to describe services that are analogous to 559 telephone calls on a circuit switched network. SIP reserves the 560 term "dialog" to refer to a signaling-only relationship between 561 User Agent peers. SIP reserves the term "transaction" to refer to 562 the brief communication between a client and a server that lasts 563 only until the final response to the SIP request. None of these 564 terms describes the entity whose performance we want to benchmark. 565 For example, the MESSAGE request does not create a dialog and can 566 be sent either within or outside of a dialog. It is not 567 associated with media, but it resembles a phone call in its 568 dependence on human rather than machine initiated responses. The 569 SUBSCRIBE method does create a dialog between the originating end- 570 user and the subscription service. It, too, is not associated 571 with a media session. 573 In light of the above observations we have extended the term 574 "session" to include SIP-based services that are not initiated by 575 INVITE requests and that do not have associated media. In this 576 extended definition, a session always has a signaling component 577 and may also have a media component. Thus, a session can be 578 defined as signaling-only or a combination of signaling and media. 579 We define the term "Associated Media", see Section 3.1.4, to 580 describe the situation in which media is associated with a SIP 581 dialog. The terminology "Invite-initiated Session" (IS) 582 Section 3.1.8 and "Non-invite-Initiated Session" (NS) 583 Section 3.1.9 are used to distinguish between these two types of 584 session. An Invite-initiated Session is a session as defined in 585 SIP. The performance of a device or system that supports Invite- 586 initiated Sessions that do not create media sessions, "Invite- 587 initiated Sessions without Associated Media", can be measured and 588 is of interest for comparison and as a limiting case. The 589 REGISTER request can be considered to be a "Non-invite-initiated 590 Session without Associated Media." A separate set of benchmarks 591 is provided for REGISTER requests since most implementations of 592 SIP-based services require this request and since a registrar may 593 be a device under test. 595 A Session in the context of this document, can be considered to be 596 a vector with three components: 598 1. A component in the signaling plane (SIP messages), sess.sig; 599 2. A media component in the media plane (RTP and SRTP streams for 600 example), sess.med (which may be null); 601 3. A control component in the media plane (RTCP messages for 602 example), sess.medc (which may be null). 604 An IS is expected to have non-null sess.sig and sess.med 605 components. The use of control protocols in the media component 606 is media dependent, thus the expected presence or absence of 607 sess.medc is media dependent and test-case dependent. An NS is 608 expected to have a non-null sess.sig component, but null sess.med 609 and sess.medc components. 611 Packets in the Signaling Plane and Media Plane will be handled by 612 different processes within the DUT. They will take different 613 paths within a SUT. These different processes and paths may 614 produce variations in performance. The terminology and benchmarks 615 defined in this document and the methodology for their use are 616 designed to enable us to compare performance of the DUT/SUT with 617 reference to the type of SIP-supported application it is handling. 619 Note that one or more sessions can simultaneously exist between 620 any participants. This can be the case, for example, when the EA 621 sets up both an IM and a voice call through the DUT/SUT. These 622 sessions are represented as an array session[x]. 624 Sessions will be represented as a vector array with three 625 components, as follows: 626 session-> 627 session[x].sig, the signaling component 628 session[x].medc[y], the media control component (e.g. RTCP) 629 session[x].med[y], an array of associated media streams (e.g. 630 RTP, SRTP, RTSP, MSRP). This media component may consist of zero 631 or more media streams. 632 Figure 12 models the vectors of the session. 634 Measurement Units: 635 N/A. 637 Issues: 638 None. 640 See Also: 641 Media Plane 642 Signaling Plane 643 Associated Media 644 Invite-initiated Session (IS) 645 Non-invite-initiated Session (NS) 646 |\ 647 | 648 | \ 649 sess.sig| 650 | \ 651 | 652 | \ 653 | o 654 | / 655 | / | 656 | / 657 | / | 658 | / 659 | / | 660 | / 661 | / | sess.medc 662 |/_____________________ 663 / / 664 / | 665 / / 666 sess.med / | 667 /_ _ _ _ _ _ _ _/ 668 / 669 / 670 / 671 / 673 Figure 12: Session components 675 3.1.2. Signaling Plane 677 Definition: 678 The plane in which SIP messages [RFC3261] are exchanged between 679 SIP Agents [RFC3261]. 681 Discussion: 682 SIP messages are used to establish sessions in several ways: 683 directly between two User Agents [RFC3261], through a Proxy Server 684 [RFC3261], or through a series of Proxy Servers. The Session 685 Description Protocol (SDP) is included in the Signaling Plane. 686 The Signaling Plane for a single Session is represented by 687 session.sig. 689 Measurement Units: 690 N/A. 692 Issues: 693 None. 695 See Also: 696 Media Plane 697 EAs 699 3.1.3. Media Plane 701 Definition: 702 The data plane in which one or more media streams and their 703 associated media control protocols are exchanged between User 704 Agents after a media connection has been created by the exchange 705 of signaling messages in the Signaling Plane. 707 Discussion: 708 Media may also be known as the "bearer channel". The Media Plane 709 MUST include the media control protocol, if one is used, and the 710 media stream(s). Examples of media are audio and video. The 711 media streams are described in the SDP of the Signaling Plane. 712 The media for a single Session is represented by session.med. The 713 media control protocol for a single media description is 714 represented by session.medc. 716 Measurement Units: 717 N/A. 719 Issues: 720 None. 722 See Also: 723 Signaling Plane 725 3.1.4. Associated Media 727 Definition: 728 Media that corresponds to an 'm' line in the SDP payload of the 729 Signaling Plane. 731 Discussion: 733 Any media protocol MAY be used. 734 For any session's signaling component, session.sig, there may be 735 zero, one, or multiple associated media streams. When there are 736 multiple media streams, these are represented be a vector array 737 session.med[y]. When there are multiple media streams there will 738 be multiple media control protocol descriptions as well. They are 739 represented by a vector array session.medc[y]. 741 Measurement Units: 742 N/A. 744 Issues: 745 None. 747 3.1.5. Overload 749 Definition: 750 Overload is defined as the state where a SIP server does not have 751 sufficient resources to process all incoming SIP messages 752 [RFC6357]. 754 Discussion: 755 The distinction between an overload condition and other failure 756 scenarios is outside the scope of black box testing and of this 757 document. Under overload conditions, all or a percentage of 758 Session Attempts will fail due to lack of resources. In black box 759 testing the cause of the failure is not explored. The fact that a 760 failure occurred for whatever reason, will trigger the tester to 761 reduce the offered load, as described in the companion methodology 762 document, [I-D.ietf-bmwg-sip-bench-meth]. SIP server resources 763 may include CPU processing capacity, network bandwidth, input/ 764 output queues, or disk resources. Any combination of resources 765 may be fully utilized when a SIP server (the DUT/SUT) is in the 766 overload condition. For proxy-only type of devices, it is 767 expected that the proxy will be driven into overload based on the 768 delivery rate of signaling requests. 769 For UA-type of network devices such as gateways, it is expected 770 that the UA will be driven into overload based on the volume of 771 media streams it is processing. 773 Measurement Units: 774 N/A. 776 Issues: 777 The issue of overload in SIP networks is currently a topic of 778 discussion in the SIPPING WG. The normal response to an overload 779 stimulus -- sending a 503 response -- is considered inadequate and 780 new response codes and behaviors may be specified in the future. 781 From the perspective of this document, all these responses will be 782 considered to be failures. There is thus no dependency between 783 this document and the ongoing work on the treatment of overload 784 failure. 786 3.1.6. Session Attempt 788 Definition: 789 A SIP request sent by the EA that has not received a final 790 response. 792 Discussion: 793 The attempted session may be Invite Initiated or Non-invite 794 Initiated. When counting the number of session attempts we 795 include all INVITEs that are rejected for lack of authentication 796 information. The EA needs to record the total number of session 797 attempts including those attempts that are routinely rejected by a 798 proxy that requires the UA to authenticate itself. The EA is 799 provisioned to deliver a specific number of session attempts per 800 second. But the EA must also count the actual number of session 801 attempts per given tie interval. 803 Measurement Units: 804 N/A. 806 Issues: 807 None. 809 See Also: 810 Session 811 Session Attempt Rate 812 Invite-initiated Session 813 Non-Invite initiated Session 815 3.1.7. Established Session 817 Definition: 818 A SIP session for which the EA acting as the UE/UA has received a 819 200 OK message. 821 Discussion: 822 An Established Session MAY be Invite Initiated or Non-invite 823 Initiated. 825 Measurement Units: 826 N/A. 828 Issues: 829 None. 831 See Also: 832 Invite-initiated Session 833 Session Attempting State 834 Session Disconnecting State 836 3.1.8. Invite-initiated Session (IS) 838 Definition: 839 A Session that is created by an exchange of messages in the 840 Signaling Plane, the first of which is a SIP INVITE request. 842 Discussion: 843 When an IS becomes an Established Session its signaling component 844 is identified by the SIP dialog parameter values, Call-ID, To-tag, 845 and From-tag (RFC3261 [RFC3261]). An IS may have zero, one or 846 multiple Associated Media descriptions in the SDP body. The 847 inclusion of media is test case dependent. An IS is successfully 848 established if the following two conditions are met: 849 1. Sess.sig is established by the end of Establishment Threshold 850 Time (c.f. Section 3.3.3), and 851 2. If a media session is described in the SDP body of the 852 signaling message, then the media session is established by 853 the end of Establishment Threshold Time (c.f. Section 3.3.3). 854 An SBC or B2BUA may receive media from a calling or called 855 party before a signaling dialog is established and certainly 856 before a confirmed dialog is established. The EA can be built 857 in such a way that it does not send early media or it needs to 858 include a parameter that indicates when it will send media. 859 This parameter must be included in the list of test setup 860 parameters in Section 5.1 of [I-D.ietf-bmwg-sip-bench-meth] 862 Measurement Units: 863 N/A. 865 Issues: 866 None. 868 See Also: 869 Session 870 Non-Invite initiated Session 871 Associated Media 873 3.1.9. Non-INVITE-initiated Session (NS) 875 Definition: 876 A session that is created by an exchange of SIP messages in the 877 Signaling Plane the first of which is not a SIP INVITE message. 879 Discussion: 880 An NS is successfully established if the Session Attempt via a 881 non- INVITE request results in the EA receiving a 2xx reply before 882 the expiration of the Establishment Threshold timer (c.f., 883 Section 3.3.3). An example of a NS is a session created by the 884 SUBSCRIBE request. 886 Measurement Units: 887 N/A. 889 Issues: 890 None. 892 See Also: 893 Session 894 Invite-initiated Session 896 3.1.10. Session Attempt Failure 898 Definition: 899 A session attempt that does not result in an Established Session. 901 Discussion: 902 The session attempt failure may be indicated by the following 903 observations at the EA: 904 1. Receipt of a SIP 4xx, 5xx, or 6xx class response to a Session 905 Attempt. 906 2. The lack of any received SIP response to a Session Attempt 907 within the Establishment Threshold Time (c.f. Section 3.3.3). 909 Measurement Units: 910 N/A. 912 Issues: 913 None. 915 See Also: 916 Session Attempt 918 3.1.11. Standing Sessions Count 920 Definition: 921 The number of Sessions currently established on the DUT/SUT at any 922 instant. 924 Discussion: 925 The number of Standing Sessions is influenced by the Session 926 Duration and the Session Attempt Rate. Benchmarks MUST be 927 reported with the maximum and average Standing Sessions for the 928 DUT/SUT for the duration of the test. In order to determine the 929 maximum and average Standing Sessions on the DUT/SUT for the 930 duration of the test it is necessary to make periodic measurements 931 of the number of Standing Sessions on the DUT/SUT. The 932 recommended value for the measurement period is 1 second. Since 933 we cannot directly poll the DUT/SUT, we take the number of 934 standing sessions on the DUT/SUT to be the number of distinct 935 calls as measured by the number of distinct Call-IDs that the EA 936 is processing at the time of measurement. The EA must make that 937 count available for viewing and recording. 939 Measurement Units: 940 Number of sessions 942 Issues: 943 None. 945 See Also: 946 Session Duration 947 Session Attempt Rate 948 Session Attempt Rate 949 Emulated Agent 951 3.2. Test Components 952 3.2.1. Emulated Agent 954 Definition: 955 A device in the test topology that initiates/responds to SIP 956 messages as one or more session endpoints and, wherever 957 applicable, sources/receives Associated Media for Established 958 Sessions. 960 Discussion: 961 The EA functions in the Signaling and Media Planes. The Tester 962 may act as multiple EAs. 964 Measurement Units: 965 N/A 967 Issues: 968 None. 970 See Also: 971 Media Plane 972 Signaling Plane 973 Established Session 974 Associated Media 976 3.2.2. Signaling Server 978 Definition: 979 Device in the test topology that acts to create sessions between 980 EAs. This device is either a DUT or a component of a SUT. 982 Discussion: 983 The DUT MUST be an RFC 3261 capable network equipment such as a 984 Registrar, Redirect Server, User Agent Server, Stateless Proxy, or 985 Stateful Proxy. A DUT MAY also include B2BUA or SBC. 987 Measurement Units: 988 NA 990 Issues: 991 None. 993 See Also: 994 Signaling Plane 996 3.2.3. SIP-Aware Stateful Firewall 997 Definition: 998 Device in the test topology that provides protection against 999 various types of security threats to which the Signaling and Media 1000 Planes of the EAs and Signaling Server are vulnerable. 1002 Discussion: 1003 Threats may include Denial-of-Service, theft of service and misuse 1004 of service. The SIP-Aware Stateful Firewall MAY be an internal 1005 component or function of the Session Server. The SIP-Aware 1006 Stateful Firewall MAY be a standalone device. If it is a 1007 standalone device it MUST be paired with a Signaling Server. If 1008 it is a standalone device it MUST be benchmarked as part of a SUT. 1009 SIP-Aware Stateful Firewalls MAY include Network Address 1010 Translation (NAT) functionality. Ideally, the inclusion of the 1011 SIP-Aware Stateful Firewall in the SUT does not lower the measured 1012 values of the performance benchmarks. 1014 Measurement Units: 1015 N/A 1017 Issues: 1018 None. 1020 See Also: 1022 3.2.4. SIP Transport Protocol 1024 Definition: 1025 The protocol used for transport of the Signaling Plane messages. 1027 Discussion: 1028 Performance benchmarks may vary for the same SIP networking device 1029 depending upon whether TCP, UDP, TLS, SCTP, or another transport 1030 layer protocol is used. For this reason it MAY be necessary to 1031 measure the SIP Performance Benchmarks using these various 1032 transport protocols. Performance Benchmarks MUST report the SIP 1033 Transport Protocol used to obtain the benchmark results. 1035 Measurement Units: 1036 TCP,UDP, SCTP, TLS over TCP, TLS over UDP, or TLS over SCTP 1038 Issues: 1039 None. 1041 See Also: 1043 3.3. Test Setup Parameters 1045 3.3.1. Session Attempt Rate 1047 Definition: 1048 Configuration of the EA for the number of sessions per second that 1049 the EA attempts to establish using the services of the DUT/SUT. 1051 Discussion: 1052 The Session Attempt Rate is the number of sessions per second that 1053 the EA sends toward the DUT/SUT. Some of the sessions attempted 1054 may not result in a session being established. A session in this 1055 case may be either an IS or an NS. 1057 Measurement Units: 1058 Session attempts per second 1060 Issues: 1061 None. 1063 See Also: 1064 Session 1065 Session Attempt 1067 3.3.2. IS Media Attempt Rate 1069 Definition: 1070 Configuration on the EA for the rate, measured in sessions per 1071 second, at which the EA attempts to establish INVITE-initiated 1072 sessions with Associated Media, using the services of the DUT/SUT. 1074 Discussion: 1075 An IS is not required to include a media description. The IS 1076 Media Attempt Rate defines the number of media sessions we are 1077 trying to create, not the number of media sessions that are 1078 actually created. Some attempts might not result in successful 1079 sessions established on the DUT. 1081 Measurement Units: 1082 session attempts per second (saps) 1084 Issues: 1086 None. 1088 See Also: 1089 IS 1091 3.3.3. Establishment Threshold Time 1093 Definition: 1094 Configuration of the EA for representing the amount of time that 1095 an EA will wait before declaring a Session Attempt Failure. 1097 Discussion: 1098 This time duration is test dependent. 1099 It is RECOMMENDED that the Establishment Threshold Time value be 1100 set to Timer B (for ISs) or Timer F (for NSs) as specified in RFC 1101 3261, Table 4 [RFC3261]. Following the default value of T1 1102 (500ms) specified in the table and a constant multiplier of 64 1103 gives a value of 32 seconds for this timer (i.e., 500ms * 64 = 1104 32s). 1106 Measurement Units: 1107 seconds 1109 Issues: 1110 None. 1112 See Also: 1113 session establishment failure 1115 3.3.4. Session Duration 1117 Definition: 1118 Configuration of the EA that represents the amount of time that 1119 the SIP dialog is intended to exist between the two EAs associated 1120 with the test. 1122 Discussion: 1123 The time at which the BYE is sent will control the Session 1124 Duration 1125 Normally the Session Duration will be the same as the Media 1126 Session Hold Time. However, it is possible that the dialog 1127 established between the two EAs can support different media 1128 sessions at different points in time. Providing both parameters 1129 allows the testing agency to explore this possibility. 1131 Measurement Units: 1132 seconds 1134 Issues: 1135 None. 1137 See Also: 1138 Media Session Hold Time 1140 3.3.5. Media Packet Size 1142 Definition: 1143 Configuration on the EA for a fixed size of packets used for media 1144 streams. 1146 Discussion: 1147 For a single benchmark test, all sessions use the same size packet 1148 for media streams. The size of packets can cause variation in 1149 performance benchmark measurements. 1151 Measurement Units: 1152 bytes 1154 Issues: 1155 None. 1157 See Also: 1159 3.3.6. Media Offered Load 1161 Definition: 1162 Configuration of the EA for the constant rate of Associated Media 1163 traffic offered by the EA to the DUT/SUT for one or more 1164 Established Sessions of type IS. 1166 Discussion: 1167 The Media Offered Load to be used for a test MUST be reported with 1168 three components: 1169 1. per Associated Media stream; 1170 2. per IS; 1171 3. aggregate. 1172 For a single benchmark test, all sessions use the same Media 1173 Offered Load per Media Stream. There may be multiple Associated 1174 Media streams per IS. The aggregate is the sum of all Associated 1175 Media for all IS. 1177 Measurement Units: 1178 packets per second (pps) 1180 Issues: 1181 None. 1183 See Also: 1184 Established Session 1185 Invite Initiated Session 1186 Associated Media 1188 3.3.7. Media Session Hold Time 1190 Definition: 1191 Parameter configured at the EA, that represents the amount of time 1192 that the Associated Media for an Established Session of type IS 1193 will last. 1195 Discussion: 1196 The Associated Media streams may be bi-directional or uni- 1197 directional as indicated in the test methodology. 1198 Normally the Media Session Hold Time will be the same as the 1199 Session Duration. However, it is possible that the dialog 1200 established between the two EAs can support different media 1201 sessions at different points in time. Providing both parameters 1202 allows the testing agency to explore this possibility. 1204 Measurement Units: 1205 seconds 1207 Issues: 1208 None. 1210 See Also: 1211 Associated Media 1212 Established Session 1213 Invite-initiated Session (IS) 1215 3.3.8. Loop Detection Option 1217 Definition: 1218 An option that causes a Proxy to check for loops in the routing of 1219 a SIP request before forwarding the request. 1221 Discussion: 1222 This is an optional process that a SIP proxy may employ; the 1223 process is described under Proxy Behavior in RFC 3261 [RFC3261] in 1224 Section 16.3 Request Validation and that section also contains 1225 suggestions as to how the option could be implemented. Any 1226 procedure to detect loops will use processor cycles and hence 1227 could impact the performance of a proxy. 1229 Measurement Units: 1230 N/A 1232 Issues: 1233 None. 1235 See Also: 1237 3.3.9. Forking Option 1239 Definition: 1240 An option that enables a Proxy to fork requests to more than one 1241 destination. 1243 Discussion: 1244 This is an process that a SIP proxy may employ to find the UAS. 1245 The option is described under Proxy Behavior in RFC 3261 in 1246 Section 16.1. A proxy that uses forking must maintain state 1247 information and this will use processor cycles and memory. Thus 1248 the use of this option could impact the performance of a proxy and 1249 different implementations could produce different impacts. 1250 SIP supports serial or parallel forking. When performing a test, 1251 the type of forking mode MUST be indicated. 1253 Measurement Units: 1254 The number of endpoints that will receive the forked invitation. 1255 A value of 1 indicates that the request is destined to only one 1256 endpoint, a value of 2 indicates that the request is forked to two 1257 endpoints, and so on. This is an integer value ranging between 1 1258 and N inclusive, where N is the maximum number of endpoints to 1259 which the invitation is sent. 1260 Type of forking used, namely parallel or serial. 1262 Issues: 1263 None. 1265 See Also: 1267 3.4. Benchmarks 1269 3.4.1. Registration Rate 1271 Definition: 1272 The maximum number of registrations that can be successfully 1273 completed by the DUT/SUT in a given time period without 1274 registration failures in that time period. 1276 Discussion: 1277 This benchmark is obtained with zero failure in which 100% of the 1278 registrations attempted by the EA are successfully completed by 1279 the DUT/SUT. The registration rate provisioned on the Emulated 1280 Agent is raised and lowered as described in the algorithm in the 1281 companion methodology draft [I-D.ietf-bmwg-sip-bench-meth] until a 1282 traffic load consisting of registrations at the given attempt rate 1283 over the sustained period of time identified by T in the algorithm 1284 completes without failure. 1286 Measurement Units: 1287 registrations per second (rps) 1289 Issues: 1290 None. 1292 See Also: 1294 3.4.2. Session Establishment Rate 1296 Definition: 1297 The maximum number of sessions that can be successfully completed 1298 by the DUT/SUT in a given time period without session 1299 establishment failures in that time period. 1301 Discussion: 1302 This benchmark is obtained with zero failure in which 100% of the 1303 sessions attempted by the Emulated Agent are successfully 1304 completed by the DUT/SUT. The session attempt rate provisioned on 1305 the EA is raised and lowered as described in the algorithm in the 1306 accompanying methodology document, until a traffic load at the 1307 given attempt rate over the sustained period of time identified by 1308 T in the algorithm completes without any failed session attempts. 1309 Sessions may be IS or NS or a mix of both and will be defined in 1310 the particular test. 1312 Measurement Units: 1313 sessions per second (sps) 1315 Issues: 1316 None. 1318 See Also: 1319 Invite-initiated Sessions 1320 Non-INVITE initiated Sessions 1321 Session Attempt Rate 1323 3.4.3. Session Capacity 1325 Definition: 1326 The maximum value of Standing Sessions Count achieved by the DUT/ 1327 SUT during a time period T in which the EA is sending session 1328 establishment messages at the Session Establishment Rate. 1330 Discussion: 1331 Sessions may be IS or NS. If they are IS they can be with or 1332 without media. When benchmarking Session Capacity for sessions 1333 with media it is required that these sessions be permanently 1334 established, i.e., they remain active for the duration of the 1335 test. In the signaling plane, this requirement means that the 1336 dialog lasts as long as the test lasts. When media is present, 1337 the Media Session Hold Time MUST be set to infinity so that 1338 sessions remain established for the duration of the test. If the 1339 DUT/SUT is dialog-stateful, then we expect its performance will be 1340 impacted by setting Media Session Hold Time to infinity, since the 1341 DUT/SUT will need to allocate resources to process and store the 1342 state information. The report of the Session Capacity must 1343 include the Session Establishment Rate at which it was measured. 1345 Measurement Units: 1346 sessions 1348 Issues: 1349 None. 1351 See Also: 1352 Established Session 1353 Session Attempt Rate 1354 Session Attempt Failure 1356 3.4.4. Session Overload Capacity 1358 Definition: 1359 The maximum number of Established Sessions that can exist 1360 simultaneously on the DUT/SUT until it stops responding to Session 1361 Attempts. 1363 Discussion: 1364 Session Overload Capacity is measured after the Session Capacity 1365 is measured. The Session Overload Capacity is greater than or 1366 equal to the Session Capacity. When benchmarking Session Overload 1367 Capacity, continue to offer Session Attempts to the DUT/SUT after 1368 the first Session Attempt Failure occurs and measure Established 1369 Sessions until there is no SIP message response for the duration 1370 of the Establishment Threshold. Note that the Session 1371 Establishment Performance is expected to decrease after the first 1372 Session Attempt Failure occurs. 1374 Units: 1375 Sessions 1377 Issues: 1378 None. 1380 See Also: 1381 Overload 1382 Session Capacity 1383 Session Attempt Failure 1385 3.4.5. Session Establishment Performance 1387 Definition: 1388 The percent of Session Attempts that become Established Sessions 1389 over the duration of a benchmarking test. 1391 Discussion: 1392 Session Establishment Performance is a benchmark to indicate 1393 session establishment success for the duration of a test. The 1394 duration for measuring this benchmark is to be specified in the 1395 Methodology. The Session Duration SHOULD be configured to 1396 infinity so that sessions remain established for the entire test 1397 duration. 1398 Session Establishment Performance is calculated as shown in the 1399 following equation: 1401 Session Establishment = Total Established Sessions 1402 Performance -------------------------- 1403 Total Session Attempts 1405 Session Establishment Performance may be monitored real-time 1406 during a benchmarking test. However, the reporting benchmark MUST 1407 be based on the total measurements for the test duration. 1409 Measurement Units: 1410 Percent (%) 1412 Issues: 1413 None. 1415 See Also: 1416 Established Session 1417 Session Attempt 1419 3.4.6. Session Attempt Delay 1421 Definition: 1422 The average time measured at the EA for a Session Attempt to 1423 result in an Established Session. 1425 Discussion: 1426 Time is measured from when the EA sends the first INVITE for the 1427 call-ID in the case of an IS. Time is measured from when the EA 1428 sends the first non-INVITE message in the case of an NS. Session 1429 Attempt Delay MUST be measured for every established session to 1430 calculate the average. Session Attempt Delay MUST be measured at 1431 the Session Establishment Rate. 1433 Measurement Units: 1434 Seconds 1436 Issues: 1437 None. 1439 See Also: 1440 Session Establishment Rate 1442 3.4.7. IM Rate 1444 Definition: 1445 Maximum number of IM messages completed by the DUT/SUT. 1447 Discussion: 1448 For a UAS, the definition of success is the receipt of an IM 1449 request and the subsequent sending of a final response. 1451 For a UAC, the definition of success is the sending of an IM 1452 request and the receipt of a final response to it. For a proxy, 1453 the definition of success is as follows: 1454 A. the number of IM requests it receives from the upstream client 1455 MUST be equal to the number of IM requests it sent to the 1456 downstream server; and 1457 B. the number of IM responses it receives from the downstream 1458 server MUST be equal to the number of IM requests sent to the 1459 downstream server; and 1460 C. the number of IM responses it sends to the upstream client 1461 MUST be equal to the number of IM requests it received from 1462 the upstream client. 1464 Measurement Units: 1465 IM messages per second 1467 Issues: 1468 None. 1470 See Also: 1472 4. IANA Considerations 1474 This document requires no IANA considerations. 1476 5. Security Considerations 1478 Documents of this type do not directly affect the security of 1479 Internet or corporate networks as long as benchmarking is not 1480 performed on devices or systems connected to production networks. 1481 Security threats and how to counter these in SIP and the media layer 1482 is discussed in RFC3261 [RFC3261], RFC 3550 [RFC3550], RFC3711 1483 [RFC3711] and various other drafts. This document attempts to 1484 formalize a set of common terminology for benchmarking SIP networks. 1485 Packets with unintended and/or unauthorized DSCP or IP precedence 1486 values may present security issues. Determining the security 1487 consequences of such packets is out of scope for this document. 1489 6. Acknowledgments 1491 The authors would like to thank Keith Drage, Cullen Jennings, Daryl 1492 Malas, Al Morton, and Henning Schulzrinne for invaluable 1493 contributions to this document. Dale Worley provided an extensive 1494 review that lead to improvements in the documents. We are grateful 1495 to Barry Constantine for providing valuable comments during the 1496 document's WGLC. 1498 7. References 1500 7.1. Normative References 1502 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1503 Requirement Levels", BCP 14, RFC 2119, March 1997. 1505 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 1506 Network Interconnect Devices", RFC 2544, March 1999. 1508 [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, 1509 A., Peterson, J., Sparks, R., Handley, M., and E. 1510 Schooler, "SIP: Session Initiation Protocol", RFC 3261, 1511 June 2002. 1513 [I-D.ietf-bmwg-sip-bench-meth] 1514 Davids, C., Gurbani, V., and S. Poretsky, "Methodology for 1515 Benchmarking SIP Networking Devices", 1516 draft-ietf-bmwg-sip-bench-meth-08 (work in progress), 1517 January 2013. 1519 7.2. Informational References 1521 [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN 1522 Switching Devices", RFC 2285, February 1998. 1524 [RFC1242] Bradner, S., "Benchmarking terminology for network 1525 interconnection devices", RFC 1242, July 1991. 1527 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 1528 Jacobson, "RTP: A Transport Protocol for Real-Time 1529 Applications", STD 64, RFC 3550, July 2003. 1531 [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 1532 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 1533 RFC 3711, March 2004. 1535 [RFC6357] Hilt, V., Noel, E., Shen, C., and A. Abdelal, "Design 1536 Considerations for Session Initiation Protocol (SIP) 1537 Overload Control", RFC 6357, August 2011. 1539 [I-D.ietf-soc-overload-control] 1540 Gurbani, V., Hilt, V., and H. Schulzrinne, "Session 1541 Initiation Protocol (SIP) Overload Control", 1542 draft-ietf-soc-overload-control-11 (work in progress), 1543 November 2012. 1545 Appendix A. White Box Benchmarking Terminology 1547 Session Attempt Arrival Rate 1549 Definition: 1550 The number of Session Attempts received at the DUT/SUT over a 1551 specified time period. 1553 Discussion: 1554 Sessions Attempts are indicated by the arrival of SIP INVITES OR 1555 SUBSCRIBE NOTIFY messages. Session Attempts Arrival Rate 1556 distribution can be any model selected by the user of this 1557 document. It is important when comparing benchmarks of different 1558 devices that same distribution model was used. Common 1559 distributions are expected to be Uniform and Poisson. 1561 Measurement Units: 1562 Session attempts/sec 1564 Issues: 1565 None. 1567 See Also: 1568 Session Attempt 1570 Authors' Addresses 1572 Carol Davids 1573 Illinois Institute of Technology 1574 201 East Loop Road 1575 Wheaton, IL 60187 1576 USA 1578 Phone: +1 630 682 6024 1579 Email: davids@iit.edu 1580 Vijay K. Gurbani 1581 Bell Laboratories, Alcatel-Lucent 1582 1960 Lucent Lane 1583 Rm 9C-533 1584 Naperville, IL 60566 1585 USA 1587 Phone: +1 630 224 0216 1588 Email: vkg@bell-labs.com 1590 Scott Poretsky 1591 Allot Communications 1592 300 TradeCenter, Suite 4680 1593 Woburn, MA 08101 1594 USA 1596 Phone: +1 508 309 2179 1597 Email: sporetsky@allot.com