idnits 2.17.1 draft-ietf-bmwg-sip-bench-term-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 496 has weird spacing: '...UTs and hop- ...' -- The document date (November 8, 2012) is 4186 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Downref: Normative reference to an Informational RFC: RFC 2544 == Outdated reference: A later version (-12) exists of draft-ietf-bmwg-sip-bench-meth-05 ** Downref: Normative reference to an Informational draft: draft-ietf-bmwg-sip-bench-meth (ref. 'I-D.ietf-bmwg-sip-bench-meth') == Outdated reference: A later version (-15) exists of draft-ietf-soc-overload-control-10 Summary: 2 errors (**), 0 flaws (~~), 5 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Methodology Working C. Davids 3 Group Illinois Institute of Technology 4 Internet-Draft V. Gurbani 5 Expires: May 12, 2013 Bell Laboratories, Alcatel-Lucent 6 S. Poretsky 7 Allot Communications 8 November 8, 2012 10 Terminology for Benchmarking Session Initiation Protocol (SIP) 11 Networking Devices 12 draft-ietf-bmwg-sip-bench-term-06 14 Abstract 16 This document provides a terminology for benchmarking the SIP 17 performance of networking devices. The term performance in this 18 context means the capacity of the device- or system-under-test to 19 process SIP messages. Terms are included for test components, test 20 setup parameters, and performance benchmark metrics for black-box 21 benchmarking of SIP networking devices. The performance benchmark 22 metrics are obtained for the SIP signaling plane only. The terms are 23 intended for use in a companion methodology document for 24 characterizing the performance of a SIP networking device under a 25 variety of conditions. The intent of the two documents is to enable 26 a comparison of the capacity of SIP networking devices. Test setup 27 parameters and a methodology document are necessary because SIP 28 allows a wide range of configuration and operational conditions that 29 can influence performance benchmark measurements. A standard 30 terminology and methodology will ensure that benchmarks have 31 consistent definition and were obtained following the same 32 procedures. 34 Status of this Memo 36 This Internet-Draft is submitted in full conformance with the 37 provisions of BCP 78 and BCP 79. 39 Internet-Drafts are working documents of the Internet Engineering 40 Task Force (IETF). Note that other groups may also distribute 41 working documents as Internet-Drafts. The list of current Internet- 42 Drafts is at http://datatracker.ietf.org/drafts/current/. 44 Internet-Drafts are draft documents valid for a maximum of six months 45 and may be updated, replaced, or obsoleted by other documents at any 46 time. It is inappropriate to use Internet-Drafts as reference 47 material or to cite them other than as "work in progress." 48 This Internet-Draft will expire on May 12, 2013. 50 Copyright Notice 52 Copyright (c) 2012 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents 57 (http://trustee.ietf.org/license-info) in effect on the date of 58 publication of this document. Please review these documents 59 carefully, as they describe your rights and restrictions with respect 60 to this document. Code Components extracted from this document must 61 include Simplified BSD License text as described in Section 4.e of 62 the Trust Legal Provisions and are provided without warranty as 63 described in the Simplified BSD License. 65 Table of Contents 67 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 68 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 6 69 2.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 7 70 2.2. Benchmarking Models . . . . . . . . . . . . . . . . . . . 9 71 3. Term Definitions . . . . . . . . . . . . . . . . . . . . . . . 14 72 3.1. Protocol Components . . . . . . . . . . . . . . . . . . . 14 73 3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 14 74 3.1.2. Signaling Plane . . . . . . . . . . . . . . . . . . . 17 75 3.1.3. Media Plane . . . . . . . . . . . . . . . . . . . . . 18 76 3.1.4. Associated Media . . . . . . . . . . . . . . . . . . . 18 77 3.1.5. Overload . . . . . . . . . . . . . . . . . . . . . . . 19 78 3.1.6. Session Attempt . . . . . . . . . . . . . . . . . . . 20 79 3.1.7. Established Session . . . . . . . . . . . . . . . . . 20 80 3.1.8. Invite-initiated Session (IS) . . . . . . . . . . . . 21 81 3.1.9. Non-INVITE-initiated Session (NS) . . . . . . . . . . 22 82 3.1.10. Session Attempt Failure . . . . . . . . . . . . . . . 22 83 3.1.11. Standing Sessions Count . . . . . . . . . . . . . . . 23 84 3.2. Test Components . . . . . . . . . . . . . . . . . . . . . 23 85 3.2.1. Emulated Agent . . . . . . . . . . . . . . . . . . . . 24 86 3.2.2. Signaling Server . . . . . . . . . . . . . . . . . . . 24 87 3.2.3. SIP-Aware Stateful Firewall . . . . . . . . . . . . . 24 88 3.2.4. SIP Transport Protocol . . . . . . . . . . . . . . . . 25 89 3.3. Test Setup Parameters . . . . . . . . . . . . . . . . . . 26 90 3.3.1. Session Attempt Rate . . . . . . . . . . . . . . . . . 26 91 3.3.2. IS Media Attempt Rate . . . . . . . . . . . . . . . . 26 92 3.3.3. Establishment Threshold Time . . . . . . . . . . . . . 27 93 3.3.4. Session Duration . . . . . . . . . . . . . . . . . . . 27 94 3.3.5. Media Packet Size . . . . . . . . . . . . . . . . . . 28 95 3.3.6. Media Offered Load . . . . . . . . . . . . . . . . . . 28 96 3.3.7. Media Session Hold Time . . . . . . . . . . . . . . . 29 97 3.3.8. Loop Detection Option . . . . . . . . . . . . . . . . 29 98 3.3.9. Forking Option . . . . . . . . . . . . . . . . . . . . 30 99 3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 31 100 3.4.1. Registration Rate . . . . . . . . . . . . . . . . . . 31 101 3.4.2. Session Establishment Rate . . . . . . . . . . . . . . 31 102 3.4.3. Session Capacity . . . . . . . . . . . . . . . . . . . 32 103 3.4.4. Session Overload Capacity . . . . . . . . . . . . . . 33 104 3.4.5. Session Establishment Performance . . . . . . . . . . 33 105 3.4.6. Session Attempt Delay . . . . . . . . . . . . . . . . 34 106 3.4.7. IM Rate . . . . . . . . . . . . . . . . . . . . . . . 34 107 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 35 108 5. Security Considerations . . . . . . . . . . . . . . . . . . . 35 109 6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 36 110 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 36 111 7.1. Normative References . . . . . . . . . . . . . . . . . . . 36 112 7.2. Informational References . . . . . . . . . . . . . . . . . 36 114 Appendix A. White Box Benchmarking Terminology . . . . . . . . . 37 115 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 37 117 1. Terminology 119 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 120 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 121 document are to be interpreted as described in BCP 14, RFC2119 122 [RFC2119]. RFC 2119 defines the use of these key words to help make 123 the intent of standards track documents as clear as possible. While 124 this document uses these keywords, this document is not a standards 125 track document. The term Throughput is defined in RFC2544 [RFC2544]. 127 For the sake of clarity and continuity, this document adopts the 128 template for definitions set out in Section 2 of RFC 1242 [RFC1242]. 130 The terms Device Under Test (DUT) and System Under Test (SUT) are 131 defined in the following BMWG documents: 133 Device Under Test (DUT) (c.f., Section 3.1.1 RFC 2285 [RFC2285]). 134 System Under Test (SUT) (c.f., Section 3.1.2, RFC 2285 [RFC2285]). 136 Many commonly used SIP terms in this document are defined in RFC 3261 137 [RFC3261]. For convenience the most important of these are 138 reproduced below. Use of these terms in this document is consistent 139 with their corresponding definition in [RFC3261]. 140 o Call Stateful: A proxy is call stateful if it retains state for a 141 dialog from the initiating INVITE to the terminating BYE request. 142 A call stateful proxy is always transaction stateful, but the 143 converse is not necessarily true. 144 o Stateful Proxy: A logical entity that maintains the client and 145 server transaction state machines defined by this specification 146 during the processing of a request, also known as a transaction 147 stateful proxy. The behavior of a stateful proxy is further 148 defined in Section 16 of RFC 3261 [RFC3261] . A transaction 149 stateful proxy is not the same as a call stateful proxy. 150 o Stateless Proxy: A logical entity that does not maintain the 151 client or server transaction state machines defined in this 152 specification when it processes requests. A stateless proxy 153 forwards every request it receives downstream and every response 154 it receives upstream. 155 o Back-to-back User Agent: A back-to-back user agent (B2BUA) is a 156 logical entity that receives a request and processes it as a user 157 agent server (UAS). In order to determine how the request should 158 be answered, it acts as a user agent client (UAC) and generates 159 requests. Unlike a proxy server, it maintains dialog state and 160 must participate in all requests sent on the dialogs it has 161 established. Since it is a concatenation of a UAC and a UAS, no 162 explicit definitions are needed for its behavior. 164 o Loop: A request that arrives at a proxy, is forwarded, and later 165 arrives back at the same proxy. When it arrives the second time, 166 its Request-URI is identical to the first time, and other header 167 fields that affect proxy operation are unchanged, so that the 168 proxy will make the same processing decision on the request it 169 made the first time. Looped requests are errors, and the 170 procedures for detecting them and handling them are described by 171 the SIP protocol[RFC3261] and also by RFC 5393 173 2. Introduction 175 Service Providers and IT Organizations deliver Voice Over IP (VoIP) 176 and Multimedia network services based on the IETF Session Initiation 177 Protocol (SIP) [RFC3261]. SIP is a signaling protocol originally 178 intended to be used to dynamically establish, disconnect and modify 179 streams of media between end users. As it has evolved it has been 180 adopted for use in a growing number of services and applications. 181 Many of these result in the creation of a media session, but some do 182 not. Examples of this latter group include text messaging and 183 subscription services. The set of benchmarking terms provided in 184 this document is intended for use with any SIP-enabled device 185 performing SIP functions in the interior of the network, whether or 186 not these result in the creation of media sessions. The performance 187 of end-user devices is outside the scope of this document. 189 A number of networking devices have been developed to support SIP- 190 based VoIP services. These include SIP Servers, Session Border 191 Controllers (SBC), Back-to-back User Agents (B2BUA), and SIP-Aware 192 Stateful Firewalls. These devices contain a mix of voice and IP 193 functions whose performance may be reported using metrics defined by 194 the equipment manufacturer or vendor. The Service Provider or IT 195 Organization seeking to compare the performance of such devices will 196 not be able to do so using these vendor-specific metrics, whose 197 conditions of test and algorithms for collection are often 198 unspecified. SIP functional elements and the devices that include 199 them can be configured many different ways and can be organized into 200 various topologies. These configuration and topological choices 201 impact the value of any chosen signaling benchmark. Unless these 202 conditions-of-test are defined, a true comparison of performance 203 metrics will not be possible. Some SIP-enabled network devices 204 terminate or relay media as well as signaling. The processing of 205 media by the device impacts the signaling performance. As a result, 206 the conditions-of-test must include information as to whether or not 207 the device under test processes media and if the device does process 208 media, a description of the media handled and the manner in which it 209 is handled. This document and its companion methodology document 210 [I-D.ietf-bmwg-sip-bench-meth] provide a set of black-box benchmarks 211 for describing and comparing the performance of devices that 212 incorporate the SIP User Agent Client and Server functions and that 213 operate in the network's core. 215 The definition of SIP performance benchmarks necessarily includes 216 definitions of Test Setup Parameters and a test methodology. These 217 enable the Tester to perform benchmarking tests on different devices 218 and to achieve comparable results. This document provides a common 219 set of definitions for Test Components, Test Setup Parameters, and 220 Benchmarks. All the benchmarks defined are black-box measurements of 221 the SIP signaling plane. The Test Setup Parameters and Benchmarks 222 defined in this document are intended for use with the companion 223 Methodology document. Benchmarks of internal DUT characteristics 224 (also known as white-box benchmarks) such as Session Attempt Arrival 225 Rate, which is measured at the DUT, are described in Appendix A to 226 allow additional characterization of DUT behavior with different 227 distribution models. 229 2.1. Scope 231 The scope of this work item is summarized as follows: 232 o This terminology document describes SIP signaling performance 233 benchmarks for black-box measurements of SIP networking devices. 234 Stress and debug scenarios are not addressed in this work item. 235 o The DUT must be an RFC 3261 capable network equipment. This may 236 be a Registrar, Redirect Server, Stateless Proxy or Stateful 237 Proxy. A DUT MAY also include a B2BUA, SBC functionality. The 238 DUT MAY be a multi-port SIP-to-switched network gateway 239 implemented as a SIP UAC or UAS. 240 o The DUT MAY include an internal SIP Application Level Gateway 241 (ALG), firewall, and/or a Network Address Translator (NAT). This 242 is referred to as the "SIP Aware Stateful Firewall." 243 o The DUT or SUT MUST NOT be end user equipment, such as personal 244 digital assistant, a computer-based client, or a user terminal. 245 o The Tester acts as multiple "Emulated Agents" (EA) that initiate 246 (or respond to) SIP messages as session endpoints and source (or 247 receive) associated media for established connections. 248 o SIP Signaling in presence of Media 249 * The media performance is not benchmarked in this work item. 250 * It is RECOMMENDED that SIP signaling plane benchmarks be 251 performed with media present, but this is optional. 252 * The SIP INVITE requests MUST include the SDP body. 253 * The type of DUT dictates whether the associated media streams 254 traverse the DUT or SUT. Both scenarios are within the scope 255 of this work item. 256 * SIP is frequently used to create media streams; the signaling 257 plane and media plane are treated as orthogonal to each other 258 in this document. While many devices support the creation of 259 media streams, benchmarks that measure the performance of these 260 streams are outside the scope of this document and its 261 companion methodology document [I-D.ietf-bmwg-sip-bench-meth]. 262 Tests may be performed with or without the creation of media 263 streams. The presence or absence of media streams MUST be 264 noted as a condition of the test as the performance of SIP 265 devices may vary accordingly. Even if the media is used during 266 benchmarking, only the SIP performance will be benchmarked, not 267 the media performance or quality. 268 o Both INVITE and non-INVITE scenarios (such as Instant Messages or 269 IM) are addressed in this document. However, benchmarking SIP 270 presence is not a part of this work item. 271 o Different transport mechanisms -- such as UDP, TCP, SCTP, or TLS 272 -- may be used. The specific transport mechanism MUST be noted as 273 a condition of the test as the performance of SIP devices may vary 274 accordingly. 275 o Looping and forking options are also considered since they impact 276 processing at SIP proxies. 277 o REGISTER and INVITE requests may be challenged or remain 278 unchallenged for authentication purpose. Whether or not the 279 REGISTER and INVITE requests are challenged is a condition of test 280 which will be recorded along with other such parameters which may 281 impact the SIP performance of the device or system under test. 282 o Re-INVITE requests are not considered in scope of this work item 283 since the benchmarks for INVITEs are based on the dialog created 284 by the INVITE and not on the transactions that take place within 285 that dialog. 286 o Only session establishment is considered for the performance 287 benchmarks. Session disconnect is not considered in the scope of 288 this work item. This is because our goal is to determine the 289 maximum throughput of the device or system under test, that is the 290 number of simultaneous SIP sessions that the device or system can 291 support. It is true that there are BYE requests being created 292 during the test process. These transactions do contribute to the 293 load on the device or system under test and thus are accounted for 294 in the metric we derive. We do not seek a separate metric for the 295 number of BYE transactions a device or system can support. 296 o SIP Overload [I-D.ietf-soc-overload-design] is within the scope of 297 this work item. We test to failure and then can continue to 298 observe and record the behavior of the system after failures are 299 recorded. The cause of failure is not within the scope of this 300 work. We note the failure and may continue to test until a 301 different failure or condition is encountered. Considerations on 302 how to handle overload are deferred to work progressing in the SOC 303 working group [I-D.ietf-soc-overload-control]. Vendors are, of 304 course, free to implement their specific overload control behavior 305 as the expected test outcome if it is different from the IETF 306 recommendations. However, such behavior MUST be documented and 307 interpreted appropriately across multiple vendor implementations. 308 This will make it more meaningful to compare the performance of 309 different SIP overload implementations. 310 o IMS-specific scenarios are not considered, but test cases can be 311 applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT. 313 2.2. Benchmarking Models 315 This section shows ten models to be used when benchmarking SIP 316 performance of a networking device. Figure 1 shows shows the 317 configuration needed to benchmark the tester itself. This model will 318 be used to establish the limitations of the test apparatus. 320 +--------+ Signaling request +--------+ 321 | +----------------------------->| | 322 | Tester | | Tester | 323 | EA | Signaling response | EA | 324 | |<-----------------------------+ | 325 +--------+ +--------+ 326 /|\ /|\ 327 | Media | 328 +=========================================+ 330 Figure 1: Baseline performance of the Emulated Agent without a DUT 331 present 333 Figure 2 shows the DUT playing the role of a user agent client (UAC), 334 initiating requests and absorbing responses. This model can be used 335 to baseline the performance of the DUT acting as an UAC without 336 associated media. 338 +--------+ Signaling request +--------+ 339 | +----------------------------->| | 340 | DUT | | Tester | 341 | | Signaling response | EA | 342 | |<-----------------------------+ | 343 +--------+ +--------+ 345 Figure 2: Baseline performance for DUT acting as a user agent client 346 without associated media 348 Figure 3 shows the DUT plays the role of a user agent server (UAS), 349 absorbing the requests and sending responses. This model can be used 350 as a baseline performance for the DUT acting as a UAS without 351 associated media. 353 +--------+ Signaling request +--------+ 354 | +----------------------------->| | 355 | Tester | | DUT | 356 | EA | Response | | 357 | |<-----------------------------+ | 358 +--------+ +--------+ 360 Figure 3: Baseline performance for DUT acting as a user agent server 361 without associated media 363 Figure 4 shows the DUT plays the role of a user agent client (UAC), 364 initiating requests and absorbing responses. This model can be used 365 as a baseline performance for the DUT acting as a UAC with associated 366 media. 368 +--------+ Signaling request +--------+ 369 | +----------------------------->| | 370 | DUT | | Tester | 371 | | Signaling response | (EA) | 372 | |<-----------------------------+ | 373 | |<============ Media =========>| | 374 +--------+ +--------+ 376 Figure 4: Baseline performance for DUT acting as a user agent client 377 with associated media 379 Figure 5 shows the DUT plays the role of a user agent server (UAS), 380 absorbing the requests and sending responses. This model can be used 381 as a baseline performance for the DUT acting as a UAS with associated 382 media. 384 +--------+ Signaling request +--------+ 385 | +----------------------------->| | 386 | Tester | | DUT | 387 | (EA) | Response | | 388 | |<-----------------------------+ | 389 | |<============ Media =========>| | 390 +--------+ +--------+ 392 Figure 5: Baseline performance for DUT acting as a user agent server 393 with associated media 395 Figure 6 shows that the Tester acts as the initiating and responding 396 EA as the DUT/SUT forwards Session Attempts. 398 +--------+ Session +--------+ Session +--------+ 399 | | Attempt | | Attempt | | 400 | |<------------+ |<------------+ | 401 | | | | | | 402 | | Response | | Response | | 403 | Tester +------------>| DUT +------------>| Tester | 404 | (EA) | | | | (EA) | 405 | | | | | | 406 +--------+ +--------+ +--------+ 408 Figure 6: DUT/SUT performance benchmark for session establishment 409 without media 411 Figure 7 is used when performing those same benchmarks with 412 Associated Media traversing the DUT/SUT. 414 +--------+ Session +--------+ Session +--------+ 415 | | Attempt | | Attempt | | 416 | |<------------+ |<------------+ | 417 | | | | | | 418 | | Response | | Response | | 419 | Tester +------------>| DUT +------------>| Tester | 420 | (EA) | | | | (EA) | 421 | | Media | | Media | | 422 | |<===========>| |<===========>| | 423 +--------+ +--------+ +--------+ 425 Figure 7: DUT/SUT performance benchmark for session establishment 426 with media traversing the DUT 428 Figure 8 is to be used when performing those same benchmarks with 429 Associated Media, but the media does not traverse the DUT/SUT. 430 Again, the benchmarking of the media is not within the scope of this 431 work item. The SIP control signaling is benchmarked in the presence 432 of Associated Media to determine if the SDP body of the signaling and 433 the handling of media impacts the performance of the DUT/SUT. 435 +--------+ Session +--------+ Session +--------+ 436 | | Attempt | | Attempt | | 437 | |<------------+ |<------------+ | 438 | | | | | | 439 | | Response | | Response | | 440 | Tester +------------>| DUT +------------>| Tester | 441 | (EA) | | | | (EA) | 442 | | | | | | 443 +--------+ +--------+ +--------+ 444 /|\ /|\ 445 | Media | 446 +=============================================+ 448 Figure 8: DUT/SUT performance benchmark for session establishment 449 with media external to the DUT 451 Figure 9 is used when performing benchmarks that require one or more 452 intermediaries to be in the signaling path. The intent is to gather 453 benchmarking statistics with a series of DUTs in place. In this 454 topology, the media is delivered end-to-end and does not traverse the 455 DUT. 457 SUT 458 ------------------^^^^^^^^------------- 459 / \ 460 +------+ Session +---+ Session +---+ Session +------+ 461 | | Attempt | | Attempt | | Attempt | | 462 | |<---------+ |<---------+ |<---------+ | 463 | | | | | | | | 464 | | Response | | Response | | Response | | 465 |Tester+--------->|DUT+--------->|DUT|--------->|Tester| 466 | (EA) | | | | | | (EA) | 467 | | | | | | | | 468 +------+ +---+ +---+ +------+ 469 /|\ /|\ 470 | Media | 471 +=============================================+ 473 Figure 9: DUT/SUT performance benchmark for session establishment 474 with multiple DUTs and end-to-end media 476 Figure 10 is used when performing benchmarks that require one or more 477 intermediaries to be in the signaling path. The intent is to gather 478 benchmarking statistics with a series of DUTs in place. In this 479 topology, the media is delivered hop-by-hop through each DUT. 481 SUT 482 -----------------^^^^^^^^------------- 483 / \ 484 +------+ Session +---+ Session +---+ Session +------+ 485 | | Attempt | | Attempt | | Attempt | | 486 | |<---------+ |<---------+ |<---------+ | 487 | | | | | | | | 488 | | Response | | Response | | Response | | 489 |Tester+--------->|DUT+--------->|DUT|--------->|Tester| 490 | (EA) | | | | | | (EA) | 491 | | | | | | | | 492 | |<========>| |<========>| |<========>| | 493 +------+ Media +---+ Media +---+ Media +------+ 495 Figure 10: DUT/SUT performance benchmark for session establishment 496 with multiple DUTs and hop- by-hop media 498 Figure 11 illustrates the SIP signaling for an Established Session. 499 The Tester acts as the EAs and initiates a Session Attempt with the 500 DUT/SUT. When the Emulated Agent (EA) receives a 200 OK from the 501 DUT/SUT that session is considered to be an Established Session. The 502 illustration indicates three states of the session bring created by 503 the EA - Attempting, Established, and Disconnecting. Sessions can be 504 one of two type: Invite-Initiated Session (IS) or Non-Invite 505 Initiated Session (NS). Failure for the DUT/SUT to successfully 506 respond within the Establishment Threshold Time is considered a 507 Session Attempt Failure. SIP Invite messages MUST include the SDP 508 body to specify the Associated Media. Use of Associated Media, to be 509 sourced from the EA, is optional. When Associated Media is used, it 510 may traverse the DUT/SUT depending upon the type of DUT/SUT. The 511 Associated Media is shown in Figure 11 as "Media" connected to media 512 ports M1 and M2 on the EA. After the EA sends a BYE, the session 513 disconnects. Performance test cases for session disconnects are not 514 considered in this work item (the BYE request is shown for 515 completeness.) 516 EA DUT/SUT M1 M2 517 | | | | 518 | INVITE | | | 519 ---------+-------------->| | | 520 | | | | 521 Attempting | | | 522 | 200 OK | | | 523 ---------+<--------------| | | 524 | ACK | | | 525 |-------------->| | | 526 | | | | 527 | | | | 528 | | | Media | 529 Established | |<=====>| 530 | | | | 531 | BYE | | | 532 --------+--------------> | | | 533 | | | | 534 Disconnecting | | | 535 | 200 OK | | | 536 --------|<-------------- | | | 537 | | | | 539 Figure 11: Invite-initiated Session States 541 3. Term Definitions 543 3.1. Protocol Components 545 3.1.1. Session 547 Definition: 548 The combination of signaling and media messages and processes that 549 support a SIP-based service. 551 Discussion: 552 SIP messages are used to create and manage services for end users. 553 Often, these services include the creation of media streams that 554 are defined in the SDP body of a SIP message and carried in RTP 555 protocol data units. However, SIP messages can also be used to 556 create Instant Message services and subscription services, and 557 such services are not associated with media streams. SIP reserves 558 the term "session" to describe services that are analogous to 559 telephone calls on a circuit switched network. SIP reserves the 560 term "dialog" to refer to a signaling-only relationship between 561 User Agent peers. SIP reserves the term "transaction" to refer to 562 the brief communication between a client and a server that lasts 563 only until the final response to the SIP request. None of these 564 terms describes the entity whose performance we want to benchmark. 565 For example, the MESSAGE request does not create a dialog and can 566 be sent either within or outside of a dialog. It is not 567 associated with media, but it resembles a phone call in its 568 dependence on human rather than machine initiated responses. The 569 SUBSCRIBE method does create a dialog between the originating end- 570 user and the subscription service. It, too, is not associated 571 with a media session. 572 In light of the above observations we have extended the term 573 "session" to include SIP-based services that are not initiated by 574 INVITE requests and that do not have associated media. In this 575 extended definition, a session always has a signaling component 576 and may also have a media component. Thus, a session can be 577 defined as signaling-only or a combination of signaling and media. 578 We define the term "Associated Media", see Section 3.1.4, to 579 describe the situation in which media is associated with a SIP 580 dialog. The terminology "Invite-initiated Session" (IS) 581 Section 3.1.8 and "Non-invite-Initiated Session" (NS) 582 Section 3.1.9 are used to distinguish between these two types of 583 session. An Invite-initiated Session is a session as defined in 584 SIP. The performance of a device or system that supports Invite- 585 initiated Sessions that do not create media sessions, "Invite- 586 initiated Sessions without Associated Media", can be measured and 587 is of interest for comparison and as a limiting case. The 588 REGISTER request can be considered to be a "Non-invite-initiated 589 Session without Associated Media." A separate set of benchmarks 590 is provided for REGISTER requests since most implementations of 591 SIP-based services require this request and since a registrar may 592 be a device under test. 594 A Session in the context of this document, can be considered to be 595 a vector with three components: 597 1. A component in the signaling plane (SIP messages), sess.sig; 598 2. A media component in the media plane (RTP and SRTP streams for 599 example), sess.med (which may be null); 600 3. A control component in the media plane (RTCP messages for 601 example), sess.medc (which may be null). 603 An IS is expected to have non-null sess.sig and sess.med 604 components. The use of control protocols in the media component 605 is media dependent, thus the expected presence or absence of 606 sess.medc is media dependent and test-case dependent. An NS is 607 expected to have a non-null sess.sig component, but null sess.med 608 and sess.medc components. 610 Packets in the Signaling Plane and Media Plane will be handled by 611 different processes within the DUT. They will take different 612 paths within a SUT. These different processes and paths may 613 produce variations in performance. The terminology and benchmarks 614 defined in this document and the methodology for their use are 615 designed to enable us to compare performance of the DUT/SUT with 616 reference to the type of SIP-supported application it is handling. 618 Note that one or more sessions can simultaneously exist between 619 any participants. This can be the case, for example, when the EA 620 sets up both an IM and a voice call through the DUT/SUT. These 621 sessions are represented as an array session[x]. 623 Sessions will be represented as a vector array with three 624 components, as follows: 625 session-> 626 session[x].sig, the signaling component 627 session[x].medc[y], the media control component (e.g. RTCP) 628 session[x].med[y], an array of associated media streams (e.g. 629 RTP, SRTP, RTSP, MSRP). This media component may consist of zero 630 or more media streams. 631 Figure 12 models the vectors of the session. 633 Measurement Units: 634 N/A. 636 Issues: 637 None. 639 See Also: 640 Media Plane 641 Signaling Plane 642 Associated Media 643 Invite-initiated Session (IS) 644 Non-invite-initiated Session (NS) 645 |\ 646 | 647 | \ 648 sess.sig| 649 | \ 650 | 651 | \ 652 | o 653 | / 654 | / | 655 | / 656 | / | 657 | / 658 | / | 659 | / 660 | / | sess.medc 661 |/_____________________ 662 / / 663 / | 664 / / 665 sess.med / | 666 /_ _ _ _ _ _ _ _/ 667 / 668 / 669 / 670 / 672 Figure 12: Session components 674 3.1.2. Signaling Plane 676 Definition: 677 The plane in which SIP messages [RFC3261] are exchanged between 678 SIP Agents [RFC3261]. 680 Discussion: 681 SIP messages are used to establish sessions in several ways: 682 directly between two User Agents [RFC3261], through a Proxy Server 683 [RFC3261], or through a series of Proxy Servers. The Session 684 Description Protocol is included in the Signaling Plane. (SDP). 685 The Signaling Plane for a single Session is represented by 686 session.sig. 688 Measurement Units: 689 N/A. 691 Issues: 692 None. 694 See Also: 695 Media Plane 696 EAs 698 3.1.3. Media Plane 700 Definition: 701 The data plane in which one or more media streams and their 702 associated media control protocols are exchanged between User 703 Agents after a media connection has been created by the exchange 704 of signaling messages in the Signaling Plane. 706 Discussion: 707 Media may also be known as the "bearer channel". The Media Plane 708 MUST include the media control protocol, if one is used, and the 709 media stream(s). Examples of media are audio and video. The 710 media streams are described in the SDP of the Signaling Plane. 711 The media for a single Session is represented by session.med. The 712 media control protocol for a single media description is 713 represented by session.medc. 715 Measurement Units: 716 N/A. 718 Issues: 719 None. 721 See Also: 722 Signaling Plane 724 3.1.4. Associated Media 726 Definition: 727 Media that corresponds to an 'm' line in the SDP payload of the 728 Signaling Plane. 730 Discussion: 732 Any media protocol MAY be used. 733 For any session's signaling component, session.sig, there may be 734 zero, one, or multiple associated media streams. When there are 735 multiple media streams, these are represented be a vector array 736 session.med[y]. When there are multiple media streams there will 737 be multiple media control protocol descriptions as well. They are 738 represented by a vector array session.medc[y]. 740 Measurement Units: 741 N/A. 743 Issues: 744 None. 746 3.1.5. Overload 748 Definition: 749 Overload is defined as the state where a SIP server does not have 750 sufficient resources to process all incoming SIP messages 751 [I-D.ietf-soc-overload-design]. 753 Discussion: 754 The distinction between an overload condition and other failure 755 scenarios is outside the scope of black box testing and of this 756 document. Under overload conditions, all or a percentage of 757 Session Attempts will fail due to lack of resources. In black box 758 testing the cause of the failure is not explored. The fact that a 759 failure occurred for whatever reason, will trigger the tester to 760 reduce the offered load, as described in the companion methodology 761 document, [I-D.ietf-bmwg-sip-bench-meth]. SIP server resources 762 may include CPU processing capacity, network bandwidth, input/ 763 output queues, or disk resources. Any combination of resources 764 may be fully utilized when a SIP server (the DUT/SUT) is in the 765 overload condition. For proxy-only type of devices, it is 766 expected that the proxy will be driven into overload based on the 767 delivery rate of signaling requests. 768 For UA-type of network devices such as gateways, it is expected 769 that the UA will be driven into overload based on the volume of 770 media streams it is processing. 772 Measurement Units: 773 N/A. 775 Issues: 776 The issue of overload in SIP networks is currently a topic of 777 discussion in the SIPPING WG. The normal response to an overload 778 stimulus -- sending a 503 response -- is considered inadequate and 779 new response codes and behaviors may be specified in the future. 780 From the perspective of this document, all these responses will be 781 considered to be failures. There is thus no dependency between 782 this document and the ongoing work on the treatment of overload 783 failure. 785 3.1.6. Session Attempt 787 Definition: 788 A SIP request sent by the EA that has not received a final 789 response. 791 Discussion: 792 The attempted session may be Invite Initiated or Non-invite 793 Initiated. When counting the number of session attempts we 794 include all INVITEs that are rejected for lack of authentication 795 information. The EA needs to record the total number of session 796 attempts including those attempts that are routinely rejected by a 797 proxy that requires the UA to authenticate itself. The EA is 798 provisioned to deliver a specific number of session attempts per 799 second. But the EA must also count the actual number of session 800 attempts per given tie interval. 802 Measurement Units: 803 N/A. 805 Issues: 806 None. 808 See Also: 809 Session 810 Session Attempt Rate 811 Invite-initiated Session 812 Non-Invite initiated Session 814 3.1.7. Established Session 816 Definition: 817 A SIP session for which the EA acting as the UE/UA has received a 818 200 OK message. 820 Discussion: 821 An Established Session MAY be Invite Initiated or Non-invite 822 Initiated. 824 Measurement Units: 825 N/A. 827 Issues: 828 None. 830 See Also: 831 Invite-initiated Session 832 Session Attempting State 833 Session Disconnecting State 835 3.1.8. Invite-initiated Session (IS) 837 Definition: 838 A Session that is created by an exchange of messages in the 839 Signaling Plane, the first of which is a SIP INVITE request. 841 Discussion: 842 When an IS becomes an Established Session its signaling component 843 is identified by the SIP dialog parameter values, Call-ID, To-tag, 844 and From-tag (RFC3261 [RFC3261]). An IS may have zero, one or 845 multiple Associated Media descriptions in the SDP body. The 846 inclusion of media is test case dependent. An IS is successfully 847 established if the following two conditions are met: 848 1. Sess.sig is established by the end of Establishment Threshold 849 Time (c.f. Section 3.3.3), and 850 2. If a media session is described in the SDP body of the 851 signaling message, then the media session is established by 852 the end of Establishment Threshold Time (c.f. Section 3.3.3). 853 An SBC or B2BUA may receive media from a calling or called 854 party before a signaling dialog is established and certainly 855 before a confirmed dialog is established. The EA can be built 856 in such a way that it does not send early media or it needs to 857 include a parameter that indicates when it will send media. 858 This parameter must be included in the list of test setup 859 parameters in Section 5.1 of [I-D.ietf-bmwg-sip-bench-meth] 861 Measurement Units: 862 N/A. 864 Issues: 865 None. 867 See Also: 868 Session 869 Non-Invite initiated Session 870 Associated Media 872 3.1.9. Non-INVITE-initiated Session (NS) 874 Definition: 875 A session that is created by an exchange of SIP messages in the 876 Signaling Plane the first of which is not a SIP INVITE message. 878 Discussion: 879 An NS is successfully established if the Session Attempt via a 880 non- INVITE request results in the EA receiving a 2xx reply before 881 the expiration of the Establishment Threshold timer (c.f., 882 Section 3.3.3). An example of a NS is a session created by the 883 SUBSCRIBE request. 885 Measurement Units: 886 N/A. 888 Issues: 889 None. 891 See Also: 892 Session 893 Invite-initiated Session 895 3.1.10. Session Attempt Failure 897 Definition: 898 A session attempt that does not result in an Established Session. 900 Discussion: 901 The session attempt failure may be indicated by the following 902 observations at the EA: 903 1. Receipt of a SIP 4xx, 5xx, or 6xx class response to a Session 904 Attempt. 905 2. The lack of any received SIP response to a Session Attempt 906 within the Establishment Threshold Time (c.f. Section 3.3.3). 908 Measurement Units: 909 N/A. 911 Issues: 912 None. 914 See Also: 915 Session Attempt 917 3.1.11. Standing Sessions Count 919 Definition: 920 The number of Sessions currently established on the DUT/SUT at any 921 instant. 923 Discussion: 924 The number of Standing Sessions is influenced by the Session 925 Duration and the Session Attempt Rate. Benchmarks MUST be 926 reported with the maximum and average Standing Sessions for the 927 DUT/SUT for the duration of the test. In order to determine the 928 maximum and average Standing Sessions on the DUT/SUT for the 929 duration of the test it is necessary to make periodic measurements 930 of the number of Standing Sessions on the DUT/SUT. The 931 recommended value for the measurement period is 1 second. Since 932 we cannot directly poll the DUT/SUT, we take the number of 933 standing sessions on the DUT/SUT to be the number of distinct 934 calls as measured by the number of distinct Call-IDs that the EA 935 is processing at the time of measurement. The EA must make that 936 count available for viewing ad recording. 938 Measurement Units: 939 Number of sessions 941 Issues: 942 None. 944 See Also: 945 Session Duration 946 Session Attempt Rate 947 Session Attempt Rate 948 Emulated Agent 950 3.2. Test Components 951 3.2.1. Emulated Agent 953 Definition: 954 A device in the test topology that initiates/responds to SIP 955 messages as one or more session endpoints and, wherever 956 applicable, sources/receives Associated Media for Established 957 Sessions. 959 Discussion: 960 The EA functions in the Signaling and Media Planes. The Tester 961 may act as multiple EAs. 963 Measurement Units: 964 N/A 966 Issues: 967 None. 969 See Also: 970 Media Plane 971 Signaling Plane 972 Established Session 973 Associated Media 975 3.2.2. Signaling Server 977 Definition: 978 Device in the test topology that acts to create sessions between 979 EAs. This device is either a DUT or a component of a SUT. 981 Discussion: 982 The DUT MUST be an RFC 3261 capable network equipment such as a 983 Registrar, Redirect Server, User Agent Server, Stateless Proxy, or 984 Stateful Proxy. A DUT MAY also include B2BUA or SBC. 986 Measurement Units: 987 NA 989 Issues: 990 None. 992 See Also: 993 Signaling Plane 995 3.2.3. SIP-Aware Stateful Firewall 996 Definition: 997 Device in the test topology that provides protection against 998 various types of security threats to which the Signaling and Media 999 Planes of the EAs and Signaling Server are vulnerable. 1001 Discussion: 1002 Threats may include Denial-of-Service, theft of service and misuse 1003 of service.he SIP-Aware Stateful Firewall MAY be an internal 1004 component or function of the Session Server. The SIP-Aware 1005 Stateful Firewall MAY be a standalone device. If it is a 1006 standalone device it MUST be paired with a Signaling Server. If 1007 it is a standalone device it MUST be benchmarked as part of a SUT. 1008 SIP-Aware Stateful Firewalls MAY include Network Address 1009 Translation (NAT) functionality. Ideally, the inclusion of the 1010 SIP-Aware Stateful Firewall in the SUT does not lower the measured 1011 values of the performance benchmarks. 1013 Measurement Units: 1014 N/A 1016 Issues: 1017 None. 1019 See Also: 1021 3.2.4. SIP Transport Protocol 1023 Definition: 1024 The protocol used for transport of the Signaling Plane messages. 1026 Discussion: 1027 Performance benchmarks may vary for the same SIP networking device 1028 depending upon whether TCP, UDP, TLS, SCTP, or another transport 1029 layer protocol is used. For this reason it MAY be necessary to 1030 measure the SIP Performance Benchmarks using these various 1031 transport protocols. Performance Benchmarks MUST report the SIP 1032 Transport Protocol used to obtain the benchmark results. 1034 Measurement Units: 1035 TCP,UDP, SCTP, TLS over TCP, TLS over UDP, or TLS over SCTP 1037 Issues: 1038 None. 1040 See Also: 1042 3.3. Test Setup Parameters 1044 3.3.1. Session Attempt Rate 1046 Definition: 1047 Configuration of the EA for the number of sessions per second that 1048 the EA attempts to establish using the services of the DUT/SUT. 1050 Discussion: 1051 The Session Attempt Rate is the number of sessions per second that 1052 the EA sends toward the DUT/SUT. Some of the sessions attempted 1053 may not result in a session being established. A session in this 1054 case may be either an IS or an NS. 1056 Measurement Units: 1057 Session attempts per second 1059 Issues: 1060 None. 1062 See Also: 1063 Session 1064 Session Attempt 1066 3.3.2. IS Media Attempt Rate 1068 Definition: 1069 Configuration on the EA for the rate, measured in sessions per 1070 second, at which the EA attempts to establish INVITE-initiated 1071 sessions with Associated Media, using the services of the DUT/SUT. 1073 Discussion: 1074 An IS is not required to include a media description. The IS 1075 Media Attempt Rate defines the number of media sessions we are 1076 trying to create, not the number of media sessions that are 1077 actually created. Some attempts might not result in successful 1078 sessions established on the DUT. 1080 Measurement Units: 1081 session attempts per second (saps) 1083 Issues: 1085 None. 1087 See Also: 1088 IS 1090 3.3.3. Establishment Threshold Time 1092 Definition: 1093 Configuration of the EA for representing the amount of time that 1094 an EA will wait before declaring a Session Attempt Failure. 1096 Discussion: 1097 This time duration is test dependent. 1098 It is RECOMMENDED that the Establishment Threshold Time value be 1099 set to Timer B (for ISs) or Timer F (for NSs) as specified in RFC 1100 3261, Table 4 [RFC3261]. Following the default value of T1 1101 (500ms) specified in the table and a constant multiplier of 64 1102 gives a value of 32 seconds for this timer (i.e., 500ms * 64 = 1103 32s). 1105 Measurement Units: 1106 seconds 1108 Issues: 1109 None. 1111 See Also: 1112 session establishment failure 1114 3.3.4. Session Duration 1116 Definition: 1117 Configuration of the EA that represents the amount of time that 1118 the SIP dialog is intended to exist between the two EAs associated 1119 with the test. 1121 Discussion: 1122 The time at which the BYE is sent will control the Session 1123 Duration 1124 Normally the Session Duration will be the same as the Media 1125 Session Hold Time. However, it is possible that the dialog 1126 established between the two EAs can support different media 1127 sessions at different points in time. Providing both parameters 1128 allows the testing agency to explore this possibility. 1130 Measurement Units: 1131 seconds 1133 Issues: 1134 None. 1136 See Also: 1137 Media Session Hold Time 1139 3.3.5. Media Packet Size 1141 Definition: 1142 Configuration on the EA for a fixed size of packets used for media 1143 streams. 1145 Discussion: 1146 For a single benchmark test, all sessions use the same size packet 1147 for media streams. The size of packets can cause variation in 1148 performance benchmark measurements. 1150 Measurement Units: 1151 bytes 1153 Issues: 1154 None. 1156 See Also: 1158 3.3.6. Media Offered Load 1160 Definition: 1161 Configuration of the EA for the constant rate of Associated Media 1162 traffic offered by the EA to the DUT/SUT for one or more 1163 Established Sessions of type IS. 1165 Discussion: 1166 The Media Offered Load to be used for a test MUST be reported with 1167 three components: 1168 1. per Associated Media stream; 1169 2. per IS; 1170 3. aggregate. 1171 For a single benchmark test, all sessions use the same Media 1172 Offered Load per Media Stream. There may be multiple Associated 1173 Media streams per IS. The aggregate is the sum of all Associated 1174 Media for all IS. 1176 Measurement Units: 1177 packets per second (pps) 1179 Issues: 1180 None. 1182 See Also: 1183 Established Session 1184 Invite Initiated Session 1185 Associated Media 1187 3.3.7. Media Session Hold Time 1189 Definition: 1190 Parameter configured at the EA, that represents the amount of time 1191 that the Associated Media for an Established Session of type IS 1192 will last. 1194 Discussion: 1195 The Associated Media streams may be bi-directional or uni- 1196 directional as indicated in the test methodology. 1197 Normally the Media Session Hold Time will be the same as the 1198 Session Duration. However, it is possible that the dialog 1199 established between the two EAs can support different media 1200 sessions at different points in time. Providing both parameters 1201 allows the testing agency to explore this possibility. 1203 Measurement Units: 1204 seconds 1206 Issues: 1207 None. 1209 See Also: 1210 Associated Media 1211 Established Session 1212 Invite-initiated Session (IS) 1214 3.3.8. Loop Detection Option 1216 Definition: 1217 An option that causes a Proxy to check for loops in the routing of 1218 a SIP request before forwarding the request. 1220 Discussion: 1221 This is an optional process that a SIP proxy may employ; the 1222 process is described under Proxy Behavior in RFC 3261 [RFC3261] in 1223 Section 16.3 Request Validation and that section also contains 1224 suggestions as to how the option could be implemented. Any 1225 procedure to detect loops will use processor cycles and hence 1226 could impact the performance of a proxy. 1228 Measurement Units: 1229 NA 1231 Issues: 1232 None. 1234 See Also: 1236 3.3.9. Forking Option 1238 Definition: 1239 An option that enables a Proxy to fork requests to more than one 1240 destination. 1242 Discussion: 1243 This is an process that a SIP proxy may employ to find the UAS. 1244 The option is described under Proxy Behavior in RFC 3261 in 1245 Section 16.1. A proxy that uses forking must maintain state 1246 information and this will use processor cycles and memory. Thus 1247 the use of this option could impact the performance of a proxy and 1248 different implementations could produce different impacts. 1249 SIP supports serial or parallel forking. When performing a test, 1250 the type of forking mode MUST be indicated. 1252 Measurement Units: 1253 The number of endpoints that will receive the forked invitation. 1254 A value of 1 indicates that the request is destined to only one 1255 endpoint, a value of 2 indicates that the request is forked to two 1256 endpoints, and so on. This is an integer value ranging between 1 1257 and N inclusive, where N is the maximum number of endpoints to 1258 which the invitation is sent. 1259 Type of forking used, namely parallel or serial. 1261 Issues: 1262 None. 1264 See Also: 1266 3.4. Benchmarks 1268 3.4.1. Registration Rate 1270 Definition: 1271 The maximum number of registrations that can be successfully 1272 completed by the DUT/SUT in a given time period without 1273 registration failures in that time period. 1275 Discussion: 1276 This benchmark is obtained with zero failure in which 100% of the 1277 registrations attempted by the EA are successfully completed by 1278 the DUT/SUT. The registration rate provisioned on the Emulated 1279 Agent is raised and lowered as described in the algorithm in the 1280 companion methodology draft [I-D.ietf-bmwg-sip-bench-meth] until a 1281 traffic load consisting of registrations at the given attempt rate 1282 over the sustained period of time identified by T in the algorithm 1283 completes without failure. 1285 Measurement Units: 1286 registrations per second (rps) 1288 Issues: 1289 None. 1291 See Also: 1293 3.4.2. Session Establishment Rate 1295 Definition: 1296 The maximum number of sessions that can be successfully completed 1297 by the DUT/SUT in a given time period without session 1298 establishment failures in that time period. 1300 Discussion: 1301 This benchmark is obtained with zero failure in which 100% of the 1302 sessions attempted by the Emulated Agent are successfully 1303 completed by the DUT/SUT. The session attempt rate provisioned on 1304 the EA is raised and lowered as described in the algorithm in the 1305 accompanying methodology document, until a traffic load at the 1306 given attempt rate over the sustained period of time identified by 1307 T in the algorithm completes without any failed session attempts. 1308 Sessions may be IS or NS or a mix of both and will be defined in 1309 the particular test. 1311 Measurement Units: 1312 sessions per second (sps) 1314 Issues: 1315 None. 1317 See Also: 1318 Invite-initiated Sessions 1319 Non-INVITE initiated Sessions 1320 Session Attempt Rate 1322 3.4.3. Session Capacity 1324 Definition: 1325 The maximum value of Standing Sessions Count achieved by the DUT/ 1326 SUT during a time period T in which the EA is sending session 1327 establishment messages at the Session Establishment Rate. 1329 Discussion: 1330 Sessions may be IS or NS. If they are IS they can be with or 1331 without media. When benchmarking Session Capacity for sessions 1332 with media it is required that these sessions be permanently 1333 established (i.e., they remain active for the duration of the 1334 test.) This can be achieved by causing the EA not to send a BYE 1335 for the duration of the testing. In the signaling plane, this 1336 requirement means that the dialog lasts as long as the test lasts. 1337 When media is present, the Media Session Hold Time MUST be set to 1338 infinity so that sessions remain established for the duration of 1339 the test. If the DUT/SUT is dialog-stateful, then we expect its 1340 performance will be impacted by setting Media Session Hold Time to 1341 infinity, since the DUT/SUT will need to allocate resources to 1342 process and store the state information. The report of the 1343 Session Capacity must include the Session Establishment Rate at 1344 which it was measured. 1346 Measurement Units: 1347 sessions 1349 Issues: 1350 None. 1352 See Also: 1353 Established Session 1354 Session Attempt Rate 1355 Session Attempt Failure 1357 3.4.4. Session Overload Capacity 1359 Definition: 1360 The maximum number of Established Sessions that can exist 1361 simultaneously on the DUT/SUT until it stops responding to Session 1362 Attempts. 1364 Discussion: 1365 Session Overload Capacity is measured after the Session Capacity 1366 is measured. The Session Overload Capacity is greater than or 1367 equal to the Session Capacity. When benchmarking Session Overload 1368 Capacity, continue to offer Session Attempts to the DUT/SUT after 1369 the first Session Attempt Failure occurs and measure Established 1370 Sessions until no there is no SIP message response for the 1371 duration of the Establishment Threshold. Note that the Session 1372 Establishment Performance is expected to decrease after the first 1373 Session Attempt Failure occurs. 1375 Units: 1376 Sessions 1378 Issues: 1379 None. 1381 See Also: 1382 Overload 1383 Session Capacity 1384 Session Attempt Failure 1386 3.4.5. Session Establishment Performance 1388 Definition: 1389 The percent of Session Attempts that become Established Sessions 1390 over the duration of a benchmarking test. 1392 Discussion: 1393 Session Establishment Performance is a benchmark to indicate 1394 session establishment success for the duration of a test. The 1395 duration for measuring this benchmark is to be specified in the 1396 Methodology. The Session Duration SHOULD be configured to 1397 infinity so that sessions remain established for the entire test 1398 duration. 1400 Session Establishment Performance is calculated as shown in the 1401 following equation: 1403 Session Establishment = Total Established Sessions 1404 Performance -------------------------- 1405 Total Session Attempts 1407 Session Establishment Performance may be monitored real-time 1408 during a benchmarking test. However, the reporting benchmark MUST 1409 be based on the total measurements for the test duration. 1411 Measurement Units: 1412 Percent (%) 1414 Issues: 1415 None. 1417 See Also: 1418 Established Session 1419 Session Attempt 1421 3.4.6. Session Attempt Delay 1423 Definition: 1424 The average time measured at the EA for a Session Attempt to 1425 result in an Established Session. 1427 Discussion: 1428 Time is measured from when the EA sends the first INVITE for the 1429 call-ID in the case of an IS. Time is measured from when the EA 1430 sends the first non-INVITE message in the case of an NS. Session 1431 Attempt Delay MUST be measured for every established session to 1432 calculate the average. Session Attempt Delay MUST be measured at 1433 the Session Establishment Rate. 1435 Measurement Units: 1436 Seconds 1438 Issues: 1439 None. 1441 See Also: 1442 Session Establishment Rate 1444 3.4.7. IM Rate 1445 Definition: 1446 Maximum number of IM messages completed by the DUT/SUT. 1448 Discussion: 1449 For a UAS, the definition of success is the receipt of an IM 1450 request and the subsequent sending of a final response. 1451 For a UAC, the definition of success is the sending of an IM 1452 request and the receipt of a final response to it. For a proxy, 1453 the definition of success is as follows: 1454 A. the number of IM requests it receives from the upstream client 1455 MUST be equal to the number of IM requests it sent to the 1456 downstream server; and 1457 B. the number of IM responses it receives from the downstream 1458 server MUST be equal to the number of IM requests sent to the 1459 downstream server; and 1460 C. the number of IM responses it sends to the upstream client 1461 MUST be equal to the number of IM requests it received from 1462 the upstream client. 1464 Measurement Units: 1465 IM messages per second 1467 Issues: 1468 None. 1470 See Also: 1472 4. IANA Considerations 1474 This document requires no IANA considerations. 1476 5. Security Considerations 1478 Documents of this type do not directly affect the security of 1479 Internet or corporate networks as long as benchmarking is not 1480 performed on devices or systems connected to production networks. 1481 Security threats and how to counter these in SIP and the media layer 1482 is discussed in RFC3261 [RFC3261], RFC 3550 [RFC3550], RFC3711 1483 [RFC3711] and various other drafts. This document attempts to 1484 formalize a set of common terminology for benchmarking SIP networks. 1485 Packets with unintended and/or unauthorized DSCP or IP precedence 1486 values may present security issues. Determining the security 1487 consequences of such packets is out of scope for this document. 1489 6. Acknowledgments 1491 The authors would like to thank Keith Drage, Cullen Jennings, Daryl 1492 Malas, Al Morton, and Henning Schulzrinne for invaluable 1493 contributions to this document. Dale Worley provided an extensive 1494 review that lead to improvements in the documents. 1496 7. References 1498 7.1. Normative References 1500 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1501 Requirement Levels", BCP 14, RFC 2119, March 1997. 1503 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 1504 Network Interconnect Devices", RFC 2544, March 1999. 1506 [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, 1507 A., Peterson, J., Sparks, R., Handley, M., and E. 1508 Schooler, "SIP: Session Initiation Protocol", RFC 3261, 1509 June 2002. 1511 [I-D.ietf-bmwg-sip-bench-meth] 1512 Davids, C., Gurbani, V., and S. Poretsky, "Methodology for 1513 Benchmarking SIP Networking Devices", 1514 draft-ietf-bmwg-sip-bench-meth-05 (work in progress), 1515 October 2012. 1517 7.2. Informational References 1519 [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN 1520 Switching Devices", RFC 2285, February 1998. 1522 [RFC1242] Bradner, S., "Benchmarking terminology for network 1523 interconnection devices", RFC 1242, July 1991. 1525 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 1526 Jacobson, "RTP: A Transport Protocol for Real-Time 1527 Applications", STD 64, RFC 3550, July 2003. 1529 [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 1530 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 1531 RFC 3711, March 2004. 1533 [I-D.ietf-soc-overload-design] 1534 Hilt, V., Noel, E., Shen, C., and A. Abdelal, "Design 1535 Considerations for Session Initiation Protocol (SIP) 1536 Overload Control", draft-ietf-soc-overload-design-08 (work 1537 in progress), July 2011. 1539 [I-D.ietf-soc-overload-control] 1540 Gurbani, V., Hilt, V., and H. Schulzrinne, "Session 1541 Initiation Protocol (SIP) Overload Control", 1542 draft-ietf-soc-overload-control-10 (work in progress), 1543 October 2012. 1545 Appendix A. White Box Benchmarking Terminology 1547 Session Attempt Arrival Rate 1549 Definition: 1550 The number of Session Attempts received at the DUT/SUT over a 1551 specified time period. 1553 Discussion: 1554 Sessions Attempts are indicated by the arrival of SIP INVITES OR 1555 SUBSCRIBE NOTIFY messages. Session Attempts Arrival Rate 1556 distribution can be any model selected by the user of this 1557 document. It is important when comparing benchmarks of different 1558 devices that same distribution model was used. Common 1559 distributions are expected to be Uniform and Poisson. 1561 Measurement Units: 1562 Session attempts/sec 1564 Issues: 1565 None. 1567 See Also: 1568 Session Attempt 1570 Authors' Addresses 1572 Carol Davids 1573 Illinois Institute of Technology 1574 201 East Loop Road 1575 Wheaton, IL 60187 1576 USA 1578 Phone: +1 630 682 6024 1579 Email: davids@iit.edu 1580 Vijay K. Gurbani 1581 Bell Laboratories, Alcatel-Lucent 1582 1960 Lucent Lane 1583 Rm 9C-533 1584 Naperville, IL 60566 1585 USA 1587 Phone: +1 630 224 0216 1588 Email: vkg@bell-labs.com 1590 Scott Poretsky 1591 Allot Communications 1592 300 TradeCenter, Suite 4680 1593 Woburn, MA 08101 1594 USA 1596 Phone: +1 508 309 2179 1597 Email: sporetsky@allot.com