idnits 2.17.1 draft-ietf-bmwg-sip-bench-term-09.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (February 14, 2014) is 3724 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-12) exists of draft-ietf-bmwg-sip-bench-meth-08 Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Methodology Working Group C. Davids 3 Internet-Draft Illinois Institute of Technology 4 Intended status: Informational V. Gurbani 5 Expires: August 18, 2014 Bell Laboratories, 6 Alcatel-Lucent 7 S. Poretsky 8 Allot Communications 9 February 14, 2014 11 Terminology for Benchmarking Session Initiation Protocol (SIP) Devices: 12 Basic session setup and registration 13 draft-ietf-bmwg-sip-bench-term-09 15 Abstract 17 This document provides a terminology for benchmarking the Session 18 Initiation Protocol (SIP) performance of devices. Methodology 19 related to benchmarking SIP devices is described in the companion 20 methodology document. Using these two documents, benchmarks can be 21 obtained and compared for different types of devices such as SIP 22 Proxy Servers, Registrars and Session Border Controllers. The term 23 "performance" in this context means the capacity of the device-under- 24 test (DUT) to process SIP messages. Media streams are used only to 25 study how they impact the signaling behavior. The intent of the two 26 documents is to provide a normalized set of tests that will enable an 27 objective comparison of the capacity of SIP devices. Test setup 28 parameters and a methodology is necessary because SIP allows a wide 29 range of configuration and operational conditions that can influence 30 performance benchmark measurements. A standard terminology and 31 methodology will ensure that benchmarks have consistent definition 32 and were obtained following the same procedures. 34 Status of this Memo 36 This Internet-Draft is submitted in full conformance with the 37 provisions of BCP 78 and BCP 79. 39 Internet-Drafts are working documents of the Internet Engineering 40 Task Force (IETF). Note that other groups may also distribute 41 working documents as Internet-Drafts. The list of current Internet- 42 Drafts is at http://datatracker.ietf.org/drafts/current/. 44 Internet-Drafts are draft documents valid for a maximum of six months 45 and may be updated, replaced, or obsoleted by other documents at any 46 time. It is inappropriate to use Internet-Drafts as reference 47 material or to cite them other than as "work in progress." 48 This Internet-Draft will expire on August 18, 2014. 50 Copyright Notice 52 Copyright (c) 2014 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents 57 (http://trustee.ietf.org/license-info) in effect on the date of 58 publication of this document. Please review these documents 59 carefully, as they describe your rights and restrictions with respect 60 to this document. Code Components extracted from this document must 61 include Simplified BSD License text as described in Section 4.e of 62 the Trust Legal Provisions and are provided without warranty as 63 described in the Simplified BSD License. 65 Table of Contents 67 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 68 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5 69 2.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 6 70 3. Term Definitions . . . . . . . . . . . . . . . . . . . . . . . 7 71 3.1. Protocol Components . . . . . . . . . . . . . . . . . . . 7 72 3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 7 73 3.1.2. Signaling Plane . . . . . . . . . . . . . . . . . . . 10 74 3.1.3. Media Plane . . . . . . . . . . . . . . . . . . . . . 11 75 3.1.4. Associated Media . . . . . . . . . . . . . . . . . . . 11 76 3.1.5. Overload . . . . . . . . . . . . . . . . . . . . . . . 12 77 3.1.6. Session Attempt . . . . . . . . . . . . . . . . . . . 12 78 3.1.7. Established Session . . . . . . . . . . . . . . . . . 13 79 3.1.8. Invite-Initiated Session (IS) . . . . . . . . . . . . 14 80 3.1.9. Non-INVITE-Initiated Session (NS) . . . . . . . . . . 14 81 3.1.10. Session Attempt Failure . . . . . . . . . . . . . . . 15 82 3.2. Test Components . . . . . . . . . . . . . . . . . . . . . 15 83 3.2.1. Emulated Agent . . . . . . . . . . . . . . . . . . . . 15 84 3.2.2. Signaling Server . . . . . . . . . . . . . . . . . . . 16 85 3.2.3. SIP Transport Protocol . . . . . . . . . . . . . . . . 16 86 3.3. Test Setup Parameters . . . . . . . . . . . . . . . . . . 17 87 3.3.1. Session Attempt Rate . . . . . . . . . . . . . . . . . 17 88 3.3.2. Establishment Threshold Time . . . . . . . . . . . . . 17 89 3.3.3. Session Duration . . . . . . . . . . . . . . . . . . . 18 90 3.3.4. Media Packet Size . . . . . . . . . . . . . . . . . . 18 91 3.3.5. Codec Type . . . . . . . . . . . . . . . . . . . . . . 19 92 3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 19 93 3.4.1. Session Establishment Rate . . . . . . . . . . . . . . 19 94 3.4.2. Registration Rate . . . . . . . . . . . . . . . . . . 20 95 3.4.3. Registration Attempt Rate . . . . . . . . . . . . . . 21 96 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 97 5. Security Considerations . . . . . . . . . . . . . . . . . . . 21 98 6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 22 99 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 22 100 7.1. Normative References . . . . . . . . . . . . . . . . . . . 22 101 7.2. Informational References . . . . . . . . . . . . . . . . . 22 102 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 23 104 1. Terminology 106 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 107 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 108 document are to be interpreted as described in BCP 14, RFC2119 109 [RFC2119]. RFC 2119 defines the use of these key words to help make 110 the intent of standards track documents as clear as possible. While 111 this document uses these keywords, this document is not a standards 112 track document. The term Throughput is defined in RFC2544 [RFC2544]. 114 For the sake of clarity and continuity, this document adopts the 115 template for definitions set out in Section 2 of RFC 1242 [RFC1242]. 117 The term Device Under Test (DUT) is defined in the following BMWG 118 documents: 120 Device Under Test (DUT) (c.f., Section 3.1.1 RFC 2285 [RFC2285]). 122 Many commonly used SIP terms in this document are defined in RFC 3261 123 [RFC3261]. For convenience the most important of these are 124 reproduced below. Use of these terms in this document is consistent 125 with their corresponding definition in the base SIP specification 126 [RFC3261] as amended by [RFC4320], [RFC5393] and [RFC6026]. 128 o Call Stateful: A proxy is call stateful if it retains state for a 129 dialog from the initiating INVITE to the terminating BYE request. 130 A call stateful proxy is always transaction stateful, but the 131 converse is not necessarily true. 132 o Stateful Proxy: A logical entity that maintains the client and 133 server transaction state machines defined by this specification 134 during the processing of a request, also known as a transaction 135 stateful proxy. The behavior of a stateful proxy is further 136 defined in Section 16 of RFC 3261 [RFC3261] . A transaction 137 stateful proxy is not the same as a call stateful proxy. 138 o Stateless Proxy: A logical entity that does not maintain the 139 client or server transaction state machines defined in this 140 specification when it processes requests. A stateless proxy 141 forwards every request it receives downstream and every response 142 it receives upstream. 143 o Back-to-back User Agent: A back-to-back user agent (B2BUA) is a 144 logical entity that receives a request and processes it as a user 145 agent server (UAS). In order to determine how the request should 146 be answered, it acts as a user agent client (UAC) and generates 147 requests. Unlike a proxy server, it maintains dialog state and 148 must participate in all requests sent on the dialogues it has 149 established. Since it is a concatenation of a UAC and a UAS, no 150 explicit definitions are needed for its behavior. 152 2. Introduction 154 Service Providers and IT Organizations deliver Voice Over IP (VoIP) 155 and Multimedia network services based on the IETF Session Initiation 156 Protocol (SIP) [RFC3261]. SIP is a signaling protocol originally 157 intended to be used to dynamically establish, disconnect and modify 158 streams of media between end users. As it has evolved it has been 159 adopted for use in a growing number of services and applications. 160 Many of these result in the creation of a media session, but some do 161 not. Examples of this latter group include text messaging and 162 subscription services. The set of benchmarking terms provided in 163 this document is intended for use with any SIP-enabled device 164 performing SIP functions in the interior of the network, whether or 165 not these result in the creation of media sessions. The performance 166 of end-user devices is outside the scope of this document. 168 A number of networking devices have been developed to support SIP- 169 based VoIP services. These include SIP Servers, Session Border 170 Controllers (SBC) and Back-to-back User Agents (B2BUA). These 171 devices contain a mix of voice and IP functions whose performance may 172 be reported using metrics defined by the equipment manufacturer or 173 vendor. The Service Provider or IT Organization seeking to compare 174 the performance of such devices will not be able to do so using these 175 vendor-specific metrics, whose conditions of test and algorithms for 176 collection are often unspecified. SIP functional elements and the 177 devices that include them can be configured many different ways and 178 can be organized into various topologies. These configuration and 179 topological choices impact the value of any chosen signaling 180 benchmark. Unless these conditions-of-test are defined, a true 181 comparison of performance metrics across multiple vendor 182 implementations will not be possible. Some SIP-enabled devices 183 terminate or relay media as well as signaling. The processing of 184 media by the device impacts the signaling performance. As a result, 185 the conditions-of-test must include information as to whether or not 186 the device under test processes media and if the device does process 187 media, a description of the media handled and the manner in which it 188 is handled. This document and its companion methodology document 189 [I-D.ietf-bmwg-sip-bench-meth] provide a set of black-box benchmarks 190 for describing and comparing the performance of devices that 191 incorporate the SIP User Agent Client and Server functions and that 192 operate in the network's core. 194 The definition of SIP performance benchmarks necessarily includes 195 definitions of Test Setup Parameters and a test methodology. These 196 enable the Tester to perform benchmarking tests on different devices 197 and to achieve comparable results. This document provides a common 198 set of definitions for Test Components, Test Setup Parameters, and 199 Benchmarks. All the benchmarks defined are black-box measurements of 200 the SIP signaling plane. The Test Setup Parameters and Benchmarks 201 defined in this document are intended for use with the companion 202 Methodology document. 204 2.1. Scope 206 The scope of this work item is summarized as follows: 207 o This terminology document describes SIP signaling performance 208 benchmarks for black-box measurements of SIP networking devices. 209 Stress and debug scenarios are not addressed in this work item. 210 o The DUT must be an RFC 3261 capable network equipment. This may 211 be a Registrar, Redirect Server, Stateless Proxy or Stateful 212 Proxy. A DUT MAY also include a B2BUA, SBC functionality. 213 o The DUT MUST NOT be end user equipment, such as personal digital 214 assistant, a computer-based client, or a user terminal. 215 o The Tester acts as multiple "Emulated Agents" (EA) that initiate 216 (or respond to) SIP messages as session endpoints and source (or 217 receive) associated media for established connections. 218 o SIP Signaling in presence of Media 219 * The media performance is not benchmarked in this work item. 220 * Some tests require media, but the use of media is limited to 221 observing the performance of SIP signaling. Tests that require 222 media will annotate the media characteristics as a condition of 223 test. 224 * The type of DUT dictates whether the associated media streams 225 traverse the DUT. Both scenarios are within the scope of this 226 work item. 227 * SIP is frequently used to create media streams; the signaling 228 plane and media plane are treated as orthogonal to each other 229 in this document. While many devices support the creation of 230 media streams, benchmarks that measure the performance of these 231 streams are outside the scope of this document and its 232 companion methodology document [I-D.ietf-bmwg-sip-bench-meth]. 233 Tests may be performed with or without the creation of media 234 streams. The presence or absence of media streams MUST be 235 noted as a condition of the test as the performance of SIP 236 devices may vary accordingly. Even if the media is used during 237 benchmarking, only the SIP performance will be benchmarked, not 238 the media performance or quality. 239 o Both INVITE and non-INVITE scenarios (registrations) are addressed 240 in this document. However, benchmarking SIP presence or 241 subscribe-notify extensions is not a part of this work item. 242 o Different transport -- such as UDP, TCP, SCTP, or TLS -- may be 243 used. The specific transport mechanism MUST be noted as a 244 condition of the test as the performance of SIP devices may vary 245 accordingly. 247 o REGISTER and INVITE requests may be challenged or remain 248 unchallenged for authentication purpose. Whether or not the 249 REGISTER and INVITE requests are challenged is a condition of test 250 which will be recorded along with other such parameters which may 251 impact the SIP performance of the device or system under test. 252 o Re-INVITE requests are not considered in scope of this work item 253 since the benchmarks for INVITEs are based on the dialog created 254 by the INVITE and not on the transactions that take place within 255 that dialog. 256 o Only session establishment is considered for the performance 257 benchmarks. Session disconnect is not considered in the scope of 258 this work item. This is because our goal is to determine the 259 maximum capacity of the device or system under test, that is the 260 number of simultaneous SIP sessions that the device or system can 261 support. It is true that there are BYE requests being created 262 during the test process. These transactions do contribute to the 263 load on the device or system under test and thus are accounted for 264 in the metric we derive. We do not seek a separate metric for the 265 number of BYE transactions a device or system can support. 266 o IMS-specific scenarios are not considered, but test cases can be 267 applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT. 269 3. Term Definitions 271 3.1. Protocol Components 273 3.1.1. Session 275 Definition: 276 The combination of signaling and media messages and processes that 277 support a SIP-based service. 279 Discussion: 280 SIP messages are used to create and manage services for end users. 281 Often, these services include the creation of media streams that 282 are defined in the SDP body of a SIP message and carried in RTP 283 protocol data units. However, SIP messages can also be used to 284 create Instant Message services and subscription services, and 285 such services are not associated with media streams. SIP reserves 286 the term "session" to describe services that are analogous to 287 telephone calls on a circuit switched network. SIP reserves the 288 term "dialog" to refer to a signaling-only relationship between 289 User Agent peers. SIP reserves the term "transaction" to refer to 290 the brief communication between a client and a server that lasts 291 only until the final response to the SIP request. None of these 292 terms describes the entity whose performance we want to benchmark. 293 For example, the MESSAGE request does not create a dialog and can 294 be sent either within or outside of a dialog. It is not 295 associated with media, but it resembles a phone call in its 296 dependence on human rather than machine initiated responses. The 297 SUBSCRIBE method does create a dialog between the originating end- 298 user and the subscription service. It, too, is not associated 299 with a media session. 301 In light of the above observations we have extended the term 302 "session" to include SIP-based services that are not initiated by 303 INVITE requests and that do not have associated media. In this 304 extended definition, a session always has a signaling component 305 and may also have a media component. Thus, a session can be 306 defined as signaling-only or a combination of signaling and media. 307 We define the term "Associated Media", see Section 3.1.4, to 308 describe the situation in which media is associated with a SIP 309 dialog. The terminology "Invite-Initiated Session" (IS) 310 Section 3.1.8 and "Non-invite-Initiated Session" (NS) 311 Section 3.1.9 are used to distinguish between these two types of 312 session. An Invite-Initiated Session is a session as defined in 313 SIP. The performance of a device or system that supports Invite- 314 Initiated Sessions that do not create media sessions, "Invite- 315 Initiated Sessions without Associated Media", can be measured and 316 is of interest for comparison and as a limiting case. The 317 REGISTER request can be considered to be a "Non-invite-Initiated 318 Session without Associated Media." A separate set of benchmarks 319 is provided for REGISTER requests since most implementations of 320 SIP-based services require this request and since a registrar may 321 be a device under test. 323 A Session in the context of this document, can be considered to be 324 a vector with three components: 326 1. A component in the signaling plane (SIP messages), sess.sig; 327 2. A media component in the media plane (RTP and SRTP streams for 328 example), sess.med (which may be null); 329 3. A control component in the media plane (RTCP messages for 330 example), sess.medc (which may be null). 332 An IS is expected to have non-null sess.sig and sess.med 333 components. The use of control protocols in the media component 334 is media dependent, thus the expected presence or absence of 335 sess.medc is media dependent and test-case dependent. An NS is 336 expected to have a non-null sess.sig component, but null sess.med 337 and sess.medc components. 339 Packets in the Signaling Plane and Media Plane will be handled by 340 different processes within the DUT. They will take different 341 paths within a network. These different processes and paths may 342 produce variations in performance. The terminology and benchmarks 343 defined in this document and the methodology for their use are 344 designed to enable us to compare performance of the DUT with 345 reference to the type of SIP-supported application it is handling. 347 Note that one or more sessions can simultaneously exist between 348 any participants. This can be the case, for example, when the EA 349 sets up both an IM and a voice call through the DUT. These 350 sessions are represented as an array session[x]. 352 Sessions will be represented as a vector array with three 353 components, as follows: 354 session-> 355 session[x].sig, the signaling component 356 session[x].medc[y], the media control component (e.g. RTCP) 357 session[x].med[y], an array of associated media streams (e.g. 358 RTP, SRTP, RTSP, MSRP). This media component may consist of zero 359 or more media streams. 360 Figure 1 models the vectors of the session. 362 Measurement Units: 363 N/A. 365 Issues: 366 None. 368 See Also: 369 Media Plane 370 Signaling Plane 371 Associated Media 372 Invite-Initiated Session (IS) 373 Non-Invite-Initiated Session (NS) 374 |\ 375 | 376 | \ 377 sess.sig| 378 | \ 379 | 380 | \ 381 | o 382 | / 383 | / | 384 | / 385 | / | 386 | / 387 | / | 388 | / 389 | / | sess.medc 390 |/_____________________ 391 / / 392 / | 393 / / 394 sess.med / | 395 /_ _ _ _ _ _ _ _/ 396 / 397 / 398 / 399 / 401 Figure 1: Session components 403 3.1.2. Signaling Plane 405 Definition: 406 The plane in which SIP messages [RFC3261] are exchanged between 407 SIP Agents [RFC3261]. 409 Discussion: 410 SIP messages are used to establish sessions in several ways: 411 directly between two User Agents [RFC3261], through a Proxy Server 412 [RFC3261], or through a series of Proxy Servers. The Session 413 Description Protocol (SDP) is included in the Signaling Plane. 414 The Signaling Plane for a single Session is represented by 415 session.sig. 417 Measurement Units: 418 N/A. 420 Issues: 421 None. 423 See Also: 424 Media Plane 425 EAs 427 3.1.3. Media Plane 429 Definition: 430 The data plane in which one or more media streams and their 431 associated media control protocols are exchanged between User 432 Agents after a media connection has been created by the exchange 433 of signaling messages in the Signaling Plane. 435 Discussion: 436 Media may also be known as the "bearer channel". The Media Plane 437 MUST include the media control protocol, if one is used, and the 438 media stream(s). Examples of media are audio and video. The 439 media streams are described in the SDP of the Signaling Plane. 440 The media for a single Session is represented by session.med. The 441 media control protocol for a single media description is 442 represented by session.medc. 444 Measurement Units: 445 N/A. 447 Issues: 448 None. 450 See Also: 451 Signaling Plane 453 3.1.4. Associated Media 455 Definition: 456 Media that corresponds to an 'm' line in the SDP payload of the 457 Signaling Plane. 459 Discussion: 461 Any media protocol MAY be used. 462 For any session's signaling component, session.sig, there may be 463 zero, one, or multiple associated media streams. When there are 464 multiple media streams, these are represented be a vector array 465 session.med[y]. When there are multiple media streams there will 466 be multiple media control protocol descriptions as well. They are 467 represented by a vector array session.medc[y]. 469 Measurement Units: 470 N/A. 472 Issues: 473 None. 475 3.1.5. Overload 477 Definition: 478 Overload is defined as the state where a SIP server does not have 479 sufficient resources to process all incoming SIP messages 480 [RFC6357]. 482 Discussion: 483 The distinction between an overload condition and other failure 484 scenarios is outside the scope of black box testing and of this 485 document. Under overload conditions, all or a percentage of 486 Session Attempts will fail due to lack of resources. In black box 487 testing the cause of the failure is not explored. The fact that a 488 failure occurred for whatever reason, will trigger the tester to 489 reduce the offered load, as described in the companion methodology 490 document, [I-D.ietf-bmwg-sip-bench-meth]. SIP server resources 491 may include CPU processing capacity, network bandwidth, input/ 492 output queues, or disk resources. Any combination of resources 493 may be fully utilized when a SIP server (the DUT) is in the 494 overload condition. For proxy-only (or intermediary) devices, it 495 is expected that the proxy will be driven into overload based on 496 the delivery rate of signaling requests. 498 Measurement Units: 499 N/A. 501 3.1.6. Session Attempt 503 Definition: 505 A SIP INVITE or REGISTER request sent by the EA that has not 506 received a final response. 508 Discussion: 509 The attempted session may be Invite Initiated or Non-invite- 510 Initiated. When counting the number of session attempts we 511 include all INVITEs that are rejected for lack of authentication 512 information. The EA needs to record the total number of session 513 attempts including those attempts that are routinely rejected by a 514 proxy that requires the UA to authenticate itself. The EA is 515 provisioned to deliver a specific number of session attempts per 516 second. But the EA must also count the actual number of session 517 attempts per given tie interval. 519 Measurement Units: 520 N/A. 522 Issues: 523 None. 525 See Also: 526 Session 527 Session Attempt Rate 528 Invite-Initiated Session 529 Non-Invite-Initiated Session 531 3.1.7. Established Session 533 Definition: 534 A SIP session for which the EA acting as the UE/UA has received a 535 200 OK message. 537 Discussion: 538 An Established Session MAY be Invite Initiated or Non-Invite- 539 Initiated. Early dialogues for INVITE requests are out of scope 540 for this work. 542 Measurement Units: 543 N/A. 545 Issues: 546 None. 548 See Also: 550 Invite-Initiated Session 552 3.1.8. Invite-Initiated Session (IS) 554 Definition: 555 A Session that is created by an exchange of messages in the 556 Signaling Plane, the first of which is a SIP INVITE request. 558 Discussion: 559 When an IS becomes an Established Session its signaling component 560 is identified by the SIP dialog parameter values, Call-ID, To-tag, 561 and From-tag (RFC3261 [RFC3261]). An IS may have zero, one or 562 multiple Associated Media descriptions in the SDP body. The 563 inclusion of media is test case dependent. An IS is successfully 564 established if the following two conditions are met: 565 1. Sess.sig is established by the end of Establishment Threshold 566 Time (c.f. Section 3.3.2), and 567 2. If the DUT is a device, such as an SBC or B2BUA, that may 568 receive media from a calling or called party before a 569 signaling dialog is established or confirmed , this fact needs 570 to be taken into account when the EA is built. Any additional 571 parameters that need to be configured in the EA must be added 572 to the list of test setup parameters in Section 5.1 of 573 [I-D.ietf-bmwg-sip-bench-meth] 575 Measurement Units: 576 N/A. 578 Issues: 579 None. 581 See Also: 582 Session 583 Non-Invite-Initiated Session 584 Associated Media 586 3.1.9. Non-INVITE-Initiated Session (NS) 588 Definition: 589 A session that is created by an exchange of SIP messages in the 590 Signaling Plane the first of which is not a SIP INVITE message. 592 Discussion: 593 An NS is successfully established if the Session Attempt via a 594 non- INVITE request results in the EA receiving a 2xx reply before 595 the expiration of the Establishment Threshold timer (c.f., 596 Section 3.3.2). For the purpose of this document, a NS is a 597 session created only by the REGISTER request and no other request. 599 Measurement Units: 600 N/A. 602 Issues: 603 None. 605 See Also: 606 Session 607 Invite-Initiated Session 609 3.1.10. Session Attempt Failure 611 Definition: 612 A session attempt that does not result in an Established Session. 614 Discussion: 615 The session attempt failure may be indicated by the following 616 observations at the EA: 617 1. Receipt of a SIP 3xx-, 4xx-, 5xx-, or 6xx-class response to a 618 Session Attempt. 619 2. The lack of any received SIP response to a Session Attempt 620 within the Establishment Threshold Time (c.f. Section 3.3.2). 622 Measurement Units: 623 N/A. 625 Issues: 626 None. 628 See Also: 629 Session Attempt 631 3.2. Test Components 633 3.2.1. Emulated Agent 635 Definition: 636 A device in the test topology that initiates/responds to SIP 637 messages as one or more session endpoints and, wherever 638 applicable, sources/receives Associated Media for Established 639 Sessions. 641 Discussion: 642 The EA functions in the Signaling and Media Planes. The Tester 643 may act as multiple EAs. 645 Measurement Units: 646 N/A 648 Issues: 649 None. 651 See Also: 652 Media Plane 653 Signaling Plane 654 Established Session 655 Associated Media 657 3.2.2. Signaling Server 659 Definition: 660 Device in the test topology that facilitates the creation of 661 sessions between EAs. This device is the DUT. 663 Discussion: 664 The DUT is a RFC3261-capable network intermediary such as a 665 Registrar, Redirect Server, User Agent Server, Stateless Proxy, 666 Stateful Proxy, B2BUA or SBC. 668 Measurement Units: 669 NA 671 Issues: 672 None. 674 See Also: 675 Signaling Plane 677 3.2.3. SIP Transport Protocol 679 Definition: 680 The protocol used for transport of the Signaling Plane messages. 682 Discussion: 683 Performance benchmarks may vary for the same SIP networking device 684 depending upon whether TCP, UDP, TLS, SCTP, websockets [RFC7118] 685 or any future transport layer protocol is used. For this reason 686 it is necessary to measure the SIP Performance Benchmarks using 687 these various transport protocols. Performance Benchmarks MUST 688 report the SIP Transport Protocol used to obtain the benchmark 689 results. 691 Measurement Units: 692 While these are not units of measure, they are attributes that are 693 one of many factors that will contribute to the value of the 694 measurements to be taken. TCP, UDP, SCTP, TLS over TCP, TLS over 695 UDP, TLS over SCTP, and websockets are among the possible values 696 to be recorded as part of the test. 698 Issues: 699 None. 701 See Also: 703 3.3. Test Setup Parameters 705 3.3.1. Session Attempt Rate 707 Definition: 708 Configuration of the EA for the number of sessions per second that 709 the EA attempts to establish using the services of the DUT. 711 Discussion: 712 The Session Attempt Rate is the number of sessions per second that 713 the EA sends toward the DUT. Some of the sessions attempted may 714 not result in a session being established. A session in this case 715 may be either an IS or an NS. 717 Measurement Units: 718 Session attempts per second 720 Issues: 721 None. 723 See Also: 724 Session 725 Session Attempt 727 3.3.2. Establishment Threshold Time 729 Definition: 730 Configuration of the EA that represents the amount of time that an 731 EA client will wait for a response from an EA server before 732 declaring a Session Attempt Failure. 734 Discussion: 735 This time duration is test dependent. 737 It is RECOMMENDED that the Establishment Threshold Time value be 738 set to Timer B (for ISs) or Timer F (for NSs) as specified in RFC 739 3261, Table 4 [RFC3261]. Following the default value of T1 740 (500ms) specified in the table and a constant multiplier of 64 741 gives a value of 32 seconds for this timer (i.e., 500ms * 64 = 742 32s). 744 Measurement Units: 745 Seconds 747 Issues: 748 None. 750 3.3.3. Session Duration 752 Definition: 753 Configuration of the EA that represents the amount of time that 754 the SIP dialog is intended to exist between the two EAs associated 755 with the test. 757 Discussion: 758 The time at which the BYE is sent will control the Session 759 Duration. 761 Measurement Units: 762 seconds 764 Issues: 765 None. 767 See Also: 769 3.3.4. Media Packet Size 771 Definition: 772 Configuration on the EA for a fixed number of frames or samples to 773 be sent in each RTP packet of the media session. 775 Discussion: 777 For a single benchmark test, media sessions use a defined number 778 of samples or frames per RTP packet. If two SBCs, for example, 779 used the same codec but one puts more frames into the RTP packet, 780 this might cause variation in the performance benchmark results. 782 Measurement Units: 783 An integer number of frames or samples, depending on whether 784 hybrid- or sample-based codec are used, respectively. 786 Issues: 787 None. 789 See Also: 791 3.3.5. Codec Type 793 Definition: 794 The name of the codec used to generate the media session. 796 Discussion 797 For a single benchmark text, all sessions use the same size packet 798 for media streams. The size of packets can cause a variation in 799 the performance benchmark measurements. 801 Measurement Units: 802 This is a textual name (alphanumeric) assigned to uniquely 803 identify the codec. 805 Issues: 806 None. 807 See Also: 809 3.4. Benchmarks 811 3.4.1. Session Establishment Rate 813 Definition: 814 The maximum value of the Session Attempt Rate that the DUT can 815 handle for an extended, pre-defined, period with zero failures. 817 Discussion: 818 This benchmark is obtained with zero failure in which 100% of the 819 sessions attempted by the Emulated Agent are successfully 820 completed by the DUT. The session attempt rate provisioned on the 821 EA is raised and lowered as described in the algorithm in the 822 accompanying methodology document, until a traffic load at the 823 given attempt rate over the sustained period of time identified by 824 T in the algorithm completes without any failed session attempts. 825 Sessions may be IS or NS or a mix of both and will be defined in 826 the particular test. 828 Measurement Units: 829 sessions per second (sps) 831 Issues: 832 None. 834 See Also: 835 Invite-Initiated Sessions 836 Non-Invite-Initiated Sessions 837 Session Attempt Rate 839 3.4.2. Registration Rate 841 Definition: 842 The maximum value of the Registration Attempt Rate that the DUT 843 can handle for an extended, pre-defined, period with zero 844 failures. 846 Discussion: 847 This benchmark is obtained with zero failures in which 100% of the 848 registrations attempted by the EA are successfully completed by 849 the DUT. The registration rate provisioned on the Emulated Agent 850 is raised and lowered as described in the algorithm in the 851 companion methodology draft [I-D.ietf-bmwg-sip-bench-meth] until a 852 traffic load consisting of registration attempts at the given 853 attempt rate over the period of time necessary to attempt N 854 registrations completes without failure, where N is a parameter 855 specified in the algorithm and recorded in the Test Setup Report. 856 This benchmark is described separately from the Session 857 Establishment Rate (Section 3.4.1), although it could be 858 considered a special case of that benchmark, since a REGISTER 859 request is a request for a Non-Invite-Initiated session. It is 860 defined separately because it is a very important benchmark for 861 most SIP installations. An example demonstrating its use is an 862 avalanche restart, where hundreds of thousands of end points 863 register simultaneously following a power outage. In such a case, 864 an authoritative measurement of the capacity of the device to 865 register endpoints is useful to the network designer. Finally, in 866 certain controlled networks, there appears to be a difference in 867 the registration rate of new endpoints registering versus existing 868 endpoints refreshing their registrations. This benchmark can 869 capture these differences as well. 871 Measurement Units: 872 registrations per second (rps) 874 Issues: 875 None. 877 See Also: 879 3.4.3. Registration Attempt Rate 881 Definition: 882 Configuration of the EA for the number of registrations per second 883 that the EA attempts to send to the DUT. 885 Discussion: 886 The Registration Attempt Rate is the number of registration 887 requests per second that the EA sends toward the DUT. 889 Measurement Units: 890 Registrations per second (rps) 892 Issues: 893 None. 895 See Also: Non-Invite-Initiated Session 897 4. IANA Considerations 899 This document requires no IANA considerations. 901 5. Security Considerations 903 Documents of this type do not directly affect the security of 904 Internet or corporate networks as long as benchmarking is not 905 performed on devices or systems connected to production networks. 906 Security threats and how to counter these in SIP and the media layer 907 is discussed in RFC3261 [RFC3261], RFC 3550 [RFC3550], RFC3711 908 [RFC3711] and various other drafts. This document attempts to 909 formalize a set of common terminology for benchmarking SIP networks. 910 Packets with unintended and/or unauthorized DSCP or IP precedence 911 values may present security issues. Determining the security 912 consequences of such packets is out of scope for this document. 914 6. Acknowledgments 916 The authors would like to thank Keith Drage, Cullen Jennings, Daryl 917 Malas, Al Morton, and Henning Schulzrinne for invaluable 918 contributions to this document. Dale Worley provided an extensive 919 review that lead to improvements in the documents. We are grateful 920 to Barry Constantine for providing valuable comments during the 921 document's WGLC. 923 7. References 925 7.1. Normative References 927 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 928 Requirement Levels", BCP 14, RFC 2119, March 1997. 930 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 931 Network Interconnect Devices", RFC 2544, March 1999. 933 [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, 934 A., Peterson, J., Sparks, R., Handley, M., and E. 935 Schooler, "SIP: Session Initiation Protocol", RFC 3261, 936 June 2002. 938 [RFC5393] Sparks, R., Lawrence, S., Hawrylyshen, A., and B. Campen, 939 "Addressing an Amplification Vulnerability in Session 940 Initiation Protocol (SIP) Forking Proxies", RFC 5393, 941 December 2008. 943 [RFC4320] Sparks, R., "Actions Addressing Identified Issues with the 944 Session Initiation Protocol's (SIP) Non-INVITE 945 Transaction", RFC 4320, January 2006. 947 [RFC6026] Sparks, R. and T. Zourzouvillys, "Correct Transaction 948 Handling for 2xx Responses to Session Initiation Protocol 949 (SIP) INVITE Requests", RFC 6026, September 2010. 951 [I-D.ietf-bmwg-sip-bench-meth] 952 Davids, C., Gurbani, V., and S. Poretsky, "Methodology for 953 Benchmarking SIP Networking Devices", 954 draft-ietf-bmwg-sip-bench-meth-08 (work in progress), 955 January 2013. 957 7.2. Informational References 959 [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN 960 Switching Devices", RFC 2285, February 1998. 962 [RFC1242] Bradner, S., "Benchmarking terminology for network 963 interconnection devices", RFC 1242, July 1991. 965 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 966 Jacobson, "RTP: A Transport Protocol for Real-Time 967 Applications", STD 64, RFC 3550, July 2003. 969 [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. 970 Norrman, "The Secure Real-time Transport Protocol (SRTP)", 971 RFC 3711, March 2004. 973 [RFC6357] Hilt, V., Noel, E., Shen, C., and A. Abdelal, "Design 974 Considerations for Session Initiation Protocol (SIP) 975 Overload Control", RFC 6357, August 2011. 977 [RFC7118] Baz Castillo, I., Millan Villegas, J., and V. Pascual, 978 "The Websocket Protocol as a Transport for the Session 979 Initiation Protocol (SIP)", RFC 7118, January 2014. 981 Authors' Addresses 983 Carol Davids 984 Illinois Institute of Technology 985 201 East Loop Road 986 Wheaton, IL 60187 987 USA 989 Phone: +1 630 682 6024 990 Email: davids@iit.edu 992 Vijay K. Gurbani 993 Bell Laboratories, Alcatel-Lucent 994 1960 Lucent Lane 995 Rm 9C-533 996 Naperville, IL 60566 997 USA 999 Phone: +1 630 224 0216 1000 Email: vkg@bell-labs.com 1001 Scott Poretsky 1002 Allot Communications 1003 300 TradeCenter, Suite 4680 1004 Woburn, MA 08101 1005 USA 1007 Phone: +1 508 309 2179 1008 Email: sporetsky@allot.com