idnits 2.17.1 draft-ietf-pmol-sip-perf-metrics-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document seems to contain a disclaimer for pre-RFC5378 work, and may have content which was first submitted before 10 November 2008. The disclaimer is necessary when there are original authors that you have been unable to contact, or if some do not wish to grant the BCP78 rights to the IETF Trust. If you are able to get all authors (current and original) to grant those rights, you can and should remove the disclaimer; otherwise, the disclaimer is needed and you can ignore this comment. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (September 21, 2010) is 4965 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Obsolete normative reference: RFC 3265 (Obsoleted by RFC 6665) Summary: 1 error (**), 0 flaws (~~), 1 warning (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 PMOL D. Malas 3 Internet-Draft CableLabs 4 Intended status: Standards Track A. Morton 5 Expires: March 25, 2011 AT&T Labs 6 September 21, 2010 8 Basic Telephony SIP End-to-End Performance Metrics 9 draft-ietf-pmol-sip-perf-metrics-07 11 This document may contain material from IETF Documents or IETF 12 Contributions published or made publicly available before November 13 10, 2008. The person(s) controlling the copyright in some of this 14 material may not have granted the IETF Trust the right to allow 15 modifications of such material outside the IETF Standards Process. 16 Without obtaining an adequate license from the person(s) controlling 17 the copyright in such materials, this document may not be modified 18 outside the IETF Standards Process, and derivative works of it may 19 not be created outside the IETF Standards Process, except to format 20 it for publication as an RFC or to translate it into languages other 21 than English. 23 Abstract 25 This document defines a set of metrics and their usage to evaluate 26 the performance of end-to-end Session Initiation Protocol (SIP) for 27 telephony services in both production and testing environments. The 28 purpose of this document is to combine a standard set of common 29 metrics, allowing interoperable performance measurements, easing the 30 comparison of industry implementations. 32 Status of this Memo 34 This Internet-Draft is submitted in full conformance with the 35 provisions of BCP 78 and BCP 79. 37 Internet-Drafts are working documents of the Internet Engineering 38 Task Force (IETF). Note that other groups may also distribute 39 working documents as Internet-Drafts. The list of current Internet- 40 Drafts is at http://datatracker.ietf.org/drafts/current/. 42 Internet-Drafts are draft documents valid for a maximum of six months 43 and may be updated, replaced, or obsoleted by other documents at any 44 time. It is inappropriate to use Internet-Drafts as reference 45 material or to cite them other than as "work in progress." 47 This Internet-Draft will expire on March 25, 2011. 49 Copyright Notice 51 Copyright (c) 2010 IETF Trust and the persons identified as the 52 document authors. All rights reserved. 54 This document is subject to BCP 78 and the IETF Trust's Legal 55 Provisions Relating to IETF Documents 56 (http://trustee.ietf.org/license-info) in effect on the date of 57 publication of this document. Please review these documents 58 carefully, as they describe your rights and restrictions with respect 59 to this document. Code Components extracted from this document must 60 include Simplified BSD License text as described in Section 4.e of 61 the Trust Legal Provisions and are provided without warranty as 62 described in the Simplified BSD License. 64 Table of Contents 66 1. Introduction and Scope . . . . . . . . . . . . . . . . . . . . 4 67 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 68 3. Time Interval Measurement and Reporting . . . . . . . . . . . 6 69 4. SIP Performance Metrics . . . . . . . . . . . . . . . . . . . 7 70 4.1. Registration Request Delay (RRD) . . . . . . . . . . . . . 8 71 4.2. Ineffective Registration Attempts (IRA) . . . . . . . . . 9 72 4.3. Session Request Delay (SRD) . . . . . . . . . . . . . . . 11 73 4.3.1. Successful Session Setup SRD . . . . . . . . . . . . . 11 74 4.3.2. Failed Session Setup SRD . . . . . . . . . . . . . . . 12 75 4.4. Session Disconnect Delay (SDD) . . . . . . . . . . . . . . 13 76 4.5. Session Duration Time (SDT) . . . . . . . . . . . . . . . 15 77 4.5.1. Successful session duration SDT . . . . . . . . . . . 16 78 4.5.2. Failed session completion SDT . . . . . . . . . . . . 17 79 4.6. Session Establishment Ratio (SER) . . . . . . . . . . . . 19 80 4.7. Session Establishment Effectiveness Ratio (SEER) . . . . . 20 81 4.8. Ineffective Session Attempts (ISA) . . . . . . . . . . . . 21 82 4.9. Session Completion Ratio (SCR) . . . . . . . . . . . . . . 22 83 5. Additional Considerations . . . . . . . . . . . . . . . . . . 23 84 5.1. Metric Correlations . . . . . . . . . . . . . . . . . . . 23 85 5.2. Back-to-back User Agent (B2BUA) . . . . . . . . . . . . . 24 86 5.3. Authorization and Authentication . . . . . . . . . . . . . 24 87 5.4. Forking . . . . . . . . . . . . . . . . . . . . . . . . . 24 88 5.5. Data Collection . . . . . . . . . . . . . . . . . . . . . 25 89 5.6. Testing Documentation . . . . . . . . . . . . . . . . . . 26 90 6. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 26 91 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 26 92 8. Security Considerations . . . . . . . . . . . . . . . . . . . 26 93 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 26 94 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 27 95 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 27 96 11.1. Normative References . . . . . . . . . . . . . . . . . . . 27 97 11.2. Informative References . . . . . . . . . . . . . . . . . . 27 98 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 28 100 1. Introduction and Scope 102 SIP has become a widely-used standard among many service providers, 103 vendors, and end users in the telecommunications industry. Although 104 there are many different standards for measuring the performance of 105 telephony signaling protocols, such as SS7, none of the metrics 106 specifically address SIP. 108 The scope of this document is limited to the definitions of a 109 standard set of metrics for measuring and reporting SIP performance 110 from an end-to-end perspective in a telephony environment. The 111 metrics introduce a common foundation for understanding and 112 quantifying performance expectations between service providers, 113 vendors, and the users of services based on SIP. The intended 114 audience for this document can be found among network operators, who 115 often collect information on the responsiveness of the network to 116 customer requests for services. 118 Measurements of the metrics described in this document are affected 119 by variables external to SIP. The following is a non-exhaustive list 120 of examples: 122 o Network connectivity 124 o Switch and router performance 126 o Server processes and hardware performance 128 This document defines a list of pertinent metrics for varying aspects 129 of a telephony environment. They may be used individually or as a 130 set based on the usage of SIP within the context of a given 131 telecommunications service. 133 The metrics defined in this document DO NOT take into consideration 134 the impairment or failure of actual application processing of a 135 request or response. The metrics do not distinguish application 136 processing time from other sources of delay, such as packet transfer 137 delay. 139 Metrics designed to quantify single device application processing 140 performance are beyond the scope of this document. 142 This document does not provide any numerical objectives or acceptance 143 threshold values for the SIP performance metrics defined below, as 144 these items are beyond the scope of IETF activities, in general. 146 The metrics defined in this document are applicable in scenarios 147 where the SIP messages launched (into a network under test) are 148 dedicated messages for testing purposes, or where the messages are 149 user-initiated and a portion of the live traffic present. These two 150 scenarios are sometimes referred to as active and passive 151 measurement, respectively. 153 2. Terminology 155 The following terms and conventions will be used throughout this 156 document: 158 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 159 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 160 document are to be interpreted as described in RFC 2119 [RFC2119]. 162 End-to-End - This is described as two or more elements utilized for 163 initiating a request, receiving the request, and responding to the 164 request. It encompasses elements as necessary to be involved in a 165 session dialog between the originating user agent client (UAC), 166 destination user agent server (UAS), and any interim proxies (may 167 also include back-to-back user agent's (B2BUA's)). This may be 168 relative to a single operator's set of elements or extend to 169 encompass all elements (if beyond a single operator's network) 170 associated with a session. 172 Session - As described in RFC 3261 [RFC3261], SIP is used primarily 173 to request, create, and conclude sessions. "These sessions include 174 Internet telephone calls, multimedia distribution, and multimedia 175 conferences." The metrics within this document measure the 176 performance associated with the SIP dialogs necessary to establish 177 these sessions; therefore, they are titled as: Session Request Delay, 178 Session Disconnect Delay, etc. Although the titles of many of the 179 metrics include this term, they are specifically measuring the 180 signaling aspects only. Each session is identified by a unique 181 Call-ID, "To" and "From" header field tag. 183 Session Establishment - Session establishment occurs when a 200 OK 184 response from the target UA has been received, in response to the 185 originating UA's INVITE setup request, indicating the session setup 186 request was successful. 188 Session Setup - As referenced within the sub-sections of 4.2 in this 189 document, session setup is the set of messages and included 190 parameters directly related to the process of a UA requesting to 191 establish a session with a corresponding UA. This is also described 192 as a set of steps in order to establish "ringing" [RFC3261]. 194 3. Time Interval Measurement and Reporting 196 Many of the metrics defined in this memo utilize a clock to assess 197 the time interval between two events. This section defines time- 198 related terms and reporting requirements. 200 t1 - start time 202 This is the time instant (when a request is sent) that begins a 203 continuous time interval. T1 occurs when the designated request has 204 been processed by the SIP application and the first bit of the 205 request packet has been sent from the UA or proxy (and is externally 206 observable at some logical or physical interface). 208 T1 represents the time at which each request-response test begins, 209 and SHALL be used to designate the time-of-day when a particular 210 measurement was conducted (e.g., The Session Request Delay at "t1" 211 and (some specific UA interface) was measured to be X ms.) 213 t4 - end time 215 This is the time instant that concludes the continuous time interval 216 begun when the related request is sent. t4 occurs when the last bit 217 of the designated response is received by the SIP application at the 218 requesting device (and is externally observable at some logical or 219 physical interface). 221 Note: The designations t2 and t3 are reserved for future use at 222 another interface involved in satisfying a request. 224 Section 10.1 of [RFC2330] describes time-related issues in 225 measurements, and defines the errors that can be attributed to the 226 clock themselves. These definitions are used in the material below. 228 Time of Day Accuracy 230 As defined above, t1 is associated with the start of a request and 231 also serves as the time-of-day stamp associated with a single 232 specific measurement. The clock offset [RFC2330] is the difference 233 between t1 and a recognized primary source of time, such as UTC 234 (offset = t1 - UTC). 236 When measurement results will be correlated with other results or 237 information using time-of-day stamps, then the time clock that 238 supplies t1 SHOULD be synchronized to a primary time source, to 239 minimize the clock's offset. The clocks used at the different 240 measurement points SHOULD be synchronized to each other, to minimize 241 the relative offset (as defined in RFC2330). The clock's offset and 242 the relative offset MUST be reported with each measurement. 244 Time Interval Accuracy 246 The accuracy of the t4-t1 interval is also critical to maintain and 247 report. The difference between a clock's offsets at t1 and t4 is one 248 source of error for the measurement and is associated with the 249 clock's skew [RFC2330]. 251 A stable and reasonably accurate clock is needed to make the time 252 interval measurements required by this memo. This source of error 253 SHOULD be constrained to less than +/- 1 ms, implying 1 part per 1000 254 frequency accuracy for a 1 second interval. This implies greater 255 stability is required as the length of the t4-t1 increases, in order 256 to constrain the error to be less than +/- 1ms. 258 There are several other important aspects of clock operation: 260 1. Synchronization protocols require some ability to make 261 adjustments to the local clock. However, these adjustments 262 (clock steps or slewing) can cause large errors if they occur 263 during the t1 to t4 measurement interval. Clock correction 264 SHOULD be suspended during a t1 to t4 measurement interval, 265 unless the time interval accuracy requirement above will be met. 266 Alternatively, a measurement SHOULD NOT be performed during clock 267 correction, unless the time interval accuracy requirement above 268 will be met. 270 2. If a free-running clock is used to make the time interval 271 measurement, then the time of day reported with the measurement 272 (which is normally timestamp t1) SHOULD be derived from a 273 different clock that meets the time of day accuracy requirements 274 described above. 276 The physical operation of reading time from a clock may be 277 constrained by the delay to service the interrupt. Therefore, if the 278 accuracy of the time stamp read at t1 or t4 includes the interrupt 279 delay, this source of error SHOULD be known and included in the error 280 assessment. 282 4. SIP Performance Metrics 284 In regards to all of the following metrics, t1 begins with the first 285 associated SIP message sent by either UA, and is not reset if the UA 286 must retransmit the same message, within the same transaction, 287 multiple times. The first associated SIP message indicates the t1 288 associated with the user or application expectation relative to the 289 request. 291 Some metrics are calculated using messages from different 292 transactions in order to measure across actions such as redirection 293 and failure recovery. The end time is typically based on a 294 successful end-to-end provisional response, a successful final 295 response, or a failure final response for which there is no recovery. 296 The individual metrics detail which message to base the end time on. 298 The authentication method used to establish the SIP dialog will 299 change the message exchanges. The example message exchanges used do 300 not attempt to describe all of the various authentication types. 301 Since, authentication is frequently used, SIP Digest authentication 302 was used for example purposes. 304 In regards to all of the metrics, the accuracy and granularity of the 305 output values are related to the accuracy and granularity of the 306 input values. Some of the metrics below are defined by a ratio. 307 When the denominator of this ratio is 0, the metric is undefined. 309 While these metrics do not specify the sample size. This should be 310 taken into consideration. These metrics will provide a better 311 indication of performance with larger sample sets. For example, some 312 SIP Service Providers (SSPs) [RFC5486] may choose to collect input 313 over an hour, daily, weekly or monthly timeframe, while another SSP 314 may choose to perform metric calculations over a varying set of SIP 315 dialogs. 317 4.1. Registration Request Delay (RRD) 319 Registration Request Delay (RRD) is a measurement of the delay in 320 responding to a UA REGISTER request. RRD SHALL be measured and 321 reported only for successful REGISTER requests, while Ineffective 322 Registration Attempts (Section 4.2) SHALL be reported for failures. 323 This metric is measured at the originating UA. The output value of 324 this metric is numerical and SHOULD be stated in units of 325 milliseconds. The RRD is calculated using the following formula: 327 RRD = Time of Final Response - Time of REGISTER Request 329 In a successful registration attempt, RRD is defined as the time 330 interval from the first bit of the initial REGISTER message 331 containing the necessary information is passed by the originating UA 332 to the intended registrar until the last bit of the 200 OK is 333 received indicating the registration attempt has completed 334 successfully. This dialog includes an expected authentication 335 challenge prior to receiving the 200 OK as described in the following 336 registration flow examples. 338 The following message exchange provides an example of identifiable 339 events necessary for inputs in calculating RRD during a successful 340 registration completion: 342 UA1 Registrar 343 | | 344 |REGISTER | 345 t1---->|--------------------->| 346 /\ | 401| 347 || |<---------------------| 348 RRD |REGISTER | 349 || |--------------------->| 350 \/ | 200| 351 t4---->|<---------------------| 352 | | 354 Note: Networks with elements using primarily Digest authentication 355 will exhibit different RRD characteristics than networks with 356 elements primarily using other authentication mechanisms (such as 357 Identity). Operators monitoring RRD in networks with a mixture of 358 authentications schemes should take note that the RRD measurements 359 will likely have a multimodal distribution. 361 4.2. Ineffective Registration Attempts (IRA) 363 Ineffective registration attempts are utilized to detect failures or 364 impairments causing an inability for a registrar to receive a UA 365 REGISTER request. This metric is measured at the originating UA. 366 The output value of this metric is numerical and SHOULD be reported 367 as a percentage of registration attempts. 369 This metric is calculated as a percentage of total REGISTER requests. 370 The IRA is calculated using the following formula: 372 # of IRA 373 IRA % = ----------------------------- x 100 374 Total # of REGISTER Requests 376 A failed registration attempt is defined as a final failure response 377 to the initial REGISTER request. It usually indicates a failure 378 received from the destination registrar, interim proxies, or due to a 379 timeout of the REGISTER request at the originating UA. A failure 380 response is described as a 4XX (excluding 401, 402, and 407 non- 381 failure challenge response codes), 5XX, or possible 6XX message. A 382 timeout failure is identified by the timer F expiring. IRA may be 383 used to detect problems in downstream signaling functions, which may 384 be impairing the REGISTER message from reaching the intended 385 registrar; or, it may indicate a registrar has become overloaded and 386 is unable to respond to the request. 388 The following message exchange provides a timeout example of an 389 identifiable event necessary for input as a failed registration 390 attempt: 392 UA1 Registrar 393 | | 394 |REGISTER | 395 |--------------------->| 396 |REGISTER | 397 |--------------------->| 398 |REGISTER | 399 |--------------------->| 400 | | 401 Failure ---->|***Timer F Expires | 402 | | 404 In the previous message exchange UA1 retries a REGISTER request 405 multiple times before the timer, indicating the failure, expires. 406 Only the first REGISTER request MUST used for input to the 407 calculation and an IRA. Subsequent REGISTER retries are identified 408 by the same transaction identifier (same topmost Via header field 409 branch parameter value) and MUST be ignored for purposes of metric 410 calculation. This ensures an accurate representation of the metric 411 output. 413 The following message exchange provides a registrar servicing failure 414 example of an identifiable event necessary for input as a failed 415 registration attempt: 417 UA1 Registrar 418 | | 419 |REGISTER | 420 |--------------------->| 421 | | 422 | | 423 | | 424 | | 425 | 503| 426 Failure ---->|<---------------------| 427 | | 429 4.3. Session Request Delay (SRD) 431 Session Request Delay (SRD) is utilized to detect failures or 432 impairments causing delays in responding to a UA session request. 433 SRD is measured for both successful and failed session setup requests 434 as this metric usually relates to a user experience; however, SRD for 435 session requests ending in a failure MUST NOT be combined in the same 436 result with successful requests. The duration associated with 437 success and failure responses will likely vary substantially, and the 438 desired output time associated with each will be significantly 439 different in many cases. This metric is similar to Post-Selection 440 Delay defined in [E.721], and it is measured at the originating UA 441 only. The output value of this metric MUST indicate whether the 442 output is for successful or failed session requests and SHOULD be 443 stated in units of seconds. The SRD is calculated using the 444 following formula: 446 SRD = Time of Status Indicative Response - Time of INVITE 448 4.3.1. Successful Session Setup SRD 450 In a successful request attempt, SRD is defined as the time interval 451 from the first bit of the initial INVITE message containing the 452 necessary information is sent by the originating user agent to the 453 intended mediation or destination agent until the last bit of the 454 first provisional response is received indicating an audible or 455 visual status of the initial session setup request. (Note: In some 456 cases, the initial INVITE may be forked. Section 5.4 provides 457 information for consideration on forking.) In SIP, the message 458 indicating status would be a non-100 Trying provisional message 459 received in response to an INVITE request. In some cases, a non-100 460 Trying provisional message is not received, but rather a 200 message 461 is received as the first status message instead. In these 462 situations, the 200 message would be used to calculate the interval. 463 In most circumstances, this metric relies on receiving a non-100 464 Trying message. The use of PRACK [RFC3262] MAY improve the quality 465 and consistency of the results. 467 The following message exchange provides an example of identifiable 468 events necessary for inputs in calculating SRD during a successful 469 session setup request without a redirect (i.e. 3XX message): 471 UA1 UA2 472 | | 473 |INVITE | 474 t1---->|--------------------->| 475 /\ | | 476 || | | 477 SRD | | 478 || | | 479 \/ | 180| 480 t4---->|<---------------------| 481 | | 483 The following message exchange provides an example of identifiable 484 events necessary for inputs in calculating SRD during a successful 485 session setup with a redirect (e.g. 302 Moved Temporarily): 487 UA1 Redirect Server UA2 488 | | | 489 |INVITE | | 490 t1---->|--------------------->| | 491 /\ | 302| | 492 || |<---------------------| | 493 || |ACK | | 494 SRD |--------------------->| | 495 || |INVITE | 496 || |------------------------------------------->| 497 \/ | 180| 498 t4---->|<-------------------------------------------| 500 4.3.2. Failed Session Setup SRD 502 In a failed request attempt, SRD is defined as the time interval from 503 the first bit of the initial INVITE message containing the necessary 504 information sent by the originating agent or user to the intended 505 mediation or destination agent until the last bit of the first 506 provisional response or a failure indication response. A failure 507 response is described as a 4XX (excluding 401, 402, and 407 non- 508 failure challenge response codes), 5XX, or possible 6XX message. A 509 change in the metric output might indicate problems in downstream 510 signaling functions, which may be impairing the INVITE message from 511 reaching the intended UA or may indicate changes in end-point 512 behavior. While this metric calculates the delay associated with a 513 failed session request, the metric Ineffective Session Attempts 514 (Section 4.10) is used for calculating a ratio of session attempt 515 failures. 517 The following message exchange provides an example of identifiable 518 events necessary for inputs in calculating SRD during a failed 519 session setup attempt without a redirect (i.e. 3XX message): 521 UA1 UA2 522 | | 523 |INVITE | 524 t1---->|--------------------->| 525 /\ | | 526 || | | 527 SRD | | 528 || | | 529 \/ | 480| 530 t4---->|<---------------------| 531 | | 533 The following message exchange provides an example of identifiable 534 events necessary for inputs in calculating SRD during a failed 535 session setup attempt with a redirect (e.g. 302 Moved Temporarily): 537 UA1 Redirect Server UA2 538 | | | 539 |INVITE | | 540 t1---->|--------------------->| | 541 /\ | 302| | 542 || |<---------------------| | 543 || |ACK | | 544 SRD |--------------------->| | 545 || |INVITE | 546 || |------------------------------------------->| 547 \/ | 480| 548 t4---->|<-------------------------------------------| 550 4.4. Session Disconnect Delay (SDD) 552 This metric is utilized to detect failures or impairments delaying 553 the time necessary to end a session. SDD is measured for both 554 successful and failed session disconnects; however, SDD for session 555 disconnects ending in a failure MUST NOT be combined in the same 556 result with successful disconnects. The duration associated with 557 success and failure results will likely vary substantially, and the 558 desired output time associated with each will be significantly 559 different in many cases. It can be measured from either end-point 560 UAs involved in the SIP dialog. The output value of this metric is 561 numerical and SHOULD be stated in units of milliseconds. The SDD is 562 calculated using the following formula: 564 SDD = Time of 2XX or Timeout - Time of Completion Message (BYE) 566 SDD is defined as the interval between the first bit of the sent 567 session completion message, such as a BYE, and the last bit of the 568 subsequently received 2XX response. In some cases a recoverable 569 error response, such as a 503 retry-after, may be received. In such 570 situations, these responses should not be used as the end time for 571 this metric calculation. Instead the successful (2XX) response 572 related to the recovery message is used. The following message 573 exchanges provide an example of identifiable events necessary for 574 inputs in calculating SDD during a successful session completion: 576 Measuring SDD at the originating UA (UA1) - 578 UA1 UA2 579 | | 580 |INVITE | 581 |--------------------->| 582 | 180| 583 |<---------------------| 584 | 200| 585 |<---------------------| 586 |ACK | 587 |--------------------->| 588 |BYE | 589 t1---->|--------------------->| 590 /\ | | 591 || | | 592 SDD | | 593 || | | 594 \/ | 200| 595 t4---->|<---------------------| 597 Measuring SDD at the target UA (UA2) - 598 UA1 UA2 599 | | 600 |INVITE | 601 |--------------------->| 602 | 180| 603 |<---------------------| 604 | 200| 605 |<---------------------| 606 |ACK | 607 |--------------------->| 608 | BYE| 609 |<---------------------|<----t1 610 | | /\ 611 | | || 612 | | SDD 613 | | || 614 |200 | \/ 615 |--------------------->|<----t4 617 In some cases, no response is received after a session completion 618 message is sent and potentially retried. In this case, the 619 completion message, such as a BYE, results in a Timer F expiration. 620 Sessions ending in this manner SHOULD be excluded from the metric 621 calculation. 623 4.5. Session Duration Time (SDT) 625 This metric is used to detect problems (e.g. poor audio quality) 626 causing short session durations. SDT is measured for both successful 627 and failed session completions. It can be measured from either end- 628 point UAs involved in the SIP dialog. This metric is similar to Call 629 Hold Time, and it is traditionally calculated as Average Call Hold 630 Time (ACHT) in telephony applications of SIP. The output value of 631 this metric is numerical and SHOULD be stated in units of seconds. 632 The SDT is calculated using the following formula: 634 SDT = Time of BYE or Timeout - Time of 200 OK response to INVITE 636 This metric does not calculate the duration of sessions leveraging 637 early media. For example, some automated response systems only use 638 early media by responding with a SIP 183 Session in Progress message 639 with SDP connecting the originating UA with the the automated 640 message. Usually, in these sessions the originating UA never 641 receives a 200 OK, and the message exchange ends with the originating 642 UA sending a CANCEL. 644 4.5.1. Successful session duration SDT 646 In a successful session completion, SDT is calculated as an average 647 and is defined as the duration of a dialog defined by the interval 648 from receipt of the first bit of a 200 OK response to an INVITE and 649 receipt of the last bit of an associated BYE message indicating 650 dialog completion. Retransmissions of the 200 OK and ACK messages 651 due to network impairments do not reset the metric timers. 653 The following message exchanges provide an example of identifiable 654 events necessary for inputs in calculating SDT during a successful 655 session completion (The message message exchanges are changed between 656 the originating and target UAs to provide varying examples.): 658 Measuring SDT at the originating UA (UA1) - 660 UA1 UA2 661 | | 662 |INVITE | 663 |--------------------->| 664 | 180| 665 |<---------------------| 666 | 200| 667 t1---->|<---------------------| 668 /\ |ACK | 669 || |--------------------->| 670 || | | 671 SDT | | 672 || | | 673 || | | 674 \/ | BYE| 675 t4---->|<---------------------| 676 | | 678 When measuring SDT at the target UA (UA2), it is defined by the 679 interval from sending the first bit of a 200 OK response to an INVITE 680 and receipt of the last bit of an associated BYE message indicating 681 dialog completion. If UA2 initiates the BYE, then it is defined by 682 the interval from sending the first bit of a 200 OK response to an 683 INVITE and sending the first bit of an associated BYE message 684 indicating dialog completion. This is illustrated in the following 685 example message exchange - 686 UA1 UA2 687 | | 688 |INVITE | 689 |--------------------->| 690 | 180| 691 |<---------------------| 692 | 200| 693 |<---------------------|<----t1 694 |ACK | /\ 695 |--------------------->| || 696 | | || 697 | | SDT 698 | | || 699 | | || 700 | BYE| \/ 701 |<---------------------|<----t4 702 | | 704 (In these two examples, t1 is the same even if either UA receives the 705 BYE instead of sending it.) 707 4.5.2. Failed session completion SDT 709 In some cases, no response is received after a session completion 710 message is sent and potentially retried. In this case, SDT is 711 defined as the interval between receiving the first bit of a 200 OK 712 response to an INVITE, and the resulting Timer F expiration. The 713 following message exchanges provide an example of identifiable events 714 necessary for inputs in calculating SDT during a failed session 715 completion attempt: 717 Measuring SDT at the originating UA (UA1) - 718 UA1 UA2 719 | | 720 |INVITE | 721 |--------------------->| 722 | 180| 723 |<---------------------| 724 | 200| 725 t1---->|<---------------------| 726 /\ |ACK | 727 || |--------------------->| 728 || |BYE | 729 SDT |--------------------->| 730 || |BYE | 731 || |--------------------->| 732 \/ | | 733 t4---->|***Timer F Expires | 735 When measuring SDT at UA2, SDT is defined as the interval between 736 sending the first bit of a 200 OK response to an INVITE, and the 737 resulting Timer F expiration. This is illustrated in the following 738 example message exchange - 740 UA1 UA2 741 | | 742 |INVITE | 743 |--------------------->| 744 | 180| 745 |<---------------------| 746 | 200| 747 |<---------------------|<----t1 748 | ACK| /\ 749 |--------------------->| || 750 | BYE| || 751 |<---------------------| SDT 752 | BYE| || 753 |<---------------------| || 754 | | \/ 755 | Timer F Expires***|<----t4 757 Note that in the presence of message loss and retransmission, the 758 value of this metric measured at UA1 may differ from the value 759 measured at UA2 up to the value of Timer F. 761 4.6. Session Establishment Ratio (SER) 763 This metric is used to detect the ability of a terminating UA or 764 downstream proxy to successfully establish sessions per new session 765 INVITE requests. SER is defined as the number of new session INVITE 766 requests resulting in a 200 OK response, to the total number of 767 attempted INVITE requests less INVITE requests resulting in a 3XX 768 response. This metric is similar to Answer Seizure Ratio (ASR) 769 defined in [E.411]. It is measured at the originating UA only. The 770 output value of this metric is numerical and SHOULD be adjusted to 771 indicate a percentage of successfully established sessions. The SER 772 is calculated using the following formula metric: 774 # of INVITE Requests w/ associated 200 775 SER = --------------------------------------------------------- x 100 776 (Total # of INVITE Requests)-(# of INVITE Requests w/ 3XX Response) 778 The following message exchange provides an example of identifiable 779 events necessary for inputs in determining session establishment as 780 described above: 782 UA1 UA2 783 | | 784 |INVITE | 785 +----------->|------------------>| 786 | | 180| 787 | |<------------------| 788 Session Established | | 789 | | | 790 | | 200| 791 +----------->|<------------------| 792 | | 794 The following is an example message exchange including a SIP 302 795 Redirect response. 797 UA1 UA2 UA3 798 | | | 799 |INVITE | | 800 +----------->|------------------>| | 801 | | | | 802 INVITE w/ 3XX Response | | | 803 | | 302| | 804 +----------->|<------------------| | 805 | | | 806 |INVITE | 807 +----------->|-------------------------------------->| 808 | | | 809 | | 180| 810 Session Established |<--------------------------------------| 811 | | | 812 | | 200| 813 +----------->|<--------------------------------------| 814 | | 816 4.7. Session Establishment Effectiveness Ratio (SEER) 818 This metric is complimentary to SER, but is intended to exclude the 819 potential effects of an individual user of the target UA from the 820 metric. SER is defined as the number of INVITE requests resulting in 821 a 200 OK response and INVITE requests resulting in a 480, 486, 600 or 822 603; to the total number of attempted INVITE requests less INVITE 823 requests resulting in a 3XX response. The response codes 480, 486, 824 600 and 603 were chosen, because they clearly indicate the effect of 825 an individual user of the UA. It is possible an individual user 826 could cause a negative effect on the UA. For example, they may have 827 misconfigured the UA causing a response code not directly related to 828 a SSP, but this cannot be easily determined from an intermediary 829 B2BUA somewhere inbetween the originating and terminating UA. With 830 this in consideration, response codes such as 401, 407 and 420 (not 831 an exhaustive list) were not included in the numerator of the metric. 832 This metric is similar to Network Effectiveness Ratio (NER) defined 833 in [E.411]. It is measured at the originating UA only. The output 834 value of this metric is numerical and SHOULD be adjusted to indicate 835 a percentage of successfully established sessions less common UAS 836 failures. 838 The SEER is calculated using the following formula: 840 # of INVITE Requests w/ associated 200, 480, 486, 600 or 603 841 SEER = -------------------------------------------------------- x 100 842 (Total # of INVITE Requests)-(# of INVITE Requests w/ 3XX Response) 844 Reference the example flows is Section 4.6. 846 4.8. Ineffective Session Attempts (ISA) 848 Ineffective session attempts occur when a proxy or agent internally 849 releases a setup request with a failed or overloaded condition. This 850 metric is similar to Ineffective Machine Attempts (IMA) in telephony 851 applications of SIP, and was adopted from Telcordia GR-512-CORE 852 [GR-512]. The output value of this metric is numerical and SHOULD be 853 adjusted to indicate a percentage of ineffective session attempts. 854 The following failure responses provide a guideline for this 855 criterion: 857 o 408 Request Timeout 859 o 500 Server Internal Error 861 o 503 Service Unavailable 863 o 504 Server Timeout 865 This set was derived in a similar manner as described in Section 4.9, 866 in addition 408 failure responses may indicate an overloaded state 867 with a downstream element; however, there are situations other than 868 overload, which may cause an increase in 408 responses. 870 This metric is calculated as a percentage of total session setup 871 requests. The ISA is calculated using the following formula: 873 # of ISA 874 ISA % = ----------------------------- x 100 875 Total # of Session Requests 877 The following dialog [RFC3665] provides an example describing message 878 exchanges of an ineffective session attempt: 880 UA1 Proxy 1 Proxy 2 UA2 881 | | | | 882 |INVITE | | | 883 |--------------->| | | 884 | 407| | | 885 |<---------------| | | 886 |ACK | | | 887 |--------------->| | | 888 |INVITE | | | 889 |--------------->|INVITE | | 890 | 100|--------------->|INVITE | 891 |<---------------| 100|--------------->| 892 | |<---------------| | 893 | | |INVITE | 894 | | |--------------->| 895 | | | | 896 | | |INVITE | 897 | | |--------------->| 898 | | | | 899 | | 408| | 900 | 408|<---------------| | 901 |<---------------|ACK | | 902 | |--------------->| | 903 |ACK | | | 904 |--------------->| | | 906 4.9. Session Completion Ratio (SCR) 908 A session completion is defined as a SIP dialog, which completes 909 without failing due to a lack of response from an intended proxy or 910 UA. This metric is similar to Call Completion Ratio (CCR) in 911 telephony applications of SIP. The output value of this metric is 912 numerical and SHOULD be adjusted to indicate a percentage of 913 successfully completed sessions. 915 This metric is calculated as a percentage of total sessions completed 916 successfully. The SCR is calculated using the following formula: 918 # of Successfully Completed Sessions 919 SCR % = --------------------------------------- x 100 920 Total # of Session Requests 922 The following dialog [RFC3665] provides an example describing the 923 necessary message exchanges of a successful session completion: 925 UA1 Proxy 1 Proxy 2 UA2 926 | | | | 927 |INVITE | | | 928 |--------------->| | | 929 | 407| | | 930 |<---------------| | | 931 |ACK | | | 932 |--------------->| | | 933 |INVITE | | | 934 |--------------->|INVITE | | 935 | 100|--------------->|INVITE | 936 |<---------------| 100|--------------->| 937 | |<---------------| | 938 | | | 180| 939 | | 180 |<---------------| 940 | 180|<---------------| | 941 |<---------------| | 200| 942 | | 200|<---------------| 943 | 200|<---------------| | 944 |<---------------| | | 945 |ACK | | | 946 |--------------->|ACK | | 947 | |--------------->|ACK | 948 | | |--------------->| 949 | Both Way RTP Media | 950 |<================================================>| 951 | | | BYE| 952 | | BYE|<---------------| 953 | BYE|<---------------| | 954 |<---------------| | | 955 |200 | | | 956 |--------------->|200 | | 957 | |--------------->|200 | 958 | | |--------------->| 959 | | | | 961 5. Additional Considerations 963 5.1. Metric Correlations 965 These metrics may be used to determine the performance of a domain 966 and/or user. This would be to provide a metric relative to one or 967 more dimensions. The following is an example subset of dimensions 968 for providing further granularity per metric: 970 o To "user" 972 o From "user" 974 o Bi-direction "user" 976 o To "domain" 978 o From "domain" 980 o Bi-direction "domain" 982 5.2. Back-to-back User Agent (B2BUA) 984 A B2BUA may impact the ability to collect these metrics with an end- 985 to-end perspective. It is necessary to realize a B2BUA may act as an 986 originating UAC and terminating UAS or it may act as a proxy. In 987 some cases, it may be necessary to consider information collected 988 from both sides of the B2BUA in order to determine the end-to-end 989 perspective. In other cases, the B2BUA may act simply as a proxy 990 allowing data to be derived as necessary for the input into any of 991 the listed calculations. 993 5.3. Authorization and Authentication 995 During the process of setting up a SIP dialog, various authentication 996 methods may be utilized. These authentication methods will add to 997 the duration as measured by the metrics, and the length of time will 998 vary based on those methods. The failures of these authentication 999 methods will also be captured by these metrics, since SIP is 1000 ultimately used to indicate the success or failure of the 1001 authorization and/or authentication attempt. The metrics in section 1002 3 are inclusive of the duration associated with this process, even if 1003 the method is external to the SIP protocol. This was included 1004 purposefully, due to its inherent impact on the protocol and the 1005 subsequent SIP dialogs. 1007 5.4. Forking 1009 Forking SHOULD be considered when determining the messages associated 1010 with the input values for the described metrics. If all of the 1011 forked dialogs were used in the metric calculations, the numbers 1012 would skew dramatically. There are two different points of forking, 1013 which MUST be considered. First, forking may occur at a proxy 1014 downstream from the UA that is being used for metric input values. 1015 The downstream proxy is responsible for forking a message. Then, 1016 this proxy will send provisional (e.g. 180) messages received from 1017 the requests and send the accepted (e.g. 200) response to the UA. 1019 Second, in the cases where the originating UA or proxy is forking the 1020 messages, then it MUST parse the message exchanges necessary for 1021 input into the metrics For example, it MAY utilize the first INVITE 1022 or set of INVITE messages sent and the first accepted 200 OK. Tags 1023 will identify this dialog as distinct from the other 200 OK 1024 responses, which are acknowledged and an immediate BYE is sent. The 1025 application responsible for capturing and/or understanding the input 1026 values MUST utilize these tags to distinguish between dialog 1027 requests. 1029 Note that if an INVITE is forked before reaching its destination, 1030 multiple early dialogs are likely and multiple confirmed dialogs are 1031 possible (though unlikely). When this occurs, an SRD measurement 1032 should be taken for each dialog that is created (early or confirmed). 1034 5.5. Data Collection 1036 The input necessary for these calculations may be collected in a 1037 number of different manners. It may be collected or retrieved from 1038 call detail records (CDR) or raw signaling information generated by a 1039 proxy or UA. When using records, time synchronization MUST be 1040 considered between applicable elements. 1042 If these metrics are calculated at individual elements (such as 1043 proxies or endpoints) instead of by a centralized management system 1044 and the individual elements use different measurement sample sizes, 1045 then the metrics reported for the same event at those elements may 1046 differ significantly. 1048 The information may also be transmitted through the use of network 1049 management protocols like Simple Network Management Protocol (SNMP) 1050 and via future extensions to the SIP Management Information Base 1051 (MIB) modules [RFC4780], or through a potential undefined new 1052 performance metric event package [RFC3265] retrieved via SUBSCRIBE 1053 requests. 1055 Data may be collected for a sample of calls or all calls, and may 1056 also be derived from test call scenarios. These metrics are flexible 1057 based on the needs of the application. 1059 For consistency in calculation of the metrics, elements should expect 1060 to reveal event inputs for use by a centralized management system, 1061 which would calculate the metrics based on a varying set sample size 1062 of inputs received from elements compliant with this specification. 1064 5.6. Testing Documentation 1066 In some cases, these metrics will be used to provide output values to 1067 signify the performance level of a specific SIP-based element. When 1068 using these metrics in a test environment, the environment MUST be 1069 accurately documented for the purposes of replicating any output 1070 values in future testing and/or validation. 1072 6. Conclusions 1074 This document provides a description of common performance metrics, 1075 and their defined use with SIP. The use of these metrics will 1076 provide a common viewpoint across all vendors, service providers, and 1077 users. These metrics will likely be utilized in production telephony 1078 SIP environments for providing input regarding Key Performance 1079 Indicators (KPI) and Service Level Agreement (SLA) indications; 1080 however, they may also be used for testing end-to-end SIP-based 1081 service environments. 1083 7. IANA Considerations 1085 There are no IANA considerations at this time. 1087 8. Security Considerations 1089 Security should be considered in the aspect of securing the relative 1090 data utilized in providing input to the above calculations. All 1091 other aspects of security should be considered as described in RFC 1092 3261 [RFC3261]. 1094 Implementers of these metrics MUST realize that these metrics could 1095 be used to describe characteristics of customer and user usage 1096 patterns, and privacy should be considered when collecting, 1097 transporting and storing them. 1099 9. Contributors 1101 The following people made substantial contributions to this work: 1103 Carol Davids Illinois Institute of Technology 1104 Marian Delkinov Ericsson 1105 Adam Uzelac Global Crossing 1106 Jean-Francois Mule CableLabs 1107 Rich Terpstra Level 3 Communications 1109 10. Acknowledgements 1111 We would like to thank Robert Sparks, John Hearty and Dean Bayless 1112 for their efforts in reviewing the draft and providing insight 1113 regarding clarification of certain aspects described throughout the 1114 draft. We also thank Dan Romascanu for his insightful comments and 1115 Vijay Gurbani for agreeing to perform the role of document shepherd. 1117 11. References 1119 11.1. Normative References 1121 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1122 Requirement Levels", BCP 14, RFC 2119, March 1997. 1124 [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, 1125 A., Peterson, J., Sparks, R., Handley, M., and E. 1126 Schooler, "SIP: Session Initiation Protocol", RFC 3261, 1127 June 2002. 1129 [RFC3262] Rosenberg, J. and H. Schulzrinne, "Reliability of 1130 Provisional Responses in Session Initiation Protocol 1131 (SIP)", RFC 3262, June 2002. 1133 [RFC3265] Roach, A., "Session Initiation Protocol (SIP)-Specific 1134 Event Notification", RFC 3265, June 2002. 1136 [RFC3665] Johnston, A., Donovan, S., Sparks, R., Cunningham, C., and 1137 K. Summers, "Session Initiation Protocol (SIP) Basic Call 1138 Flow Examples", BCP 75, RFC 3665, December 2003. 1140 [RFC4780] Lingle, K., Mule, J-F., Maeng, J., and D. Walker, 1141 "Management Information Base for the Session Initiation 1142 Protocol (SIP)", RFC 4780, April 2007. 1144 11.2. Informative References 1146 [E.411] ITU-T, "Series E: Overall Network Operation, Telephone 1147 Service, Service Operation and Human Factors", E.411 , 1148 March 2000. 1150 [E.721] ITU-T, "Series E: Overall Network Operation, Telephone 1151 Service, Service Operation and Human Factors", E.411 , 1152 May 1999. 1154 [GR-512] Telcordia, "LSSGR: Reliability, Section 12", GR-512- 1155 CORE Issue 2, January 1998. 1157 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 1158 "Framework for IP Performance Metrics", RFC 2330, 1159 May 1998. 1161 [RFC5486] Malas, D. and D. Meyer, "Session Peering for Multimedia 1162 Interconnect (SPEERMINT) Terminology", RFC 5486, 1163 March 2009. 1165 Authors' Addresses 1167 Daryl Malas 1168 CableLabs 1169 858 Coal Creek Circle 1170 Louisville, CO 80027 1171 US 1173 Phone: +1 303 661 3302 1174 Email: d.malas@cablelabs.com 1176 Al Morton 1177 AT&T Labs 1178 200 Laurel Avenue South 1179 Middletown, NJ 07748 1180 US 1182 Phone: +1 732 420 1571 1183 Email: acmorton@att.com