idnits 2.17.1 draft-malas-performance-metrics-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 14. -- Found old boilerplate from RFC 3978, Section 5.5, updated by RFC 4748 on line 1264. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 1241. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1248. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1254. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (December 6, 2007) is 5986 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: 'RFC 1157' on line 1095 ** Obsolete normative reference: RFC 3265 (ref. '3') (Obsoleted by RFC 6665) == Outdated reference: A later version (-12) exists of draft-ietf-sip-mib-10 -- Possible downref: Non-RFC (?) normative reference: ref. '7' -- Possible downref: Non-RFC (?) normative reference: ref. '8' -- Possible downref: Non-RFC (?) normative reference: ref. '9' Summary: 2 errors (**), 0 flaws (~~), 4 warnings (==), 12 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 PMOL Working Group D. Malas 2 Internet-Draft Level 3 Communications 3 Expires: June 2008 December 6, 2007 5 SIP End-to-End Performance Metrics 6 draft-malas-performance-metrics-08.txt 8 Status of this Memo 10 By submitting this Internet-Draft, each author represents that 11 any applicable patent or other IPR claims of which he or she is 12 aware have been or will be disclosed, and any of which he or she 13 becomes aware will be disclosed, in accordance with Section 6 of 14 BCP 79. 16 Internet-Drafts are working documents of the Internet Engineering 17 Task Force (IETF), its areas, and its working groups. Note that 18 other groups may also distribute working documents as Internet- 19 Drafts. 21 Internet-Drafts are draft documents valid for a maximum of six months 22 and may be updated, replaced, or obsoleted by other documents at any 23 time. It is inappropriate to use Internet-Drafts as reference 24 material or to cite them other than as "work in progress." 26 The list of current Internet-Drafts can be accessed at 27 http://www.ietf.org/ietf/1id-abstracts.txt 29 The list of Internet-Draft Shadow Directories can be accessed at 30 http://www.ietf.org/shadow.html 32 This Internet-Draft will expire on May 6, 2008. 34 Abstract 36 This document defines a set of metrics and their usage to evaluate 37 the performance of end-to-end Session Initiation Protocol (SIP) based 38 services in both production and testing environments. The purpose of 39 this document is to combine a set of common metrics, allowing 40 interoperable performance measurements, easing the comparison of 41 industry implementations. 43 Table of Contents 45 1. Introduction...................................................2 46 2. Terminology....................................................3 47 3. SIP Performance Metrics........................................4 48 3.1. Registration Request Delay (RRD)..........................5 49 3.1.1. Successful REGISTER Completion RRD...................5 50 3.1.2. Failed REGISTER Attempt RRD..........................6 52 3.2. Session Request Delay (SRD)...............................7 53 3.2.1. Successful Session Setup SRD.........................7 54 3.2.2. Failed Session Setup SRD.............................8 55 3.2.3. Instant Messaging....................................9 56 3.3. Session Disconnect Delay (SDD)............................9 57 3.3.1. Successful session completion SDD...................10 58 3.3.2. Failed session completion SDD.......................11 59 3.4. Session Duration Time (SDT)..............................12 60 3.4.1. Successful session completion SDT...................13 61 3.4.2. Failed session completion SDT.......................14 62 3.5. Average Hops per Request (AHR)...........................15 63 3.6. Session Establishment Rate (SER).........................17 64 3.6.1. Instant Messaging...................................18 65 3.7. Session Establishment Efficiency Rate (SEER).............18 66 3.8. Session Defects (SD).....................................19 67 3.9. Ineffective Session Attempts (ISA).......................19 68 3.10. Session Disconnect Failures (SDF).......................20 69 3.11. Session Completion Rate (SCR)...........................20 70 3.11.1. Successful Session Completion......................21 71 3.11.2. Failed Session Completion..........................22 72 3.12. Session Success Rate (SSR)..............................22 73 4. Metric Correlations...........................................23 74 5. Additional Considerations.....................................23 75 5.1. Back-to-back User Agent (B2BUA)..........................23 76 5.2. Authorization and Authentication.........................23 77 5.3. Forking..................................................24 78 5.4. Data Collection..........................................24 79 5.5. Testing Documentation....................................24 80 5.6. Metric Template..........................................25 81 6. Security Considerations.......................................25 82 7. IANA Considerations...........................................25 83 8. Conclusions...................................................25 84 9. Contributors..................................................26 85 10. Acknowledgments..............................................26 86 11. References...................................................26 87 11.1. Normative References....................................26 88 11.2. Informative References..................................27 89 Author's Addresses...............................................27 90 Intellectual Property Statement..................................27 91 Disclaimer of Validity...........................................28 92 Copyright Statement..............................................28 93 Acknowledgment...................................................28 95 1. Introduction 97 SIP has become a widely-used standard among many service providers, 98 vendors, and end users. Although there are many different standards 99 for measuring the performance of signaling protocols, none of them 100 specifically address SIP. 102 The scope of this document is limited to the definitions of a 103 standard set of metrics for measuring and reporting SIP performance 104 from an end-to-end perspective. The metrics introduce a common 105 foundation for understanding and quantifying performance expectations 106 between service providers, vendors, and the users of services based 107 on SIP. 109 Measurements of the metrics described in this document are affected 110 by variables external to SIP. The following is a non-exhaustive list 111 of examples: 113 . Network connectivity 115 . Switch and router performance 117 . Server processes and hardware performance 119 Note that some metrics in this document may not apply to all 120 applications of SIP. This document provides an overview of pertinent 121 metrics, which may be used individually or as a set based on the 122 usage of SIP within the context of a given service. 124 The metrics defined in this document DO NOT take into consideration 125 the impairment or failure of actual application processing of a 126 request or response. The metrics do not distinguish application 127 processing time from other sources of delay, such as packet transfer 128 delay. 130 Metrics designed to quantify single device application processing 131 performance are beyond the scope of this document. 133 This document does not provide any numerical objectives or acceptance 134 threshold values for the SIP performance metrics defined below, as 135 these items are beyond the scope of IETF activities, in general. 137 2. Terminology 139 The following terms and conventions will be used throughout this 140 document: 142 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 143 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 144 document are to be interpreted as described in RFC-2119 [1]. 146 End-to-End - This is described as two or more elements utilized for 147 initiating a request, receiving the request, and responding to the 148 request. It encompasses elements as necessary to be involved in a 149 session dialog between the originating user agent client (UAC), 150 destination user agent server (UAS), and any interim proxies (may 151 also include back-to-back user agent's (B2BUA's)). This may be 152 relative to a single operator's set of elements or extend to 153 encompass all elements (if beyond a single operator's network) 154 associated with a session. 156 Session - As described in RFC 3261, SIP is used primarily to request, 157 create, and conclude sessions. "These sessions include Internet 158 telephone calls, multimedia distribution, and multimedia conferences 159 [2]." The metrics within this document measure the performance 160 associated with the processes necessary to establish these sessions; 161 therefore, they are titled as: Session Request Delay, Session 162 Disconnect Delay, etc. Although the titles of many of the metrics 163 include this term, they are specifically measuring the signaling 164 aspects only. 166 Session Establishment - Session establishment occurs when a 200 OK 167 response from the UAS has been received, in response to a 168 corresponding UAC's INVITE setup request, indicating the session 169 setup request was successful. 171 Session Setup - As referenced within the sub-sections of 3.2 in this 172 document, session setup is the set of messages and included 173 parameters directly related to the process of a UAC requesting to 174 establish a session with a corresponding UAS. This is also described 175 as a set of steps in order to establish "ringing" [2]. 177 Time Begin (TB) - This is the time instant that starts a continuous 178 time interval running when a request is sent. TB occurs when the 179 designated request has been processed by the SIP application and the 180 first bit of the request packet has been sent from the proxy or UA 181 (and is externally observable at some logical or physical interface). 183 Time Stop (TS) - This is the time instant that ends a continuous time 184 interval running from when the related request is sent. TS occurs 185 when the last bit of the designated response is received by the SIP 186 application at the requesting device (and is externally observable at 187 some logical or physical interface). 189 3. SIP Performance Metrics 191 In regards to all of the following metrics, TB begins with the first 192 associated SIP message sent by the UAC or UAS, and is not reset if 193 the UAC or UAS must retransmit the same request multiple times. The 194 first associated SIP message indicates the TB associated with the 195 user or application expectation relative to the request. 197 Some metrics are calculated based on the final response message. 198 These metrics do not take into consideration route advances to 199 additional signaling functions based on "final" failure responses. 201 In these unique cases, the final response related to the initial 202 setup attempt should be utilized for input to the metric. 204 In regards to all of the metrics, the output values are directly 205 related to the accuracy and the equivalent level of granularity of 206 the input values. 208 The following metrics may be utilized for many different SIP 209 applications. 211 3.1. Registration Request Delay (RRD) 213 Registration Request Delay is utilized to detect failures or 214 impairments causing delays in responding to a UAC REGISTER request. 215 RRD is measured for both successful and failed REGISTER requests. 216 This metric is measured at the UAC only. The output value of this 217 metric is numerical and should be adjusted to indicate milliseconds. 218 The following represents the calculation for this metric: 220 RRD = Time of Final Response - Time of REGISTER Request 222 An average is one of the uses of this metric. The following 223 represents the calculation for this metric as an average: 225 SUM (Time of Final Response - Time of REGISTER Request) 226 ARRD = ------------------------------------------------------- 227 SUM # of REGISTER Requests 229 3.1.1. Successful REGISTER Completion RRD 231 In a successful registration attempt, RRD is defined as the time 232 interval from the moment the initial REGISTER message containing the 233 necessary information is passed by the originating UAC to the 234 intended registrar until the 200OK is received indicating the 235 registration attempt has completed successfully. This dialog 236 includes an expected authentication challenge prior to receiving the 237 200OK as describe in the following registration flow examples. 239 The following flow provides an example of identifiable events 240 necessary for inputs in calculating RRD during a successful 241 registration completion: 243 UA1 Registrar 244 | | 245 |REGISTER | 246 TB---->|--------------------->| 247 /\ | 401| 248 || |<---------------------| 249 RRD |REGISTER | 250 || |--------------------->| 251 \/ | 200| 252 TS---->|<---------------------| 253 | | 255 3.1.2. Failed REGISTER Attempt RRD 257 In a failed registration attempt, the interval is defined from the 258 initial REGISTER request and the final response indicating a failure 259 received from the destination registrar or interim proxies (Note: 260 Many times registration attempts are repeated; therefore, a failed 261 scenario must identify a failure response associated with the final 262 attempt). A failure response is described as a 4XX, 5XX, or possible 263 6XX message. RRD may be used to detect problems in downstream 264 signaling functions, which may be impairing the REGISTER message from 265 reaching the intended registrar. 267 The following flow provides an example of identifiable events 268 necessary for inputs in calculating RRD during a failed registration 269 attempt: 271 UA1 Registrar 272 | | 273 |REGISTER | 274 TB---->|--------------------->| 275 /\ | 401| 276 || |<---------------------| 277 RRD |REGISTER | 278 || |--------------------->| 279 \/ | 401| 280 TS---->|<---------------------| 281 | | 283 Note: A second 401 was used due to a common failure related to 284 incorrect authentication credentials related to the second REGISTER 285 request after the initial request failed. In this case, the UAC gives 286 up on attempting to REGISTER with the registrar. 288 3.2. Session Request Delay (SRD) 290 Session Request Delay is utilized to detect failures or impairments 291 causing delays in responding to a UA session request. SRD is 292 measured for both successful and failed session setup requests. 293 This metric is similar to Post-Selection Delay [9] or Post-Dial Delay 294 (PDD) in telephony applications of SIP, and it is measured at the UAC 295 only. The output value of this metric is numerical and should be 296 adjusted to indicate milliseconds. The following represents the 297 calculation for this metric: 299 SRD = Time of Status Indicative Response - Time of INVITE 301 An average is one of the uses of this metric. The following 302 represents the calculation for this metric as an average: 304 SUM (Time of Status Indicative Response - Time of INVITE) 305 ASRD = --------------------------------------------------------- 306 SUM # of INVITE Requests 308 3.2.1. Successful Session Setup SRD 310 In a successful request attempt, SRD is defined as the time interval 311 from the moment the INVITE message containing the necessary 312 information is passed by the originating agent or user to the 313 intended mediation or destination agent until the first provisional 314 response is received indicating an audible or visual status of the 315 initial session setup request. In SIP, the message indicating status 316 would be a non-100 Trying provisional message received in response to 317 an INVITE request. In some cases, a non-100 Trying provisional 318 message is not received, but rather a 200 message is received as the 319 first status message instead. In these situations, the 200 message 320 would be used to calculate the interval. 322 The following flow provides an example of identifiable events 323 necessary for inputs in calculating SRD during a successful session 324 setup request without a redirect (i.e. 3XX message): 326 UA1 UA2 327 | | 328 |INVITE | 329 TB---->|--------------------->| 330 /\ | | 331 || | | 332 SRD | | 333 || | | 334 \/ | 180| 335 TS---->|<---------------------| 336 | | 338 The following flow provides an example of identifiable events 339 necessary for inputs in calculating SRD during a successful session 340 setup with a redirect (e.g. 302 Moved Temporarily): 342 UA1 Redirect Server UA2 343 | | | 344 |INVITE | | 345 TB---->|--------------------->| | 346 /\ | 302| | 347 || |<---------------------| | 348 || |ACK | | 349 SRD |--------------------->| | 350 || |INVITE | 351 || |------------------------------------------->| 352 \/ | 180| 353 TS---->|<-------------------------------------------| 355 3.2.2. Failed Session Setup SRD 357 In a failed request attempt, the interval is defined from the initial 358 session request and a non-100 Trying provisional message or a failure 359 indication response. A failure response is described as a 4XX, 5XX, 360 or possible 6XX message. SRD may be used to detect problems in 361 downstream signaling functions, which may be impairing the INVITE 362 message from reaching the intended UA. 364 The following flow provides an example of identifiable events 365 necessary for inputs in calculating SRD during a failed session setup 366 attempt without a redirect (i.e. 3XX message): 368 UA1 UA2 369 | | 370 |INVITE | 371 TB---->|--------------------->| 372 /\ | | 373 || | | 374 SRD | | 375 || | | 376 \/ | 480| 377 TS---->|<---------------------| 378 | | 380 The following flow provides an example of identifiable events 381 necessary for inputs in calculating SRD during a failed session setup 382 attempt with a redirect (e.g. 302 Moved Temporarily): 384 UA1 Redirect Server UA2 385 | | | 386 |INVITE | | 387 TB---->|--------------------->| | 388 /\ | 302| | 389 || |<---------------------| | 390 || |ACK | | 391 SRD |--------------------->| | 392 || |INVITE | 393 || |------------------------------------------->| 394 \/ | 480| 395 TS---->|<-------------------------------------------| 397 3.2.3. Instant Messaging 399 This metric is also applicable to MESSAGE [10] requests. In the 400 above metric, INVITE can be replaced with MESSAGE to provide SRD for 401 instant messaging (IM). The dialog will vary slightly as described 402 in [10]. The inputs for this metric should be utilized regardless of 403 whether a prior SIP dialog was utilized to setup the session. In 404 that case both the SIP dialog and the MESSAGE requests are measured 405 independently. 407 The following flow provides an example of identifiable events 408 necessary for inputs in calculating SRD during a successful session 409 MESSAGE request: 411 UA1 UA2 412 | | 413 |MESSAGE | 414 TB---->|--------------------->| 415 /\ | | 416 || | | 417 SRD | | 418 || | | 419 \/ | 200| 420 TS---->|<---------------------| 421 | | 423 Failure requests occur similarly as to those described in section 424 3.2.2 with MESSAGE in replacement of INVITE as the IM session request 425 method. 427 3.3. Session Disconnect Delay (SDD) 429 This metric is utilized to detect failures or impairments delaying 430 the time necessary to end a session. It can be measured from both a 431 UAC and UAS perspective. SDD is measured for both successful and 432 failed session completions. The output value of this metric is 433 numerical and should be adjusted to indicate milliseconds. The 434 following represents the calculation for this metric: 436 SDD = Time of 2XX or Timeout - Time of Completion Message (BYE) 438 An average is one of the uses of this metric. The following 439 represents the calculation for this metric as an average: 441 SUM (Time of 2XX or Timeout - Time of Completion Message) 442 ASDD = --------------------------------------------------------- 443 SUM # of Completed Sessions 445 3.3.1. Successful session completion SDD 447 In a successful session completion, SDD is defined as the interval 448 between sending a session completion message, such as a BYE, and 449 receiving the subsequent 2XX acknowledgement. The following flows 450 provide an example of identifiable events necessary for inputs in 451 calculating SDD during a successful session completion: 453 Measuring SDD at the UAC - 455 UA1 UA2 456 | | 457 |INVITE | 458 |--------------------->| 459 | 180| 460 |<---------------------| 461 | 200| 462 |<---------------------| 463 |ACK | 464 |--------------------->| 465 |BYE | 466 TB---->|--------------------->| 467 /\ | | 468 || | | 469 SDD | | 470 || | | 471 \/ | 200| 472 TS---->|<---------------------| 474 Measuring SDD at the UAS - 476 UA1 UA2 477 | | 478 |INVITE | 479 |--------------------->| 480 | 180| 481 |<---------------------| 482 | 200| 483 |<---------------------| 484 |ACK | 485 |--------------------->| 486 | BYE| 487 |<---------------------|<----TB 488 | | /\ 489 | | || 490 | | SDD 491 | | || 492 |200 | \/ 493 |--------------------->|<----TS 495 (In these two examples, TB and TS are the same even if the UAC/UAS 496 receives the indicated messages instead of sending them.) 498 3.3.2. Failed session completion SDD 500 In some cases, no response is received after a session completion 501 message is sent and potentially retried. In this case, SDD is 502 defined as the interval between sending a session completion message, 503 such as a BYE, and the resulting Timer F expiration. The following 504 flows provide an example of identifiable events necessary for inputs 505 in calculating SDD during a failed session completion attempt: 507 Measuring SDD at the UAC - 509 UA1 UA2 510 | | 511 |INVITE | 512 |--------------------->| 513 | 180| 514 |<---------------------| 515 | 200| 516 |<---------------------| 517 |ACK | 518 |--------------------->| 519 |BYE | 520 TB---->|--------------------->| 521 /\ |BYE | 522 || |--------------------->| 523 SDD |BYE | 524 || |--------------------->| 525 \/ | | 526 TS---->|***Timer F Expires | 528 Measuring SDD at the UAS - 530 UA1 UA2 531 | | 532 |INVITE | 533 |--------------------->| 534 | 180| 535 |<---------------------| 536 | 200| 537 |<---------------------| 538 |ACK | 539 |--------------------->| 540 | BYE| 541 |<---------------------|<----TB 542 | BYE| /\ 543 |<---------------------| || 544 | BYE| SDD 545 |<---------------------| || 546 | | \/ 547 | Timer F Expires***|<----TS 549 3.4. Session Duration Time (SDT) 551 This metric is used to detect problems (e.g. poor audio quality) 552 causing short session durations. SDT is measured for both successful 553 and failed session completions. It can be measured from both a UAC 554 and UAS perspective. This metric is similar to Call Hold Time, and 555 is traditionally calculated as Average Call Hold Time (ACHT) in 556 telephony applications of SIP. The output value of this metric is 557 numerical and should be adjusted to indicate minutes and seconds. 558 The following represents the calculation for this metric: 560 SDT = Time of BYE or Timeout - Time of 200 OK response to INVITE 562 An average is one of the uses of this metric. The following 563 represents the calculation for this metric as an average: 565 SUM (Time of BYE or Timeout - Time of 200 OK response to INVITE) 566 ASDT = -------------------------------------------------------------- 567 SUM # of INVITE w/ 200OK & BYE or Timeout 569 3.4.1. Successful session completion SDT 571 In a successful session completion, SDT is calculated as an average 572 and is defined as the duration of a dialog from receipt of a 200 OK 573 response to an INVITE and an associated BYE message indicating dialog 574 completion. 576 The following flows provide an example of identifiable events 577 necessary for inputs in calculating SDT during a successful session 578 completion (The message flows are changed between the UAC and UAS to 579 provide varying examples.): 581 Measuring SDT at the UAC - 583 UA1 UA2 584 | | 585 |INVITE | 586 |--------------------->| 587 | 180| 588 |<---------------------| 589 | 200| 590 TB---->|<---------------------| 591 /\ |ACK | 592 || |--------------------->| 593 || | | 594 SDT | | 595 || | | 596 || | | 597 \/ |BYE | 598 TS---->|--------------------->| 599 | | 601 Measuring SDT at the UAS - 603 UA1 UA2 604 | | 605 |INVITE | 606 |--------------------->| 607 | 180| 608 |<---------------------| 609 | 200| 610 |<---------------------|<----TB 611 |ACK | /\ 612 |--------------------->| || 613 | | || 614 | | SDT 615 | | || 616 | | || 617 | BYE| \/ 618 |<---------------------|<----TS 619 | | 621 (In these two examples, TS is the same even if the UAC/UAS receives 622 the BYE instead of sending it.) 624 3.4.2. Failed session completion SDT 626 In some cases, no response is received after a session completion 627 message is sent and potentially retried. In this case, SDT is 628 defined as the interval between sending a session completion message, 629 such as a BYE, and the resulting Timer F expiration. The following 630 flows provide an example of identifiable events necessary for inputs 631 in calculating SDT during a failed session completion attempt: 633 Measuring SDT at the UAC - 635 UA1 UA2 636 | | 637 |INVITE | 638 |--------------------->| 639 | 180| 640 |<---------------------| 641 | 200| 642 TB---->|<---------------------| 643 /\ |BYE | 644 || |--------------------->| 645 || |BYE | 646 SDT |--------------------->| 647 || |BYE | 648 || |--------------------->| 649 \/ | | 650 TS---->|***Timer F Expires | 652 Measuring SDT at the UAS - 654 UA1 UA2 655 | | 656 |INVITE | 657 |--------------------->| 658 | 180| 659 |<---------------------| 660 | 200| 661 |<---------------------|<----TB 662 | BYE| /\ 663 |<---------------------| || 664 | BYE| || 665 |<---------------------| SDT 666 | BYE| || 667 |<---------------------| || 668 | | \/ 669 | Timer F Expires***|<----TS 671 3.5. Average Hops per Request (AHR) 673 This metric is used to indicate potential inefficient routing and to 674 detect failure occurrences related to the number of elements 675 traversed by a single SIP INVITE or MESSAGE request. AHR is defined 676 as the number of hops traversed by an INVITE or MESSAGE request. It 677 is calculated as an average. This metric requires the Max-Forwards 678 value to be captured at both the originating UAC or proxy and the 679 terminating UAS or proxy perspective as relative to the end-to-end 680 network under measurement. The output value of this metric is 681 measured in a numerical value indicating a number of hops. 683 Variables = 685 a = Initial INVITE/MESSAGE "Max-Forwards" value 687 b = Initial INVITE/MESSAGE received by terminating UAS "Max- 688 Forwards" value 690 c = # of Hops for INVITE/MESSAGE requests 692 d = SUM # of INVITE/MESSAGE requests 694 c = a - b 696 This metric is calculated as an average. The following represents 697 the calculation for this metric: 699 AHR = (SUM of aggregate c's / d) 701 The following dialog provides an example describing the inputs 702 necessary for this calculation. Although this example is of an 703 INVITE SIP dialog request, a MESSAGE request is similar in its use of 704 the Max-Forwards header. (The dialog continuation was omitted for 705 clarity): 707 UA1 Proxy 1 Proxy 2 UA2 708 | | | | 709 |INVITE | | | 710 |--------------->| | | 711 | 407| | | 712 |<---------------| | | 713 |ACK | | | 714 |--------------->| | | 715 |INVITE (F4) | | | 716 |--------------->|INVITE (F5) | | 717 | 100|--------------->|INVITE (F6) | 718 |<---------------| 100|--------------->| 719 | |<---------------| | 721 Message Details (Only the message details of the INVITE messages have 722 been included for clarity. Also, some headers after Max-Forwards 723 have been omitted for additional clarity.): 725 (F4) INVITE UA1 -> Proxy 1 727 INVITE sip:ua2@biloxi.example.com SIP/2.0 728 Via: SIP/2.0/TCP 729 client.atlanta.example.com:5060;branch=z9hG4bK74bf9 730 Max-Forwards: 70 731 Route: 732 From: UA1 ;tag=9fxced76sl 733 To: UA2 735 (F5) INVITE Proxy 1 -> Proxy 2 737 INVITE sip:ua2@biloxi.example.com SIP/2.0 738 Via: SIP/2.0/TCP 739 ss1.atlanta.example.com:5060;branch=z9hG4bK2d4790.1 740 Via: SIP/2.0/TCP 741 client.atlanta.example.com:5060;branch=z9hG4bK74bf9 742 ;received=192.0.2.101 743 Max-Forwards: 69 744 Record-Route: 745 From: UA1 ;tag=9fxced76sl 746 To: UA2 748 (F6) INVITE Proxy 2 -> UA2 750 INVITE sip:ua2@client.biloxi.example.com SIP/2.0 751 Via: SIP/2.0/TCP ss2.biloxi.example.com:5060;branch=z9hG4bK721e4.1 752 Via: SIP/2.0/TCP 753 ss1.atlanta.example.com:5060;branch=z9hG4bK2d4790.1 754 ;received=192.0.2.111 755 Via: SIP/2.0/TCP 756 client.atlanta.example.com:5060;branch=z9hG4bK74bf9 757 ;received=192.0.2.101 758 Max-Forwards: 68 759 Record-Route: , 760 761 From: UA1 ;tag=9fxced76sl 762 To: UA2 764 3.6. Session Establishment Rate (SER) 766 This metric is used to detect the ability of a terminating UA or 767 downstream proxy to successfully establish sessions per INVITE 768 request. SER is defined as the number of INVITE requests resulting 769 in a 200 OK response, to the total number of attempted INVITE 770 requests less INVITE requests resulting in a 3XX response. This 771 metric is similar to Answer Seizure Rate (ASR) [8] in telephony 772 applications of SIP. It is measured at the UAC only. The output 773 value of this metric is numerical and should be adjusted to indicate 774 a percentage of successfully established sessions. The following 775 represents the calculation for this metric: 777 # of INVITE Requests w/ associated 200OK 778 SER = --------------------------------------------------------------- 779 (Total # of INVITE Requests)-(# of INVITE Requests w/ 3XX Response) 781 The following flow provides an example of identifiable events 782 necessary for inputs in determining session establishment as 783 described above: 785 UA1 UA2 786 | | 787 |INVITE | 788 +----------->|------------------>| 789 | | 180| 790 | |<------------------| 791 Session Established | | 792 | | | 793 | | 200| 794 +----------->|<------------------| 795 | | 796 3.6.1. Instant Messaging 798 This metric is also applicable to MESSAGE [10] requests. In the 799 above metric, INVITE can be replaced with MESSAGE to provide SER for 800 IM. The dialog will vary slightly as described in [10]. 802 The following flow provides an example of identifiable events 803 necessary for inputs in calculating SER for MESSAGE requests: 805 UA1 UA2 806 | | 807 |MESSAGE | 808 +----------->|------------------>| 809 | | | 810 Session Established | | 811 | | 200| 812 +----------->|<------------------| 813 | | 815 3.7. Session Establishment Efficiency Rate (SEER) 817 This metric is complimentary to SER, but is intended to exclude the 818 potential effects of the terminating UAS from the metric. SEER is 819 defined as the number of INVITE requests resulting in a 200 OK 820 response and INVITE requests resulting in a 480, 486, or 600; to the 821 total number of attempted INVITE requests less INVITE requests 822 resulting in a 3XX response. This metric is similar to Network 823 Efficiency Rate (NER) [8] in telephony applications of SIP. It is 824 measured at the UAC only. The output value of this metric is 825 numerical and should be adjusted to indicate a percentage of 826 successfully established sessions less common UAS failures. The 827 following represents the calculation for this metric: 829 # of INVITE Requests w/ associated 200OK, 480, 486, or 600 830 SER = --------------------------------------------------------------- 831 (Total # of INVITE Requests)-(# of INVITE Requests w/ 3XX Response) 833 Reference the example flow is Section 3.6. 835 3.8. Session Defects (SD) 837 Session defects provide a subset of SIP failure responses, which 838 consistently indicate a failure in dialog processing. Defects are 839 necessary to provide input to calculations such as Defects per 840 Million (DPM) or other similar metrics. These failure responses are 841 in response to initial session setup requests, such as a new INVITE. 842 The output value of this metric is numerical and should be adjusted 843 to indicate a percentage of defective sessions. The following 844 failure responses provide a guideline for defective criterion: 846 . 500 Server Internal Error 848 . 503 Service Unavailable 850 . 504 Server Timeout 852 This set of failure responses was derived through correlating more 853 granular ISUP failure responses as described in RFC 3398. 855 3.9. Ineffective Session Attempts (ISA) 857 Ineffective session attempts occur when a proxy or agent internally 858 releases a setup request with a failed or congested condition. This 859 metric is similar to Ineffective Machine Attempts (IMA) in telephony 860 applications of SIP, and was adopted from Telcordia GR-512-CORE [7]. 861 The output value of this metric is numerical and should be adjusted 862 to indicate a percentage of ineffective session attempts. The 863 following failure responses provide a guideline for this criterion: 865 . 408 Request Timeout 867 . 500 Server Internal Error 869 . 503 Service Unavailable 870 . 504 Server Timeout 872 This set was derived in a similar manner as described in Section 3.6, 873 in addition 408 failure responses is indicative a congested state 874 with a downstream element. 876 This metric is calculated as a percentage of total session setup 877 requests. The following represents the calculation for this metric: 879 # of ISA 880 ISA % = ----------------------------- 881 Total # of Session Requests 883 3.10. Session Disconnect Failures (SDF) 885 Session disconnect failures occur when an active session is 886 terminated due to a failure condition that can be identified by a 887 REASON header [5] in a BYE message. This occurs, for example, when a 888 user agent (UA) is controlling an IP or TDM (Time Division 889 Multiplexing) media gateway, and the media gateway notifies the UA of 890 a failure condition causing the loss of media related to an 891 established session. The UA will release the session with a BYE, but 892 should include a REASON header indicating the session was 893 disconnected abnormally. The REASON value is utilized to determine 894 the disconnect was a failure. This metric is similar to Cutoff Calls 895 (CC) in telephony applications of SIP, and was adopted from Telcordia 896 GR-512-CORE [7]. The input variables for this metric are captured 897 from the originating UAC or proxy perspective as relative to the end- 898 to-end network under measurement. The output value of this metric is 899 numerical and should be adjusted to indicate a percentage of session 900 disconnect failures. 902 This metric is calculated as a percentage of total session completed 903 successfully as defined in Section 3.5. The following represents the 904 calculation for this metric: 906 # of SDF's 907 SDF % = ------------------------------- 908 Total # of Session Requests 910 3.11. Session Completion Rate (SCR) 912 A session completion is defined as a SIP dialog, which completes 913 without failing due to a lack of response from an intended proxy or 914 UA. This metric is only used when at least one proxy is involved in 915 the dialog. This metric is similar to Call Completion Rate (CCR) in 916 telephony applications of SIP. The output value of this metric is 917 numerical and should be adjusted to indicate a percentage of 918 successfully completed sessions. 920 This metric is calculated as a percentage of total sessions completed 921 successfully. The following represents the calculation for this 922 metric: 924 # of Successfully Completed Sessions 925 SCR % = --------------------------------------- 926 Total # of Session Requests 928 3.11.1. Successful Session Completion 930 A session completes successfully when it begins with a setup request 931 and ends with a session completion message. 933 The following dialog [4] provides an example describing the necessary 934 events of a successful session completion: 936 UA1 Proxy 1 Proxy 2 UA2 937 | | | | 938 |INVITE | | | 939 |--------------->| | | 940 | 407| | | 941 |<---------------| | | 942 |ACK | | | 943 |--------------->| | | 944 |INVITE | | | 945 |--------------->|INVITE | | 946 | 100|--------------->|INVITE | 947 |<---------------| 100|--------------->| 948 | |<---------------| | 949 | | | 180| 950 | | 180 |<---------------| 951 | 180|<---------------| | 952 |<---------------| | 200| 953 | | 200|<---------------| 954 | 200|<---------------| | 955 |<---------------| | | 956 |ACK | | | 957 |--------------->|ACK | | 958 | |--------------->|ACK | 959 | | |--------------->| 960 | Both Way RTP Media | 961 |<================================================>| 962 | | | BYE| 963 | | BYE|<---------------| 964 | BYE|<---------------| | 965 |<---------------| | | 966 |200 | | | 967 |--------------->|200 | | 968 | |--------------->|200 | 969 | | |--------------->| 970 | | | | 972 3.11.2. Failed Session Completion 974 Session completion fails when an INVITE is sent from a UAC, but there 975 is no indication the INVITE reached the intended UAS. This can also 976 occur if the intended UAS does not respond to the UAC or the response 977 never reaches the UAC associated with the session. 979 The following dialog provides an example describing the necessary 980 events of an unsuccessful session completion: 982 UA1 Proxy 1 Proxy 2 UA2 983 | | | | 984 |INVITE | | | 985 |--------------->| | | 986 | 407| | | 987 |<---------------| | | 988 |ACK | | | 989 |--------------->| | | 990 |INVITE | | | 991 |--------------->|INVITE | | 992 | 100|--------------->|INVITE | 993 |<---------------| 100|--------------->| 994 | |<---------------| | 995 | | |INVITE | 996 | | |--------------->| 997 | | | | 998 | | |INVITE | 999 | | |--------------->| 1000 | | | | 1001 | | 408| | 1002 | 408|<---------------| | 1003 |<---------------|ACK | | 1004 | |--------------->| | 1005 |ACK | | | 1006 |--------------->| | | 1008 3.12. Session Success Rate (SSR) 1010 Session success rate is defined as the percentage of successfully 1011 completed sessions compared to sessions, which fail due to ISA or 1012 SDF. This metric is also known as Call Success Rate (CSR) in 1013 telephony applications of SIP. The output value of this metric is 1014 numerical and should be adjusted to indicate a percentage of 1015 successful sessions. The following represents the calculation for 1016 this metric: 1018 SSR = 100% - (ISA% + SDF%) 1020 4. Metric Correlations 1022 These metrics may be used to determine the performance of a domain 1023 and/or user. This would be to provide a metric relative to one or 1024 more dimensions. The following is a subset of dimensions for 1025 providing further granularity per metric: 1027 . To "user" 1029 . From "user" 1031 . Bi-direction "user" 1033 . To "domain" 1035 . From "domain" 1037 . Bi-direction "domain" 1039 5. Additional Considerations 1041 5.1. Back-to-back User Agent (B2BUA) 1043 A B2BUA may impact the ability to collect these metrics with an end- 1044 to-end perspective. It is necessary to realize a B2BUA may act as an 1045 originating UAC and terminating UAS or it may act as a proxy. In 1046 some cases, it may be necessary to consider information collected 1047 from both sides of the B2BUA in order to determine the end-to-end 1048 perspective. In other cases, the B2BUA may act simply as a proxy 1049 allowing data to be derived as necessary for the input into any of 1050 the listed calculations. 1052 5.2. Authorization and Authentication 1054 During the process of setting up a SIP dialog, various authentication 1055 methods may be utilized. These authentication methods will add to 1056 the duration as measured by the metrics, and the length of time will 1057 vary based on those methods. The failures of these authentication 1058 methods will also be captured by these metrics, since SIP is 1059 ultimately used to indicate the success or failure of the 1060 authorization and/or authentication attempt. The metrics in section 1061 3 are inclusive of the duration associated with this process, even if 1062 the method is external to the SIP protocol. This was included 1063 purposefully, due to its inherent impact on the protocol and the 1064 subsequent SIP dialogs. 1066 5.3. Forking 1068 Forking should be considered when determining the messages associated 1069 with the input values for the described metrics. If all of the 1070 forked dialogs were used in the metric calculations, the numbers 1071 would skew dramatically. There are two different points of forking, 1072 which must be considered. First, forking may occur at a proxy 1073 downstream from the UAC that is being used for metric input values. 1074 Since, the downstream proxy is responsible for forking a message and 1075 then only sending the accepted response to the UAC, the UAC will only 1076 see messages as indicated in the described metrics. Second, in the 1077 cases where the observed UAC or proxy is forking the messages, then 1078 it must utilize the first INVITE or set of INVITE messages sent and 1079 the first accepted 200 OK. A tag will identify this dialog as 1080 distinct from the other 200 OK responses, which are acknowledged and 1081 an immediate BYE is sent. The application responsible for capturing 1082 and/or understanding the input values must utilize this tag to 1083 distinguish between dialogs. 1085 5.4. Data Collection 1087 The input necessary for these calculations may be collected in a 1088 number of different manners. It may be collected or retrieved from 1089 call detail records (CDR) or raw signaling information generated by a 1090 proxy or UA. When using records, time synchronization must be 1091 considered between applicable elements. 1093 The information may also be transmitted through the use of network 1094 management protocols like Simple Network Management Protocol (SNMP) 1095 [RFC 1157] and via future extensions to the SIP Management 1096 Information Base (MIB) modules [6], or through a potential undefined 1097 new performance metric event package [3] retrieved via SUBSCRIBE 1098 requests. 1100 Data may be collected for a sample of calls or all calls, and may 1101 also be derived from test call scenarios. These metrics are flexible 1102 based on the needs of the application. 1104 5.5. Testing Documentation 1106 In some cases, these metrics will be used to provide output values to 1107 signify the performance level of a specific SIP-based element. When 1108 using these metrics in a test environment, the environment must be 1109 accurately documented for the purposes of replicating any output 1110 values in future testing and/or validation. 1112 5.6. Metric Template 1114 Although, this document provides details for a foundational set of 1115 pertinent metrics, other metrics may be necessary as the SIP protocol 1116 evolves or as deemed necessary by an individual or set of companies 1117 and/or vendors. This section details a template, which should be 1118 used when creating new performance metrics. This template will allow 1119 for a common structure of information, which will enable a more 1120 adaptable method of understanding and incorporating metrics defined 1121 beyond the scope of this document. 1123 All metrics should include the following characteristics: 1125 . A metric should have a common 2-4 word summary description, 1126 which can be identified as a 2-4 letter acronym. 1128 . The metric must describe the problem or motivation for which 1129 it is attempting to detect or identify. 1131 . The metric must describe what is measured as indicated 1132 specifically by defined SIP messages. 1134 . The metric must identify the unit(s) of measure described in 1135 the associated output. 1137 . The metric must define the time at which the inputs are 1138 captured, including both beginning and end. 1140 . The metric must describe if the outputs can be utilized in a 1141 manner other than the raw output (e.g. average, high/low, 1142 etc.), and if so, how. 1144 6. Security Considerations 1146 Security should be considered in the aspect of securing the relative 1147 data utilized in providing input to the above calculations. All 1148 other aspects of security should be considered as described in [2]. 1150 7. IANA Considerations 1152 There are no IANA considerations at this time. 1154 8. Conclusions 1156 The proposed guideline provides a description of common performance 1157 metrics, and their defined use with SIP. The use of these metrics 1158 will provide a common viewpoint across all vendors, service 1159 providers, and customers. These metrics will likely be utilized in 1160 production SIP environments for providing input regarding Key 1161 Performance Indicators (KPI) and Service Level Agreement (SLA) 1162 indications; however, they may also be used for testing end-to-end 1163 SIP-based service environments. 1165 9. Contributors 1167 The following people made substantial contributions to this work: 1169 Carol Davids Illinois Institute of Technology 1170 Al Morton AT&T Labs 1171 Marian Delkinov Ericsson 1172 Adam Uzelac Global Crossing 1173 Jean-Francois Mule CableLabs 1174 Rich Terpstra Level 3 Communications 1176 10. Acknowledgments 1178 I would like to thank John Hearty and Dean Bayless for their efforts 1179 in reviewing the draft and providing insight regarding clarification 1180 of certain aspects described throughout the draft. 1182 11. References 1184 11.1. Normative References 1186 [1] Bradner, S., "Key words for use in RFCs to Indicate Requirement 1187 Levels", BCP 14, RFC 2119, March 1997. 1189 [2] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., 1190 Peterson, J., Sparks, R., Handley, M. and E. Schooler, "SIP: 1191 Session Initiation Protocol", RFC 3261, June 2002. 1193 [3] Roach, A., "Session Initiation Protocol (SIP)-Specific Event 1194 Notification", RFC 3265, June 2002. 1196 [4] Johnston, A., Donovan, S., Sparks, R., Cunningham, C., and K. 1197 Summers, "Session Initiation Protocol (SIP) Basic Call 1198 Flow Examples", BCP 75, RFC 3665, December 2003. 1200 [5] Schulzrinne, H., Oran, D., Camarillo, G., "The Reason Header 1201 Field for the Sessions Initiation Protocol (SIP)", RFC 3326, 1202 December 2002. 1204 [6] Lingle, K., Mule, J., Maeng, J., Walker, D., "Management 1205 Information Base for the Session Initiation Protocol (SIP)", 1206 draft-ietf-sip-mib-10, Work in Progress. 1208 [7] Telcordia, "LSSGR: Reliability, Section 12", GR-512-CORE, Issue 1209 2, January 1998. 1211 [8] ITU-T, "Series E: Overall Network Operation, Telephone Service, 1212 Service Operation and Human Factors", E.411, March 2000. 1214 [9] ITU-T, "Series E: Overall Network Operation, Telephone Service, 1215 Service Operation and Human Factors", E.721, May 1999. 1217 [10] B. Campbell, Ed, Rosenberg, J., Schulzrinne, H., Huitema, C., 1218 Gurle, D., "Session Initiation Protocol (SIP) Extension for 1219 Instant Messaging", RFC 3428, December 2002. 1221 11.2. Informative References 1223 Author's Addresses 1225 Daryl Malas 1226 Level 3 Communications LLC 1227 1025 Eldorado Blvd. 1228 Broomfield, CO 80021 1229 USA 1230 EMail: daryl.malas@level3.com 1232 Intellectual Property Statement 1234 The IETF takes no position regarding the validity or scope of any 1235 Intellectual Property Rights or other rights that might be claimed to 1236 pertain to the implementation or use of the technology described in 1237 this document or the extent to which any license under such rights 1238 might or might not be available; nor does it represent that it has 1239 made any independent effort to identify any such rights. Information 1240 on the procedures with respect to rights in RFC documents can be 1241 found in BCP 78 and BCP 79. 1243 Copies of IPR disclosures made to the IETF Secretariat and any 1244 assurances of licenses to be made available, or the result of an 1245 attempt made to obtain a general license or permission for the use of 1246 such proprietary rights by implementers or users of this 1247 specification can be obtained from the IETF on-line IPR repository at 1248 http://www.ietf.org/ipr. 1250 The IETF invites any interested party to bring to its attention any 1251 copyrights, patents or patent applications, or other proprietary 1252 rights that may cover technology that may be required to implement 1253 this standard. Please address the information to the IETF at 1254 ietf-ipr@ietf.org. 1256 Disclaimer of Validity 1258 This document and the information contained herein are provided on an 1259 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 1260 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND 1261 THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS 1262 OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF 1263 THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 1264 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 1266 Copyright Statement 1268 Copyright (C) The IETF Trust (2007). 1270 This document is subject to the rights, licenses and restrictions 1271 contained in BCP 78, and except as set forth therein, the authors 1272 retain all their rights. 1274 Acknowledgment 1276 Funding for the RFC Editor function is currently provided by the 1277 Internet Society.