idnits 2.17.1 draft-morton-ippm-reporting-metrics-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 15. -- Found old boilerplate from RFC 3978, Section 5.5 on line 537. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 514. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 521. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 527. ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (June 18, 2006) is 6522 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'I-D.ietf-ippm-reordering' is defined on line 441, but no explicit reference was found in the text ** Downref: Normative reference to an Informational RFC: RFC 2330 ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) Summary: 6 errors (**), 0 flaws (~~), 4 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group A. Morton 3 Internet-Draft G. Ramachandran 4 Expires: December 20, 2006 AT&T Labs 5 June 18, 2006 7 Reporting Metrics: Different Points of View 8 draft-morton-ippm-reporting-metrics-00 10 Status of this Memo 12 By submitting this Internet-Draft, each author represents that any 13 applicable patent or other IPR claims of which he or she is aware 14 have been or will be disclosed, and any of which he or she becomes 15 aware will be disclosed, in accordance with Section 6 of BCP 79. 17 Internet-Drafts are working documents of the Internet Engineering 18 Task Force (IETF), its areas, and its working groups. Note that 19 other groups may also distribute working documents as Internet- 20 Drafts. 22 Internet-Drafts are draft documents valid for a maximum of six months 23 and may be updated, replaced, or obsoleted by other documents at any 24 time. It is inappropriate to use Internet-Drafts as reference 25 material or to cite them other than as "work in progress." 27 The list of current Internet-Drafts can be accessed at 28 http://www.ietf.org/ietf/1id-abstracts.txt. 30 The list of Internet-Draft Shadow Directories can be accessed at 31 http://www.ietf.org/shadow.html. 33 This Internet-Draft will expire on December 20, 2006. 35 Copyright Notice 37 Copyright (C) The Internet Society (2006). 39 Abstract 41 Consumers of IP network performance metrics have many different uses 42 in mind. This memo categorizes the different audience points of 43 view. It describes how the categories affect the selection of metric 44 parameters and options when seeking info that serves their needs. 46 Requirements Language 48 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 49 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 50 document are to be interpreted as described in RFC 2119 [RFC2119]. 52 Table of Contents 54 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 55 2. Purpose and Scope . . . . . . . . . . . . . . . . . . . . . . 3 56 3. Effect of POV on the Loss Metric . . . . . . . . . . . . . . . 4 57 3.1. Loss Threshold . . . . . . . . . . . . . . . . . . . . . . 4 58 3.2. Errored Packet Designation . . . . . . . . . . . . . . . . 5 59 3.3. Causes of Lost Packets . . . . . . . . . . . . . . . . . . 6 60 4. Effect of POV on the Delay Metric . . . . . . . . . . . . . . 6 61 4.1. Treatment of Lost Packets . . . . . . . . . . . . . . . . 6 62 4.1.1. Application Performance . . . . . . . . . . . . . . . 7 63 4.1.2. Network Characterization . . . . . . . . . . . . . . . 7 64 4.1.3. Delay Variation . . . . . . . . . . . . . . . . . . . 8 65 4.1.4. Reordering . . . . . . . . . . . . . . . . . . . . . . 8 66 4.2. Preferred Statistics . . . . . . . . . . . . . . . . . . . 8 67 4.3. Summary for Delay . . . . . . . . . . . . . . . . . . . . 9 68 5. Sampling: Test Stream Characteristics . . . . . . . . . . . . 9 69 6. Reporting Results . . . . . . . . . . . . . . . . . . . . . . 9 70 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 71 8. Security Considerations . . . . . . . . . . . . . . . . . . . 10 72 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 10 73 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 10 74 10.1. Normative References . . . . . . . . . . . . . . . . . . . 10 75 10.2. Informative References . . . . . . . . . . . . . . . . . . 11 76 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 12 77 Intellectual Property and Copyright Statements . . . . . . . . . . 13 79 1. Introduction 81 When designing measurements of IP networks and presenting the 82 results, knowledge of the audience is a key consideration. To 83 present a useful and relevant portrait of network conditions, one 84 must answer the following question: 86 "How will the results be used?" 88 There are two main audience categories: 90 1. Network Characterization - describes conditions in an IP network 91 for quality assurance, troubleshooting, modeling, etc. The 92 point-of-view looks inward, toward the network, and the consumer 93 intends their actions there. 95 2. Application Performance Estimation - describes the network 96 conditions in a way that facilitates determining affects on user 97 applications, and ultimately the users themselves. This point- 98 of-view looks outward, toward the user(s), accepting the network 99 as-is. This consumer intends to estimate a network-dependent 100 aspect of performance, or design some aspect of an application's 101 accommodation of the network. (These are *not* application 102 metrics, they are defined at the IP layer.) 104 This memo considers how these different points-of-view affect both 105 the measurement design (parameters and options of the metrics) and 106 statistics reported when serving their needs. 108 The IPPM framework [RFC2330] and other RFCs describing IPPM metrics 109 provide a background for this memo. 111 2. Purpose and Scope 113 The purpose of this memo is to clearly delineate two points-of-view 114 (POV) for using measurements, and describe their effect on the test 115 and measurement design, including the selection of metric parameters 116 and reporting the results. 118 The current scope of this memo is primarily limited to design and 119 reporting of the loss and delay metrics [add refs] , but will also 120 discuss the delay variation and reordering metrics where applicable. 121 Sampling, or the design of the active packet stream that is the basis 122 for the measurements, is also discussed. 124 3. Effect of POV on the Loss Metric 126 This section describes the ways in which the Loss metric can be tuned 127 to reflect the preferences of the two audience categories, or 128 different POV. 130 3.1. Loss Threshold 132 RFC 2680 [RFC2680] defines the concept of a waiting time for packets 133 to arrive, beyond which they are declared lost. The text of the RFC 134 declines to recommend a value, instead saying that "good engineering, 135 including an understanding of packet lifetimes, will be needed in 136 practice." Later, in the methodology, they give reasons for waiting 137 "a reasonable period of time", and leaving the definition of 138 "reasonable" intentionally vague. 140 Practical measurement experience has shown that unusual network 141 circumstances can cause long delays. One such circumstance is when 142 routing loops form during IGP re-convergence following a failure or 143 drastic link cost change. Packets will loop between two routers 144 until new routes are installed, or until the Time-to-Live (TTL) field 145 decrements to zero. Very long delays on the order of several seconds 146 have been measured [Casner] [Cia03]. 148 Therefore, network characterization activities prefer a long waiting 149 time in order to distinguish these events from other causes of loss 150 (such as packet discard at a full queue, or tail drop). This way, 151 the metric design helps to distinguish more reliably between packets 152 that might yet arrive, and those that are no longer traversing the 153 network. 155 It is possible to calculate a worst-case waiting time, assuming that 156 a routing loop is the cause. We model the path between Source and 157 Destination as a series of delays in links (t) and queues (q), as 158 these two are the dominant contributors to delay. The normal path 159 delay across n hops without encountering a loop, D, is 161 n 162 --- 163 \ 164 D = t + > t + q 165 0 / i i 166 --- 167 i = 1 169 and the time spent in the loop with L hops, is 170 i + L 171 --- 172 \ (TTL - n) 173 R = C > t + q where C = --------- 174 / i i max L 175 --- 176 i 178 and where C is the number of times a packet circles the loop. 180 If we take the delays of all links and queues as 100ms each, the 181 TTL=255, the number of hops n=5 and the hops in the loop L=4, then 183 D = 1.1 sec and R ~= 51.2 sec, and D + R ~= 52.3 seconds 185 We note that the link delays of 100ms would span most continents, and 186 a constant queue length of 100ms is also very generous. When a loop 187 occurs, it is almost certain to be resolved in 10 seconds or less. 188 The value calculated above is an upper limit for almost any realistic 189 circumstance. 191 A waiting time threshold parameter, dT, set consistent with this 192 calculation would not truncate the delay distribution (possibly 193 causing a change in its mathematical properties), because the packets 194 that might arrive have been given sufficient time to traverse the 195 network. 197 It is worth noting that packets that are stored and deliberately 198 forwarded at a much later time constitute a replay attack on the 199 measurement system, and are beyond the scope of normal performance 200 reporting. 202 Fortunately, application performance estimation activities are not 203 adversely affected by the estimated worst-case transfer time. 204 Although the designer's tendency might be to set the Loss Threshold 205 at a value equivalent to a particular application's threshold, this 206 specific threshold can be applied when post-processing the 207 measurements. A shorter waiting time can be enforced by locating 208 packets with delays longer than the application's threshold, and re- 209 designating such packets as lost. 211 3.2. Errored Packet Designation 213 RFC 2680 designates packets that arrive containing errors as lost 214 packets. Many packets that are corrupted by bit errors are discarded 215 within the network and do not reach their intended destination. 217 This is consistent with applications that would check the payload 218 integrity at higher layers, and discard the packet. However, some 219 applications prefer to deal with errored payloads on their own, and 220 even a corrupted payload is better than no packet at all. 222 To address this possibility, and to make network characterization 223 more complete, it is recommended to distinguish between packets that 224 do not arrive (lost) and errored packets that arrive (conditionally 225 lost). 227 3.3. Causes of Lost Packets 229 Although many measurement systems use a waiting time to determine if 230 a packet is lost or not, most of the waiting is in vain. The packets 231 are no-longer traversing the network, and have not reached their 232 destination. 234 There are many causes of packet loss, including: 236 1. Queue drop, or discard 238 2. Corruption of the IP header, or other essential header info 240 3. TTL expiration (or use of a TTL value that is too small) 242 4. Link or router failure 244 After waiting sufficient time, packet loss can probably be attributed 245 to one of these causes. 247 4. Effect of POV on the Delay Metric 249 This section describes the ways in which the Delay metric can be 250 tuned to reflect the preferences of the two consumer categories, or 251 different POV. 253 4.1. Treatment of Lost Packets 255 The Delay Metric RFC 2679[RFC2679] specifies the treatment of packets 256 that do not successfully traverse the network: their delay is 257 undefined. 259 " >>The *Type-P-One-way-Delay* from Src to Dst at T is undefined 260 (informally, infinite)<< means that Src sent the first bit of a 261 Type-P packet to Dst at wire-time T and that Dst did not receive that 262 packet." 264 It is an accepted, but informal practice to assign infinite delay to 265 lost packets. We next look at how these two different treatments 266 align with the needs of measurement consumers who wish to 267 characterize networks or estimate application performance. Also, we 268 look at the way that lost packets have been treated in other metrics: 269 delay variation and reordering. 271 4.1.1. Application Performance 273 Applications need to perform different functions, dependent on 274 whether or not each packet arrives within some finite tolerance. In 275 other words, a receivers' packet processing forks on packet arrival: 277 o Packets that arrive within expected tolerance are handled by 278 processes that remove headers, restore smooth delivery timing (as 279 in a de-jitter buffer), restore sending order, check for errors in 280 payloads, and many other operations. 282 o Packets that do not arrive when expected spawn other processes 283 that attempt recovery from the apparent loss, such as 284 retransmission requests, loss concealment, or forward error 285 correction to replace the missing packet. 287 So, it is important to maintain a distinction between packets that 288 actually arrive, and those that do not. Therefore, it is preferable 289 to leave the delay of lost packets undefined, and to characterize the 290 delay distribution as a conditional distribution (conditioned on 291 arrival). 293 4.1.2. Network Characterization 295 In this discussion, we assume that both loss and delay metrics will 296 be reported for network characterization (at least). 298 Assume packets that do not arrive are reported as Lost, usually as a 299 percentage of all sent packets. If these lost packets are assigned 300 undefined delay, then network's inability to deliver them in a timely 301 way is captured only in the loss metric. 303 However, if we assign infinite delay to all lost packets, then: 305 o The delay metric results are influenced by packets that arrive and 306 those that do not. 308 o The delay singleton and the loss singleton do not appear to be 309 orthogonal. 311 o The network is penalized in both the loss and delay metrics, 312 effectively double-counting the lost packets. 314 Although infinity is a familiar mathematical concept, it is somewhat 315 disconcerting to see any time-related metric reported as infinity, in 316 the opinion of the authors. Questions are bound to arise, and tend 317 to detract from the goal of informing the consumer with a performance 318 report. 320 4.1.3. Delay Variation 322 [RFC3393] excludes lost packets from samples, effectively assigning 323 an undefined delay to packets that do not arrive in a reasonable 324 time. Section 4.1 describes this specification and its rationale. 326 "The treatment of lost packets as having "infinite" or "undefined" 327 delay complicates the derivation of statistics for ipdv. 328 Specifically, when packets in the measurement sequence are lost, 329 simple statistics such as sample mean cannot be computed. One 330 possible approach to handling this problem is to reduce the event 331 space by conditioning. That is, we consider conditional statistics; 332 namely we estimate the mean ipdv (or other derivative statistic) 333 conditioned on the event that selected packet pairs arrive at the 334 destination (within the given timeout). While this itself is not 335 without problems (what happens, for example, when every other packet 336 is lost), it offers a way to make some (valid) statements about ipdv, 337 at the same time avoiding events with undefined outcomes." 339 4.1.4. Reordering 341 draft-ietf-ippm-reordering [I-D.ietf-ippm-reordering]defines metrics 342 that are based on evaluation of packet arrival order, and include a 343 waiting time to declare a packet lost (to exclude them from further 344 processing). 346 If packets are assigned a delay value, then the reordering metric 347 would declare any packets with infinite delay to be reordered, 348 because their sequence numbers will surely be less than the "Next 349 Expected" threshold when (or if) they arrive. But this practice 350 would fail to maintain orthogonality between the reordering metric 351 and the loss metric. Confusion can be avoided by designating the 352 delay of non-arriving packets as undefined, and reserving delay 353 values only for packets that arrive within a sufficiently long 354 waiting time. 356 4.2. Preferred Statistics 358 Today in network characterization, the sample mean is one statistic 359 that is almost ubiquitously reported. It is easily computed and 360 understood by virtually everyone in this audience category. Also, 361 the sample is usually filtered on packet arrival, so that the mean is 362 based a conditional distribution. 364 The median is another statistic that summarizes a distribution, 365 having somewhat different properties from the sample mean. 367 When both the sample mean and median are available, a comparison will 368 sometimes be informative, because these two statistics are equal only 369 when the delay distribution is perfectly symmetrical. 371 4.3. Summary for Delay 373 From the perspectives of: 375 1. application/receiver analysis, where processing forks on packet 376 arrival or time out, 378 2. straightforward network characterization without double-counting 379 defects, and 381 3. consistency with Delay variation and Reordering metric 382 definitions, 384 the most efficient practice is to distinguish between truly lost and 385 delayed packets with a sufficiently long waiting time, and to 386 designate the delay of non-arriving packets as undefined. 388 5. Sampling: Test Stream Characteristics 390 Network Characterization has traditionally used Poisson-distributed 391 interpacket spacing, as this provides an unbiased sample. The 392 average inter-packet spacing may be selected to allow observation of 393 specific network phenomena. Other test streams are designed to 394 sample some property of the network, such as the presence of 395 congestion, link bandwidth, or packet reordering. 397 If measuring a network in order to make inferences about applications 398 or receiver performance, then there are usually efficiencies derived 399 from a test stream that has similar characteristics to the sender. 400 In some cases, it is essential to synthesize the sender stream, as 401 with Bulk Transfer Capacity estimates. In other cases, it may be 402 sufficient to sample with a "known bias", e.g., a Periodic stream to 403 estimate real-time application performance. 405 6. Reporting Results 407 >>>>>>>>Note: this section will have sub-sections that address the 408 different audience categories, for now it gives an overview for the 409 loss and delay metrics discussed above. 411 For Packet Loss, the loss ratio defined in RFC 2680 is a sufficient 412 starting point. We note that a loss ratio calculated according to 413 [Y.1540] would exclude errored packets form the numerator. In 414 practice, the difference between these two loss metrics is small if 415 any, depending on whether the last link prior to the destination 416 contributes errored packets. 418 For Packet Delay, we currently recommend providing both the mean 419 delay and the median delay with lost packets designated as undefined. 420 These would both be statistics on a conditional distribution, and the 421 condition is arrival prior to a waiting time dT, where dT has been 422 set to take maximum packet lifetimes into account. 424 7. IANA Considerations 426 This document makes no request of IANA. 428 Note to RFC Editor: this section may be removed on publication as an 429 RFC. 431 8. Security Considerations 433 9. Acknowledgements 435 The authors would like to thank 437 10. References 439 10.1. Normative References 441 [I-D.ietf-ippm-reordering] 442 Morton, A., "Packet Reordering Metric for IPPM", 443 draft-ietf-ippm-reordering-13 (work in progress), 444 May 2006. 446 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 447 Requirement Levels", BCP 14, RFC 2119, March 1997. 449 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 450 "Framework for IP Performance Metrics", RFC 2330, 451 May 1998. 453 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 454 Delay Metric for IPPM", RFC 2679, September 1999. 456 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 457 Packet Loss Metric for IPPM", RFC 2680, September 1999. 459 [RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation 460 Metric for IP Performance Metrics (IPPM)", RFC 3393, 461 November 2002. 463 10.2. Informative References 465 [Casner] "A Fine-Grained View of High Performance Networking, NANOG 466 22 Conf.; http://www.nanog.org/mtg-0105/agenda.html", May 467 20-22 2001. 469 [Cia03] "Standardized Active Measurements on a Tier 1 IP Backbone, 470 IEEE Communications Mag., pp 90-97.", June 2003. 472 [Y.1540] ITU-T Recommendation Y.1540, "Internet protocol data 473 communication service - IP packet transfer and availability 474 performance parameters", December 2002. 476 Authors' Addresses 478 Al Morton 479 AT&T Labs 480 200 Laurel Avenue South 481 Middletown,, NJ 07748 482 USA 484 Phone: +1 732 420 1571 485 Fax: +1 732 368 1192 486 Email: acmorton@att.com 487 URI: http://home.comcast.net/~acmacm/ 489 Gomathi Ramachandran 490 AT&T Labs 491 200 Laurel Avenue South 492 Middletown,, NJ 07748 493 USA 495 Phone: +1 732 420 2353 496 Fax: +1 732 368 1192 497 Email: gomathi@att.com 498 URI: 500 Phone: 501 Fax: 502 Email: 503 URI: 505 Intellectual Property Statement 507 The IETF takes no position regarding the validity or scope of any 508 Intellectual Property Rights or other rights that might be claimed to 509 pertain to the implementation or use of the technology described in 510 this document or the extent to which any license under such rights 511 might or might not be available; nor does it represent that it has 512 made any independent effort to identify any such rights. Information 513 on the procedures with respect to rights in RFC documents can be 514 found in BCP 78 and BCP 79. 516 Copies of IPR disclosures made to the IETF Secretariat and any 517 assurances of licenses to be made available, or the result of an 518 attempt made to obtain a general license or permission for the use of 519 such proprietary rights by implementers or users of this 520 specification can be obtained from the IETF on-line IPR repository at 521 http://www.ietf.org/ipr. 523 The IETF invites any interested party to bring to its attention any 524 copyrights, patents or patent applications, or other proprietary 525 rights that may cover technology that may be required to implement 526 this standard. Please address the information to the IETF at 527 ietf-ipr@ietf.org. 529 Disclaimer of Validity 531 This document and the information contained herein are provided on an 532 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 533 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET 534 ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, 535 INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE 536 INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 537 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 539 Copyright Statement 541 Copyright (C) The Internet Society (2006). This document is subject 542 to the rights, licenses and restrictions contained in BCP 78, and 543 except as set forth therein, the authors retain all their rights. 545 Acknowledgment 547 Funding for the RFC Editor function is currently provided by the 548 Internet Society.