idnits 2.17.1 draft-ietf-ippm-reporting-metrics-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to contain a disclaimer for pre-RFC5378 work, but was first submitted on or after 10 November 2008. The disclaimer is usually necessary only for documents that revise or obsolete older RFCs, and that take significant amounts of text from those RFCs. If you can contact all authors of the source material and they are willing to grant the BCP78 rights to the IETF Trust, you can and should remove the disclaimer. Otherwise, the disclaimer is needed and you can ignore this comment. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (May 30, 2010) is 5078 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) == Outdated reference: A later version (-06) exists of draft-ietf-ippm-reporting-04 Summary: 2 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group A. Morton 3 Internet-Draft G. Ramachandran 4 Intended status: Informational G. Maguluri 5 Expires: December 1, 2010 AT&T Labs 6 May 30, 2010 8 Reporting Metrics: Different Points of View 9 draft-ietf-ippm-reporting-metrics-02 11 Abstract 13 Consumers of IP network performance metrics have many different uses 14 in mind. This memo categorizes the different audience points of 15 view. It describes how the categories affect the selection of metric 16 parameters and options when seeking info that serves their needs. 17 The memo then proceeds to discuss "long-term" reporting 18 considerations (e.g, days, weeks or months, as opposed to 10 19 seconds). 21 Requirements Language 23 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 24 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 25 document are to be interpreted as described in RFC 2119 [RFC2119]. 27 Status of this Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at http://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on December 1, 2010. 44 Copyright Notice 46 Copyright (c) 2010 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents 51 (http://trustee.ietf.org/license-info) in effect on the date of 52 publication of this document. Please review these documents 53 carefully, as they describe your rights and restrictions with respect 54 to this document. Code Components extracted from this document must 55 include Simplified BSD License text as described in Section 4.e of 56 the Trust Legal Provisions and are provided without warranty as 57 described in the Simplified BSD License. 59 This document may contain material from IETF Documents or IETF 60 Contributions published or made publicly available before November 61 10, 2008. The person(s) controlling the copyright in some of this 62 material may not have granted the IETF Trust the right to allow 63 modifications of such material outside the IETF Standards Process. 64 Without obtaining an adequate license from the person(s) controlling 65 the copyright in such materials, this document may not be modified 66 outside the IETF Standards Process, and derivative works of it may 67 not be created outside the IETF Standards Process, except to format 68 it for publication as an RFC or to translate it into languages other 69 than English. 71 Table of Contents 73 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 74 2. Purpose and Scope . . . . . . . . . . . . . . . . . . . . . . 4 75 3. Reporting Results . . . . . . . . . . . . . . . . . . . . . . 5 76 3.1. Overview of Metric Statistics . . . . . . . . . . . . . . 5 77 3.2. Long-Term Reporting Considerations . . . . . . . . . . . . 6 78 4. Effect of POV on the Loss Metric . . . . . . . . . . . . . . . 8 79 4.1. Loss Threshold . . . . . . . . . . . . . . . . . . . . . . 8 80 4.1.1. Network Characterization . . . . . . . . . . . . . . . 8 81 4.1.2. Application Performance . . . . . . . . . . . . . . . 10 82 4.2. Errored Packet Designation . . . . . . . . . . . . . . . . 10 83 4.3. Causes of Lost Packets . . . . . . . . . . . . . . . . . . 10 84 4.4. Summary for Loss . . . . . . . . . . . . . . . . . . . . . 11 85 5. Effect of POV on the Delay Metric . . . . . . . . . . . . . . 11 86 5.1. Treatment of Lost Packets . . . . . . . . . . . . . . . . 11 87 5.1.1. Application Performance . . . . . . . . . . . . . . . 11 88 5.1.2. Network Characterization . . . . . . . . . . . . . . . 12 89 5.1.3. Delay Variation . . . . . . . . . . . . . . . . . . . 13 90 5.1.4. Reordering . . . . . . . . . . . . . . . . . . . . . . 14 91 5.2. Preferred Statistics . . . . . . . . . . . . . . . . . . . 14 92 5.3. Summary for Delay . . . . . . . . . . . . . . . . . . . . 15 93 6. Effect of POV on Raw Capacity Metrics . . . . . . . . . . . . 15 94 6.1. Type-P Parameter . . . . . . . . . . . . . . . . . . . . . 15 95 6.2. a priori Factors . . . . . . . . . . . . . . . . . . . . . 16 96 6.3. IP-layer Capacity . . . . . . . . . . . . . . . . . . . . 16 97 6.4. IP-layer Utilization . . . . . . . . . . . . . . . . . . . 17 98 6.5. IP-layer Available Capacity . . . . . . . . . . . . . . . 17 99 6.6. Variability in Utilization and Avail. Capacity . . . . . . 18 100 7. Test Streams and Sample Size . . . . . . . . . . . . . . . . . 18 101 7.1. Test Stream Characteristics . . . . . . . . . . . . . . . 18 102 7.2. Sample Size . . . . . . . . . . . . . . . . . . . . . . . 19 103 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 104 9. Security Considerations . . . . . . . . . . . . . . . . . . . 19 105 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 19 106 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 20 107 11.1. Normative References . . . . . . . . . . . . . . . . . . . 20 108 11.2. Informative References . . . . . . . . . . . . . . . . . . 21 109 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 21 111 1. Introduction 113 When designing measurements of IP networks and presenting the 114 results, knowledge of the audience is a key consideration. To 115 present a useful and relevant portrait of network conditions, one 116 must answer the following question: 118 "How will the results be used?" 120 There are two main audience categories: 122 1. Network Characterization - describes conditions in an IP network 123 for quality assurance, troubleshooting, modeling, Service Level 124 Agreements (SLA), etc. The point-of-view looks inward, toward 125 the network, and the consumer intends their actions there. 127 2. Application Performance Estimation - describes the network 128 conditions in a way that facilitates determining affects on user 129 applications, and ultimately the users themselves. This point- 130 of-view looks outward, toward the user(s), accepting the network 131 as-is. This consumer intends to estimate a network-dependent 132 aspect of performance, or design some aspect of an application's 133 accommodation of the network. (These are *not* application 134 metrics, they are defined at the IP layer.) 136 This memo considers how these different points-of-view affect both 137 the measurement design (parameters and options of the metrics) and 138 statistics reported when serving their needs. 140 The IPPM framework [RFC2330] and other RFCs describing IPPM metrics 141 provide a background for this memo. 143 2. Purpose and Scope 145 The purpose of this memo is to clearly delineate two points-of-view 146 (POV) for using measurements, and describe their effects on the test 147 design, including the selection of metric parameters and reporting 148 the results. 150 The current scope of this memo primarily covers the design and 151 reporting of the loss and delay metrics [RFC2680] [RFC2679]. It will 152 also discuss the delay variation [RFC3393] and reordering metrics 153 [RFC4737] where applicable. 155 With capacity metrics growing in relevance to the industry, the memo 156 also covers POV and reporting considerations for metrics resulting 157 from the Bulk Transfer Capacity Framework [RFC3148] and Network 158 Capacity Definitions [RFC5136]. These memos effectively describe two 159 different categories of metrics, [RFC3148] with congestion flow- 160 control and the notion of unique data bits delivered, and [RFC5136] 161 using a definition of raw capacity without the restrictions of data 162 uniqueness or congestion-awareness. It might seem at first glance 163 that each of these metrics has an obvious audience (Raw = Network 164 Characterization, Restricted = Application Performance), but reality 165 is more complex and consistent with the overall topic of capacity 166 measurement and reporting. For example, TCP is usually used in 167 Restricted capacity measurement methods, while UDP appears in Raw 168 capacity measurement. The Raw and Restricted capacity metrics will 169 be treated in separate sections, although they share one common 170 reporting issue: representing variability in capacity metric results. 172 Sampling, or the design of the active packet stream that is the basis 173 for the measurements, is also discussed. 175 3. Reporting Results 177 This section gives an overview of recommendations, followed by 178 additional considerations for reporting results in the "long-term". 180 3.1. Overview of Metric Statistics 182 This section gives an overview of reporting recommendations for the 183 loss, delay, and delay variation metrics based on the discussion and 184 conclusions of the preceding sections. 186 The minimal report on measurements MUST include both Loss and Delay 187 Metrics. 189 For Packet Loss, the loss ratio defined in [RFC2680] is a sufficient 190 starting point, especially the guidance for setting the loss 191 threshold waiting time. We have calculated a waiting time above that 192 should be sufficient to differentiate between packets that are truly 193 lost or have long finite delays under general measurement 194 circumstances, 51 seconds. Knowledge of specific conditions can help 195 to reduce this threshold, but 51 seconds is considered to be 196 manageable in practice. 198 We note that a loss ratio calculated according to [Y.1540] would 199 exclude errored packets form the numerator. In practice, the 200 difference between these two loss metrics is small if any, depending 201 on whether the last link prior to the destination contributes errored 202 packets. 204 For Packet Delay, we recommend providing both the mean delay and the 205 median delay with lost packets designated undefined (as permitted by 206 [RFC2679]). Both statistics are based on a conditional distribution, 207 and the condition is packet arrival prior to a waiting time dT, where 208 dT has been set to take maximum packet lifetimes into account, as 209 discussed above. Using a long dT helps to ensure that delay 210 distributions are not truncated. 212 For Packet Delay Variation (PDV), the minimum delay of the 213 conditional distribution should be used as the reference delay for 214 computing PDV according to [Y.1540] or [RFC3393]. A useful value to 215 report is a pseudo range of delay variation based on calculating the 216 difference between a high percentile of delay and the minimum delay. 217 For example, the 99.9%-ile minus the minimum will give a value that 218 can be compared with objectives in [Y.1541]. 220 3.2. Long-Term Reporting Considerations 222 [I-D.ietf-ippm-reporting] describes methods to conduct measurements 223 and report the results on a near-immediate time scale (10 seconds, 224 which we consider to be "short-term"). 226 Measurement intervals and reporting intervals need not be the same 227 length. Sometimes, the user is only concerned with the performance 228 levels achieved over a relatively long interval of time (e.g, days, 229 weeks, or months, as opposed to 10 seconds). However, there can be 230 risks involved with running a measurement continuously over a long 231 period without recording intermediate results: 233 o Temporary power failure may cause loss of all the results to date. 235 o Measurement system timing synchronization signals may experience a 236 temporary outage, causing sub-sets of measurements to be in error 237 or invalid. 239 o Maintenance may be necessary on the measurement system, or its 240 connectivity to the network under test. 242 For these and other reasons, such as 244 o the constraint to collect measurements on intervals similar to 245 user session length, or 247 o the dual-use of measurements in monitoring activities where 248 results are needed on a period of a few minutes, 250 there is value in conducting measurements on intervals that are much 251 shorter than the reporting interval. 253 There are several approaches for aggregating a series of measurement 254 results over time in order to make a statement about the longer 255 reporting interval. One approach requires the storage of all metric 256 singletons collected throughout the reporting interval, even though 257 the measurement interval stops and starts many times. 259 Another approach is described in [RFC5835] as "temporal aggregation". 260 This approach would estimate the results for the reporting interval 261 based on many individual measurement interval statistics (results) 262 alone. The result would ideally appear in the same form as though a 263 continuous measurement was conducted. A memo to address the details 264 of temporal aggregation is yet to be prepared. 266 Yet another approach requires a numerical objective for the metric, 267 and the results of each measurement interval are compared with the 268 objective. Every measurement interval where the results meet the 269 objective contribute to the fraction of time with performance as 270 specified. When the reporting interval contains many measurement 271 intervals it is possible to present the results as "metric A was less 272 than or equal to objective X during Y% of time. 274 NOTE that numerical thresholds are not set in IETF performance work 275 and are explicitly excluded from the IPPM charter. 277 In all measurement, it is important to avoid unintended 278 synchronization with network events. This topic is treated in 279 [RFC2330] for Poisson-distributed inter-packet time streams, and 280 [RFC3432] for Periodic streams. Both avoid synchronization through 281 use of random start times. 283 There are network conditions where it is simply more useful to report 284 the connectivity status of the Source-Destination path, and to 285 distinguish time intervals where connectivity can be demonstrated 286 from other time intervals (where connectivity does not appear to 287 exist). [RFC2678] specifies a number of one-way and two connectivity 288 metrics of increasing complexity. In this memo, we RECOMMEND that 289 long term reporting of loss, delay, and other metrics be limited to 290 time intervals where connectivity can be demonstrated, and other 291 intervals be summarized as percent of time where connectivity does 292 not appear to exist. We note that this same approach has been 293 adopted in ITU-T Recommendation [Y.1540] where performance parameters 294 are only valid during periods of service "availability" (evaluated 295 according to a function based on packet loss, and sustained periods 296 of loss ratio greater than a threshold are declared "unavailable"). 298 4. Effect of POV on the Loss Metric 300 This section describes the ways in which the Loss metric can be tuned 301 to reflect the preferences of the two audience categories, or 302 different POV. The waiting time to declare a packet lost, or loss 303 threshold is one area where there would appear to be a difference, 304 but the ability to post-process the results may resolve it. 306 4.1. Loss Threshold 308 RFC 2680 [RFC2680] defines the concept of a waiting time for packets 309 to arrive, beyond which they are declared lost. The text of the RFC 310 declines to recommend a value, instead saying that "good engineering, 311 including an understanding of packet lifetimes, will be needed in 312 practice." Later, in the methodology, they give reasons for waiting 313 "a reasonable period of time", and leaving the definition of 314 "reasonable" intentionally vague. 316 4.1.1. Network Characterization 318 Practical measurement experience has shown that unusual network 319 circumstances can cause long delays. One such circumstance is when 320 routing loops form during IGP re-convergence following a failure or 321 drastic link cost change. Packets will loop between two routers 322 until new routes are installed, or until the IPv4 Time-to-Live (TTL) 323 field (or the IPv6 Hop Limit) decrements to zero. Very long delays 324 on the order of several seconds have been measured [Casner] [Cia03]. 326 Therefore, network characterization activities prefer a long waiting 327 time in order to distinguish these events from other causes of loss 328 (such as packet discard at a full queue, or tail drop). This way, 329 the metric design helps to distinguish more reliably between packets 330 that might yet arrive, and those that are no longer traversing the 331 network. 333 It is possible to calculate a worst-case waiting time, assuming that 334 a routing loop is the cause. We model the path between Source and 335 Destination as a series of delays in links (t) and queues (q), as 336 these two are the dominant contributors to delay. The normal path 337 delay across n hops without encountering a loop, D, is 338 n 339 --- 340 \ 341 D = t + > t + q 342 0 / i i 343 --- 344 i = 1 346 Figure 1: Normal Path Delay 348 and the time spent in the loop with L hops, is 350 i + L-1 351 --- 352 \ (TTL - n) 353 R = C > t + q where C = --------- 354 / i i max L 355 --- 356 i 358 Figure 2: Delay due to Rotations in a Loop 360 and where C is the number of times a packet circles the loop. 362 If we take the delays of all links and queues as 100ms each, the 363 TTL=255, the number of hops n=5 and the hops in the loop L=4, then 365 D = 1.1 sec and R ~= 50 sec, and D + R ~= 51.1 seconds 367 We note that the link delays of 100ms would span most continents, and 368 a constant queue length of 100ms is also very generous. When a loop 369 occurs, it is almost certain to be resolved in 10 seconds or less. 370 The value calculated above is an upper limit for almost any realistic 371 circumstance. 373 A waiting time threshold parameter, dT, set consistent with this 374 calculation would not truncate the delay distribution (possibly 375 causing a change in its mathematical properties), because the packets 376 that might arrive have been given sufficient time to traverse the 377 network. 379 It is worth noting that packets that are stored and deliberately 380 forwarded at a much later time constitute a replay attack on the 381 measurement system, and are beyond the scope of normal performance 382 reporting. 384 4.1.2. Application Performance 386 Fortunately, application performance estimation activities are not 387 adversely affected by the estimated worst-case transfer time. 388 Although the designer's tendency might be to set the Loss Threshold 389 at a value equivalent to a particular application's threshold, this 390 specific threshold can be applied when post-processing the 391 measurements. A shorter waiting time can be enforced by locating 392 packets with delays longer than the application's threshold, and re- 393 designating such packets as lost. Thus, the measurement system can 394 use a single loss threshold and support both application and network 395 performance POVs simultaneously. 397 4.2. Errored Packet Designation 399 RFC 2680 designates packets that arrive containing errors as lost 400 packets. Many packets that are corrupted by bit errors are discarded 401 within the network and do not reach their intended destination. 403 This is consistent with applications that would check the payload 404 integrity at higher layers, and discard the packet. However, some 405 applications prefer to deal with errored payloads on their own, and 406 even a corrupted payload is better than no packet at all. 408 To address this possibility, and to make network characterization 409 more complete, it is recommended to distinguish between packets that 410 do not arrive (lost) and errored packets that arrive (conditionally 411 lost). 413 4.3. Causes of Lost Packets 415 Although many measurement systems use a waiting time to determine if 416 a packet is lost or not, most of the waiting is in vain. The packets 417 are no-longer traversing the network, and have not reached their 418 destination. 420 There are many causes of packet loss, including: 422 1. Queue drop, or discard 424 2. Corruption of the IP header, or other essential header info 426 3. TTL expiration (or use of a TTL value that is too small) 428 4. Link or router failure 430 After waiting sufficient time, packet loss can probably be attributed 431 to one of these causes. 433 4.4. Summary for Loss 435 Given that measurement post-processing is possible (even encouraged 436 in the definitions of IPPM metrics), measurements of loss can easily 437 serve both points of view: 439 o Use a long waiting time to serve network characterization and 440 revise results for specific application delay thresholds as 441 needed. 443 o Distinguish between errored packets and lost packets when possible 444 to aid network characterization, and combine the results for 445 application performance if appropriate. 447 5. Effect of POV on the Delay Metric 449 This section describes the ways in which the Delay metric can be 450 tuned to reflect the preferences of the two consumer categories, or 451 different POV. 453 5.1. Treatment of Lost Packets 455 The Delay Metric [RFC2679] specifies the treatment of packets that do 456 not successfully traverse the network: their delay is undefined. 458 " >>The *Type-P-One-way-Delay* from Src to Dst at T is undefined 459 (informally, infinite)<< means that Src sent the first bit of a 460 Type-P packet to Dst at wire-time T and that Dst did not receive that 461 packet." 463 It is an accepted, but informal practice to assign infinite delay to 464 lost packets. We next look at how these two different treatments 465 align with the needs of measurement consumers who wish to 466 characterize networks or estimate application performance. Also, we 467 look at the way that lost packets have been treated in other metrics: 468 delay variation and reordering. 470 5.1.1. Application Performance 472 Applications need to perform different functions, dependent on 473 whether or not each packet arrives within some finite tolerance. In 474 other words, a receivers' packet processing takes one of two 475 directions (or "forks" in the road): 477 o Packets that arrive within expected tolerance are handled by 478 processes that remove headers, restore smooth delivery timing (as 479 in a de-jitter buffer), restore sending order, check for errors in 480 payloads, and many other operations. 482 o Packets that do not arrive when expected spawn other processes 483 that attempt recovery from the apparent loss, such as 484 retransmission requests, loss concealment, or forward error 485 correction to replace the missing packet. 487 So, it is important to maintain a distinction between packets that 488 actually arrive, and those that do not. Therefore, it is preferable 489 to leave the delay of lost packets undefined, and to characterize the 490 delay distribution as a conditional distribution (conditioned on 491 arrival). 493 5.1.2. Network Characterization 495 In this discussion, we assume that both loss and delay metrics will 496 be reported for network characterization (at least). 498 Assume packets that do not arrive are reported as Lost, usually as a 499 fraction of all sent packets. If these lost packets are assigned 500 undefined delay, then network's inability to deliver them (in a 501 timely way) is captured only in the loss metric when we report 502 statistics on the Delay distribution conditioned on the event of 503 packet arrival (within the Loss waiting time threshold). We can say 504 that the Delay and Loss metrics are Orthogonal, in that they convey 505 non-overlapping information about the network under test. 507 However, if we assign infinite delay to all lost packets, then: 509 o The delay metric results are influenced both by packets that 510 arrive and those that do not. 512 o The delay singleton and the loss singleton do not appear to be 513 orthogonal (Delay is finite when Loss=0, Delay is infinite when 514 Loss=1). 516 o The network is penalized in both the loss and delay metrics, 517 effectively double-counting the lost packets. 519 As further evidence of overlap, consider the Cumulative Distribution 520 Function (CDF) of Delay when the value positive infinity is assigned 521 to all lost packets. Figure 3 shows a CDF where a small fraction of 522 packets are lost. 524 1 | - - - - - - - - - - - - - - - - - -+ 525 | | 526 | _..----'''''''''''''''''''' 527 | ,-'' 528 | ,' 529 | / Mass at 530 | / +infinity 531 | / = fraction 532 || lost 533 |/ 534 0 |_____________________________________ 536 0 Delay +o0 538 Figure 3: Cumulative Distribution Function for Delay when Loss = 539 +Infinity 541 We note that a Delay CDF that is conditioned on packet arrival would 542 not exhibit this apparent overlap with loss. 544 Although infinity is a familiar mathematical concept, it is somewhat 545 disconcerting to see any time-related metric reported as infinity, in 546 the opinion of the authors. Questions are bound to arise, and tend 547 to detract from the goal of informing the consumer with a performance 548 report. 550 5.1.3. Delay Variation 552 [RFC3393] excludes lost packets from samples, effectively assigning 553 an undefined delay to packets that do not arrive in a reasonable 554 time. Section 4.1 describes this specification and its rationale 555 (ipdv = inter-packet delay variation in the quote below). 557 "The treatment of lost packets as having "infinite" or "undefined" 558 delay complicates the derivation of statistics for ipdv. 559 Specifically, when packets in the measurement sequence are lost, 560 simple statistics such as sample mean cannot be computed. One 561 possible approach to handling this problem is to reduce the event 562 space by conditioning. That is, we consider conditional statistics; 563 namely we estimate the mean ipdv (or other derivative statistic) 564 conditioned on the event that selected packet pairs arrive at the 565 destination (within the given timeout). While this itself is not 566 without problems (what happens, for example, when every other packet 567 is lost), it offers a way to make some (valid) statements about ipdv, 568 at the same time avoiding events with undefined outcomes." 570 We note that the argument above applies to all forms of packet delay 571 variation that can be constructed using the "selection function" 572 concept of [RFC3393]. In recent work the two main forms of delay 573 variation metrics have been compared and the results are summarized 574 in [RFC5481]. 576 5.1.4. Reordering 578 [RFC4737]defines metrics that are based on evaluation of packet 579 arrival order, and include a waiting time to declare a packet lost 580 (to exclude them from further processing). 582 If packets are assigned a delay value, then the reordering metric 583 would declare any packets with infinite delay to be reordered, 584 because their sequence numbers will surely be less than the "Next 585 Expected" threshold when (or if) they arrive. But this practice 586 would fail to maintain orthogonality between the reordering metric 587 and the loss metric. Confusion can be avoided by designating the 588 delay of non-arriving packets as undefined, and reserving delay 589 values only for packets that arrive within a sufficiently long 590 waiting time. 592 5.2. Preferred Statistics 594 Today in network characterization, the sample mean is one statistic 595 that is almost ubiquitously reported. It is easily computed and 596 understood by virtually everyone in this audience category. Also, 597 the sample is usually filtered on packet arrival, so that the mean is 598 based a conditional distribution. 600 The median is another statistic that summarizes a distribution, 601 having somewhat different properties from the sample mean. The 602 median is stable in distributions with a few outliers or without 603 them. However, the median's stability prevents it from indicating 604 when a large fraction of the distribution changes value. 50% or more 605 values would need to change for the median to capture the change. 607 Both the median and sample mean have difficulty with bimodal 608 distributions. The median will reside in only one of the modes, and 609 the mean may not lie in either mode range. For this and other 610 reasons, additional statistics such as the minimum, maximum, and 95%- 611 ile have value when summarizing a distribution. 613 When both the sample mean and median are available, a comparison will 614 sometimes be informative, because these two statistics are equal only 615 when the delay distribution is perfectly symmetrical. 617 Also, these statistics are generally useful from the Application 618 Performance POV, so there is a common set that should satisfy 619 audiences. 621 Plots of the delay distribution may also be useful when single-value 622 statistics indicate that new conditions are present. An empirically- 623 derived probability distribution function will usually describe 624 multiple modes more efficiently than any other form of result. 626 5.3. Summary for Delay 628 From the perspectives of: 630 1. application/receiver analysis, where subsequent processing 631 depends on whether the packet arrives or times-out, 633 2. straightforward network characterization without double-counting 634 defects, and 636 3. consistency with Delay variation and Reordering metric 637 definitions, 639 the most efficient practice is to distinguish between truly lost and 640 delayed packets with a sufficiently long waiting time, and to 641 designate the delay of non-arriving packets as undefined. 643 6. Effect of POV on Raw Capacity Metrics 645 This section describes the ways that raw capacity metrics can be 646 tuned to reflect the preferences of the two audiences, or different 647 Points-of-View (POV). Raw capacity refers to the metrics defined in 648 [RFC5136] which do not include restrictions such as data uniqueness 649 or flow-control response to congestion. 651 In summary, the metrics considered are IP-layer Capacity, Utilization 652 (or used capacity), and Available Capacity, for individual links and 653 complete paths. These three metrics form a triad: knowing one metric 654 constrains the other two (within their allowed range), and knowing 655 two determines the third. The link metrics have another key aspect 656 in common: they are single-measurement-point metrics at the egress of 657 a link. The path Capacity and Available Capacity are derived by 658 examining the set of single-point link measurements and taking the 659 minimum value. 661 6.1. Type-P Parameter 663 The concept of "packets of type-P" is defined in [RFC2330]. The 664 type-P categorization has critical relevance in all forms of capacity 665 measurement and reporting. The ability to categorize packets based 666 on header fields for assignment to different queues and scheduling 667 mechanisms is now common place. When un-used resources are shared 668 across queues, the conditions in all packet categories will affect 669 capacity and related measurements. This is one source of variability 670 in the results that all audiences would prefer to see reported in a 671 useful and easily understood way. 673 Type-P in OWAMP and TWAMP is essentially confined to the Diffserv 674 Codepoint [ref]. DSCP is the most common qualifier for type-P. 676 Each audience will have a set of type-P qualifications and value 677 combinations that are of interest. Measurements and reports SHOULD 678 have the flexibility to per-type and aggregate performance. 680 6.2. a priori Factors 682 The audience for Network Characterization may have detailed 683 information about each link that comprises a complete path (due to 684 ownership, for example), or some of the links in the path but not 685 others, or none of the links. 687 There are cases where the measurement audience only has information 688 on one of the links (the local access link), and wishes to measure 689 one or more of the raw capacity metrics. This scenario is quite 690 common, and has spawned a substantial number of experimental 691 measurement methods [ref to CAIDA survey page, etc.]. Many of these 692 methods respect that their users want a result fairly quickly and in 693 a one-trial. Thus, the measurement interval is kept short (a few 694 seconds to a minute). 696 6.3. IP-layer Capacity 698 For links, this metric's theoretical maximum value can be determined 699 from the physical layer bit rate and the bit rate reduction due to 700 the layers between the physical layer and IP. When measured, this 701 metric takes additional factors into account, such as the ability of 702 the sending device to process and forward traffic under various 703 conditions. For example, the arrival of routing updates may spawn 704 high priority processes that reduce the sending rate temporarily. 705 Thus, the measured capacity of a link will be variable, and the 706 maximum capacity observed applies to a specific time, time interval, 707 and other relevant circumstances. 709 For paths composed of a series of links, it is easy to see how the 710 sources of variability for the results grow with each link in the 711 path. Results variability will be discussed in more detail below. 713 6.4. IP-layer Utilization 715 The ideal metric definition of Link Utilization [RFC5136] is based on 716 the actual usage (bits successfully received during a time interval) 717 and the Maximum Capacity for the same interval. 719 In practice, Link Utilization can be calculated by counting the IP- 720 layer (or other layer) octets received over a time interval and 721 dividing by the theoretical maximum of octets that could have been 722 delivered in the same interval. A commonly used time interval is 5 723 minutes, and this interval has been sufficient to support network 724 operations and design for some time. 5 minutes is somewhat long 725 compared with the expected download time for web pages, but short 726 with respect to large file transfers and TV program viewing. It is 727 fair to say that considerable variability is concealed by reporting a 728 single (average) Utilization value for each 5 minute interval. Some 729 performance management systems have begun to make 1 minute averages 730 available. 732 There is also a limit on the smallest useful measurement interval. 733 Intervals on the order of the serialization time for a single Maximum 734 Transmission Unit (MTU) packet will observe on/off behavior and 735 report 100% or 0%. The smallest interval needs to be some multiple 736 of MTU serialization time for averaging to be effective. 738 6.5. IP-layer Available Capacity 740 The Available Capacity of a link can be calculated using the Capacity 741 and Utilization metrics. 743 When Available capacity of a link or path is estimated through some 744 measurement technique, the following parameters SHOULD be reported: 746 o Name and reference to the exact method of measurement 748 o IP packet length, octets (including IP header) 750 o Maximum Capacity that can be assessed in the measurement 751 configuration 753 o The time a duration of the measurement 755 o All other parameters specific to the measurement method 757 Many methods of Available capacity measurement have a maximum 758 capacity that they can measure, and this maximum may be less than the 759 actual Available capacity of the link or path. Therefore, it is 760 important to know the capacity value beyond which there will be no 761 measured improvement. 763 The Application Design audience may have a target capacity value and 764 simply wish to assess whether there is sufficient Available Capacity. 765 This case simplifies measurement of link and path capacity to some 766 degree, as long as the measurable maximum exceeds the target 767 capacity. 769 6.6. Variability in Utilization and Avail. Capacity 771 As with most metrics and measurements, assessing the consistency or 772 variability in the results gives a the user an intuitive feel for the 773 degree (or confidence) that any one value is representative of other 774 results, or the underlying distribution from which these singleton 775 measurements have come. 777 Two questions are raised here for further discussion: 779 What ways can Utilization be measured and summarized to describe the 780 potential variability in a useful way? 782 How can the variability in Available Capacity estimates be reported, 783 so that the confidence in the results is also conveyed? 785 7. Test Streams and Sample Size 787 This section discusses two key aspects of measurement that are 788 sometimes omitted from the report: the description of the test stream 789 on which the measurements are based, and the sample size. 791 7.1. Test Stream Characteristics 793 Network Characterization has traditionally used Poisson-distributed 794 inter-packet spacing, as this provides an unbiased sample. The 795 average inter-packet spacing may be selected to allow observation of 796 specific network phenomena. Other test streams are designed to 797 sample some property of the network, such as the presence of 798 congestion, link bandwidth, or packet reordering. 800 If measuring a network in order to make inferences about applications 801 or receiver performance, then there are usually efficiencies derived 802 from a test stream that has similar characteristics to the sender. 803 In some cases, it is essential to synthesize the sender stream, as 804 with Bulk Transfer Capacity estimates. In other cases, it may be 805 sufficient to sample with a "known bias", e.g., a Periodic stream to 806 estimate real-time application performance. 808 7.2. Sample Size 810 Sample size is directly related to the accuracy of the results, and 811 plays a critical role in the report. Even if only the sample size 812 (in terms of number of packets) is given for each value or summary 813 statistic, it imparts a notion of the confidence in the result. 815 In practice, the sample size will be selected taking both statistical 816 and practical factors into account. Among these factors are: 818 1. The estimated variability of the quantity being measured 820 2. The desired confidence in the result (although this may be 821 dependent on assumption of the underlying distribution of the 822 measured quantity). 824 3. The effects of active measurement traffic on user traffic 826 4. etc. 828 A sample size may sometimes be referred to as "large". This is a 829 relative, and qualitative term. It is preferable to describe what 830 one is attempting to achieve with their sample. For example, stating 831 an implication may be helpful: this sample is large enough such that 832 a single outlying value at ten times the "typical" sample mean (the 833 mean without the outlying value) would influence the mean by no more 834 than X. 836 8. IANA Considerations 838 This document makes no request of IANA. 840 Note to RFC Editor: this section may be removed on publication as an 841 RFC. 843 9. Security Considerations 845 The security considerations that apply to any active measurement of 846 live networks are relevant here as well. See [RFC4656]. 848 10. Acknowledgements 850 The authors thank: Phil Chimento for his suggestion to employ 851 conditional distributions for Delay, Steve Konish Jr. for his careful 852 review and suggestions, Dave Mcdysan and Don McLachlan for useful 853 comments based on their long experience with measurement and 854 reporting. 856 11. References 858 11.1. Normative References 860 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 861 Requirement Levels", BCP 14, RFC 2119, March 1997. 863 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 864 "Framework for IP Performance Metrics", RFC 2330, 865 May 1998. 867 [RFC2678] Mahdavi, J. and V. Paxson, "IPPM Metrics for Measuring 868 Connectivity", RFC 2678, September 1999. 870 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 871 Delay Metric for IPPM", RFC 2679, September 1999. 873 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 874 Packet Loss Metric for IPPM", RFC 2680, September 1999. 876 [RFC3148] Mathis, M. and M. Allman, "A Framework for Defining 877 Empirical Bulk Transfer Capacity Metrics", RFC 3148, 878 July 2001. 880 [RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation 881 Metric for IP Performance Metrics (IPPM)", RFC 3393, 882 November 2002. 884 [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network 885 performance measurement with periodic streams", RFC 3432, 886 November 2002. 888 [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. 889 Zekauskas, "A One-way Active Measurement Protocol 890 (OWAMP)", RFC 4656, September 2006. 892 [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, 893 S., and J. Perser, "Packet Reordering Metrics", RFC 4737, 894 November 2006. 896 [RFC5136] Chimento, P. and J. Ishac, "Defining Network Capacity", 897 RFC 5136, February 2008. 899 11.2. Informative References 901 [Casner] "A Fine-Grained View of High Performance Networking, NANOG 902 22 Conf.; http://www.nanog.org/mtg-0105/agenda.html", May 903 20-22 2001. 905 [Cia03] "Standardized Active Measurements on a Tier 1 IP Backbone, 906 IEEE Communications Mag., pp 90-97.", June 2003. 908 [I-D.ietf-ippm-reporting] 909 Shalunov, S. and M. Swany, "Reporting IP Performance 910 Metrics to Users", draft-ietf-ippm-reporting-04 (work in 911 progress), July 2009. 913 [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation 914 Applicability Statement", RFC 5481, March 2009. 916 [RFC5835] Morton, A. and S. Van den Berghe, "Framework for Metric 917 Composition", RFC 5835, April 2010. 919 [Y.1540] ITU-T Recommendation Y.1540, "Internet protocol data 920 communication service - IP packet transfer and 921 availability performance parameters", December 2002. 923 [Y.1541] ITU-T Recommendation Y.1540, "Network Performance 924 Objectives for IP-Based Services", February 2006. 926 Authors' Addresses 928 Al Morton 929 AT&T Labs 930 200 Laurel Avenue South 931 Middletown, NJ 07748 932 USA 934 Phone: +1 732 420 1571 935 Fax: +1 732 368 1192 936 Email: acmorton@att.com 937 URI: http://home.comcast.net/~acmacm/ 938 Gomathi Ramachandran 939 AT&T Labs 940 200 Laurel Avenue South 941 Middletown, New Jersey 07748 942 USA 944 Phone: +1 732 420 2353 945 Fax: 946 Email: gomathi@att.com 947 URI: 949 Ganga Maguluri 950 AT&T Labs 951 200 Laurel Avenue 952 Middletown, New Jersey 07748 953 USA 955 Phone: 732-420-2486 956 Fax: 957 Email: gmaguluri@att.com 958 URI: