idnits 2.17.1 draft-ietf-ippm-reporting-metrics-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** You're using the IETF Trust Provisions' Section 6.b License Notice from 12 Sep 2009 rather than the newer Notice from 28 Dec 2009. (See https://trustee.ietf.org/license-info/) Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to contain a disclaimer for pre-RFC5378 work, but was first submitted on or after 10 November 2008. The disclaimer is usually necessary only for documents that revise or obsolete older RFCs, and that take significant amounts of text from those RFCs. If you can contact all authors of the source material and they are willing to grant the BCP78 rights to the IETF Trust, you can and should remove the disclaimer. Otherwise, the disclaimer is needed and you can ignore this comment. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (March 5, 2010) is 5156 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC4737' is defined on line 855, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) == Outdated reference: A later version (-06) exists of draft-ietf-ippm-reporting-04 Summary: 3 errors (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group A. Morton 3 Internet-Draft G. Ramachandran 4 Intended status: Informational G. Maguluri 5 Expires: September 6, 2010 AT&T Labs 6 March 5, 2010 8 Reporting Metrics: Different Points of View 9 draft-ietf-ippm-reporting-metrics-01 11 Abstract 13 Consumers of IP network performance metrics have many different uses 14 in mind. This memo categorizes the different audience points of 15 view. It describes how the categories affect the selection of metric 16 parameters and options when seeking info that serves their needs. 17 The memo then proceeds to discuss "long-term" reporting 18 considerations (e.g, days, weeks or months, as opposed to 10 19 seconds). 21 Requirements Language 23 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 24 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 25 document are to be interpreted as described in RFC 2119 [RFC2119]. 27 Status of this Memo 29 This Internet-Draft is submitted to IETF in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF), its areas, and its working groups. Note that 34 other groups may also distribute working documents as Internet- 35 Drafts. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 The list of current Internet-Drafts can be accessed at 43 http://www.ietf.org/ietf/1id-abstracts.txt. 45 The list of Internet-Draft Shadow Directories can be accessed at 46 http://www.ietf.org/shadow.html. 48 This Internet-Draft will expire on September 6, 2010. 50 Copyright Notice 52 Copyright (c) 2010 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents 57 (http://trustee.ietf.org/license-info) in effect on the date of 58 publication of this document. Please review these documents 59 carefully, as they describe your rights and restrictions with respect 60 to this document. Code Components extracted from this document must 61 include Simplified BSD License text as described in Section 4.e of 62 the Trust Legal Provisions and are provided without warranty as 63 described in the BSD License. 65 This document may contain material from IETF Documents or IETF 66 Contributions published or made publicly available before November 67 10, 2008. The person(s) controlling the copyright in some of this 68 material may not have granted the IETF Trust the right to allow 69 modifications of such material outside the IETF Standards Process. 70 Without obtaining an adequate license from the person(s) controlling 71 the copyright in such materials, this document may not be modified 72 outside the IETF Standards Process, and derivative works of it may 73 not be created outside the IETF Standards Process, except to format 74 it for publication as an RFC or to translate it into languages other 75 than English. 77 Table of Contents 79 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 80 2. Purpose and Scope . . . . . . . . . . . . . . . . . . . . . . 4 81 3. Effect of POV on the Loss Metric . . . . . . . . . . . . . . . 5 82 3.1. Loss Threshold . . . . . . . . . . . . . . . . . . . . . . 5 83 3.1.1. Network Characterization . . . . . . . . . . . . . . . 5 84 3.1.2. Application Performance . . . . . . . . . . . . . . . 7 85 3.2. Errored Packet Designation . . . . . . . . . . . . . . . . 7 86 3.3. Causes of Lost Packets . . . . . . . . . . . . . . . . . . 7 87 3.4. Summary for Loss . . . . . . . . . . . . . . . . . . . . . 8 88 4. Effect of POV on the Delay Metric . . . . . . . . . . . . . . 8 89 4.1. Treatment of Lost Packets . . . . . . . . . . . . . . . . 8 90 4.1.1. Application Performance . . . . . . . . . . . . . . . 9 91 4.1.2. Network Characterization . . . . . . . . . . . . . . . 9 92 4.1.3. Delay Variation . . . . . . . . . . . . . . . . . . . 10 93 4.1.4. Reordering . . . . . . . . . . . . . . . . . . . . . . 11 94 4.2. Preferred Statistics . . . . . . . . . . . . . . . . . . . 11 95 4.3. Summary for Delay . . . . . . . . . . . . . . . . . . . . 12 96 5. Effect of POV on Raw Capacity Metrics . . . . . . . . . . . . 12 97 5.1. Type-P Parameter . . . . . . . . . . . . . . . . . . . . . 12 98 5.2. a priori Factors . . . . . . . . . . . . . . . . . . . . . 13 99 5.3. IP-layer Capacity . . . . . . . . . . . . . . . . . . . . 13 100 5.4. IP-layer Utilization . . . . . . . . . . . . . . . . . . . 13 101 5.5. IP-layer Available Capacity . . . . . . . . . . . . . . . 14 102 5.6. Variability in Utilization and Avail. Capacity . . . . . . 15 103 6. Test Streams and Sample Size . . . . . . . . . . . . . . . . . 15 104 6.1. Test Stream Characteristics . . . . . . . . . . . . . . . 15 105 6.2. Sample Size . . . . . . . . . . . . . . . . . . . . . . . 15 106 7. Reporting Results . . . . . . . . . . . . . . . . . . . . . . 16 107 7.1. Overview of Metric Statistics . . . . . . . . . . . . . . 16 108 7.2. Long-Term Reporting Considerations . . . . . . . . . . . . 17 109 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 110 9. Security Considerations . . . . . . . . . . . . . . . . . . . 18 111 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 18 112 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 19 113 11.1. Normative References . . . . . . . . . . . . . . . . . . . 19 114 11.2. Informative References . . . . . . . . . . . . . . . . . . 19 115 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20 117 1. Introduction 119 When designing measurements of IP networks and presenting the 120 results, knowledge of the audience is a key consideration. To 121 present a useful and relevant portrait of network conditions, one 122 must answer the following question: 124 "How will the results be used?" 126 There are two main audience categories: 128 1. Network Characterization - describes conditions in an IP network 129 for quality assurance, troubleshooting, modeling, etc. The 130 point-of-view looks inward, toward the network, and the consumer 131 intends their actions there. 133 2. Application Performance Estimation - describes the network 134 conditions in a way that facilitates determining affects on user 135 applications, and ultimately the users themselves. This point- 136 of-view looks outward, toward the user(s), accepting the network 137 as-is. This consumer intends to estimate a network-dependent 138 aspect of performance, or design some aspect of an application's 139 accommodation of the network. (These are *not* application 140 metrics, they are defined at the IP layer.) 142 This memo considers how these different points-of-view affect both 143 the measurement design (parameters and options of the metrics) and 144 statistics reported when serving their needs. 146 The IPPM framework [RFC2330] and other RFCs describing IPPM metrics 147 provide a background for this memo. 149 2. Purpose and Scope 151 The purpose of this memo is to clearly delineate two points-of-view 152 (POV) for using measurements, and describe their effects on the test 153 design, including the selection of metric parameters and reporting 154 the results. 156 The current scope of this memo primarily covers the design and 157 reporting of the loss and delay metrics [RFC2680] [RFC2679]. It will 158 also discuss the delay variation and reordering metrics where 159 applicable. 161 With capacity metrics growing in relevance to the industry, the memo 162 also covers POV and reporting considerations for metrics resulting 163 from the Bulk Transfer Capacity Framework [RFC3148] and Network 164 Capacity Definitions [RFC5136]. These memos effectively describe two 165 different categories of metrics, [RFC3148] with congestion flow- 166 control and the notion of unique data bits delivered, and [RFC5136] 167 using a definition of raw capacity without the restrictions of data 168 uniqueness or congestion-awareness. It might seem at first glance 169 that each of these metrics has an obvious audience (Raw = Network 170 Characterization, Restricted = Application Performance), but reality 171 is more complex and consistent with the overall topic of capacity 172 measurement and reporting. The Raw and Restricted capacity metrics 173 will be treated in separate sections, although they share one common 174 reporting issue: representing variability in capacity metric results. 176 Sampling, or the design of the active packet stream that is the basis 177 for the measurements, is also discussed. 179 3. Effect of POV on the Loss Metric 181 This section describes the ways in which the Loss metric can be tuned 182 to reflect the preferences of the two audience categories, or 183 different POV. The waiting time to declare a packet lost, or loss 184 threshold is one area where there would appear to be a difference, 185 but the ability to post-process the results may resolve it. 187 3.1. Loss Threshold 189 RFC 2680 [RFC2680] defines the concept of a waiting time for packets 190 to arrive, beyond which they are declared lost. The text of the RFC 191 declines to recommend a value, instead saying that "good engineering, 192 including an understanding of packet lifetimes, will be needed in 193 practice." Later, in the methodology, they give reasons for waiting 194 "a reasonable period of time", and leaving the definition of 195 "reasonable" intentionally vague. 197 3.1.1. Network Characterization 199 Practical measurement experience has shown that unusual network 200 circumstances can cause long delays. One such circumstance is when 201 routing loops form during IGP re-convergence following a failure or 202 drastic link cost change. Packets will loop between two routers 203 until new routes are installed, or until the IPv4 Time-to-Live (TTL) 204 field (or the IPv6 Hop Limit) decrements to zero. Very long delays 205 on the order of several seconds have been measured [Casner] [Cia03]. 207 Therefore, network characterization activities prefer a long waiting 208 time in order to distinguish these events from other causes of loss 209 (such as packet discard at a full queue, or tail drop). This way, 210 the metric design helps to distinguish more reliably between packets 211 that might yet arrive, and those that are no longer traversing the 212 network. 214 It is possible to calculate a worst-case waiting time, assuming that 215 a routing loop is the cause. We model the path between Source and 216 Destination as a series of delays in links (t) and queues (q), as 217 these two are the dominant contributors to delay. The normal path 218 delay across n hops without encountering a loop, D, is 220 n 221 --- 222 \ 223 D = t + > t + q 224 0 / i i 225 --- 226 i = 1 228 Figure 1: Normal Path Delay 230 and the time spent in the loop with L hops, is 232 i + L-1 233 --- 234 \ (TTL - n) 235 R = C > t + q where C = --------- 236 / i i max L 237 --- 238 i 240 Figure 2: Delay due to Rotations in a Loop 242 and where C is the number of times a packet circles the loop. 244 If we take the delays of all links and queues as 100ms each, the 245 TTL=255, the number of hops n=5 and the hops in the loop L=4, then 247 D = 1.1 sec and R ~= 50 sec, and D + R ~= 51.1 seconds 249 We note that the link delays of 100ms would span most continents, and 250 a constant queue length of 100ms is also very generous. When a loop 251 occurs, it is almost certain to be resolved in 10 seconds or less. 252 The value calculated above is an upper limit for almost any realistic 253 circumstance. 255 A waiting time threshold parameter, dT, set consistent with this 256 calculation would not truncate the delay distribution (possibly 257 causing a change in its mathematical properties), because the packets 258 that might arrive have been given sufficient time to traverse the 259 network. 261 It is worth noting that packets that are stored and deliberately 262 forwarded at a much later time constitute a replay attack on the 263 measurement system, and are beyond the scope of normal performance 264 reporting. 266 3.1.2. Application Performance 268 Fortunately, application performance estimation activities are not 269 adversely affected by the estimated worst-case transfer time. 270 Although the designer's tendency might be to set the Loss Threshold 271 at a value equivalent to a particular application's threshold, this 272 specific threshold can be applied when post-processing the 273 measurements. A shorter waiting time can be enforced by locating 274 packets with delays longer than the application's threshold, and re- 275 designating such packets as lost. Thus, the measurement system can 276 use a single loss threshold and support both application and network 277 performance POVs simultaneously. 279 3.2. Errored Packet Designation 281 RFC 2680 designates packets that arrive containing errors as lost 282 packets. Many packets that are corrupted by bit errors are discarded 283 within the network and do not reach their intended destination. 285 This is consistent with applications that would check the payload 286 integrity at higher layers, and discard the packet. However, some 287 applications prefer to deal with errored payloads on their own, and 288 even a corrupted payload is better than no packet at all. 290 To address this possibility, and to make network characterization 291 more complete, it is recommended to distinguish between packets that 292 do not arrive (lost) and errored packets that arrive (conditionally 293 lost). 295 3.3. Causes of Lost Packets 297 Although many measurement systems use a waiting time to determine if 298 a packet is lost or not, most of the waiting is in vain. The packets 299 are no-longer traversing the network, and have not reached their 300 destination. 302 There are many causes of packet loss, including: 304 1. Queue drop, or discard 305 2. Corruption of the IP header, or other essential header info 307 3. TTL expiration (or use of a TTL value that is too small) 309 4. Link or router failure 311 After waiting sufficient time, packet loss can probably be attributed 312 to one of these causes. 314 3.4. Summary for Loss 316 Given that measurement post-processing is possible (even encouraged 317 in the definitions of IPPM metrics), measurements of loss can easily 318 serve both points of view: 320 o Use a long waiting time to serve network characterization and 321 revise results for specific application delay thresholds as 322 needed. 324 o Distinguish between errored packets and lost packets when possible 325 to aid network characterization, and combine the results for 326 application performance if appropriate. 328 4. Effect of POV on the Delay Metric 330 This section describes the ways in which the Delay metric can be 331 tuned to reflect the preferences of the two consumer categories, or 332 different POV. 334 4.1. Treatment of Lost Packets 336 The Delay Metric [RFC2679] specifies the treatment of packets that do 337 not successfully traverse the network: their delay is undefined. 339 " >>The *Type-P-One-way-Delay* from Src to Dst at T is undefined 340 (informally, infinite)<< means that Src sent the first bit of a 341 Type-P packet to Dst at wire-time T and that Dst did not receive that 342 packet." 344 It is an accepted, but informal practice to assign infinite delay to 345 lost packets. We next look at how these two different treatments 346 align with the needs of measurement consumers who wish to 347 characterize networks or estimate application performance. Also, we 348 look at the way that lost packets have been treated in other metrics: 349 delay variation and reordering. 351 4.1.1. Application Performance 353 Applications need to perform different functions, dependent on 354 whether or not each packet arrives within some finite tolerance. In 355 other words, a receivers' packet processing takes one of two 356 directions (or "forks" in the road): 358 o Packets that arrive within expected tolerance are handled by 359 processes that remove headers, restore smooth delivery timing (as 360 in a de-jitter buffer), restore sending order, check for errors in 361 payloads, and many other operations. 363 o Packets that do not arrive when expected spawn other processes 364 that attempt recovery from the apparent loss, such as 365 retransmission requests, loss concealment, or forward error 366 correction to replace the missing packet. 368 So, it is important to maintain a distinction between packets that 369 actually arrive, and those that do not. Therefore, it is preferable 370 to leave the delay of lost packets undefined, and to characterize the 371 delay distribution as a conditional distribution (conditioned on 372 arrival). 374 4.1.2. Network Characterization 376 In this discussion, we assume that both loss and delay metrics will 377 be reported for network characterization (at least). 379 Assume packets that do not arrive are reported as Lost, usually as a 380 fraction of all sent packets. If these lost packets are assigned 381 undefined delay, then network's inability to deliver them (in a 382 timely way) is captured only in the loss metric when we report 383 statistics on the Delay distribution conditioned on the event of 384 packet arrival (within the Loss waiting time threshold). We can say 385 that the Delay and Loss metrics are Orthogonal, in that they convey 386 non-overlapping information about the network under test. 388 However, if we assign infinite delay to all lost packets, then: 390 o The delay metric results are influenced both by packets that 391 arrive and those that do not. 393 o The delay singleton and the loss singleton do not appear to be 394 orthogonal (Delay is finite when Loss=0, Delay is infinite when 395 Loss=1). 397 o The network is penalized in both the loss and delay metrics, 398 effectively double-counting the lost packets. 400 As further evidence of overlap, consider the Cumulative Distribution 401 Function (CDF) of Delay when the value positive infinity is assigned 402 to all lost packets. Figure 3 shows a CDF where a small fraction of 403 packets are lost. 405 1 | - - - - - - - - - - - - - - - - - -+ 406 | | 407 | _..----'''''''''''''''''''' 408 | ,-'' 409 | ,' 410 | / Mass at 411 | / +infinity 412 | / = fraction 413 || lost 414 |/ 415 0 |_____________________________________ 417 0 Delay +o0 419 Figure 3: Cumulative Distribution Function for Delay when Loss = 420 +Infinity 422 We note that a Delay CDF that is conditioned on packet arrival would 423 not exhibit this apparent overlap with loss. 425 Although infinity is a familiar mathematical concept, it is somewhat 426 disconcerting to see any time-related metric reported as infinity, in 427 the opinion of the authors. Questions are bound to arise, and tend 428 to detract from the goal of informing the consumer with a performance 429 report. 431 4.1.3. Delay Variation 433 [RFC3393] excludes lost packets from samples, effectively assigning 434 an undefined delay to packets that do not arrive in a reasonable 435 time. Section 4.1 describes this specification and its rationale 436 (ipdv = inter-packet delay variation in the quote below). 438 "The treatment of lost packets as having "infinite" or "undefined" 439 delay complicates the derivation of statistics for ipdv. 440 Specifically, when packets in the measurement sequence are lost, 441 simple statistics such as sample mean cannot be computed. One 442 possible approach to handling this problem is to reduce the event 443 space by conditioning. That is, we consider conditional statistics; 444 namely we estimate the mean ipdv (or other derivative statistic) 445 conditioned on the event that selected packet pairs arrive at the 446 destination (within the given timeout). While this itself is not 447 without problems (what happens, for example, when every other packet 448 is lost), it offers a way to make some (valid) statements about ipdv, 449 at the same time avoiding events with undefined outcomes." 451 4.1.4. Reordering 453 [RFC4737]defines metrics that are based on evaluation of packet 454 arrival order, and include a waiting time to declare a packet lost 455 (to exclude them from further processing). 457 If packets are assigned a delay value, then the reordering metric 458 would declare any packets with infinite delay to be reordered, 459 because their sequence numbers will surely be less than the "Next 460 Expected" threshold when (or if) they arrive. But this practice 461 would fail to maintain orthogonality between the reordering metric 462 and the loss metric. Confusion can be avoided by designating the 463 delay of non-arriving packets as undefined, and reserving delay 464 values only for packets that arrive within a sufficiently long 465 waiting time. 467 4.2. Preferred Statistics 469 Today in network characterization, the sample mean is one statistic 470 that is almost ubiquitously reported. It is easily computed and 471 understood by virtually everyone in this audience category. Also, 472 the sample is usually filtered on packet arrival, so that the mean is 473 based a conditional distribution. 475 The median is another statistic that summarizes a distribution, 476 having somewhat different properties from the sample mean. The 477 median is stable in distributions with a few outliers or without 478 them. However, the median's stability prevents it from indicating 479 when a large fraction of the distribution changes value. 50% or more 480 values would need to change for the median to capture the change. 482 Both the median and sample mean have difficulty with bimodal 483 distributions. The median will reside in only one of the modes, and 484 the mean may not lie in either mode range. For this and other 485 reasons, additional statistics such as the minimum, maximum, and 95%- 486 ile have value when summarizing a distribution. 488 When both the sample mean and median are available, a comparison will 489 sometimes be informative, because these two statistics are equal only 490 when the delay distribution is perfectly symmetrical. 492 Also, these statistics are generally useful from the Application 493 Performance POV, so there is a common set that should satisfy 494 audiences. 496 4.3. Summary for Delay 498 From the perspectives of: 500 1. application/receiver analysis, where subsequent processing 501 depends on whether the packet arrives or times-out, 503 2. straightforward network characterization without double-counting 504 defects, and 506 3. consistency with Delay variation and Reordering metric 507 definitions, 509 the most efficient practice is to distinguish between truly lost and 510 delayed packets with a sufficiently long waiting time, and to 511 designate the delay of non-arriving packets as undefined. 513 5. Effect of POV on Raw Capacity Metrics 515 This section describes the ways that raw capacity metrics can be 516 tuned to reflect the preferences of the two audiences, or different 517 Points-of-View (POV). Raw capacity refers to the metrics defined in 518 [RFC5136] which do not include restrictions such as data uniqueness 519 or flow-control response to congestion. 521 In summary, the metrics considered are IP-layer Capacity, Utilization 522 (or used capacity), and Available Capacity, for individual links and 523 complete paths. These three metrics form a triad: knowing one metric 524 constrains the other two (within their allowed range), and knowing 525 two determines the third. The link metrics have another key aspect 526 in common: they are single-measurement-point metrics at the egress of 527 a link. The path Capacity and Available Capacity are derived by 528 examining the set of single-point link measurements and taking the 529 minimum value. 531 5.1. Type-P Parameter 533 The concept of "packets of type-P" is defined in [RFC2330]. The 534 type-P categorization has critical relevance in all forms of capacity 535 measurement and reporting. The ability to categorize packets based 536 on header fields for assignment to different queues and scheduling 537 mechanisms is now common place. When un-used resources are shared 538 across queues, the conditions in all packet categories will affect 539 capacity and related measurements. This is one source of variability 540 in the results that all audiences would prefer to see reported in a 541 useful and easily understood way. 543 Type-P in OWAMP and TWAMP is essentially confined to the Diffserv 544 Codepoint [ref]. DSCP is the most common qualifier for type-P. 546 Each audience will have a set of type-P qualifications and value 547 combinations that are of interest. Measurements and reports SHOULD 548 have the flexibility to per-type and aggregate performance. 550 5.2. a priori Factors 552 The audience for Network Characterization may have detailed 553 information about each link that comprises a complete path (due to 554 ownership, for example), or some of the links in the path but not 555 others, or none of the links. 557 There are cases where the measurement audience only has information 558 on one of the links (the local access link), and wishes to measure 559 one or more of the raw capacity metrics. This scenario is quite 560 common, and has spawned a substantial number of experimental 561 measurement methods [ref to CAIDA survey page, etc.]. Many of these 562 methods respect that their users want a result fairly quickly and in 563 a one-trial. Thus, the measurement interval is kept short (a few 564 seconds to a minute). 566 5.3. IP-layer Capacity 568 For links, this metric's theoretical maximum value can be determined 569 from the physical layer bit rate and the bit rate reduction due to 570 the layers between the physical layer and IP. When measured, this 571 metric takes additional factors into account, such as the ability of 572 the sending device to process and forward traffic under various 573 conditions. For example, the arrival of routing updates may spawn 574 high priority processes that reduce the sending rate temporarily. 575 Thus, the measured capacity of a link will be variable, and the 576 maximum capacity observed applies to a specific time, time interval, 577 and other relevant circumstances. 579 For paths composed of a series of links, it is easy to see how the 580 sources of variability for the results grow with each link in the 581 path. Results variability will be discussed in more detail below. 583 5.4. IP-layer Utilization 585 The ideal metric definition of Link Utilization [RFC5136] is based on 586 the actual usage (bits successfully received during a time interval) 587 and the Maximum Capacity for the same interval. 589 In practice, Link Utilization can be calculated by counting the IP- 590 layer (or other layer) octets received over a time interval and 591 dividing by the theoretical maximum of octets that could have been 592 delivered in the same interval. A commonly used time interval is 5 593 minutes, and this interval has been sufficient to support network 594 operations and design for some time. 5 minutes is somewhat long 595 compared with the expected download time for web pages, but short 596 with respect to large file transfers and TV program viewing. It is 597 fair to say that considerable variability is concealed by reporting a 598 single (average) Utilization value for each 5 minute interval. Some 599 performance management systems have begun to make 1 minute averages 600 available. 602 There is also a limit on the smallest useful measurement interval. 603 Intervals on the order of the serialization time for a single Maximum 604 Transmission Unit (MTU) packet will observe on/off behavior and 605 report 100% or 0%. The smallest interval needs to be some multiple 606 of MTU serialization time for averaging to be effective. 608 5.5. IP-layer Available Capacity 610 The Available Capacity of a link can be calculated using the Capacity 611 and Utilization metrics. 613 When Available capacity of a link or path is estimated through some 614 measurement technique, the following parameters SHOULD be reported: 616 o Name and reference to the exact method of measurement 618 o IP packet length, octets (including IP header) 620 o Maximum Capacity that can be assessed in the measurement 621 configuration 623 o The time a duration of the measurement 625 o All other parameters specific to the measurement method 627 Many methods of Available capacity measurement have a maximum 628 capacity that they can measure, and this maximum may be less than the 629 actual Available capacity of the link or path. Therefore, it is 630 important to know the capacity value beyond which there will be no 631 measured improvement. 633 The Application Design audience may have a target capacity value and 634 simply wish to assess whether there is sufficient Available Capacity. 635 This case simplifies measurement of link and path capacity to some 636 degree, as long as the measurable maximum exceeds the target 637 capacity. 639 5.6. Variability in Utilization and Avail. Capacity 641 As with most metrics and measurements, assessing the consistency or 642 variability in the results gives a the user an intuitive feel for the 643 degree (or confidence) that any one value is representative of other 644 results, or the underlying distribution from which these singleton 645 measurements have come. 647 Two questions are raised here for further discussion: 649 What ways can Utilization be measured and summarized to describe the 650 potential variability in a useful way? 652 How can the variability in Available Capacity estimates be reported, 653 so that the confidence in the results is also conveyed? 655 6. Test Streams and Sample Size 657 This section discusses two key aspects of measurement that are 658 sometimes omitted from the report: the description of the test stream 659 on which the measurements are based, and the sample size. 661 6.1. Test Stream Characteristics 663 Network Characterization has traditionally used Poisson-distributed 664 inter-packet spacing, as this provides an unbiased sample. The 665 average inter-packet spacing may be selected to allow observation of 666 specific network phenomena. Other test streams are designed to 667 sample some property of the network, such as the presence of 668 congestion, link bandwidth, or packet reordering. 670 If measuring a network in order to make inferences about applications 671 or receiver performance, then there are usually efficiencies derived 672 from a test stream that has similar characteristics to the sender. 673 In some cases, it is essential to synthesize the sender stream, as 674 with Bulk Transfer Capacity estimates. In other cases, it may be 675 sufficient to sample with a "known bias", e.g., a Periodic stream to 676 estimate real-time application performance. 678 6.2. Sample Size 680 Sample size is directly related to the accuracy of the results, and 681 plays a critical role in the report. Even if only the sample size 682 (in terms of number of packets) is given for each value or summary 683 statistic, it imparts a notion of the confidence in the result. 685 In practice, the sample size will be selected taking both statistical 686 and practical factors into account. Among these factors are: 688 1. The estimated variability of the quantity being measured 690 2. The desired confidence in the result (although this may be 691 dependent on assumption of the underlying distribution of the 692 measured quantity). 694 3. The effects of active measurement traffic on user traffic 696 4. etc. 698 A sample size may sometimes be referred to as "large". This is a 699 relative, and qualitative term. It is preferable to describe what 700 one is attempting to achieve with their sample. For example, stating 701 an implication may be helpful: this sample is large enough such that 702 a single outlying value at ten times the "typical" sample mean (the 703 mean without the outlying value) would influence the mean by no more 704 than X. 706 7. Reporting Results 708 This section gives an overview of recommendations, followed by 709 additional considerations for reporting results in the "long-term". 711 7.1. Overview of Metric Statistics 713 This section gives an overview of reporting recommendations for the 714 loss, delay, and delay variation metrics based on the discussion and 715 conclusions of the preceding sections. 717 The minimal report on measurements MUST include both Loss and Delay 718 Metrics. 720 For Packet Loss, the loss ratio defined in [RFC2680] is a sufficient 721 starting point, especially the guidance for setting the loss 722 threshold waiting time. We have calculated a waiting time above that 723 should be sufficient to differentiate between packets that are truly 724 lost or have long finite delays under general measurement 725 circumstances, 51 seconds. Knowledge of specific conditions can help 726 to reduce this threshold, but 51 seconds is considered to be 727 manageable in practice. 729 We note that a loss ratio calculated according to [Y.1540] would 730 exclude errored packets form the numerator. In practice, the 731 difference between these two loss metrics is small if any, depending 732 on whether the last link prior to the destination contributes errored 733 packets. 735 For Packet Delay, we recommend providing both the mean delay and the 736 median delay with lost packets designated undefined (as permitted by 737 [RFC2679]). Both statistics are based on a conditional distribution, 738 and the condition is packet arrival prior to a waiting time dT, where 739 dT has been set to take maximum packet lifetimes into account, as 740 discussed above. Using a long dT helps to ensure that delay 741 distributions are not truncated. 743 For Packet Delay Variation (PDV), the minimum delay of the 744 conditional distribution should be used as the reference delay for 745 computing PDV according to [Y.1540] or [RFC3393]. A useful value to 746 report is a pseudo range of delay variation based on calculating the 747 difference between a high percentile of delay and the minimum delay. 748 For example, the 99.9%-ile minus the minimum will give a value that 749 can be compared with objectives in [Y.1541]. 751 7.2. Long-Term Reporting Considerations 753 [I-D.ietf-ippm-reporting] describes methods to conduct measurements 754 and report the results on a near-immediate time scale (10 seconds, 755 which we consider to be "short-term"). 757 Measurement intervals and reporting intervals need not be the same 758 length. Sometimes, the user is only concerned with the performance 759 levels achieved over a relatively long interval of time (e.g, days, 760 weeks, or months, as opposed to 10 seconds). However, there can be 761 risks involved with running a measurement continuously over a long 762 period without recording intermediate results: 764 o Temporary power failure may cause loss of all the results to date. 766 o Measurement system timing synchronization signals may experience a 767 temporary outage, causing sub-sets of measurements to be in error 768 or invalid. 770 o Maintenance may be necessary on the measurement system, or its 771 connectivity to the network under test. 773 For these and other reasons, such as 775 o the constraint to collect measurements on intervals similar to 776 user session length, or 778 o the dual-use of measurements in monitoring activities where 779 results are needed on a period of a few minutes, 781 there is value in conducting measurements on intervals that are much 782 shorter than the reporting interval. 784 There are several approaches for aggregating a series of measurement 785 results over time in order to make a statement about the longer 786 reporting interval. One approach requires the storage of all metric 787 singletons collected throughout the reporting interval, even though 788 the measurement interval stops and starts many times. 790 Another approach is described in [I-D.ietf-ippm-framework-compagg] as 791 "temporal aggregation". This approach would estimate the results for 792 the reporting interval based on many individual measurement interval 793 statistics (results) alone. The result would ideally appear in the 794 same form as though a continuous measurement was conducted. A memo 795 to address the details of temporal aggregation is yet to be prepared. 797 Yet another approach requires a numerical objective for the metric, 798 and the results of each measurement interval are compared with the 799 objective. Every measurement interval where the results meet the 800 objective contribute to the fraction of time with performance as 801 specified. When the reporting interval contains many measurement 802 intervals it is possible to present the results as "metric A was less 803 than or equal to objective X during Y% of time. 805 NOTE that numerical thresholds are not set in IETF performance work 806 and are explicitly excluded from the IPPM charter. 808 8. IANA Considerations 810 This document makes no request of IANA. 812 Note to RFC Editor: this section may be removed on publication as an 813 RFC. 815 9. Security Considerations 817 The security considerations that apply to any active measurement of 818 live networks are relevant here as well. See [RFC4656]. 820 10. Acknowledgements 822 The authors would like to thank Phil Chimento for his suggestion to 823 employ conditional distributions for Delay, and Steve Konish Jr. for 824 his careful review and suggestions. 826 11. References 828 11.1. Normative References 830 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 831 Requirement Levels", BCP 14, RFC 2119, March 1997. 833 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 834 "Framework for IP Performance Metrics", RFC 2330, 835 May 1998. 837 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 838 Delay Metric for IPPM", RFC 2679, September 1999. 840 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 841 Packet Loss Metric for IPPM", RFC 2680, September 1999. 843 [RFC3148] Mathis, M. and M. Allman, "A Framework for Defining 844 Empirical Bulk Transfer Capacity Metrics", RFC 3148, 845 July 2001. 847 [RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation 848 Metric for IP Performance Metrics (IPPM)", RFC 3393, 849 November 2002. 851 [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. 852 Zekauskas, "A One-way Active Measurement Protocol 853 (OWAMP)", RFC 4656, September 2006. 855 [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, 856 S., and J. Perser, "Packet Reordering Metrics", RFC 4737, 857 November 2006. 859 [RFC5136] Chimento, P. and J. Ishac, "Defining Network Capacity", 860 RFC 5136, February 2008. 862 11.2. Informative References 864 [Casner] "A Fine-Grained View of High Performance Networking, NANOG 865 22 Conf.; http://www.nanog.org/mtg-0105/agenda.html", May 866 20-22 2001. 868 [Cia03] "Standardized Active Measurements on a Tier 1 IP Backbone, 869 IEEE Communications Mag., pp 90-97.", June 2003. 871 [I-D.ietf-ippm-framework-compagg] 872 Morton, A., "Framework for Metric Composition", 873 draft-ietf-ippm-framework-compagg-09 (work in progress), 874 December 2009. 876 [I-D.ietf-ippm-reporting] 877 Shalunov, S. and M. Swany, "Reporting IP Performance 878 Metrics to Users", draft-ietf-ippm-reporting-04 (work in 879 progress), July 2009. 881 [Y.1540] ITU-T Recommendation Y.1540, "Internet protocol data 882 communication service - IP packet transfer and 883 availability performance parameters", December 2002. 885 [Y.1541] ITU-T Recommendation Y.1540, "Network Performance 886 Objectives for IP-Based Services", February 2006. 888 Authors' Addresses 890 Al Morton 891 AT&T Labs 892 200 Laurel Avenue South 893 Middletown, NJ 07748 894 USA 896 Phone: +1 732 420 1571 897 Fax: +1 732 368 1192 898 Email: acmorton@att.com 899 URI: http://home.comcast.net/~acmacm/ 901 Gomathi Ramachandran 902 AT&T Labs 903 200 Laurel Avenue South 904 Middletown, New Jersey 07748 905 USA 907 Phone: +1 732 420 2353 908 Fax: 909 Email: gomathi@att.com 910 URI: 912 Ganga Maguluri 913 AT&T Labs 914 200 Laurel Avenue 915 Middletown, New Jersey 07748 916 USA 918 Phone: 732-420-2486 919 Fax: 920 Email: gmaguluri@att.com 921 URI: