idnits 2.17.1 draft-ietf-ippm-testplan-rfc2679-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 7 instances of lines with non-RFC2606-compliant FQDNs in the document. == There are 11 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 801 has weird spacing: '...ean adj raw...' == Line 982 has weird spacing: '...Payload s1 ...' == Line 988 has weird spacing: '...Payload p1 ...' == Line 1117 has weird spacing: '...centile no ...' == The document seems to contain a disclaimer for pre-RFC5378 work, but was first submitted on or after 10 November 2008. The disclaimer is usually necessary only for documents that revise or obsolete older RFCs, and that take significant amounts of text from those RFCs. If you can contact all authors of the source material and they are willing to grant the BCP78 rights to the IETF Trust, you can and should remove the disclaimer. Otherwise, the disclaimer is needed and you can ignore this comment. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (September 6, 2012) is 4222 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'AS 7018' is mentioned on line 339, but not defined == Missing Reference: 'AS 3320' is mentioned on line 343, but not defined ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) Summary: 2 errors (**), 0 flaws (~~), 10 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group L. Ciavattone 3 Internet-Draft AT&T Labs 4 Intended status: Informational R. Geib 5 Expires: March 10, 2013 Deutsche Telekom 6 A. Morton 7 AT&T Labs 8 M. Wieser 9 Technical University Darmstadt 10 September 6, 2012 12 Test Plan and Results Supporting Advancement of RFC 2679 on the 13 Standards Track 14 draft-ietf-ippm-testplan-rfc2679-03 16 Abstract 18 This memo provides the supporting test plan and results to advance 19 RFC 2679 on One-way Delay Metrics along the standards track, 20 following the process in RFC 6576. Observing that the metric 21 definitions themselves should be the primary focus rather than the 22 implementations of metrics, this memo describes the test procedures 23 to evaluate specific metric requirement clauses to determine if the 24 requirement has been interpreted and implemented as intended. Two 25 completely independent implementations have been tested against the 26 key specifications of RFC 2679. This memo also provides direct input 27 for development of RFC 2679bis. 29 Requirements Language 31 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 32 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 33 document are to be interpreted as described in RFC 2119 [RFC2119]. 35 Status of this Memo 37 This Internet-Draft is submitted in full conformance with the 38 provisions of BCP 78 and BCP 79. 40 Internet-Drafts are working documents of the Internet Engineering 41 Task Force (IETF). Note that other groups may also distribute 42 working documents as Internet-Drafts. The list of current Internet- 43 Drafts is at http://datatracker.ietf.org/drafts/current/. 45 Internet-Drafts are draft documents valid for a maximum of six months 46 and may be updated, replaced, or obsoleted by other documents at any 47 time. It is inappropriate to use Internet-Drafts as reference 48 material or to cite them other than as "work in progress." 49 This Internet-Draft will expire on March 10, 2013. 51 Copyright Notice 53 Copyright (c) 2012 IETF Trust and the persons identified as the 54 document authors. All rights reserved. 56 This document is subject to BCP 78 and the IETF Trust's Legal 57 Provisions Relating to IETF Documents 58 (http://trustee.ietf.org/license-info) in effect on the date of 59 publication of this document. Please review these documents 60 carefully, as they describe your rights and restrictions with respect 61 to this document. Code Components extracted from this document must 62 include Simplified BSD License text as described in Section 4.e of 63 the Trust Legal Provisions and are provided without warranty as 64 described in the Simplified BSD License. 66 This document may contain material from IETF Documents or IETF 67 Contributions published or made publicly available before November 68 10, 2008. The person(s) controlling the copyright in some of this 69 material may not have granted the IETF Trust the right to allow 70 modifications of such material outside the IETF Standards Process. 71 Without obtaining an adequate license from the person(s) controlling 72 the copyright in such materials, this document may not be modified 73 outside the IETF Standards Process, and derivative works of it may 74 not be created outside the IETF Standards Process, except to format 75 it for publication as an RFC or to translate it into languages other 76 than English. 78 Table of Contents 80 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 81 2. A Definition-centric metric advancement process . . . . . . . 5 82 3. Test configuration . . . . . . . . . . . . . . . . . . . . . . 5 83 4. Error Calibration, RFC 2679 . . . . . . . . . . . . . . . . . 9 84 4.1. NetProbe Error and Type-P . . . . . . . . . . . . . . . . 10 85 4.2. Perfas+ Error and Type-P . . . . . . . . . . . . . . . . . 12 86 5. Pre-determined Limits on Equivalence . . . . . . . . . . . . . 13 87 6. Tests to evaluate RFC 2679 Specifications . . . . . . . . . . 13 88 6.1. One-way Delay, ADK Sample Comparison - Same & Cross 89 Implementation . . . . . . . . . . . . . . . . . . . . . . 14 90 6.1.1. NetProbe Same-implementation results . . . . . . . . . 15 91 6.1.2. Perfas+ Same-implementation results . . . . . . . . . 16 92 6.1.3. One-way Delay, Cross-Implementation ADK Comparison . . 17 93 6.1.4. Conclusions on the ADK Results for One-way Delay . . . 17 94 6.1.5. Additional Investigations . . . . . . . . . . . . . . 18 95 6.2. One-way Delay, Loss threshold, RFC 2679 . . . . . . . . . 21 96 6.2.1. NetProbe results for Loss Threshold . . . . . . . . . 22 97 6.2.2. Perfas+ Results for Loss Threshold . . . . . . . . . . 22 98 6.2.3. Conclusions for Loss Threshold . . . . . . . . . . . . 22 99 6.3. One-way Delay, First-bit to Last bit, RFC 2679 . . . . . . 22 100 6.3.1. NetProbe and Perfas+ Results for Serialization . . . . 23 101 6.3.2. Conclusions for Serialization . . . . . . . . . . . . 24 102 6.4. One-way Delay, Difference Sample Metric (Lab) . . . . . . 25 103 6.4.1. NetProbe results for Differential Delay . . . . . . . 25 104 6.4.2. Perfas+ results for Differential Delay . . . . . . . . 26 105 6.4.3. Conclusions for Differential Delay . . . . . . . . . . 26 106 6.5. Implementation of Statistics for One-way Delay . . . . . . 26 107 7. Conclusions and RFC 2679 Errata . . . . . . . . . . . . . . . 27 108 8. Security Considerations . . . . . . . . . . . . . . . . . . . 27 109 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 28 110 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 28 111 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 28 112 11.1. Normative References . . . . . . . . . . . . . . . . . . . 28 113 11.2. Informative References . . . . . . . . . . . . . . . . . . 29 114 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 30 116 1. Introduction 118 The IETF IP Performance Metrics (IPPM) working group has considered 119 how to advance their metrics along the standards track since 2001, 120 with the initial publication of Bradner/Paxson/Mankin's memo 121 [I-D.bradner-metricstest]. The original proposal was to compare the 122 performance of metric implementations. This was similar to the usual 123 procedures for advancing protocols, which did not directly apply. It 124 was found to be difficult to achieve consensus on exactly how to 125 compare implementations, since there were many legitimate sources of 126 variation that would emerge in the results despite the best attempts 127 to keep the network paths equal, and because considerable variation 128 was allowed in the parameters (and therefore implementation) of each 129 metric. Flexibility in metric definitions, essential for 130 customization and broad appeal, made the comparison task quite 131 difficult. 133 A renewed work effort investigated ways in which the measurement 134 variability could be reduced and thereby simplify the problem of 135 comparison for equivalence. 137 The consensus process documented in [RFC6576] is that metric 138 definitions should be the primary focus of evaluation rather than the 139 implementations of metrics. Equivalent test results are deemed to be 140 evidence that the metric specifications are clear and unambiguous. 141 This is now the metric specification equivalent of protocol 142 interoperability. The [RFC6576] advancement process either produces 143 confidence that the metric definitions and supporting material are 144 clearly worded and unambiguous, OR, identifies ways in which the 145 metric definitions should be revised to achieve clarity. 147 The metric RFC advancement process requires documentation of the 148 testing and results. [RFC6576] retains the testing requirement of 149 the original standards track advancement process described in 150 [RFC2026] and [RFC5657], because widespread deployment is 151 insufficient to determine whether RFCs that define performance 152 metrics result in consistent implementations. 154 The process also permits identification of options that were not 155 implemented, so that they can be removed from the advancing 156 specification (this is a similar aspect to protocol advancement along 157 the standards track). All errata must also be considered. 159 This memo's purpose is to implement the advancement process of 160 [RFC6576] for [RFC2679]. It supplies the documentation that 161 accompanies the protocol action request submitted to the Area 162 Director, including description of the test set-up, results for each 163 implementation, evaluation of each metric specification, and 164 conclusions. 166 In particular, this memo documents the consensus on the extent of 167 tolerable errors when assessing equivalence in the results. The IPPM 168 working group agreed that the test plan and procedures should include 169 the threshold for determining equivalence, and that this aspect 170 should be decided in advance of cross-implementation comparisons. 171 This memo includes procedures for same-implementation comparisons 172 that may influence the equivalence threshold. 174 Although the conclusion reached through testing is that [RFC2679] 175 should be advanced on the Standards Track with modifications, the 176 revised text of RFC 2679bis is not yet ready for review. Therefore, 177 this memo documents the information to support [RFC2679] advancement, 178 and the approval of RFC2769bis is left for future action. 180 2. A Definition-centric metric advancement process 182 The process described in Section 3.5 of [RFC6576] takes as a first 183 principle that the metric definitions, embodied in the text of the 184 RFCs, are the objects that require evaluation and possible revision 185 in order to advance to the next step on the standards track. This 186 memo follows that process. 188 3. Test configuration 190 One metric implementation used was NetProbe version 5.8.5, (an 191 earlier version is used in the AT&T's IP network performance 192 measurement system and deployed world-wide [WIPM]). NetProbe uses 193 UDP packets of variable size, and can produce test streams with 194 Periodic [RFC3432] or Poisson [RFC2330] sample distributions. 196 The other metric implementation used was Perfas+ version 3.1, 197 developed by Deutsche Telekom [Perfas]. Perfas+ uses UDP unicast 198 packets of variable size (but supports also TCP and multicast). Test 199 streams with Periodic, Poisson or uniform sample distributions may be 200 used. 202 Figure 2 shows a view of the test path as each Implementation's test 203 flows pass through the Internet and the L2TPv3 tunnel IDs (1 and 2), 204 based on Figures 2 and 3 of [RFC6576]. 206 +----+ +----+ +----+ +----+ 207 |Imp1| |Imp1| ,---. |Imp2| |Imp2| 208 +----+ +----+ / \ +-------+ +----+ +----+ 209 | V100 | V200 / \ | Tunnel| | V300 | V400 210 | | ( ) | Head | | | 211 +--------+ +------+ | |__| Router| +----------+ 212 |Ethernet| |Tunnel| |Internet | +---B---+ |Ethernet | 213 |Switch |--|Head |-| | | |Switch | 214 +-+--+---+ |Router| | | +---+---+--+--+--+----+ 215 |__| +--A---+ ( ) |Network| |__| 216 \ / |Emulat.| 217 U-turn \ / |"netem"| U-turn 218 V300 to V400 `-+-' +-------+ V100 to V200 220 Implementations ,---. +--------+ 221 +~~~~~~~~~~~/ \~~~~~~| Remote | 222 +------->-----F2->-| / \ |->---. | 223 | +---------+ | Tunnel ( ) | | | 224 | | transmit|-F1->-| ID 1 ( ) |->. | | 225 | | Imp 1 | +~~~~~~~~~| |~~~~| | | | 226 | | receive |-<--+ ( ) | F1 F2 | 227 | +---------+ | |Internet | | | | | 228 *-------<-----+ F1 | | | | | | 229 +---------+ | | +~~~~~~~~~| |~~~~| | | | 230 | transmit|-* *-| | | |<-* | | 231 | Imp 2 | | Tunnel ( ) | | | 232 | receive |-<-F2-| ID 2 \ / |<----* | 233 +---------+ +~~~~~~~~~~~\ /~~~~~~| Switch | 234 `-+-' +--------+ 236 Illustrations of a test setup with a bi-directional tunnel. The 237 upper diagram emphasizes the VLAN connectivity and geographical 238 location. The lower diagram shows example flows traveling between 239 two measurement implementations (for simplicity, only two flows are 240 shown). 242 Figure 1 244 The testing employs the Layer 2 Tunnel Protocol, version 3 (L2TPv3) 245 [RFC3931] tunnel between test sites on the Internet. The tunnel IP 246 and L2TPv3 headers are intended to conceal the test equipment 247 addresses and ports from hash functions that would tend to spread 248 different test streams across parallel network resources, with likely 249 variation in performance as a result. 251 At each end of the tunnel, one pair of VLANs encapsulated in the 252 tunnel are looped-back so that test traffic is returned to each test 253 site. Thus, test streams traverse the L2TP tunnel twice, but appear 254 to be one-way tests from the test equipment point of view. 256 The network emulator is a host running Fedora 14 Linux [Fedora14] 257 with IP forwarding enabled and the "netem" Network emulator [netem] 258 loaded and operating as part of the Fedora Kernel 2.6.35.11. 259 Connectivity across the netem/Fedora host was accomplished by 260 bridging Ethernet VLAN interfaces together with "brctl" commands 261 (e.g., eth1.100 <-> eth2.100). The netem emulator was activated on 262 one interface (eth1) and only operates on test streams traveling in 263 one direction. In some tests, independent netem instances operated 264 separately on each VLAN. 266 The links between the netem emulator host and router and switch were 267 found to be 100baseTx-HD (100Mbps half duplex) when the testing was 268 complete. Use of Half Duplex was not intended, but probably added a 269 small amount of delay variation that could have been avoided in full 270 duplex mode. 272 Each individual test was run with common packet rates (1 pps, 10pps) 273 Poisson/Periodic distributions, and IP packet sizes of 64, 340, and 274 500 Bytes. These sizes cover a reasonable range while avoiding 275 fragmentation and the complexities it causes, and thus complying with 276 the notion of "standard formed packets" described in Section 15 of 277 [RFC2330]. 279 For these tests, a stream of at least 300 packets were sent from 280 Source to Destination in each implementation. Periodic streams (as 281 per [RFC3432]) with 1 second spacing were used, except as noted. 283 With the L2TPv3 tunnel in use, the metric name for the testing 284 configured here (with respect to the IP header exposed to Internet 285 processing) is: 287 Type-IP-protocol-115-One-way-Delay--Stream 289 With (Section 4.2. [RFC2679]) Metric Parameters: 291 + Src, the IP address of a host (12.3.167.16 or 193.159.144.8) 293 + Dst, the IP address of a host (193.159.144.8 or 12.3.167.16) 295 + T0, a time 297 + Tf, a time 299 + lambda, a rate in reciprocal seconds 300 + Thresh, a maximum waiting time in seconds (see Section 3.8.2 of 301 [RFC2679]) And (Section 4.3. [RFC2679]) 303 Metric Units: A sequence of pairs; the elements of each pair are: 305 + T, a time, and 307 + dT, either a real number or an undefined number of seconds. 309 The values of T in the sequence are monotonic increasing. Note that 310 T would be a valid parameter to Type-P-One-way-Delay, and that dT 311 would be a valid value of Type-P-One-way-Delay. 313 Also, Section 3.8.4 of [RFC2679] recommends that the path SHOULD be 314 reported. In this test set-up, most of the path details will be 315 concealed from the implementations by the L2TPv3 tunnels, thus a more 316 informative path trace route can be conducted by the routers at each 317 location. 319 When NetProbe is used in production, a traceroute is conducted in 320 parallel with, and at the outset of measurements. 322 Perfas+ does not support traceroute. 324 IPLGW#traceroute 193.159.144.8 326 Type escape sequence to abort. 327 Tracing the route to 193.159.144.8 329 1 12.126.218.245 [AS 7018] 0 msec 0 msec 4 msec 330 2 cr84.n54ny.ip.att.net (12.123.2.158) [AS 7018] 4 msec 4 msec 331 cr83.n54ny.ip.att.net (12.123.2.26) [AS 7018] 4 msec 332 3 cr1.n54ny.ip.att.net (12.122.105.49) [AS 7018] 4 msec 333 cr2.n54ny.ip.att.net (12.122.115.93) [AS 7018] 0 msec 334 cr1.n54ny.ip.att.net (12.122.105.49) [AS 7018] 0 msec 335 4 n54ny02jt.ip.att.net (12.122.80.225) [AS 7018] 4 msec 0 msec 336 n54ny02jt.ip.att.net (12.122.80.237) [AS 7018] 4 msec 337 5 192.205.34.182 [AS 7018] 0 msec 338 192.205.34.150 [AS 7018] 0 msec 339 192.205.34.182 [AS 7018] 4 msec 340 6 da-rg12-i.DA.DE.NET.DTAG.DE (62.154.1.30) [AS 3320] 88 msec 88 msec 341 88 msec 342 7 217.89.29.62 [AS 3320] 88 msec 88 msec 88 msec 343 8 217.89.29.55 [AS 3320] 88 msec 88 msec 88 msec 344 9 * * * 346 It was only possible to conduct the traceroute for the measured path 347 on one of the tunnel-head routers (the normal trace facilities of the 348 measurement systems are confounded by the L2TPv3 tunnel 349 encapsulation). 351 4. Error Calibration, RFC 2679 353 An implementation is required to report on its error calibration in 354 Section 3.8 of [RFC2679] (also required in Section 4.8 for sample 355 metrics). Sections 3.6, 3.7, and 3.8 of [RFC2679] give the detailed 356 formulation of the errors and uncertainties for calibration. In 357 summary, Section 3.7.1 of [RFC2679] describes the total time-varying 358 uncertainty as: 360 Esynch(t)+ Rsource + Rdest 362 where: 364 Esynch(t) denotes an upper bound on the magnitude of clock 365 synchronization uncertainty. 367 Rsource and Rdest denote the resolution of the source clock and the 368 destination clock, respectively. 370 Further, Section 3.7.2 of [RFC2679] describes the total wire-time 371 uncertainty as 373 Hsource + Hdest 375 referring to the upper bounds on host-time to wire-time for source 376 and destination, respectively. 378 Section 3.7.3 of [RFC2679] describes a test with small packets over 379 an isolated minimal network where the results can be used to estimate 380 systematic and random components of the sum of the above errors or 381 uncertainties. In a test with hundreds of singletons, the median is 382 the systematic error and when the median is subtracted from all 383 singletons, the remaining variability is the random error. 385 The test context, or Type-P of the test packets, must also be 386 reported, as required in Section 3.8 of [RFC2679] and all metrics 387 defined there. Type-P is defined in Section 13 of [RFC2330] (as are 388 many terms used below). 390 4.1. NetProbe Error and Type-P 392 Type-P for this test was IP-UDP with Best Effort DSCP. These headers 393 were encapsulated according to the L2TPv3 specifications [RFC3931], 394 and thus may not influence the treatment received as the packets 395 traversed the Internet. 397 In general, NetProbe error is dependent on the specific version and 398 installation details. 400 NetProbe operates using host time above the UDP layer, which is 401 different from the wire-time preferred in [RFC2330], but can be 402 identified as a source of error according to Section 3.7.2 of 403 [RFC2679]. 405 Accuracy of NetProbe measurements is usually limited by NTP 406 synchronization performance (which is typically taken as ~+/-1ms 407 error or greater), although the installation used in this testing 408 often exhibits errors much less than typical for NTP. The primary 409 stratum 1 NTP server is closely located on a sparsely utilized 410 network management LAN, thus it avoids many concerns raised in 411 Section 10 of[RFC2330] (in fact, smooth adjustment, long-term drift 412 analysis and compensation, and infrequent adjustment all lead to 413 stability during measurement intervals, the main concern). 415 The resolution of the reported results is 1us (us = microsecond) in 416 the version of NetProbe tested here, which contributes to at least 417 +/-1us error. 419 NetProbe implements a time-keeping sanity check on sending and 420 receiving time-stamping processes. When the significant process 421 interruption takes place, individual test packets are flagged as 422 possibly containing unusual time errors, and are excluded from the 423 sample used for all "time" metrics. 425 We performed a NetProbe calibration of the type described in Section 426 3.7.3 of [RFC2679], using 64 Byte packets over a cross-connect cable. 427 The results estimate systematic and random components of the sum of 428 the Hsource + Hdest errors or uncertainties. In a test with 300 429 singletons conducted over 30 seconds (periodic sample with 100ms 430 spacing), the median is the systematic error and the remaining 431 variability is the random error. One set of results is tabulated 432 below: 434 (Results from the "R" software environment for statistical computing 435 and graphics - http://www.r-project.org/ ) 436 > summary(XD4CAL) 437 CAL1 CAL2 CAL3 438 Min. : 89.0 Min. : 68.00 Min. : 54.00 439 1st Qu.: 99.0 1st Qu.: 77.00 1st Qu.: 63.00 440 Median :110.0 Median : 79.00 Median : 65.00 441 Mean :116.8 Mean : 83.74 Mean : 69.65 442 3rd Qu.:127.0 3rd Qu.: 88.00 3rd Qu.: 74.00 443 Max. :205.0 Max. :177.00 Max. :163.00 444 > 445 NetProbe Calibration with Cross-Connect Cable, one-way delay values 446 in microseconds (us) 448 The median or systematic error can be as high as 110 us, and the 449 range of the random error is also on the order of 116 us for all 450 streams. 452 Also, anticipating the Anderson-Darling K-sample (ADK) [ADK] 453 comparisons to follow, we corrected the CAL2 values for the 454 difference between means between CAL2 and CAL3 (as specified in 455 [RFC6576]), and found strong support for the (Null Hypothesis that) 456 the samples are from the same distribution (resolution of 1 us and 457 alpha equal 0.05 and 0.01) 458 > XD4CVCAL2 <- XD4CAL$CAL2 - (mean(XD4CAL$CAL2)-mean(XD4CAL$CAL3)) 459 > boxplot(XD4CVCAL2,XD4CAL$CAL3) 460 > XD4CV2_ADK <- adk.test(XD4CVCAL2, XD4CAL$CAL3) 461 > XD4CV2_ADK 462 Anderson-Darling k-sample test. 464 Number of samples: 2 465 Sample sizes: 300 300 466 Total number of values: 600 467 Number of unique values: 97 469 Mean of Anderson Darling Criterion: 1 470 Standard deviation of Anderson Darling Criterion: 0.75896 472 T = (Anderson Darling Criterion - mean)/sigma 474 Null Hypothesis: All samples come from a common population. 476 t.obs P-value extrapolation 477 not adj. for ties 0.71734 0.17042 0 478 adj. for ties -0.39553 0.44589 1 479 > 480 using [Rtool] and [Radk]. 482 4.2. Perfas+ Error and Type-P 484 Perfas+ is configured to use GPS synchronisation and uses NTP 485 synchronization as a fall-back or default. GPS synchronisation 486 worked throughout this test with the exception of the calibration 487 stated here (one implementation was NTP synchronised only). The time 488 stamp accuracy typically is 0.1 ms. 490 The resolution of the results reported by Perfas+ is 1us (us = 491 microsecond) in the version tested here, which contributes to at 492 least +/-1us error. 494 Port 5001 5002 5003 495 Min. -227 -226 294 496 Median -169 -167 323 497 Mean -159 -157 335 498 Max. 6 -52 376 499 s 102 102 93 500 Perfas+ Calibration with Cross-Connect Cable, one-way delay values in 501 microseconds (us) 503 The median or systematic error can be as high as 323 us, and the 504 range of the random error is also less than 232 us for all streams. 506 5. Pre-determined Limits on Equivalence 508 This section provides the numerical limits on comparisons between 509 implementations, in order to declare that the results are equivalent 510 and therefore, the tested specification is clear. These limits have 511 their basis in Section 3.1 of [RFC6576] and the Appendix of 512 [RFC2330], with additional limits representing IPPM consensus prior 513 to publication of results. 515 A key point is that the allowable errors, corrections, and confidence 516 levels only need to be sufficient to detect mis-interpretation of the 517 tested specification resulting in diverging implementations. 519 Also, the allowable error must be sufficient to compensate for 520 measured path differences. It was simply not possible to measure 521 fully identical paths in the VLAN-loopback test configuration used, 522 and this practical compromise must be taken into account. 524 For Anderson-Darling K-sample (ADK) comparisons, the required 525 confidence factor for the cross-implementation comparisons SHALL be 526 the smallest of: 528 o 0.95 confidence factor at 1ms resolution, or 530 o the smallest confidence factor (in combination with resolution) of 531 the two same-implementation comparisons for the same test 532 conditions. 534 A constant time accuracy error of as much as +/-0.5ms MAY be removed 535 from one implementation's distributions (all singletons) before the 536 ADK comparison is conducted. 538 A constant propagation delay error (due to use of different sub-nets 539 between the switch and measurement devices at each location) of as 540 much as +2ms MAY be removed from one implementation's distributions 541 (all singletons) before the ADK comparison is conducted. 543 For comparisons involving the mean of a sample or other central 544 statistics, the limits on both the time accuracy error and the 545 propagation delay error constants given above also apply. 547 6. Tests to evaluate RFC 2679 Specifications 549 This section describes some results from real-world (cross-Internet) 550 tests with measurement devices implementing IPPM metrics and a 551 network emulator to create relevant conditions, to determine whether 552 the metric definitions were interpreted consistently by implementors. 554 The procedures are slightly modified from the original procedures 555 contained in Appendix A.1 of [RFC6576]. The modifications include 556 the use of the mean statistic for comparisons. 558 Note that there are only five instances of the requirement term 559 "MUST" in [RFC2679] outside of the boilerplate and [RFC2119] 560 reference. 562 6.1. One-way Delay, ADK Sample Comparison - Same & Cross Implementation 564 This test determines if implementations produce results that appear 565 to come from a common delay distribution, as an overall evaluation of 566 Section 4 of [RFC2679], "A Definition for Samples of One-way Delay". 567 Same-implementation comparison results help to set the threshold of 568 equivalence that will be applied to cross-implementation comparisons. 570 This test is intended to evaluate measurements in sections 3 and 4 of 571 [RFC2679]. 573 By testing the extent to which the distributions of one-way delay 574 singletons from two implementations of [RFC2679] appear to be from 575 the same distribution, we economize on comparisons, because comparing 576 a set of individual summary statistics (as defined in Section 5 of 577 [RFC2679]) would require another set of individual evaluations of 578 equivalence. Instead, we can simply check which statistics were 579 implemented, and report on those facts. 581 1. Configure an L2TPv3 path between test sites, and each pair of 582 measurement devices to operate tests in their designated pair of 583 VLANs. 585 2. Measure a sample of one-way delay singletons with 2 or more 586 implementations, using identical options and network emulator 587 settings (if used). 589 3. Measure a sample of one-way delay singletons with *four* 590 instances of the *same* implementations, using identical options, 591 noting that connectivity differences SHOULD be the same as for 592 the cross implementation testing. 594 4. Apply the ADK comparison procedures (see Appendix C of [RFC6576]) 595 and determine the resolution and confidence factor for 596 distribution equivalence of each same-implementation comparison 597 and each cross-implementation comparison. 599 5. Take the coarsest resolution and confidence factor for 600 distribution equivalence from the same-implementation pairs, or 601 the limit defined in Section 5 above, as a limit on the 602 equivalence threshold for these experimental conditions. 604 6. Apply constant correction factors to all singletons of the sample 605 distributions, as described and limited in Section 5 above. 607 7. Compare the cross-implementation ADK performance with the 608 equivalence threshold determined in step 5 to determine if 609 equivalence can be declared. 611 The common parameters used for tests in this section are: 613 o IP header + payload = 64 octets 615 o Periodic sampling at 1 packet per second 617 o Test duration = 300 seconds (March 29, 2011) 619 The netem emulator was set for 100ms average delay, with uniform 620 delay variation of +/-50ms. In this experiment, the netem emulator 621 was configured to operate independently on each VLAN and thus the 622 emulator itself is a potential source of error when comparing streams 623 that traverse the test path in different directions. 625 In the result analysis of this section: 627 o All comparisons used 1 microsecond resolution. 629 o No Correction Factors were applied. 631 o The 0.95 confidence factor (1.960 for paired stream comparison) 632 was used. 634 6.1.1. NetProbe Same-implementation results 636 A single same-implementation comparison fails the ADK criterion (s1 637 <-> sB). We note that these streams traversed the test path in 638 opposite directions, making the live network factors a possibility to 639 explain the difference. 641 All other pair comparisons pass the ADK criterion. 643 +------------------------------------------------------+ 644 | | | | | 645 | ti.obs (P) | s1 | s2 | sA | 646 | | | | | 647 .............|.............|.............|.............| 648 | | | | | 649 | s2 | 0.25 (0.28) | | | 650 | | | | | 651 ...........................|.............|.............| 652 | | | | | 653 | sA | 0.60 (0.19) |-0.80 (0.57) | | 654 | | | | | 655 ...........................|.............|.............| 656 | | | | | 657 | sB | 2.64 (0.03) | 0.07 (0.31) |-0.52 (0.48) | 658 | | | | | 659 +------------+-------------+-------------+-------------+ 661 NetProbe ADK Results for same-implementation 663 6.1.2. Perfas+ Same-implementation results 665 All pair comparisons pass the ADK criterion. 667 +------------------------------------------------------+ 668 | | | | | 669 | ti.obs (P) | p1 | p2 | p3 | 670 | | | | | 671 .............|.............|.............|.............| 672 | | | | | 673 | p2 | 0.06 (0.32) | | | 674 | | | | | 675 .........................................|.............| 676 | | | | | 677 | p3 | 1.09 (0.12) | 0.37 (0.24) | | 678 | | | | | 679 ...........................|.............|.............| 680 | | | | | 681 | p4 |-0.81 (0.57) |-0.13 (0.37) | 1.36 (0.09) | 682 | | | | | 683 +------------+-------------+-------------+-------------+ 685 Perfas+ ADK Results for same-implementation 687 6.1.3. One-way Delay, Cross-Implementation ADK Comparison 689 The cross-implementation results are compared using a combined ADK 690 analysis [Radk], where all NetProbe results are compared with all 691 Perfas+ results after testing that the combined same-implementation 692 results pass the ADK criterion. 694 When 4 (same) samples are compared, the ADK criterion for 0.95 695 confidence is 1.915, and when all 8 (cross) samples are compared it 696 is 1.85. 698 Combination of Anderson-Darling K-Sample Tests. 700 Sample sizes within each data set: 701 Data set 1 : 299 297 298 300 (NetProbe) 702 Data set 2 : 300 300 298 300 (Perfas+) 703 Total sample size per data set: 1194 1198 704 Number of unique values per data set: 1188 1192 705 ... 706 Null Hypothesis: 707 All samples within a data set come from a common distribution. 708 The common distribution may change between data sets. 710 NetProbe ti.obs P-value extrapolation 711 not adj. for ties 0.64999 0.21355 0 712 adj. for ties 0.64833 0.21392 0 713 Perfas+ 714 not adj. for ties 0.55968 0.23442 0 715 adj. for ties 0.55840 0.23473 0 717 Combined Anderson-Darling Criterion: 718 tc.obs P-value extrapolation 719 not adj. for ties 0.85537 0.17967 0 720 adj. for ties 0.85329 0.18010 0 722 The combined same-implementation samples and the combined cross- 723 implementation comparison all pass the ADK criteria at P>=0.18 and 724 support the Null Hypothesis (both data sets come from a common 725 distribution). 727 We also see that the paired ADK comparisons are rather critical. 728 Although the NetProbe s1-sB comparison failed, the combined data set 729 from 4 streams passed the ADK criterion easily. 731 6.1.4. Conclusions on the ADK Results for One-way Delay 733 Similar testing was repeated many times in the months of March and 734 April 2011. There were many experiments where a single test stream 735 from NetProbe or Perfas+ proved to be different from the others in 736 paired comparisons (even same implementation comparisons). When the 737 outlier stream was removed from the comparison, the remaining streams 738 passed combined ADK criterion. Also, the application of correction 739 factors resulted in higher comparison success. 741 We conclude that the two implementations are capable of producing 742 equivalent one-way delay distributions based on their interpretation 743 of [RFC2679]. 745 6.1.5. Additional Investigations 747 On the final day of testing, we performed a series of measurements to 748 evaluate the amount of emulated delay variation necessary to achieve 749 successful ADK comparisons. The need for Correction factors (as 750 permitted by Section 5) and the size of the measurement sample 751 (obtained as sub-sets of the complete measurement sample) were also 752 evaluated. 754 The common parameters used for tests in this section are: 756 o IP header + payload = 64 octets 758 o Periodic sampling at 1 packet per second 760 o Test duration = 300 seconds at each delay variation setting, for a 761 total of 1200 seconds (May 2, 2011 at 1720 UTC) 763 The netem emulator was set for 100ms average delay, with (emulated) 764 uniform delay variation of: 766 o +/-7.5 ms 768 o +/-5.0 ms 770 o +/-2.5 ms 772 o 0 ms 774 In this experiment, the netem emulator was configured to operate 775 independently on each VLAN and thus the emulator itself is a 776 potential source of error when comparing streams that traverse the 777 test path in different directions. 779 In the result analysis of this section: 781 o All comparisons used 1 microsecond resolution. 783 o Correction Factors *were* applied as noted (under column heading 784 "mean adj"). The difference between each sample mean and the 785 lowest mean of the NetProbe or Perfas+ stream samples was 786 subtracted from all values in the sample. ("raw" indicates no 787 correction factors were used.) All correction factors applied met 788 the limits described in Section 5. 790 o The 0.95 confidence factor (1.960 for paired stream comparison) 791 was used. 793 When 8 (cross) samples are compared, the ADK criterion for 0.95 794 confidence is 1.85. The Combined ADK test statistic ("TC observed") 795 must be less than 1.85 to accept the Null Hypothesis (all samples in 796 the data set are from a common distribution). 798 Emulated Delay Sub-Sample size 799 Variation 0ms 800 adk.combined (all) 300 values 75 values 801 Adj. for ties raw mean adj raw mean adj 802 TC observed 226.6563 67.51559 54.01359 21.56513 803 P-value 0 0 0 0 804 Mean std dev (all),us 719 635 805 Mean diff of means,us 649 0 606 0 807 Variation +/- 2.5ms 808 adk.combined (all) 300 values 75 values 809 Adj. for ties raw mean adj raw mean adj 810 TC observed 14.50436 -1.60196 3.15935 -1.72104 811 P-value 0 0.873 0.00799 0.89038 812 Mean std dev (all),us 1655 1702 813 Mean diff of means,us 471 0 513 0 815 Variation +/- 5ms 816 adk.combined (all) 300 values 75 values 817 Adj. for ties raw mean adj raw mean adj 818 TC observed 8.29921 -1.28927 0.37878 -1.81881 819 P-value 0 0.81601 0.29984 0.90305 820 Mean std dev (all),us 3023 2991 821 Mean diff of means,us 582 0 513 0 823 Variation +/- 7.5ms 824 adk.combined (all) 300 values 75 values 825 Adj. for ties raw mean adj raw mean adj 826 TC observed 2.53759 -0.72985 0.29241 -1.15840 827 P-value 0.01950 0.66942 0.32585 0.78686 828 Mean std dev (all),us 4449 4506 829 Mean diff of means,us 426 0 856 0 831 From the table above, we conclude the following: 833 1. None of the raw or mean adjusted results pass the ADK criterion 834 with 0 ms emulated delay variation. Use of the 75 value sub- 835 sample yielded the same conclusion. (We note the same results 836 when comparing same implementation samples for both NetProbe and 837 Perfas+.) 839 2. When the smallest emulated delay variation was inserted 840 (+/-2.5ms), the mean adjusted samples pass the ADK criterion and 841 the high P-value supports the result. The raw results do not 842 pass. 844 3. At higher values of emulated delay variation (+/-5.0ms and 845 +/-7.5ms), again the mean adjusted values pass ADK. We also see 846 that the 75-value sub-sample passed the ADK in both raw and mean 847 adjusted cases. This indicates that sample size may have played 848 a role in our results, as noted in the Appendix of [RFC2680] for 849 Goodness-of-Fit testing. 851 We note that 150 value sub-samples were also evaluated, with ADK 852 conclusions that followed the results for 300 values. Also, same- 853 implementation analysis was conducted with results similar to the 854 above, except that more of the "raw" or uncorrected samples passed 855 the ADK criterion. 857 6.2. One-way Delay, Loss threshold, RFC 2679 859 This test determines if implementations use the same configured 860 maximum waiting time delay from one measurement to another under 861 different delay conditions, and correctly declare packets arriving in 862 excess of the waiting time threshold as lost. 864 See Section 3.5 of [RFC2679], 3rd bullet point and also Section 3.8.2 865 of [RFC2679]. 867 1. configure an L2TPv3 path between test sites, and each pair of 868 measurement devices to operate tests in their designated pair of 869 VLANs. 871 2. configure the network emulator to add 1.0 sec one-way constant 872 delay in one direction of transmission. 874 3. measure (average) one-way delay with 2 or more implementations, 875 using identical waiting time thresholds (Thresh) for loss set at 876 3 seconds. 878 4. configure the network emulator to add 3 sec one-way constant 879 delay in one direction of transmission equivalent to 2 seconds of 880 additional one-way delay (or change the path delay while test is 881 in progress, when there are sufficient packets at the first delay 882 setting) 884 5. repeat/continue measurements 886 6. observe that the increase measured in step 5 caused all packets 887 with 2 sec additional delay to be declared lost, and that all 888 packets that arrive successfully in step 3 are assigned a valid 889 one-way delay. 891 The common parameters used for tests in this section are: 893 o IP header + payload = 64 octets 895 o Poisson sampling at lambda = 1 packet per second 897 o Test duration = 900 seconds total (March 21, 2011) 899 The netem emulator was set to add constant delays as specified in the 900 procedure above. 902 6.2.1. NetProbe results for Loss Threshold 904 In NetProbe, the Loss Threshold is implemented uniformly over all 905 packets as a post-processing routine. With the Loss Threshold set at 906 3 seconds, all packets with one-way delay >3 seconds are marked 907 "Lost" and included in the Lost Packet list with their transmission 908 time (as required in Section 3.3 of [RFC2680]). This resulted in 342 909 packets designated as lost in one of the test streams (with average 910 delay = 3.091 sec). 912 6.2.2. Perfas+ Results for Loss Threshold 914 Perfas+ uses a fixed Loss Threshold which was not adjustable during 915 this study. The Loss Threshold is approximately one minute, and 916 emulation of a delay of this size was not attempted. However, it is 917 possible to implement any delay threshold desired with a post- 918 processing routine and subsequent analysis. Using this method, 195 919 packets would be declared lost (with average delay = 3.091 sec). 921 6.2.3. Conclusions for Loss Threshold 923 Both implementations assume that any constant delay value desired can 924 be used as the Loss Threshold, since all delays are stored as a pair 925 as required in [RFC2679] . This is a simple way to 926 enforce the constant loss threshold envisioned in [RFC2679] (see 927 specific section references above). We take the position that the 928 assumption of post-processing is compliant, and that the text of the 929 RFC should be revised slightly to include this point. 931 6.3. One-way Delay, First-bit to Last bit, RFC 2679 933 This test determines if implementations register the same relative 934 change in delay from one packet size to another, indicating that the 935 first-to-last time-stamping convention has been followed. This test 936 tends to cancel the sources of error which may be present in an 937 implementation. 939 See Section 3.7.2 of [RFC2679], and Section 10.2 of [RFC2330]. 941 1. configure an L2TPv3 path between test sites, and each pair of 942 measurement devices to operate tests in their designated pair of 943 VLANs, and ideally including a low-speed link (it was not 944 possible to change the link configuration during testing, so the 945 lowest speed link present was the basis for serialization time 946 comparisons). 948 2. measure (average) one-way delay with 2 or more implementations, 949 using identical options and equal size small packets (64 octet IP 950 header and payload) 952 3. maintain the same path with additional emulated 100 ms one-way 953 delay 955 4. measure (average) one-way delay with 2 or more implementations, 956 using identical options and equal size large packets (500 octet 957 IP header and payload) 959 5. observe that the increase measured between steps 2 and 4 is 960 equivalent to the increase in ms expected due to the larger 961 serialization time for each implementation. Most of the 962 measurement errors in each system should cancel, if they are 963 stationary. 965 The common parameters used for tests in this section are: 967 o IP header + payload = 64 octets 969 o Periodic sampling at l packet per second 971 o Test duration = 300 seconds total (April 12) 973 The netem emulator was set to add constant 100ms delay. 975 6.3.1. NetProbe and Perfas+ Results for Serialization 977 When the IP header + payload size was increased from 64 octets to 500 978 octets, there was a delay increase observed. 980 Mean Delays in us 981 NetProbe 982 Payload s1 s2 sA sB 983 500 190893 191179 190892 190971 984 64 189642 189785 189747 189467 985 Diff 1251 1394 1145 1505 987 Perfas 988 Payload p1 p2 p3 p4 989 500 190908 190911 191126 190709 990 64 189706 189752 189763 190220 991 Diff 1202 1159 1363 489 993 Serialization tests, all values in microseconds 995 The typical delay increase when the larger packets were used was 1.1 996 to 1.5 ms (with one outlier). The typical measurements indicate that 997 a link with approximately 3 Mbit/s capacity is present on the path. 999 Through investigation of the facilities involved, it was determined 1000 that the lowest speed link was approximately 45 Mbit/s, and therefore 1001 the estimated difference should be about 0.077 ms. The observed 1002 differences are much higher. 1004 The unexpected large delay difference was also the outcome when 1005 testing serialization times in a lab environment, using the NIST Net 1006 Emulator and NetProbe [I-D.morton-ippm-advance-metrics]. 1008 6.3.2. Conclusions for Serialization 1010 Since it was not possible to confirm the estimated serialization time 1011 increases in field tests, we resort to examination of the 1012 implementations to determine compliance. 1014 NetProbe performs all time stamping above the IP-layer, accepting 1015 that some compromises must be made to achieve extreme portability and 1016 measurement scale. Therefore, the first-to-last bit convention is 1017 supported because the serialization time is included in the one-way 1018 delay measurement, enabling comparison with other implementations. 1020 Perfas+ is optimized for its purpose and performs all time stamping 1021 close to the interface hardware. The first-to-last bit convention is 1022 supported because the serialization time is included in the one-way 1023 delay measurement, enabling comparison with other implementations. 1025 6.4. One-way Delay, Difference Sample Metric (Lab) 1027 This test determines if implementations register the same relative 1028 increase in delay from one measurement to another under different 1029 delay conditions. This test tends to cancel the sources of error 1030 which may be present in an implementation. 1032 This test is intended to evaluate measurements in sections 3 and 4 of 1033 [RFC2679]. 1035 1. configure an L2TPv3 path between test sites, and each pair of 1036 measurement devices to operate tests in their designated pair of 1037 VLANs. 1039 2. measure (average) one-way delay with 2 or more implementations, 1040 using identical options 1042 3. configure the path with X+Y ms one-way delay 1044 4. repeat measurements 1046 5. observe that the (average) increase measured in steps 2 and 4 is 1047 ~Y ms for each implementation. Most of the measurement errors in 1048 each system should cancel, if they are stationary. 1050 In this test, X=1000ms and Y=1000ms. 1052 The common parameters used for tests in this section are: 1054 o IP header + payload = 64 octets 1056 o Poisson sampling at lambda = 1 packet per second 1058 o Test duration = 900 seconds total (March 21, 2011) 1060 The netem emulator was set to add constant delays as specified in the 1061 procedure above. 1063 6.4.1. NetProbe results for Differential Delay 1065 Average pre-increase delay, microseconds 1089868.0 1066 Average post 1s additional, microseconds 2089686.0 1067 Difference (should be ~= Y = 1s) 999818.0 1069 Average delays before/after 1 second increase 1071 The NetProbe implementation observed a 1 second increase with a 182 1072 microsecond error (assuming that the netem emulated delay difference 1073 is exact). 1075 We note that this differential delay test has been run under lab 1076 conditions and published in prior work 1077 [I-D.morton-ippm-advance-metrics]. The error was 6 microseconds. 1079 6.4.2. Perfas+ results for Differential Delay 1081 Average pre-increase delay, microseconds 1089794.0 1082 Average post 1s additional, microseconds 2089801.0 1083 Difference (should be ~= Y = 1s) 1000007.0 1085 Average delays before/after 1 second increase 1087 The Perfas+ implementation observed a 1 second increase with a 7 1088 microsecond error. 1090 6.4.3. Conclusions for Differential Delay 1092 Again, the live network conditions appear to have influenced the 1093 results, but both implementations measured the same delay increase 1094 within their calibration accuracy. 1096 6.5. Implementation of Statistics for One-way Delay 1098 The ADK tests the extent to which the sample distributions of one-way 1099 delay singletons from two implementations of [RFC2679] appear to be 1100 from the same overall distribution. By testing this way, we 1101 economize on the number of comparisons, because comparing a set of 1102 individual summary statistics (as defined in Section 5 of [RFC2679]) 1103 would require another set of individual evaluations of equivalence. 1104 Instead, we can simply check which statistics were implemented, and 1105 report on those facts, noting that Section 5 of [RFC2679] does not 1106 specify the calculations exactly, and gives only some illustrative 1107 examples. 1109 NetProbe Perfas+ 1111 5.1. Type-P-One-way-Delay-Percentile yes no 1113 5.2. Type-P-One-way-Delay-Median yes no 1115 5.3. Type-P-One-way-Delay-Minimum yes yes 1117 5.4. Type-P-One-way-Delay-Inverse-Percentile no no 1119 Implementation of Section 5 Statistics 1121 Only the Type-P-One-way-Delay-Inverse-Percentile has been ignored in 1122 both implementations, so it is a candidate for removal or deprecation 1123 in RFC2679bis (this small discrepancy does not affect candidacy for 1124 advancement). 1126 7. Conclusions and RFC 2679 Errata 1128 The conclusions throughout Section 6 support the advancement of 1129 [RFC2679] to the next step of the standards track, because its 1130 requirements are deemed to be clear and unambiguous based on 1131 evaluation of the test results for two implementations. The results 1132 indicate that these implementations produced statistically equivalent 1133 results under network conditions that were configured to be as close 1134 to identical as possible. 1136 Sections 6.2.3 and 6.5 indicate areas where minor revisions are 1137 warranted in RFC2679bis. The IETF has reached consensus on guidance 1138 for reporting metrics in [RFC6703], and this memo should be 1139 referenced in RFC2679bis to incorporate recent experience where 1140 appropriate. 1142 We note that there is currently one erratum with status "Held for 1143 document update" for [RFC2679], and it appears this minor revision 1144 and additional text should be incorporated in RFC2679bis. 1146 8. Security Considerations 1148 The security considerations that apply to any active measurement of 1149 live networks are relevant here as well. See [RFC4656] and 1150 [RFC5357]. 1152 9. IANA Considerations 1154 This memo makes no requests of IANA, and hopes that IANA will welcome 1155 our new computer overlords as willingly as the authors. 1157 10. Acknowledgements 1159 The authors thank Lars Eggert for his continued encouragement to 1160 advance the IPPM metrics during his tenure as AD Advisor. 1162 Nicole Kowalski supplied the needed CPE router for the NetProbe side 1163 of the test set-up, and graciously managed her testing in spite of 1164 issues caused by dual-use of the router. Thanks Nicole! 1166 The "NetProbe Team" also acknowledges many useful discussions with 1167 Ganga Maguluri. 1169 11. References 1171 11.1. Normative References 1173 [RFC2026] Bradner, S., "The Internet Standards Process -- Revision 1174 3", BCP 9, RFC 2026, October 1996. 1176 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1177 Requirement Levels", BCP 14, RFC 2119, March 1997. 1179 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 1180 "Framework for IP Performance Metrics", RFC 2330, 1181 May 1998. 1183 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1184 Delay Metric for IPPM", RFC 2679, September 1999. 1186 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1187 Packet Loss Metric for IPPM", RFC 2680, September 1999. 1189 [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network 1190 performance measurement with periodic streams", RFC 3432, 1191 November 2002. 1193 [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. 1194 Zekauskas, "A One-way Active Measurement Protocol 1195 (OWAMP)", RFC 4656, September 2006. 1197 [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. 1199 Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", 1200 RFC 5357, October 2008. 1202 [RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation 1203 and Implementation Reports for Advancement to Draft 1204 Standard", BCP 9, RFC 5657, September 2009. 1206 [RFC6576] Geib, R., Morton, A., Fardid, R., and A. Steinmitz, "IP 1207 Performance Metrics (IPPM) Standard Advancement Testing", 1208 BCP 176, RFC 6576, March 2012. 1210 [RFC6703] Morton, A., Ramachandran, G., and G. Maguluri, "Reporting 1211 IP Network Performance Metrics: Different Points of View", 1212 RFC 6703, August 2012. 1214 11.2. Informative References 1216 [ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling 1217 Tests of fit, for continuous and discrete cases", 1218 University of Washington, Technical Report No. 81, 1219 May 1986. 1221 [Fedora14] 1222 "Fedora Project Home Page", http://fedoraproject.org/, 1223 2012. 1225 [I-D.bradner-metricstest] 1226 Bradner, S. and V. Paxson, "Advancement of metrics 1227 specifications on the IETF Standards Track", 1228 draft-bradner-metricstest-03 (work in progress), 1229 August 2007. 1231 [I-D.morton-ippm-advance-metrics] 1232 Morton, A., "Lab Test Results for Advancing Metrics on the 1233 Standards Track", draft-morton-ippm-advance-metrics-02 1234 (work in progress), October 2010. 1236 [Perfas] Heidemann, C., "Qualitaet in IP-Netzen Messverfahren", 1237 published by ITG Fachgruppe, 2nd meeting 5.2.3 (NGN) http: 1238 //www.itg523.de/oeffentlich/01nov/ 1239 Heidemann_QOS_Messverfahren.pdf , November 2001. 1241 [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling 1242 Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. 1244 [Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and 1245 Combinations of Such Tests. R package version 1.0.", , 1246 2008. 1248 [Rtool] R Development Core Team, "R: A language and environment 1249 for statistical computing. R Foundation for Statistical 1250 Computing, Vienna, Austria. ISBN 3-900051-07-0, URL 1251 http://www.R-project.org/", , 2011. 1253 [WIPM] "AT&T Global IP Network", 1254 http://ipnetwork.bgtmo.ip.att.net/pws/index.html, 2012. 1256 [netem] ""netem" Documentation", http://www.linuxfoundation.org/ 1257 collaborate/workgroups/networking/netem, 2009. 1259 Authors' Addresses 1261 Len Ciavattone 1262 AT&T Labs 1263 200 Laurel Avenue South 1264 Middletown, NJ 07748 1265 USA 1267 Phone: +1 732 420 1239 1268 Fax: 1269 Email: lencia@att.com 1270 URI: 1272 Ruediger Geib 1273 Deutsche Telekom 1274 Heinrich Hertz Str. 3-7 1275 Darmstadt, 64295 1276 Germany 1278 Phone: +49 6151 58 12747 1279 Email: Ruediger.Geib@telekom.de 1281 Al Morton 1282 AT&T Labs 1283 200 Laurel Avenue South 1284 Middletown, NJ 07748 1285 USA 1287 Phone: +1 732 420 1571 1288 Fax: +1 732 368 1192 1289 Email: acmorton@att.com 1290 URI: http://home.comcast.net/~acmacm/ 1291 Matthias Wieser 1292 Technical University Darmstadt 1293 Darmstadt, 1294 Germany 1296 Phone: 1297 Email: matthias_michael.wieser@stud.tu-darmstadt.de