idnits 2.17.1 draft-ietf-ippm-testplan-rfc2679-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 7 instances of lines with non-RFC2606-compliant FQDNs in the document. == There are 11 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 791 has weird spacing: '...ean adj raw...' == Line 972 has weird spacing: '...Payload s1 ...' == Line 978 has weird spacing: '...Payload p1 ...' == Line 1107 has weird spacing: '...centile no ...' == The document seems to contain a disclaimer for pre-RFC5378 work, but was first submitted on or after 10 November 2008. The disclaimer is usually necessary only for documents that revise or obsolete older RFCs, and that take significant amounts of text from those RFCs. If you can contact all authors of the source material and they are willing to grant the BCP78 rights to the IETF Trust, you can and should remove the disclaimer. Otherwise, the disclaimer is needed and you can ignore this comment. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (June 24, 2012) is 4324 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'AS 7018' is mentioned on line 329, but not defined == Missing Reference: 'AS 3320' is mentioned on line 333, but not defined ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) Summary: 2 errors (**), 0 flaws (~~), 10 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group L. Ciavattone 3 Internet-Draft AT&T Labs 4 Intended status: Informational R. Geib 5 Expires: December 26, 2012 Deutsche Telekom 6 A. Morton 7 AT&T Labs 8 M. Wieser 9 Technical University Darmstadt 10 June 24, 2012 12 Test Plan and Results for Advancing RFC 2679 on the Standards Track 13 draft-ietf-ippm-testplan-rfc2679-02 15 Abstract 17 This memo proposes to advance a performance metric RFC along the 18 standards track, specifically RFC 2679 on One-way Delay Metrics. 19 Observing that the metric definitions themselves should be the 20 primary focus rather than the implementations of metrics, this memo 21 describes the test procedures to evaluate specific metric requirement 22 clauses to determine if the requirement has been interpreted and 23 implemented as intended. Two completely independent implementations 24 have been tested against the key specifications of RFC 2679. 26 Requirements Language 28 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 29 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 30 document are to be interpreted as described in RFC 2119 [RFC2119]. 32 Status of this Memo 34 This Internet-Draft is submitted in full conformance with the 35 provisions of BCP 78 and BCP 79. 37 Internet-Drafts are working documents of the Internet Engineering 38 Task Force (IETF). Note that other groups may also distribute 39 working documents as Internet-Drafts. The list of current Internet- 40 Drafts is at http://datatracker.ietf.org/drafts/current/. 42 Internet-Drafts are draft documents valid for a maximum of six months 43 and may be updated, replaced, or obsoleted by other documents at any 44 time. It is inappropriate to use Internet-Drafts as reference 45 material or to cite them other than as "work in progress." 47 This Internet-Draft will expire on December 26, 2012. 49 Copyright Notice 51 Copyright (c) 2012 IETF Trust and the persons identified as the 52 document authors. All rights reserved. 54 This document is subject to BCP 78 and the IETF Trust's Legal 55 Provisions Relating to IETF Documents 56 (http://trustee.ietf.org/license-info) in effect on the date of 57 publication of this document. Please review these documents 58 carefully, as they describe your rights and restrictions with respect 59 to this document. Code Components extracted from this document must 60 include Simplified BSD License text as described in Section 4.e of 61 the Trust Legal Provisions and are provided without warranty as 62 described in the Simplified BSD License. 64 This document may contain material from IETF Documents or IETF 65 Contributions published or made publicly available before November 66 10, 2008. The person(s) controlling the copyright in some of this 67 material may not have granted the IETF Trust the right to allow 68 modifications of such material outside the IETF Standards Process. 69 Without obtaining an adequate license from the person(s) controlling 70 the copyright in such materials, this document may not be modified 71 outside the IETF Standards Process, and derivative works of it may 72 not be created outside the IETF Standards Process, except to format 73 it for publication as an RFC or to translate it into languages other 74 than English. 76 Table of Contents 78 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 79 2. A Definition-centric metric advancement process . . . . . . . 5 80 3. Test configuration . . . . . . . . . . . . . . . . . . . . . . 5 81 4. Error Calibration, RFC 2679 . . . . . . . . . . . . . . . . . 9 82 4.1. NetProbe Error and Type-P . . . . . . . . . . . . . . . . 10 83 4.2. Perfas+ Error and Type-P . . . . . . . . . . . . . . . . . 12 84 5. Pre-determined Limits on Equivalence . . . . . . . . . . . . . 13 85 6. Tests to evaluate RFC 2679 Specifications . . . . . . . . . . 13 86 6.1. One-way Delay, ADK Sample Comparison - Same & Cross 87 Implementation . . . . . . . . . . . . . . . . . . . . . . 14 88 6.1.1. NetProbe Same-implementation results . . . . . . . . . 15 89 6.1.2. Perfas+ Same-implementation results . . . . . . . . . 16 90 6.1.3. One-way Delay, Cross-Implementation ADK Comparison . . 17 91 6.1.4. Conclusions on the ADK Results for One-way Delay . . . 17 92 6.1.5. Additional Investigations . . . . . . . . . . . . . . 18 93 6.2. One-way Delay, Loss threshold, RFC 2679 . . . . . . . . . 21 94 6.2.1. NetProbe results for Loss Threshold . . . . . . . . . 22 95 6.2.2. Perfas+ Results for Loss Threshold . . . . . . . . . . 22 96 6.2.3. Conclusions for Loss Threshold . . . . . . . . . . . . 22 97 6.3. One-way Delay, First-bit to Last bit, RFC 2679 . . . . . . 22 98 6.3.1. NetProbe and Perfas+ Results for Serialization . . . . 23 99 6.3.2. Conclusions for Serialization . . . . . . . . . . . . 24 100 6.4. One-way Delay, Difference Sample Metric (Lab) . . . . . . 25 101 6.4.1. NetProbe results for Differential Delay . . . . . . . 25 102 6.4.2. Perfas+ results for Differential Delay . . . . . . . . 26 103 6.4.3. Conclusions for Differential Delay . . . . . . . . . . 26 104 6.5. Implementation of Statistics for One-way Delay . . . . . . 26 105 7. Security Considerations . . . . . . . . . . . . . . . . . . . 27 106 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27 107 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 27 108 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 28 109 10.1. Normative References . . . . . . . . . . . . . . . . . . . 28 110 10.2. Informative References . . . . . . . . . . . . . . . . . . 28 111 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 29 113 1. Introduction 115 The IETF IP Performance Metrics (IPPM) working group has considered 116 how to advance their metrics along the standards track since 2001, 117 with the initial publication of Bradner/Paxson/Mankin's memo 118 [I-D.bradner-metricstest]. The original proposal was to compare the 119 results of implementations of the metrics, because the usual 120 procedures for advancing protocols did not appear to apply. It was 121 found to be difficult to achieve consensus on exactly how to compare 122 implementations, since there were many legitimate sources of 123 variation that would emerge in the results despite the best attempts 124 to keep the network paths equal, and because considerable variation 125 was allowed in the parameters (and therefore implementation) of each 126 metric. Flexibility in metric definitions, essential for 127 customization and broad appeal, made the comparison task quite 128 difficult. 130 A renewed work effort sought to investigate ways in which the 131 measurement variability could be reduced and thereby simplify the 132 problem of comparison for equivalence. 134 There is consensus represented in [RFC6576] that the metric 135 definitions should be the primary focus of evaluation rather than the 136 implementations of metrics, and equivalent results are deemed to be 137 evidence that the metric specifications are clear and unambiguous. 138 This is the metric specification equivalent of protocol 139 interoperability. The advancement process either produces confidence 140 that the metric definitions and supporting material are clearly 141 worded and unambiguous, OR, identifies ways in which the metric 142 definitions should be revised to achieve clarity. 144 The process should also permit identification of options that were 145 not implemented, so that they can be removed from the advancing 146 specification (this is an aspect more typical of protocol advancement 147 along the standards track). 149 This memo's purpose is to implement the current approach for 150 [RFC2679]. It was prepared to help progress discussions on the topic 151 of metric advancement, both through e-mail and at the upcoming IPPM 152 meeting at IETF. 154 In particular, consensus is sought on the extent of tolerable errors 155 when assessing equivalence in the results. In discussions, the IPPM 156 working group agreed that test plan and procedures should include the 157 threshold for determining equivalence, and this information should be 158 available in advance of cross-implementation comparisons. This memo 159 includes procedures for same-implementation comparisons to help set 160 the equivalence threshold. 162 Another aspect of the metric RFC advancement process is the 163 requirement to document the work and results. The procedures of 164 [RFC2026] are expanded in [RFC5657], including sample implementation 165 and interoperability reports. This memo expands on these RFCs and 166 the examples in Appendix A of [RFC6576] for the procedure and report 167 that accompanies the protocol action request submitted to the Area 168 Director, including description of the test set-up, results for each 169 implementation, and conclusions. 171 2. A Definition-centric metric advancement process 173 The process described in Section 3.5 of [RFC6576] takes as a first 174 principle that the metric definitions, embodied in the text of the 175 RFCs, are the objects that require evaluation and possible revision 176 in order to advance to the next step on the standards track. This 177 memo follows that process. 179 3. Test configuration 181 One metric implementation used was NetProbe version 5.8.5, (an 182 earlier version is used in the AT&T's IP network performance 183 measurement system and deployed world-wide [WIPM]). NetProbe uses 184 UDP packets of variable size, and can produce test streams with 185 Periodic [RFC3432] or Poisson [RFC2330] sample distributions. 187 The other metric implementation used was Perfas+ version 3.1, 188 developed by Deutsche Telekom. Perfas+ uses UDP unicast packets of 189 variable size (but supports also TCP and multicast). Test streams 190 with Periodic, Poisson or uniform sample distributions may be used. 192 Figure 2 shows a view of the test path as each Implementation's test 193 flows pass through the Internet and the L2TPv3 tunnel IDs (1 and 2), 194 based on Figures 2 and 3 of [RFC6576]. 196 +----+ +----+ +----+ +----+ 197 |Imp1| |Imp1| ,---. |Imp2| |Imp2| 198 +----+ +----+ / \ +-------+ +----+ +----+ 199 | V100 | V200 / \ | Tunnel| | V300 | V400 200 | | ( ) | Head | | | 201 +--------+ +------+ | |__| Router| +----------+ 202 |Ethernet| |Tunnel| |Internet | +---B---+ |Ethernet | 203 |Switch |--|Head |-| | | |Switch | 204 +-+--+---+ |Router| | | +---+---+--+--+--+----+ 205 |__| +--A---+ ( ) |Network| |__| 206 \ / |Emulat.| 207 U-turn \ / |"netem"| U-turn 208 V300 to V400 `-+-' +-------+ V100 to V200 210 Implementations ,---. +--------+ 211 +~~~~~~~~~~~/ \~~~~~~| Remote | 212 +------->-----F2->-| / \ |->---. | 213 | +---------+ | Tunnel ( ) | | | 214 | | transmit|-F1->-| ID 1 ( ) |->. | | 215 | | Imp 1 | +~~~~~~~~~| |~~~~| | | | 216 | | receive |-<--+ ( ) | F1 F2 | 217 | +---------+ | |Internet | | | | | 218 *-------<-----+ F1 | | | | | | 219 +---------+ | | +~~~~~~~~~| |~~~~| | | | 220 | transmit|-* *-| | | |<-* | | 221 | Imp 2 | | Tunnel ( ) | | | 222 | receive |-<-F2-| ID 2 \ / |<----* | 223 +---------+ +~~~~~~~~~~~\ /~~~~~~| Switch | 224 `-+-' +--------+ 226 Illustrations of a test setup with a bi-directional tunnel. The 227 upper diagram emphasizes the VLAN connectivity and geographical 228 location. The lower diagram shows example flows traveling between 229 two measurement implementations (for simplicity, only two flows are 230 shown). 232 Figure 1 234 The testing employs the Layer 2 Tunnel Protocol, version 3 (L2TPv3) 235 [RFC3931] tunnel between test sites on the Internet. The tunnel IP 236 and L2TPv3 headers are intended to conceal the test equipment 237 addresses and ports from hash functions that would tend to spread 238 different test streams across parallel network resources, with likely 239 variation in performance as a result. 241 At each end of the tunnel, one pair of VLANs encapsulated in the 242 tunnel are looped-back so that test traffic is returned to each test 243 site. Thus, test streams traverse the L2TP tunnel twice, but appear 244 to be one-way tests from the test equipment point of view. 246 The network emulator is a host running Fedora 14 Linux [Fedora14] 247 with IP forwarding enabled and the "netem" Network emulator [netem] 248 loaded and operating as part of the Fedora Kernel 2.6.35.11. 249 Connectivity across the netem/Fedora host was accomplished by 250 bridging Ethernet VLAN interfaces together with "brctl" commands 251 (e.g., eth1.100 <-> eth2.100). The netem emulator was activated on 252 one interface (eth1) and only operates on test streams traveling in 253 one direction. In some tests, independent netem instances operated 254 separately on each VLAN. 256 The links between the netem emulator host and router and switch were 257 found to be 100baseTx-HD (100Mbps half duplex) when the testing was 258 complete. Use of Half Duplex was not intended, but probably added a 259 small amount of delay variation that could have been avoided in full 260 duplex mode. 262 Each individual test was run with common packet rates (1 pps, 10pps) 263 Poisson/Periodic distributions, and IP packet sizes of 64, 340, and 264 500 Bytes. These sizes cover a reasonable range while avoiding 265 fragmentation and the complexities it causes, and thus complying with 266 the notion of "standard formed packets" described in Section 15 of 267 [RFC2330]. 269 For these tests, a stream of at least 300 packets were sent from 270 Source to Destination in each implementation. Periodic streams (as 271 per [RFC3432]) with 1 second spacing were used, except as noted. 273 With the L2TPv3 tunnel in use, the metric name for the testing 274 configured here (with respect to the IP header exposed to Internet 275 processing) is: 277 Type-IP-protocol-115-One-way-Delay--Stream 279 With (Section 4.2. [RFC2679]) Metric Parameters: 281 + Src, the IP address of a host (12.3.167.16 or 193.159.144.8) 283 + Dst, the IP address of a host (193.159.144.8 or 12.3.167.16) 285 + T0, a time 287 + Tf, a time 289 + lambda, a rate in reciprocal seconds 290 + Thresh, a maximum waiting time in seconds (see Section 3.8.2 of 291 [RFC2679]) And (Section 4.3. [RFC2679]) 293 Metric Units: A sequence of pairs; the elements of each pair are: 295 + T, a time, and 297 + dT, either a real number or an undefined number of seconds. 299 The values of T in the sequence are monotonic increasing. Note that 300 T would be a valid parameter to Type-P-One-way-Delay, and that dT 301 would be a valid value of Type-P-One-way-Delay. 303 Also, Section 3.8.4 of [RFC2679] recommends that the path SHOULD be 304 reported. In this test set-up, most of the path details will be 305 concealed from the implementations by the L2TPv3 tunnels, thus a more 306 informative path trace route can be conducted by the routers at each 307 location. 309 When NetProbe is used in production, a traceroute is conducted in 310 parallel with, and at the outset of measurements. 312 Perfas+ does not support traceroute. 314 IPLGW#traceroute 193.159.144.8 316 Type escape sequence to abort. 317 Tracing the route to 193.159.144.8 319 1 12.126.218.245 [AS 7018] 0 msec 0 msec 4 msec 320 2 cr84.n54ny.ip.att.net (12.123.2.158) [AS 7018] 4 msec 4 msec 321 cr83.n54ny.ip.att.net (12.123.2.26) [AS 7018] 4 msec 322 3 cr1.n54ny.ip.att.net (12.122.105.49) [AS 7018] 4 msec 323 cr2.n54ny.ip.att.net (12.122.115.93) [AS 7018] 0 msec 324 cr1.n54ny.ip.att.net (12.122.105.49) [AS 7018] 0 msec 325 4 n54ny02jt.ip.att.net (12.122.80.225) [AS 7018] 4 msec 0 msec 326 n54ny02jt.ip.att.net (12.122.80.237) [AS 7018] 4 msec 327 5 192.205.34.182 [AS 7018] 0 msec 328 192.205.34.150 [AS 7018] 0 msec 329 192.205.34.182 [AS 7018] 4 msec 330 6 da-rg12-i.DA.DE.NET.DTAG.DE (62.154.1.30) [AS 3320] 88 msec 88 msec 331 88 msec 332 7 217.89.29.62 [AS 3320] 88 msec 88 msec 88 msec 333 8 217.89.29.55 [AS 3320] 88 msec 88 msec 88 msec 334 9 * * * 336 It was only possible to conduct the traceroute for the measured path 337 on one of the tunnel-head routers (the normal trace facilities of the 338 measurement systems are confounded by the L2TPv3 tunnel 339 encapsulation). 341 4. Error Calibration, RFC 2679 343 An implementation is required to report on its error calibration in 344 Section 3.8 of [RFC2679] (also required in Section 4.8 for sample 345 metrics). Sections 3.6, 3.7, and 3.8 of [RFC2679] give the detailed 346 formulation of the errors and uncertainties for calibration. In 347 summary, Section 3.7.1 of [RFC2679] describes the total time-varying 348 uncertainty as: 350 Esynch(t)+ Rsource + Rdest 352 where: 354 Esynch(t) denotes an upper bound on the magnitude of clock 355 synchronization uncertainty. 357 Rsource and Rdest denote the resolution of the source clock and the 358 destination clock, respectively. 360 Further, Section 3.7.2 of [RFC2679] describes the total wire-time 361 uncertainty as 363 Hsource + Hdest 365 referring to the upper bounds on host-time to wire-time for source 366 and destination, respectively. 368 Section 3.7.3 of [RFC2679] describes a test with small packets over 369 an isolated minimal network where the results can be used to estimate 370 systematic and random components of the sum of the above errors or 371 uncertainties. In a test with hundreds of singletons, the median is 372 the systematic error and when the median is subtracted from all 373 singletons, the remaining variability is the random error. 375 The test context, or Type-P of the test packets, must also be 376 reported, as required in Section 3.8 of [RFC2679] and all metrics 377 defined there. Type-P is defined in Section 13 of [RFC2330] (as are 378 many terms used below). 380 4.1. NetProbe Error and Type-P 382 Type-P for this test was IP-UDP with Best Effort DSCP. These headers 383 were encapsulated according to the L2TPv3 specifications [RFC3931], 384 and thus may not influence the treatment received as the packets 385 traversed the Internet. 387 In general, NetProbe error is dependent on the specific version and 388 installation details. 390 NetProbe operates using host time above the UDP layer, which is 391 different from the wire-time preferred in [RFC2330], but can be 392 identified as a source of error according to Section 3.7.2 of 393 [RFC2679]. 395 Accuracy of NetProbe measurements is usually limited by NTP 396 synchronization performance (which is typically taken as ~+/-1ms 397 error or greater), although the installation used in this testing 398 often exhibits errors much less than typical for NTP. The primary 399 stratum 1 NTP server is closely located on a sparsely utilized 400 network management LAN, thus it avoids many concerns raised in 401 Section 10 of[RFC2330] (in fact, smooth adjustment, long-term drift 402 analysis and compensation, and infrequent adjustment all lead to 403 stability during measurement intervals, the main concern). 405 The resolution of the reported results is 1us (us = microsecond) in 406 the version of NetProbe tested here, which contributes to at least 407 +/-1us error. 409 NetProbe implements a time-keeping sanity check on sending and 410 receiving time-stamping processes. When the significant process 411 interruption takes place, individual test packets are flagged as 412 possibly containing unusual time errors, and are excluded from the 413 sample used for all "time" metrics. 415 We performed a NetProbe calibration of the type described in Section 416 3.7.3 of [RFC2679], using 64 Byte packets over a cross-connect cable. 417 The results estimate systematic and random components of the sum of 418 the Hsource + Hdest errors or uncertainties. In a test with 300 419 singletons conducted over 30 seconds (periodic sample with 100ms 420 spacing), the median is the systematic error and the remaining 421 variability is the random error. One set of results is tabulated 422 below: 424 (Results from the "R" software environment for statistical computing 425 and graphics - http://www.r-project.org/ ) 426 > summary(XD4CAL) 427 CAL1 CAL2 CAL3 428 Min. : 89.0 Min. : 68.00 Min. : 54.00 429 1st Qu.: 99.0 1st Qu.: 77.00 1st Qu.: 63.00 430 Median :110.0 Median : 79.00 Median : 65.00 431 Mean :116.8 Mean : 83.74 Mean : 69.65 432 3rd Qu.:127.0 3rd Qu.: 88.00 3rd Qu.: 74.00 433 Max. :205.0 Max. :177.00 Max. :163.00 434 > 435 NetProbe Calibration with Cross-Connect Cable, one-way delay values 436 in microseconds (us) 438 The median or systematic error can be as high as 110 us, and the 439 range of the random error is also on the order of 116 us for all 440 streams. 442 Also, anticipating the Anderson-Darling K-sample (ADK) [ADK] 443 comparisons to follow, we corrected the CAL2 values for the 444 difference between means between CAL2 and CAL3 (as specified in 445 [RFC6576]), and found strong support for the (Null Hypothesis that) 446 the samples are from the same distribution (resolution of 1 us and 447 alpha equal 0.05 and 0.01) 448 > XD4CVCAL2 <- XD4CAL$CAL2 - (mean(XD4CAL$CAL2)-mean(XD4CAL$CAL3)) 449 > boxplot(XD4CVCAL2,XD4CAL$CAL3) 450 > XD4CV2_ADK <- adk.test(XD4CVCAL2, XD4CAL$CAL3) 451 > XD4CV2_ADK 452 Anderson-Darling k-sample test. 454 Number of samples: 2 455 Sample sizes: 300 300 456 Total number of values: 600 457 Number of unique values: 97 459 Mean of Anderson Darling Criterion: 1 460 Standard deviation of Anderson Darling Criterion: 0.75896 462 T = (Anderson Darling Criterion - mean)/sigma 464 Null Hypothesis: All samples come from a common population. 466 t.obs P-value extrapolation 467 not adj. for ties 0.71734 0.17042 0 468 adj. for ties -0.39553 0.44589 1 469 > 470 using [Rtool] and [Radk]. 472 4.2. Perfas+ Error and Type-P 474 Perfas+ is configured to use GPS synchronisation and uses NTP 475 synchronization as a fall-back or default. GPS synchronisation 476 worked throughout this test with the exception of the calibration 477 stated here (one implementation was NTP synchronised only). The time 478 stamp accuracy typically is 0.1 ms. 480 The resolution of the results reported by Perfas+ is 1us (us = 481 microsecond) in the version tested here, which contributes to at 482 least +/-1us error. 484 Port 5001 5002 5003 485 Min. -227 -226 294 486 Median -169 -167 323 487 Mean -159 -157 335 488 Max. 6 -52 376 489 s 102 102 93 490 Perfas+ Calibration with Cross-Connect Cable, one-way delay values in 491 microseconds (us) 493 The median or systematic error can be as high as 323 us, and the 494 range of the random error is also less than 232 us for all streams. 496 5. Pre-determined Limits on Equivalence 498 This section provides the numerical limits on comparisons between 499 implementations, in order to declare that the results are equivalent 500 and therefore, the tested specification is clear. These limits have 501 their basis in Section 3.1 of [RFC6576] and the Appendix of 502 [RFC2330], with additional limits representing IPPM consensus prior 503 to publication of results. 505 A key point is that the allowable errors, corrections, and confidence 506 levels only need to be sufficient to detect mis-interpretation of the 507 tested specification resulting in diverging implementations. 509 Also, the allowable error must be sufficient to compensate for 510 measured path differences. It was simply not possible to measure 511 fully identical paths in the VLAN-loopback test configuration used, 512 and this practical compromise must be taken into account. 514 For Anderson-Darling K-sample (ADK) comparisons, the required 515 confidence factor for the cross-implementation comparisons SHALL be 516 the smallest of: 518 o 0.95 confidence factor at 1ms resolution, or 520 o the smallest confidence factor (in combination with resolution) of 521 the two same-implementation comparisons for the same test 522 conditions. 524 A constant time accuracy error of as much as +/-0.5ms MAY be removed 525 from one implementation's distributions (all singletons) before the 526 ADK comparison is conducted. 528 A constant propagation delay error (due to use of different sub-nets 529 between the switch and measurement devices at each location) of as 530 much as +2ms MAY be removed from one implementation's distributions 531 (all singletons) before the ADK comparison is conducted. 533 For comparisons involving the mean of a sample or other central 534 statistics, the limits on both the time accuracy error and the 535 propagation delay error constants given above also apply. 537 6. Tests to evaluate RFC 2679 Specifications 539 This section describes some results from real-world (cross-Internet) 540 tests with measurement devices implementing IPPM metrics and a 541 network emulator to create relevant conditions, to determine whether 542 the metric definitions were interpreted consistently by implementors. 544 The procedures are slightly modified from the original procedures 545 contained in Appendix A.1 of [RFC6576]. The modifications include 546 the use of the mean statistic for comparisons. 548 Note that there are only five instances of the requirement term 549 "MUST" in [RFC2679] outside of the boilerplate and [RFC2119] 550 reference. 552 6.1. One-way Delay, ADK Sample Comparison - Same & Cross Implementation 554 This test determines if implementations produce results that appear 555 to come from a common delay distribution, as an overall evaluation of 556 Section 4 of [RFC2679], "A Definition for Samples of One-way Delay". 557 Same-implementation comparison results help to set the threshold of 558 equivalence that will be applied to cross-implementation comparisons. 560 This test is intended to evaluate measurements in sections 3 and 4 of 561 [RFC2679]. 563 By testing the extent to which the distributions of one-way delay 564 singletons from two implementations of [RFC2679] appear to be from 565 the same distribution, we economize on comparisons, because comparing 566 a set of individual summary statistics (as defined in Section 5 of 567 [RFC2679]) would require another set of individual evaluations of 568 equivalence. Instead, we can simply check which statistics were 569 implemented, and report on those facts. 571 1. Configure an L2TPv3 path between test sites, and each pair of 572 measurement devices to operate tests in their designated pair of 573 VLANs. 575 2. Measure a sample of one-way delay singletons with 2 or more 576 implementations, using identical options and network emulator 577 settings (if used). 579 3. Measure a sample of one-way delay singletons with *four* 580 instances of the *same* implementations, using identical options, 581 noting that connectivity differences SHOULD be the same as for 582 the cross implementation testing. 584 4. Apply the ADK comparison procedures (see Appendix C of [RFC6576]) 585 and determine the resolution and confidence factor for 586 distribution equivalence of each same-implementation comparison 587 and each cross-implementation comparison. 589 5. Take the coarsest resolution and confidence factor for 590 distribution equivalence from the same-implementation pairs, or 591 the limit defined in Section 5 above, as a limit on the 592 equivalence threshold for these experimental conditions. 594 6. Apply constant correction factors to all singletons of the sample 595 distributions, as described and limited in Section 5 above. 597 7. Compare the cross-implementation ADK performance with the 598 equivalence threshold determined in step 5 to determine if 599 equivalence can be declared. 601 The common parameters used for tests in this section are: 603 o IP header + payload = 64 octets 605 o Periodic sampling at 1 packet per second 607 o Test duration = 300 seconds (March 29) 609 The netem emulator was set for 100ms average delay, with uniform 610 delay variation of +/-50ms. In this experiment, the netem emulator 611 was configured to operate independently on each VLAN and thus the 612 emulator itself is a potential source of error when comparing streams 613 that traverse the test path in different directions. 615 In the result analysis of this section: 617 o All comparisons used 1 microsecond resolution. 619 o No Correction Factors were applied. 621 o The 0.95 confidence factor (1.960 for paired stream comparison) 622 was used. 624 6.1.1. NetProbe Same-implementation results 626 A single same-implementation comparison fails the ADK criterion (s1 627 <-> sB). We note that these streams traversed the test path in 628 opposite directions, making the live network factors a possibility to 629 explain the difference. 631 All other pair comparisons pass the ADK criterion. 633 +------------------------------------------------------+ 634 | | | | | 635 | ti.obs (P) | s1 | s2 | sA | 636 | | | | | 637 .............|.............|.............|.............| 638 | | | | | 639 | s2 | 0.25 (0.28) | | | 640 | | | | | 641 ...........................|.............|.............| 642 | | | | | 643 | sA | 0.60 (0.19) |-0.80 (0.57) | | 644 | | | | | 645 ...........................|.............|.............| 646 | | | | | 647 | sB | 2.64 (0.03) | 0.07 (0.31) |-0.52 (0.48) | 648 | | | | | 649 +------------+-------------+-------------+-------------+ 651 NetProbe ADK Results for same-implementation 653 6.1.2. Perfas+ Same-implementation results 655 All pair comparisons pass the ADK criterion. 657 +------------------------------------------------------+ 658 | | | | | 659 | ti.obs (P) | p1 | p2 | p3 | 660 | | | | | 661 .............|.............|.............|.............| 662 | | | | | 663 | p2 | 0.06 (0.32) | | | 664 | | | | | 665 .........................................|.............| 666 | | | | | 667 | p3 | 1.09 (0.12) | 0.37 (0.24) | | 668 | | | | | 669 ...........................|.............|.............| 670 | | | | | 671 | p4 |-0.81 (0.57) |-0.13 (0.37) | 1.36 (0.09) | 672 | | | | | 673 +------------+-------------+-------------+-------------+ 675 Perfas+ ADK Results for same-implementation 677 6.1.3. One-way Delay, Cross-Implementation ADK Comparison 679 The cross-implementation results are compared using a combined ADK 680 analysis [Radk], where all NetProbe results are compared with all 681 Perfas+ results after testing that the combined same-implementation 682 results pass the ADK criterion. 684 When 4 (same) samples are compared, the ADK criterion for 0.95 685 confidence is 1.915, and when all 8 (cross) samples are compared it 686 is 1.85. 688 Combination of Anderson-Darling K-Sample Tests. 690 Sample sizes within each data set: 691 Data set 1 : 299 297 298 300 (NetProbe) 692 Data set 2 : 300 300 298 300 (Perfas+) 693 Total sample size per data set: 1194 1198 694 Number of unique values per data set: 1188 1192 695 ... 696 Null Hypothesis: 697 All samples within a data set come from a common distribution. 698 The common distribution may change between data sets. 700 NetProbe ti.obs P-value extrapolation 701 not adj. for ties 0.64999 0.21355 0 702 adj. for ties 0.64833 0.21392 0 703 Perfas+ 704 not adj. for ties 0.55968 0.23442 0 705 adj. for ties 0.55840 0.23473 0 707 Combined Anderson-Darling Criterion: 708 tc.obs P-value extrapolation 709 not adj. for ties 0.85537 0.17967 0 710 adj. for ties 0.85329 0.18010 0 712 The combined same-implementation samples and the combined cross- 713 implementation comparison all pass the ADK criteria at P>=0.18 and 714 support the Null Hypothesis (both data sets come from a common 715 distribution). 717 We also see that the paired ADK comparisons are rather critical. 718 Although the NetProbe s1-sB comparison failed, the combined data set 719 from 4 streams passed the ADK criterion easily. 721 6.1.4. Conclusions on the ADK Results for One-way Delay 723 Similar testing was repeated many times in the months of March and 724 April 2011. There were many experiments where a single test stream 725 from NetProbe or Perfas+ proved to be different from the others in 726 paired comparisons (even same implementation comparisons). When the 727 outlier stream was removed from the comparison, the remaining streams 728 passed combined ADK criterion. Also, the application of correction 729 factors resulted in higher comparison success. 731 We conclude that the two implementations are capable of producing 732 equivalent one-way delay distributions based on their interpretation 733 of [RFC2679]. 735 6.1.5. Additional Investigations 737 On the final day of testing, we performed a series of measurements to 738 evaluate the amount of emulated delay variation necessary to achieve 739 successful ADK comparisons. The need for Correction factors (as 740 permitted by Section 5) and the size of the measurement sample 741 (obtained as sub-sets of the complete measurement sample) were also 742 evaluated. 744 The common parameters used for tests in this section are: 746 o IP header + payload = 64 octets 748 o Periodic sampling at 1 packet per second 750 o Test duration = 300 seconds at each delay variation setting, for a 751 total of 1200 seconds (May 2, 2011 at 1720 UTC) 753 The netem emulator was set for 100ms average delay, with (emulated) 754 uniform delay variation of: 756 o +/-7.5 ms 758 o +/-5.0 ms 760 o +/-2.5 ms 762 o 0 ms 764 In this experiment, the netem emulator was configured to operate 765 independently on each VLAN and thus the emulator itself is a 766 potential source of error when comparing streams that traverse the 767 test path in different directions. 769 In the result analysis of this section: 771 o All comparisons used 1 microsecond resolution. 773 o Correction Factors *were* applied as noted (under column heading 774 "mean adj"). The difference between each sample mean and the 775 lowest mean of the NetProbe or Perfas+ stream samples was 776 subtracted from all values in the sample. ("raw" indicates no 777 correction factors were used.) All correction factors applied met 778 the limits described in Section 5. 780 o The 0.95 confidence factor (1.960 for paired stream comparison) 781 was used. 783 When 8 (cross) samples are compared, the ADK criterion for 0.95 784 confidence is 1.85. The Combined ADK test statistic ("TC observed") 785 must be less than 1.85 to accept the Null Hypothesis (all samples in 786 the data set are from a common distribution). 788 Emulated Delay Sub-Sample size 789 Variation 0ms 790 adk.combined (all) 300 values 75 values 791 Adj. for ties raw mean adj raw mean adj 792 TC observed 226.6563 67.51559 54.01359 21.56513 793 P-value 0 0 0 0 794 Mean std dev (all),us 719 635 795 Mean diff of means,us 649 0 606 0 797 Variation +/- 2.5ms 798 adk.combined (all) 300 values 75 values 799 Adj. for ties raw mean adj raw mean adj 800 TC observed 14.50436 -1.60196 3.15935 -1.72104 801 P-value 0 0.873 0.00799 0.89038 802 Mean std dev (all),us 1655 1702 803 Mean diff of means,us 471 0 513 0 805 Variation +/- 5ms 806 adk.combined (all) 300 values 75 values 807 Adj. for ties raw mean adj raw mean adj 808 TC observed 8.29921 -1.28927 0.37878 -1.81881 809 P-value 0 0.81601 0.29984 0.90305 810 Mean std dev (all),us 3023 2991 811 Mean diff of means,us 582 0 513 0 813 Variation +/- 7.5ms 814 adk.combined (all) 300 values 75 values 815 Adj. for ties raw mean adj raw mean adj 816 TC observed 2.53759 -0.72985 0.29241 -1.15840 817 P-value 0.01950 0.66942 0.32585 0.78686 818 Mean std dev (all),us 4449 4506 819 Mean diff of means,us 426 0 856 0 821 From the table above, we conclude the following: 823 1. None of the raw or mean adjusted results pass the ADK criterion 824 with 0 ms emulated delay variation. Use of the 75 value sub- 825 sample yielded the same conclusion. (We note the same results 826 when comparing same implementation samples for both NetProbe and 827 Perfas+.) 829 2. When the smallest emulated delay variation was inserted 830 (+/-2.5ms), the mean adjusted samples pass the ADK criterion and 831 the high P-value supports the result. The raw results do not 832 pass. 834 3. At higher values of emulated delay variation (+/-5.0ms and 835 +/-7.5ms), again the mean adjusted values pass ADK. We also see 836 that the 75-value sub-sample passed the ADK in both raw and mean 837 adjusted cases. This indicates that sample size may have played 838 a role in our results, as noted in the Appendix of [RFC2680] for 839 Goodness-of-Fit testing. 841 We note that 150 value sub-samples were also evaluated, with ADK 842 conclusions that followed the results for 300 values. Also, same- 843 implementation analysis was conducted with results similar to the 844 above, except that more of the "raw" or uncorrected samples passed 845 the ADK criterion. 847 6.2. One-way Delay, Loss threshold, RFC 2679 849 This test determines if implementations use the same configured 850 maximum waiting time delay from one measurement to another under 851 different delay conditions, and correctly declare packets arriving in 852 excess of the waiting time threshold as lost. 854 See Section 3.5 of [RFC2679], 3rd bullet point and also Section 3.8.2 855 of [RFC2679]. 857 1. configure an L2TPv3 path between test sites, and each pair of 858 measurement devices to operate tests in their designated pair of 859 VLANs. 861 2. configure the network emulator to add 1.0 sec one-way constant 862 delay in one direction of transmission. 864 3. measure (average) one-way delay with 2 or more implementations, 865 using identical waiting time thresholds (Thresh) for loss set at 866 3 seconds. 868 4. configure the network emulator to add 3 sec one-way constant 869 delay in one direction of transmission equivalent to 2 seconds of 870 additional one-way delay (or change the path delay while test is 871 in progress, when there are sufficient packets at the first delay 872 setting) 874 5. repeat/continue measurements 876 6. observe that the increase measured in step 5 caused all packets 877 with 2 sec additional delay to be declared lost, and that all 878 packets that arrive successfully in step 3 are assigned a valid 879 one-way delay. 881 The common parameters used for tests in this section are: 883 o IP header + payload = 64 octets 885 o Poisson sampling at lambda = 1 packet per second 887 o Test duration = 900 seconds total (March 21) 889 The netem emulator was set to add constant delays as specified in the 890 procedure above. 892 6.2.1. NetProbe results for Loss Threshold 894 In NetProbe, the Loss Threshold is implemented uniformly over all 895 packets as a post-processing routine. With the Loss Threshold set at 896 3 seconds, all packets with one-way delay >3 seconds are marked 897 "Lost" and included in the Lost Packet list with their transmission 898 time (as required in Section 3.3 of [RFC2680]). This resulted in 342 899 packets designated as lost in one of the test streams (with average 900 delay = 3.091 sec). 902 6.2.2. Perfas+ Results for Loss Threshold 904 Perfas+ uses a fixed Loss Threshold which was not adjustable during 905 this study. The Loss Threshold is approximately one minute, and 906 emulation of a delay of this size was not attempted. However, it is 907 possible to implement any delay threshold desired with a post- 908 processing routine and subsequent analysis. Using this method, 195 909 packets would be declared lost (with average delay = 3.091 sec). 911 6.2.3. Conclusions for Loss Threshold 913 Both implementations assume that any constant delay value desired can 914 be used as the Loss Threshold, since all delays are stored as a pair 915 as required in [RFC2679] . This is a simple way to 916 enforce the constant loss threshold envisioned in [RFC2679] (see 917 specific section references above). We take the position that the 918 assumption of post-processing is compliant, and that the text of the 919 RFC should be revised slightly to include this point. 921 6.3. One-way Delay, First-bit to Last bit, RFC 2679 923 This test determines if implementations register the same relative 924 change in delay from one packet size to another, indicating that the 925 first-to-last time-stamping convention has been followed. This test 926 tends to cancel the sources of error which may be present in an 927 implementation. 929 See Section 3.7.2 of [RFC2679], and Section 10.2 of [RFC2330]. 931 1. configure an L2TPv3 path between test sites, and each pair of 932 measurement devices to operate tests in their designated pair of 933 VLANs, and ideally including a low-speed link (it was not 934 possible to change the link configuration during testing, so the 935 lowest speed link present was the basis for serialization time 936 comparisons). 938 2. measure (average) one-way delay with 2 or more implementations, 939 using identical options and equal size small packets (64 octet IP 940 header and payload) 942 3. maintain the same path with additional emulated 100 ms one-way 943 delay 945 4. measure (average) one-way delay with 2 or more implementations, 946 using identical options and equal size large packets (500 octet 947 IP header and payload) 949 5. observe that the increase measured between steps 2 and 4 is 950 equivalent to the increase in ms expected due to the larger 951 serialization time for each implementation. Most of the 952 measurement errors in each system should cancel, if they are 953 stationary. 955 The common parameters used for tests in this section are: 957 o IP header + payload = 64 octets 959 o Periodic sampling at l packet per second 961 o Test duration = 300 seconds total (April 12) 963 The netem emulator was set to add constant 100ms delay. 965 6.3.1. NetProbe and Perfas+ Results for Serialization 967 When the IP header + payload size was increased from 64 octets to 500 968 octets, there was a delay increase observed. 970 Mean Delays in us 971 NetProbe 972 Payload s1 s2 sA sB 973 500 190893 191179 190892 190971 974 64 189642 189785 189747 189467 975 Diff 1251 1394 1145 1505 977 Perfas 978 Payload p1 p2 p3 p4 979 500 190908 190911 191126 190709 980 64 189706 189752 189763 190220 981 Diff 1202 1159 1363 489 983 Serialization tests, all values in microseconds 985 The typical delay increase when the larger packets were used was 1.1 986 to 1.5 ms (with one outlier). The typical measurements indicate that 987 a link with approximately 3 Mbit/s capacity is present on the path. 989 Through investigation of the facilities involved, it was determined 990 that the lowest speed link was approximately 45 Mbit/s, and therefore 991 the estimated difference should be about 0.077 ms. The observed 992 differences are much higher. 994 The unexpected large delay difference was also the outcome when 995 testing serialization times in a lab environment, using the NIST Net 996 Emulator and NetProbe [I-D.morton-ippm-advance-metrics]. 998 6.3.2. Conclusions for Serialization 1000 Since it was not possible to confirm the estimated serialization time 1001 increases in field tests, we resort to examination of the 1002 implementations to determine compliance. 1004 NetProbe performs all time stamping above the IP-layer, accepting 1005 that some compromises must be made to achieve extreme portability and 1006 measurement scale. Therefore, the first-to-last bit convention is 1007 supported because the serialization time is included in the one-way 1008 delay measurement, enabling comparison with other implementations. 1010 Perfas+ is optimized for its purpose and performs all time stamping 1011 close to the interface hardware. The first-to-last bit convention is 1012 supported because the serialization time is included in the one-way 1013 delay measurement, enabling comparison with other implementations. 1015 6.4. One-way Delay, Difference Sample Metric (Lab) 1017 This test determines if implementations register the same relative 1018 increase in delay from one measurement to another under different 1019 delay conditions. This test tends to cancel the sources of error 1020 which may be present in an implementation. 1022 This test is intended to evaluate measurements in sections 3 and 4 of 1023 [RFC2679]. 1025 1. configure an L2TPv3 path between test sites, and each pair of 1026 measurement devices to operate tests in their designated pair of 1027 VLANs. 1029 2. measure (average) one-way delay with 2 or more implementations, 1030 using identical options 1032 3. configure the path with X+Y ms one-way delay 1034 4. repeat measurements 1036 5. observe that the (average) increase measured in steps 2 and 4 is 1037 ~Y ms for each implementation. Most of the measurement errors in 1038 each system should cancel, if they are stationary. 1040 In this test, X=1000ms and Y=1000ms. 1042 The common parameters used for tests in this section are: 1044 o IP header + payload = 64 octets 1046 o Poisson sampling at lambda = 1 packet per second 1048 o Test duration = 900 seconds total (March 21) 1050 The netem emulator was set to add constant delays as specified in the 1051 procedure above. 1053 6.4.1. NetProbe results for Differential Delay 1055 Average pre-increase delay, microseconds 1089868.0 1056 Average post 1s additional, microseconds 2089686.0 1057 Difference (should be ~= Y = 1s) 999818.0 1059 Average delays before/after 1 second increase 1061 The NetProbe implementation observed a 1 second increase with a 182 1062 microsecond error (assuming that the netem emulated delay difference 1063 is exact). 1065 We note that this differential delay test has been run under lab 1066 conditions and published in prior work [ref to "advance metrics" 1067 draft]. The error was 6 microseconds. 1069 6.4.2. Perfas+ results for Differential Delay 1071 Average pre-increase delay, microseconds 1089794.0 1072 Average post 1s additional, microseconds 2089801.0 1073 Difference (should be ~= Y = 1s) 1000007.0 1075 Average delays before/after 1 second increase 1077 The Perfas+ implementation observed a 1 second increase with a 7 1078 microsecond error. 1080 6.4.3. Conclusions for Differential Delay 1082 Again, the live network conditions appear to have influenced the 1083 results, but both implementations measured the same delay increase 1084 within their calibration accuracy. 1086 6.5. Implementation of Statistics for One-way Delay 1088 The ADK tests the extent to which the sample distributions of one-way 1089 delay singletons from two implementations of [RFC2679] appear to be 1090 from the same overall distribution. By testing this way, we 1091 economize on the number of comparisons, because comparing a set of 1092 individual summary statistics (as defined in Section 5 of [RFC2679]) 1093 would require another set of individual evaluations of equivalence. 1094 Instead, we can simply check which statistics were implemented, and 1095 report on those facts, noting that Section 5 of [RFC2679] does not 1096 specify the calculations exactly, and gives only some illustrative 1097 examples. 1099 NetProbe Perfas+ 1101 5.1. Type-P-One-way-Delay-Percentile yes no 1103 5.2. Type-P-One-way-Delay-Median yes no 1105 5.3. Type-P-One-way-Delay-Minimum yes yes 1107 5.4. Type-P-One-way-Delay-Inverse-Percentile no no 1109 Implementation of Section 5 Statistics 1111 Only the Type-P-One-way-Delay-Inverse-Percentile has been ignored in 1112 both implementations, so it is a candidate for removal in RFC2679bis. 1114 7. Security Considerations 1116 The security considerations that apply to any active measurement of 1117 live networks are relevant here as well. See [RFC4656] and 1118 [RFC5357]. 1120 8. IANA Considerations 1122 This memo makes no requests of IANA, and hopes that IANA will welcome 1123 our new computer overlords as willingly as the authors. 1125 9. Acknowledgements 1127 The authors thank Lars Eggert for his continued encouragement to 1128 advance the IPPM metrics during his tenure as AD Advisor. 1130 Nicole Kowalski supplied the needed CPE router for the NetProbe side 1131 of the test set-up, and graciously managed her testing in spite of 1132 issues caused by dual-use of the router. Thanks Nicole! 1134 The "NetProbe Team" also acknowledges many useful discussions with 1135 Ganga Maguluri. 1137 10. References 1138 10.1. Normative References 1140 [RFC2026] Bradner, S., "The Internet Standards Process -- Revision 1141 3", BCP 9, RFC 2026, October 1996. 1143 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1144 Requirement Levels", BCP 14, RFC 2119, March 1997. 1146 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 1147 "Framework for IP Performance Metrics", RFC 2330, 1148 May 1998. 1150 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1151 Delay Metric for IPPM", RFC 2679, September 1999. 1153 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1154 Packet Loss Metric for IPPM", RFC 2680, September 1999. 1156 [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network 1157 performance measurement with periodic streams", RFC 3432, 1158 November 2002. 1160 [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. 1161 Zekauskas, "A One-way Active Measurement Protocol 1162 (OWAMP)", RFC 4656, September 2006. 1164 [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. 1165 Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", 1166 RFC 5357, October 2008. 1168 [RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation 1169 and Implementation Reports for Advancement to Draft 1170 Standard", BCP 9, RFC 5657, September 2009. 1172 [RFC6576] Geib, R., Morton, A., Fardid, R., and A. Steinmitz, "IP 1173 Performance Metrics (IPPM) Standard Advancement Testing", 1174 BCP 176, RFC 6576, March 2012. 1176 10.2. Informative References 1178 [ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling 1179 Tests of fit, for continuous and discrete cases", 1180 University of Washington, Technical Report No. 81, 1181 May 1986. 1183 [Fedora14] 1184 "Fedora Project Home Page", http://fedoraproject.org/, 1185 2012. 1187 [I-D.bradner-metricstest] 1188 Bradner, S. and V. Paxson, "Advancement of metrics 1189 specifications on the IETF Standards Track", 1190 draft-bradner-metricstest-03 (work in progress), 1191 August 2007. 1193 [I-D.morton-ippm-advance-metrics] 1194 Morton, A., "Lab Test Results for Advancing Metrics on the 1195 Standards Track", draft-morton-ippm-advance-metrics-02 1196 (work in progress), October 2010. 1198 [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling 1199 Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. 1201 [Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and 1202 Combinations of Such Tests. R package version 1.0.", , 1203 2008. 1205 [Rtool] R Development Core Team, "R: A language and environment 1206 for statistical computing. R Foundation for Statistical 1207 Computing, Vienna, Austria. ISBN 3-900051-07-0, URL 1208 http://www.R-project.org/", , 2011. 1210 [WIPM] "AT&T Global IP Network", 1211 http://ipnetwork.bgtmo.ip.att.net/pws/index.html, 2012. 1213 [netem] ""netem" Documentation", http://www.linuxfoundation.org/ 1214 collaborate/workgroups/networking/netem, 2009. 1216 Authors' Addresses 1218 Len Ciavattone 1219 AT&T Labs 1220 200 Laurel Avenue South 1221 Middletown, NJ 07748 1222 USA 1224 Phone: +1 732 420 1239 1225 Fax: 1226 Email: lencia@att.com 1227 URI: 1229 Ruediger Geib 1230 Deutsche Telekom 1231 Heinrich Hertz Str. 3-7 1232 Darmstadt, 64295 1233 Germany 1235 Phone: +49 6151 58 12747 1236 Email: Ruediger.Geib@telekom.de 1238 Al Morton 1239 AT&T Labs 1240 200 Laurel Avenue South 1241 Middletown, NJ 07748 1242 USA 1244 Phone: +1 732 420 1571 1245 Fax: +1 732 368 1192 1246 Email: acmorton@att.com 1247 URI: http://home.comcast.net/~acmacm/ 1249 Matthias Wieser 1250 Technical University Darmstadt 1251 Darmstadt, 1252 Germany 1254 Phone: 1255 Email: matthias_michael.wieser@stud.tu-darmstadt.de