idnits 2.17.1 draft-ietf-ippm-testplan-rfc2679-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 1 character in excess of 72. == There are 7 instances of lines with non-RFC2606-compliant FQDNs in the document. == There are 11 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 833 has weird spacing: '...ean adj raw...' == Line 1021 has weird spacing: '...Payload s1 ...' == Line 1027 has weird spacing: '...Payload p1 ...' == Line 1156 has weird spacing: '...centile no ...' == The document seems to contain a disclaimer for pre-RFC5378 work, but was first submitted on or after 10 November 2008. The disclaimer is usually necessary only for documents that revise or obsolete older RFCs, and that take significant amounts of text from those RFCs. If you can contact all authors of the source material and they are willing to grant the BCP78 rights to the IETF Trust, you can and should remove the disclaimer. Otherwise, the disclaimer is needed and you can ignore this comment. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (March 10, 2012) is 4430 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'AS 7018' is mentioned on line 373, but not defined == Missing Reference: 'AS 3320' is mentioned on line 377, but not defined == Unused Reference: 'RFC4814' is defined on line 1219, but no explicit reference was found in the text == Unused Reference: 'RFC5226' is defined on line 1223, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) ** Obsolete normative reference: RFC 5226 (Obsoleted by RFC 8126) Summary: 4 errors (**), 0 flaws (~~), 12 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group L. Ciavattone 3 Internet-Draft AT&T Labs 4 Intended status: Informational R. Geib 5 Expires: September 11, 2012 Deutsche Telekom 6 A. Morton 7 AT&T Labs 8 M. Wieser 9 Technical University Darmstadt 10 March 10, 2012 12 Test Plan and Results for Advancing RFC 2679 on the Standards Track 13 draft-ietf-ippm-testplan-rfc2679-01 15 Abstract 17 This memo proposes to advance a performance metric RFC along the 18 standards track, specifically RFC 2679 on One-way Delay Metrics. 19 Observing that the metric definitions themselves should be the 20 primary focus rather than the implementations of metrics, this memo 21 describes the test procedures to evaluate specific metric requirement 22 clauses to determine if the requirement has been interpreted and 23 implemented as intended. Two completely independent implementations 24 have been tested against the key specifications of RFC 2679. 26 Requirements Language 28 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 29 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 30 document are to be interpreted as described in RFC 2119 [RFC2119]. 32 Status of this Memo 34 This Internet-Draft is submitted in full conformance with the 35 provisions of BCP 78 and BCP 79. 37 Internet-Drafts are working documents of the Internet Engineering 38 Task Force (IETF). Note that other groups may also distribute 39 working documents as Internet-Drafts. The list of current Internet- 40 Drafts is at http://datatracker.ietf.org/drafts/current/. 42 Internet-Drafts are draft documents valid for a maximum of six months 43 and may be updated, replaced, or obsoleted by other documents at any 44 time. It is inappropriate to use Internet-Drafts as reference 45 material or to cite them other than as "work in progress." 47 This Internet-Draft will expire on September 11, 2012. 49 Copyright Notice 51 Copyright (c) 2012 IETF Trust and the persons identified as the 52 document authors. All rights reserved. 54 This document is subject to BCP 78 and the IETF Trust's Legal 55 Provisions Relating to IETF Documents 56 (http://trustee.ietf.org/license-info) in effect on the date of 57 publication of this document. Please review these documents 58 carefully, as they describe your rights and restrictions with respect 59 to this document. Code Components extracted from this document must 60 include Simplified BSD License text as described in Section 4.e of 61 the Trust Legal Provisions and are provided without warranty as 62 described in the Simplified BSD License. 64 This document may contain material from IETF Documents or IETF 65 Contributions published or made publicly available before November 66 10, 2008. The person(s) controlling the copyright in some of this 67 material may not have granted the IETF Trust the right to allow 68 modifications of such material outside the IETF Standards Process. 69 Without obtaining an adequate license from the person(s) controlling 70 the copyright in such materials, this document may not be modified 71 outside the IETF Standards Process, and derivative works of it may 72 not be created outside the IETF Standards Process, except to format 73 it for publication as an RFC or to translate it into languages other 74 than English. 76 Table of Contents 78 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 79 2. A Definition-centric metric advancement process . . . . . . . 5 80 3. Test configuration . . . . . . . . . . . . . . . . . . . . . . 6 81 4. Error Calibration, RFC 2679 . . . . . . . . . . . . . . . . . 10 82 4.1. NetProbe Error and Type-P . . . . . . . . . . . . . . . . 11 83 4.2. Perfas Error and Type-P . . . . . . . . . . . . . . . . . 13 84 5. Pre-determined Limits on Equivalence . . . . . . . . . . . . . 14 85 6. Tests to evaluate RFC 2679 Specifications . . . . . . . . . . 14 86 6.1. One-way Delay, ADK Sample Comparison - Same & Cross 87 Implementation . . . . . . . . . . . . . . . . . . . . . . 15 88 6.1.1. NetProbe Same-implementation results . . . . . . . . . 16 89 6.1.2. Perfas Same-implementation results . . . . . . . . . . 17 90 6.1.3. One-way Delay, Cross-Implementation ADK Comparison . . 18 91 6.1.4. Conclusions on the ADK Results for One-way Delay . . . 18 92 6.1.5. Additional Investigations . . . . . . . . . . . . . . 19 93 6.2. One-way Delay, Loss threshold, RFC 2679 . . . . . . . . . 22 94 6.2.1. NetProbe results for Loss Threshold . . . . . . . . . 23 95 6.2.2. Perfas Results for Loss Threshold . . . . . . . . . . 23 96 6.2.3. Conclusions for Loss Threshold . . . . . . . . . . . . 23 97 6.3. One-way Delay, First-bit to Last bit, RFC 2679 . . . . . . 24 98 6.3.1. NetProbe and Perfas Results for Serialization . . . . 24 99 6.3.2. Conclusions for Serialization . . . . . . . . . . . . 25 100 6.4. One-way Delay, Difference Sample Metric (Lab) . . . . . . 26 101 6.4.1. NetProbe results for Differential Delay . . . . . . . 26 102 6.4.2. Perfas results for Differential Delay . . . . . . . . 27 103 6.4.3. Conclusions for Differential Delay . . . . . . . . . . 27 104 6.5. Implementation of Statistics for One-way Delay . . . . . . 27 105 7. Security Considerations . . . . . . . . . . . . . . . . . . . 28 106 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 28 107 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 28 108 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 29 109 10.1. Normative References . . . . . . . . . . . . . . . . . . . 29 110 10.2. Informative References . . . . . . . . . . . . . . . . . . 30 111 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 30 113 1. Introduction 115 The IETF (IP Performance Metrics working group, IPPM) has considered 116 how to advance their metrics along the standards track since 2001, 117 with the initial publication of Bradner/Paxson/Mankin's memo [ref to 118 work in progress, draft-bradner-metricstest-]. The original proposal 119 was to compare the results of implementations of the metrics, because 120 the usual procedures for advancing protocols did not appear to apply. 121 It was found to be difficult to achieve consensus on exactly how to 122 compare implementations, since there were many legitimate sources of 123 variation that would emerge in the results despite the best attempts 124 to keep the network paths equal, and because considerable variation 125 was allowed in the parameters (and therefore implementation) of each 126 metric. Flexibility in metric definitions, essential for 127 customization and broad appeal, made the comparison task quite 128 difficult. 130 A renewed work effort sought to investigate ways in which the 131 measurement variability could be reduced and thereby simplify the 132 problem of comparison for equivalence. 134 There is consensus represented in [I-D.ietf-ippm-metrictest] that the 135 metric definitions should be the primary focus of evaluation rather 136 than the implementations of metrics, and equivalent results are 137 deemed to be evidence that the metric specifications are clear and 138 unambiguous. This is the metric specification equivalent of protocol 139 interoperability. The advancement process either produces confidence 140 that the metric definitions and supporting material are clearly 141 worded and unambiguous, OR, identifies ways in which the metric 142 definitions should be revised to achieve clarity. 144 The process should also permit identification of options that were 145 not implemented, so that they can be removed from the advancing 146 specification (this is an aspect more typical of protocol advancement 147 along the standards track). 149 This memo's purpose is to implement the current approach for 150 [RFC2679]. It was prepared to help progress discussions on the topic 151 of metric advancement, both through e-mail and at the upcoming IPPM 152 meeting at IETF. 154 In particular, consensus is sought on the extent of tolerable errors 155 when assessing equivalence in the results. In discussions, the IPPM 156 working group agreed that test plan and procedures should include the 157 threshold for determining equivalence, and this information should be 158 available in advance of cross-implementation comparisons. This memo 159 includes procedures for same-implementation comparisons to help set 160 the equivalence threshold. 162 Another aspect of the metric RFC advancement process is the 163 requirement to document the work and results. The procedures of 164 [RFC2026] are expanded in[RFC5657], including sample implementation 165 and interoperability reports. This memo follows the template in 166 [I-D.morton-ippm-advance-metrics] for the report that accompanies the 167 protocol action request submitted to the Area Director, including 168 description of the test set-up, procedures, results for each 169 implementation and conclusions. 171 2. A Definition-centric metric advancement process 173 The process described in Section 3.5 of [I-D.ietf-ippm-metrictest] 174 takes as a first principle that the metric definitions, embodied in 175 the text of the RFCs, are the objects that require evaluation and 176 possible revision in order to advance to the next step on the 177 standards track. 179 IF two implementations do not measure an equivalent singleton or 180 sample, or produce the an equivalent statistic, 182 AND sources of measurement error do not adequately explain the lack 183 of agreement, 185 THEN the details of each implementation should be audited along with 186 the exact definition text, to determine if there is a lack of clarity 187 that has caused the implementations to vary in a way that affects the 188 correspondence of the results. 190 IF there was a lack of clarity or multiple legitimate interpretations 191 of the definition text, 193 THEN the text should be modified and the resulting memo proposed for 194 consensus and advancement along the standards track. 196 Finally, all the findings MUST be documented in a report that can 197 support advancement on the standards track, similar to those 198 described in [RFC5657]. The list of measurement devices used in 199 testing satisfies the implementation requirement, while the test 200 results provide information on the quality of each specification in 201 the metric RFC (the surrogate for feature interoperability). 203 The figure below illustrates this process: 205 ,---. 206 / \ 207 ( Start ) 208 \ / Implementations 209 `-+-' +-------+ 210 | /| 1 `. 211 +---+----+ / +-------+ `.-----------+ ,-------. 212 | RFC | / |Check for | ,' was RFC `. YES 213 | | / |Equivalence..... clause x -------+ 214 | |/ +-------+ |under | `. clear? ,' | 215 | Metric \.....| 2 ....relevant | `---+---' +----+---+ 216 | Metric |\ +-------+ |identical | No | |Report | 217 | Metric | \ |network | +---+---. |results+| 218 | ... | \ |conditions | |Modify | |Advance | 219 | | \ +-------+ | | |Spec +----+ RFC | 220 +--------+ \| n |.'+-----------+ +-------+ |request?| 221 +-------+ +--------+ 223 3. Test configuration 225 One metric implementation used was NetProbe version 5.8.5, (an 226 earlier version is used in the WIPM system and deployed world-wide). 227 NetProbe uses UDP packets of variable size, and can produce test 228 streams with Periodic [RFC3432] or Poisson [RFC2330] sample 229 distributions. 231 The other metric implementation used was Perfas+ version 3.1, 232 developed by Deutsche Telekom. Perfas+ uses UDP unicast packets of 233 variable size (but supports also TCP and multicast). Test streams 234 with periodic, Poisson or uniform sample distributions may be used. 236 Figure 2 shows a view of the test path as each Implementation's test 237 flows pass through the Internet and the L2TPv3 tunnel IDs (1 and 2), 238 based on Figure 1 of [I-D.ietf-ippm-metrictest]. 240 +----+ +----+ +----+ +----+ 241 |Imp1| |Imp1| ,---. |Imp2| |Imp2| 242 +----+ +----+ / \ +-------+ +----+ +----+ 243 | V100 | V200 / \ | Tunnel| | V300 | V400 244 | | ( ) | Head | | | 245 +--------+ +------+ | |__| Router| +----------+ 246 |Ethernet| |Tunnel| |Internet | +---B---+ |Ethernet | 247 |Switch |--|Head |-| | | |Switch | 248 +-+--+---+ |Router| | | +---+---+--+--+--+----+ 249 |__| +--A---+ ( ) |Network| |__| 250 \ / |Emulat.| 251 U-turn \ / |"netem"| U-turn 252 V300 to V400 `-+-' +-------+ V100 to V200 254 Implementations ,---. +--------+ 255 +~~~~~~~~~~~/ \~~~~~~| Remote | 256 +------->-----F2->-| / \ |->---. | 257 | +---------+ | Tunnel ( ) | | | 258 | | transmit|-F1->-| ID 1 ( ) |->. | | 259 | | Imp 1 | +~~~~~~~~~| |~~~~| | | | 260 | | receive |-<--+ ( ) | F1 F2 | 261 | +---------+ | |Internet | | | | | 262 *-------<-----+ F1 | | | | | | 263 +---------+ | | +~~~~~~~~~| |~~~~| | | | 264 | transmit|-* *-| | | |<-* | | 265 | Imp 2 | | Tunnel ( ) | | | 266 | receive |-<-F2-| ID 2 \ / |<----* | 267 +---------+ +~~~~~~~~~~~\ /~~~~~~| Switch | 268 `-+-' +--------+ 270 Illustrations of a test setup with a bi-directional tunnel. The 271 upper diagram emphasizes the VLAN connectivity and geographical 272 location. The lower diagram shows example flows traveling between 273 two measurement implementations (for simplicity, only two flows are 274 shown). 276 Figure 1 278 The testing employs the Layer 2 Tunnel Protocol, version 3 (L2TPv3) 279 [RFC3931] tunnel between test sites on the Internet. The tunnel IP 280 and L2TPv3 headers are intended to conceal the test equipment 281 addresses and ports from hash functions that would tend to spread 282 different test streams across parallel network resources, with likely 283 variation in performance as a result. 285 At each end of the tunnel, one pair of VLANs encapsulated in the 286 tunnel are looped-back so that test traffic is returned to each test 287 site. Thus, test streams traverse the L2TP tunnel twice, but appear 288 to be one-way tests from the test equipment point of view. 290 The network emulator is a host running Fedora 14 Linux 291 [http://fedoraproject.org/] with IP forwarding enabled and the 292 "netem" Network emulator as part of the Fedora Kernel 2.6.35.11 [http 293 ://www.linuxfoundation.org/collaborate/workgroups/networking/netem] 294 loaded and operating. Connectivity across the netem/Fedora host was 295 accomplished by bridging Ethernet VLAN interfaces together with 296 "brctl" commands (e.g., eth1.100 <-> eth2.100). The netem emulator 297 was activated on one interface (eth1) and only operates on test 298 streams traveling in one direction. In some tests, independent netem 299 instances operated separately on each VLAN. 301 The links between the netem emulator host and router and switch were 302 found to be 100baseTx-HD (100Mbps half duplex) as reported by "mii- 303 tool"when the testing was complete. Use of Half Duplex was not 304 intended, but probably added a small amount of delay variation that 305 could have been avoided in full duplex mode. 307 Each individual test was run with common packet rates (1 pps, 10pps) 308 Poisson/Periodic distributions, and IP packet sizes of 64, 340, and 309 500 Bytes. 311 For these tests, a stream of at least 300 packets were sent from 312 Source to Destination in each implementation. Periodic streams (as 313 per [RFC3432]) with 1 second spacing were used, except as noted. 315 With the L2TPv3 tunnel in use, the metric name for the testing 316 configured here (with respect to the IP header exposed to Internet 317 processing) is: 319 Type-IP-protocol-115-One-way-Delay--Stream 321 With (Section 4.2. [RFC2679]) Metric Parameters: 323 + Src, the IP address of a host (12.3.167.16 or 193.159.144.8) 325 + Dst, the IP address of a host (193.159.144.8 or 12.3.167.16) 327 + T0, a time 329 + Tf, a time 331 + lambda, a rate in reciprocal seconds 333 + Thresh, a maximum waiting time in seconds (see Section 3.8.2 of 335 [RFC2679]) And (Section 4.3. [RFC2679]) 337 Metric Units: A sequence of pairs; the elements of each pair are: 339 + T, a time, and 341 + dT, either a real number or an undefined number of seconds. 343 The values of T in the sequence are monotonic increasing. Note that 344 T would be a valid parameter to Type-P-One-way-Delay, and that dT 345 would be a valid value of Type-P-One-way-Delay. 347 Also, Section 3.8.4 of [RFC2679] recommends that the path SHOULD be 348 reported. In this test set-up, most of the path details will be 349 concealed from the implementations by the L2TPv3 tunnels, thus a more 350 informative path trace route can be conducted by the routers at each 351 location. 353 When NetProbe is used in production, a traceroute is conducted in 354 parallel with, and at the outset of measurements. 356 Perfas+ does not support traceroute. 358 IPLGW#traceroute 193.159.144.8 360 Type escape sequence to abort. 361 Tracing the route to 193.159.144.8 363 1 12.126.218.245 [AS 7018] 0 msec 0 msec 4 msec 364 2 cr84.n54ny.ip.att.net (12.123.2.158) [AS 7018] 4 msec 4 msec 365 cr83.n54ny.ip.att.net (12.123.2.26) [AS 7018] 4 msec 366 3 cr1.n54ny.ip.att.net (12.122.105.49) [AS 7018] 4 msec 367 cr2.n54ny.ip.att.net (12.122.115.93) [AS 7018] 0 msec 368 cr1.n54ny.ip.att.net (12.122.105.49) [AS 7018] 0 msec 369 4 n54ny02jt.ip.att.net (12.122.80.225) [AS 7018] 4 msec 0 msec 370 n54ny02jt.ip.att.net (12.122.80.237) [AS 7018] 4 msec 371 5 192.205.34.182 [AS 7018] 0 msec 372 192.205.34.150 [AS 7018] 0 msec 373 192.205.34.182 [AS 7018] 4 msec 374 6 da-rg12-i.DA.DE.NET.DTAG.DE (62.154.1.30) [AS 3320] 88 msec 88 msec 375 88 msec 376 7 217.89.29.62 [AS 3320] 88 msec 88 msec 88 msec 377 8 217.89.29.55 [AS 3320] 88 msec 88 msec 88 msec 378 9 * * * 380 It was only possible to conduct the traceroute for the measured path 381 on one of the tunnel-head routers (the normal trace facilities of the 382 measurement systems are confounded by the L2TPv3 tunnel 383 encapsulation). 385 4. Error Calibration, RFC 2679 387 An implementation is required to report on its error calibration in 388 Section 3.8 of [RFC2679] (also required in Section 4.8 for sample 389 metrics). Sections 3.6, 3.7, and 3.8 of [RFC2679] give the detailed 390 formulation of the errors and uncertainties for calibration. In 391 summary, Section 3.7.1 of [RFC2679] describes the total time-varying 392 uncertainty as: 394 Esynch(t)+ Rsource + Rdest 396 where: 398 Esynch(t) denotes an upper bound on the magnitude of clock 399 synchronization uncertainty. 401 Rsource and Rdest denote the resolution of the source clock and the 402 destination clock, respectively. 404 Further, Section 3.7.2 of [RFC2679] describes the total wire-time 405 uncertainty as 407 Hsource + Hdest 409 referring to the upper bounds on host-time to wire-time for source 410 and destination, respectively. 412 Section 3.7.3 of [RFC2679] describes a test with small packets over 413 an isolated minimal network where the results can be used to estimate 414 systematic and random components of the sum of the above errors or 415 uncertainties. In a test with hundreds of singletons, the median is 416 the systematic error and when the median is subtracted from all 417 singletons, the remaining variability is the random error. 419 The test context, or Type-P of the test packets, must also be 420 reported, as required in Section 3.8 of [RFC2679] and all metrics 421 defined there. Type-P is defined in Section 13 of [RFC2330] (as are 422 many terms used below). 424 4.1. NetProbe Error and Type-P 426 Type-P for this test was IP-UDP with Best Effort DCSP. These headers 427 were encapsulated according to the L2TPv3 specifications [RFC3931], 428 and thus may not influence the treatment received as the packets 429 traversed the Internet. 431 In general, NetProbe error is dependent on the specific version and 432 installation details. 434 NetProbe operates using host time above the UDP layer, which is 435 different from the wire-time preferred in [RFC2330], but can be 436 identified as a source of error according to Section 3.7.2 of 437 [RFC2679]. 439 Accuracy of NetProbe measurements is usually limited by NTP 440 synchronization performance (which is typically taken as ~+/-1ms 441 error or greater), although the installation used in this testing 442 often exhibits errors much less than typical for NTP. The primary 443 stratum 1 NTP server is closely located on a sparsely utilized 444 network management LAN, thus it avoids many concerns raised in 445 Section 10 of[RFC2330] (in fact, smooth adjustment, long-term drift 446 analysis and compensation, and infrequent adjustment all lead to 447 stability during measurement intervals, the main concern). 449 The resolution of the reported results is 1us (us = microsecond) in 450 the version of NetProbe tested here, which contributes to at least 451 +/-1us error. 453 NetProbe implements a time-keeping sanity check on sending and 454 receiving time-stamping processes. When the significant process 455 interruption takes place, individual test packets are flagged as 456 possibly containing unusual time errors, and are excluded from the 457 sample used for all "time" metrics. 459 We performed a NetProbe calibration of the type described in Section 460 3.7.3 of [RFC2679], using 64 Byte packets over a cross-connect cable. 461 The results estimate systematic and random components of the sum of 462 the Hsource + Hdest errors or uncertainties. In a test with 300 463 singletons conducted over 30 seconds (periodic sample with 100ms 464 spacing), the median is the systematic error and the remaining 465 variability is the random error. One set of results is tabulated 466 below: 468 (Results from the "R" software environment for statistical computing 469 and graphics - http://www.r-project.org/ ) 470 > summary(XD4CAL) 471 CAL1 CAL2 CAL3 472 Min. : 89.0 Min. : 68.00 Min. : 54.00 473 1st Qu.: 99.0 1st Qu.: 77.00 1st Qu.: 63.00 474 Median :110.0 Median : 79.00 Median : 65.00 475 Mean :116.8 Mean : 83.74 Mean : 69.65 476 3rd Qu.:127.0 3rd Qu.: 88.00 3rd Qu.: 74.00 477 Max. :205.0 Max. :177.00 Max. :163.00 478 > 479 NetProbe Calibration with Cross-Connect Cable, one-way delay values 480 in microseconds (us) 482 The median or systematic error can be as high as 110 us, and the 483 range of the random error is also on the order of 116 us for all 484 streams. 486 Also, anticipating the Anderson-Darling K-sample (ADK) [ADK] 487 comparisons to follow, we corrected the CAL2 values for the 488 difference between means between CAL2 and CAL3 (as specified in 489 [I-D.ietf-ippm-metrictest]), and found strong support for the (Null 490 Hypothesis that) the samples are from the same distribution 491 (resolution of 1 us and alpha equal 0.05 and 0.01) 492 > XD4CVCAL2 <- XD4CAL$CAL2 - (mean(XD4CAL$CAL2)-mean(XD4CAL$CAL3)) 493 > boxplot(XD4CVCAL2,XD4CAL$CAL3) 494 > XD4CV2_ADK <- adk.test(XD4CVCAL2, XD4CAL$CAL3) 495 > XD4CV2_ADK 496 Anderson-Darling k-sample test. 498 Number of samples: 2 499 Sample sizes: 300 300 500 Total number of values: 600 501 Number of unique values: 97 503 Mean of Anderson Darling Criterion: 1 504 Standard deviation of Anderson Darling Criterion: 0.75896 506 T = (Anderson Darling Criterion - mean)/sigma 508 Null Hypothesis: All samples come from a common population. 510 t.obs P-value extrapolation 511 not adj. for ties 0.71734 0.17042 0 512 adj. for ties -0.39553 0.44589 1 513 > 514 using [Rtool] and [Radk]. 516 4.2. Perfas Error and Type-P 518 Perfas+ is configured to use GPS synchronisation and uses NTP 519 synchronization as a fall-back or default. GPS synchronisation 520 worked throughout this test with the exception of the calibration 521 stated here (one implementation was NTP synchronised only). The time 522 stamp accuracy typically is 0.1 ms. 524 The resolution of the results reported by Perfas+ is 1us (us = 525 microsecond) in the version tested here, which contributes to at 526 least +/-1us error. 528 Port 5001 5002 5003 529 Min. -227 -226 294 530 Median -169 -167 323 531 Mean -159 -157 335 532 Max. 6 -52 376 533 s 102 102 93 534 Perfas Calibration with Cross-Connect Cable, one-way delay values in 535 microseconds (us) 537 The median or systematic error can be as high as 323 us, and the 538 range of the random error is also less than 232 us for all streams. 540 5. Pre-determined Limits on Equivalence 542 In this section, we provide the numerical limits on comparisons 543 between implementations, in order to declare that the results are 544 equivalent and therefore, the tested specification is clear. 546 A key point is that the allowable errors, corrections, and confidence 547 levels only need to be sufficient to detect mis-interpretation of the 548 tested specification resulting in diverging implementations. 550 Also, the allowable error must be sufficient to compensate for 551 measured path differences. It was simply not possible to measure 552 fully identical paths in the VLAN-loopback test configuration used, 553 and this practical compromise must be taken into account. 555 For Anderson-Darling K-sample (ADK) comparisons, the required 556 confidence factor for the cross-implementation comparisons SHALL be 557 the smallest of: 559 o 0.95 confidence factor at 1ms resolution, or 561 o the smallest confidence factor (in combination with resolution) of 562 the two same-implementation comparisons for the same test 563 conditions. 565 A constant time accuracy error of as much as +/-0.5ms MAY be removed 566 from one implementation's distributions (all singletons) before the 567 ADK comparison is conducted. 569 A constant propagation delay error (due to use of different sub-nets 570 between the switch and measurement devices at each location) of as 571 much as +2ms MAY be removed from one implementation's distributions 572 (all singletons) before the ADK comparison is conducted. 574 For comparisons involving the mean of a sample or other central 575 statistics, the limits on both the time accuracy error and the 576 propagation delay error constants given above also apply. 578 6. Tests to evaluate RFC 2679 Specifications 580 This section describes some results from real-world (cross-Internet) 581 tests with measurement devices implementing IPPM metrics and a 582 network emulator to create relevant conditions, to determine whether 583 the metric definitions were interpreted consistently by implementors. 585 The procedures are slightly modified from the original procedures 586 contained in Appendix A.1 of [I-D.ietf-ippm-metrictest]. The 587 modifications include the use of the mean statistic for comparisons. 589 Note that there are only five instances of the requirement term 590 "MUST" in [RFC2679] outside of the boilerplate and [RFC2119] 591 reference. 593 6.1. One-way Delay, ADK Sample Comparison - Same & Cross Implementation 595 This test determines if implementations produce results that appear 596 to come from a common delay distribution, as an overall evaluation of 597 Section 4 of [RFC2679], "A Definition for Samples of One-way Delay". 598 Same-implementation comparison results help to set the threshold of 599 equivalence that will be applied to cross-implementation comparisons. 601 This test is intended to evaluate measurements in sections 3 and 4 of 602 [RFC2679]. 604 By testing the extent to which the distributions of one-way delay 605 singletons from two implementations of [RFC2679] appear to be from 606 the same distribution, we economize on comparisons, because comparing 607 a set of individual summary statistics (as defined in Section 5 of 608 [RFC2679]) would require another set of individual evaluations of 609 equivalence. Instead, we can simply check which statistics were 610 implemented, and report on those facts. 612 1. Configure an L2TPv3 path between test sites, and each pair of 613 measurement devices to operate tests in their designated pair of 614 VLANs. 616 2. Measure a sample of one-way delay singletons with 2 or more 617 implementations, using identical options and network emulator 618 settings (if used). 620 3. Measure a sample of one-way delay singletons with *four* 621 instances of the *same* implementations, using identical options, 622 noting that connectivity differences SHOULD be the same as for 623 the cross implementation testing. 625 4. Apply the ADK comparison procedures (see Appendix C of 626 [I-D.ietf-ippm-metrictest]) and determine the resolution and 627 confidence factor for distribution equivalence of each same- 628 implementation comparison and each cross-implementation 629 comparison. 631 5. Take the coarsest resolution and confidence factor for 632 distribution equivalence from the same-implementation pairs, or 633 the limit defined in Section 5 above, as a limit on the 634 equivalence threshold for these experimental conditions. 636 6. Apply constant correction factors to all singletons of the sample 637 distributions, as described and limited in Section 5 above. 639 7. Compare the cross-implementation ADK performance with the 640 equivalence threshold determined in step 5 to determine if 641 equivalence can be declared. 643 The common parameters used for tests in this section are: 645 o IP header + payload = 64 octets 647 o Periodic sampling at 1 packet per second 649 o Test duration = 300 seconds (March 29) 651 The netem emulator was set for 100ms average delay, with uniform 652 delay variation of +/-50ms. In this experiment, the netem emulator 653 was configured to operate independently on each VLAN and thus the 654 emulator itself is a potential source of error when comparing streams 655 that traverse the test path in different directions. 657 In the result analysis of this section: 659 o All comparisons used 1 microsecond resolution. 661 o No Correction Factors were applied. 663 o The 0.95 confidence factor (1.960 for paired stream comparison) 664 was used. 666 6.1.1. NetProbe Same-implementation results 668 A single same-implementation comparison fails the ADK criterion (s1 669 <-> sB). We note that these streams traversed the test path in 670 opposite directions, making the live network factors a possibility to 671 explain the difference. 673 All other pair comparisons pass the ADK criterion. 675 +------------------------------------------------------+ 676 | | | | | 677 | ti.obs (P) | s1 | s2 | sA | 678 | | | | | 679 .............|.............|.............|.............| 680 | | | | | 681 | s2 | 0.25 (0.28) | | | 682 | | | | | 683 ...........................|.............|.............| 684 | | | | | 685 | sA | 0.60 (0.19) |-0.80 (0.57) | | 686 | | | | | 687 ...........................|.............|.............| 688 | | | | | 689 | sB | 2.64 (0.03) | 0.07 (0.31) |-0.52 (0.48) | 690 | | | | | 691 +------------+-------------+-------------+-------------+ 693 NetProbe ADK Results for same-implementation 695 6.1.2. Perfas Same-implementation results 697 All pair comparisons pass the ADK criterion. 699 +------------------------------------------------------+ 700 | | | | | 701 | ti.obs (P) | p1 | p2 | p3 | 702 | | | | | 703 .............|.............|.............|.............| 704 | | | | | 705 | p2 | 0.06 (0.32) | | | 706 | | | | | 707 .........................................|.............| 708 | | | | | 709 | p3 | 1.09 (0.12) | 0.37 (0.24) | | 710 | | | | | 711 ...........................|.............|.............| 712 | | | | | 713 | p4 |-0.81 (0.57) |-0.13 (0.37) | 1.36 (0.09) | 714 | | | | | 715 +------------+-------------+-------------+-------------+ 717 Perfas ADK Results for same-implementation 719 6.1.3. One-way Delay, Cross-Implementation ADK Comparison 721 The cross-implementation results are compared using a combined ADK 722 analysis [Radk], where all NetProbe results are compared with all 723 Perfas results after testing that the combined same-implementation 724 results pass the ADK criterion. 726 When 4 (same) samples are compared, the ADK criterion for 0.95 727 confidence is 1.915, and when all 8 (cross) samples are compared it 728 is 1.85. 730 Combination of Anderson-Darling K-Sample Tests. 732 Sample sizes within each data set: 733 Data set 1 : 299 297 298 300 (NetProbe) 734 Data set 2 : 300 300 298 300 (Perfas) 735 Total sample size per data set: 1194 1198 736 Number of unique values per data set: 1188 1192 737 ... 738 Null Hypothesis: 739 All samples within a data set come from a common distribution. 740 The common distribution may change between data sets. 742 NetProbe ti.obs P-value extrapolation 743 not adj. for ties 0.64999 0.21355 0 744 adj. for ties 0.64833 0.21392 0 745 Perfas 746 not adj. for ties 0.55968 0.23442 0 747 adj. for ties 0.55840 0.23473 0 749 Combined Anderson-Darling Criterion: 750 tc.obs P-value extrapolation 751 not adj. for ties 0.85537 0.17967 0 752 adj. for ties 0.85329 0.18010 0 754 The combined same-implementation samples and the combined cross- 755 implementation comparison all pass the ADK criteria at P>=0.18 and 756 support the Null Hypothesis (both data sets come from a common 757 distribution). 759 We also see that the paired ADK comparisons are rather critical. 760 Although the NetProbe s1-sB comparison failed, the combined data set 761 from 4 streams passed the ADK criterion easily. 763 6.1.4. Conclusions on the ADK Results for One-way Delay 765 Similar testing was repeated many times in the months of March and 766 April 2011. There were many experiments where a single test stream 767 from NetProbe or Perfas proved to be different from the others in 768 paired comparisons (even same comparisons). When the out lier stream 769 was removed from the comparison, the remaining streams passed 770 combined ADK criterion. Also, the application of correction factors 771 resulted in higher comparison success. 773 We conclude that the two implementations are capable of producing 774 equivalent one-way delay distributions based on their interpretation 775 of [RFC2679] . 777 6.1.5. Additional Investigations 779 On the final day of testing, we performed a series of measurements to 780 evaluate the amount of emulated delay variation necessary to achieve 781 successful ADK comparisons. The need for Correction factors (as 782 permitted by Section 5) and the size of the measurement sample 783 (obtained as sub-sets of the complete measurement sample) were also 784 evaluated. 786 The common parameters used for tests in this section are: 788 o IP header + payload = 64 octets 790 o Periodic sampling at 1 packet per second 792 o Test duration = 300 seconds at each delay variation setting, for a 793 total of 1200 seconds (May 2, 2011 at 1720 UTC) 795 The netem emulator was set for 100ms average delay, with (emulated) 796 uniform delay variation of: 798 o +/-7.5 ms 800 o +/-5.0 ms 802 o +/-2.5 ms 804 o 0 ms 806 In this experiment, the netem emulator was configured to operate 807 independently on each VLAN and thus the emulator itself is a 808 potential source of error when comparing streams that traverse the 809 test path in different directions. 811 In the result analysis of this section: 813 o All comparisons used 1 microsecond resolution. 815 o Correction Factors *were* applied as noted (under column heading 816 "mean adj"). The difference between each sample mean and the 817 lowest mean of the NetProbe or Perfas stream samples was 818 subtracted from all values in the sample. ("raw" indicates no 819 correction factors were used.) 821 o The 0.95 confidence factor (1.960 for paired stream comparison) 822 was used. 824 When 8 (cross) samples are compared, the ADK criterion for 0.95 825 confidence is 1.85. The Combined ADK test statistic ("TC observed") 826 must be less than 1.85 to accept the Null Hypothesis (all samples in 827 the data set are from a common distribution). 829 012345678901234567890123456789012345678901234567890123456789012345678901 830 Emulated Delay Sub-Sample size 831 Variation 0ms 832 adk.combined (all) 300 values 75 values 833 Adj. for ties raw mean adj raw mean adj 834 TC observed 226.6563 67.51559 54.01359 21.56513 835 P-value 0 0 0 0 836 Mean std dev (all),us 719 635 837 Mean diff of means,us 649 0 606 0 839 Variation +/- 2.5ms 840 adk.combined (all) 300 values 75 values 841 Adj. for ties raw mean adj raw mean adj 842 TC observed 14.50436 -1.60196 3.15935 -1.72104 843 P-value 0 0.873 0.00799 0.89038 844 Mean std dev (all),us 1655 1702 845 Mean diff of means,us 471 0 513 0 847 Variation +/- 5ms 848 adk.combined (all) 300 values 75 values 849 Adj. for ties raw mean adj raw mean adj 850 TC observed 8.29921 -1.28927 0.37878 -1.81881 851 P-value 0 0.81601 0.29984 0.90305 852 Mean std dev (all),us 3023 2991 853 Mean diff of means,us 582 0 513 0 855 Variation +/- 7.5ms 856 adk.combined (all) 300 values 75 values 857 Adj. for ties raw mean adj raw mean adj 858 TC observed 2.53759 -0.72985 0.29241 -1.15840 859 P-value 0.01950 0.66942 0.32585 0.78686 860 Mean std dev (all),us 4449 4506 861 Mean diff of means,us 426 0 856 0 863 From the table above, we conclude the following: 865 1. None of the raw or mean adjusted results pass the ADK criterion 866 with 0 ms emulated delay variation. Use of the 75 value sub- 867 sample yielded the same conclusion. (We note the same results 868 when comparing same implementation samples for both NetProbe and 869 Perfas.) 871 2. When the smallest emulated delay variation was inserted 872 (+/-2.5ms), the mean adjusted samples pass the ADK criterion and 873 the high P-value supports the result. The raw results do not 874 pass. 876 3. At higher values of emulated delay variation (+/-5.0ms and 877 +/-7.5ms), again the mean adjusted values pass ADK. We also see 878 that the 75-value sub-sample passed the ADK in both raw and mean 879 adjusted cases. This indicates that sample size may have played 880 a role in our results, as noted in the Appendix of [RFC2680] for 881 Goodness-of-Fit testing. 883 We note that 150 value sub-samples were also evaluated, with ADK 884 conclusions that followed the results for 300 values. Also, same- 885 implementation analysis was conducted with results similar to the 886 above, except that more of the "raw" or uncorrected samples passed 887 the ADK criterion. 889 >>>> To be provided: 891 >>>> Overall statement about Correction Factors w.r.t. section 5 892 limits. 894 >>>> Appendix with more details ??? 896 6.2. One-way Delay, Loss threshold, RFC 2679 898 This test determines if implementations use the same configured 899 maximum waiting time delay from one measurement to another under 900 different delay conditions, and correctly declare packets arriving in 901 excess of the waiting time threshold as lost. 903 See Section 3.5 of [RFC2679], 3rd bullet point and also Section 3.8.2 904 of [RFC2679]. 906 1. configure an L2TPv3 path between test sites, and each pair of 907 measurement devices to operate tests in their designated pair of 908 VLANs. 910 2. configure the network emulator to add 1.0 sec one-way constant 911 delay in one direction of transmission. 913 3. measure (average) one-way delay with 2 or more implementations, 914 using identical waiting time thresholds (Thresh) for loss set at 915 3 seconds. 917 4. configure the network emulator to add 3 sec one-way constant 918 delay in one direction of transmission equivalent to 2 seconds of 919 additional one-way delay (or change the path delay while test is 920 in progress, when there are sufficient packets at the first delay 921 setting) 923 5. repeat/continue measurements 925 6. observe that the increase measured in step 5 caused all packets 926 with 2 sec additional delay to be declared lost, and that all 927 packets that arrive successfully in step 3 are assigned a valid 928 one-way delay. 930 The common parameters used for tests in this section are: 932 o IP header + payload = 64 octets 934 o Poisson sampling at lambda = 1 packet per second 936 o Test duration = 900 seconds total (March 21) 938 The netem emulator was set to add constant delays as specified in the 939 procedure above. 941 6.2.1. NetProbe results for Loss Threshold 943 In NetProbe, the Loss Threshold is implemented uniformly over all 944 packets as a post-processing routine. With the Loss Threshold set at 945 3 seconds, all packets with one-way delay >3 seconds are marked 946 "Lost" and included in the Lost Packet list with their transmission 947 time (as required in Section 3.3 of [RFC2680]). This resulted in 342 948 packets designated as lost in one of the test streams (with average 949 delay = 3.091 sec). 951 6.2.2. Perfas Results for Loss Threshold 953 Perfas uses a fixed Loss Threshold which was not adjustable during 954 this study. The Loss Threshold is approximately one minute, and 955 emulation of a delay of this size was not attempted. However, it is 956 possible to implement any delay threshold desired with a post- 957 processing routine and subsequent analysis. Using this method, 195 958 packets would be declared lost (with average delay = 3.091 sec). 960 6.2.3. Conclusions for Loss Threshold 962 Both implementations assume that any constant delay value desired can 963 be used as the Loss Threshold, since all delays are stored as a pair 964 as required in [RFC2679] . This is a simple way to 965 enforce the constant loss threshold envisioned in [RFC2679] (see 966 specific section references above). We take the position that the 967 assumption of post-processing is compliant, and that the text of the 968 RFC should be revised slightly to include this point. 970 6.3. One-way Delay, First-bit to Last bit, RFC 2679 972 This test determines if implementations register the same relative 973 change in delay from one packet size to another, indicating that the 974 first-to-last time-stamping convention has been followed. This test 975 tends to cancel the sources of error which may be present in an 976 implementation. 978 See Section 3.7.2 of [RFC2679], and Section 10.2 of [RFC2330]. 980 1. configure an L2TPv3 path between test sites, and each pair of 981 measurement devices to operate tests in their designated pair of 982 VLANs, and ideally including a low-speed link (it was not 983 possible to change the link configuration during testing, so the 984 lowest speed link present was the basis for serialization time 985 comparisons). 987 2. measure (average) one-way delay with 2 or more implementations, 988 using identical options and equal size small packets (64 octet IP 989 header and payload) 991 3. maintain the same path with additional emulated 100 ms one-way 992 delay 994 4. measure (average) one-way delay with 2 or more implementations, 995 using identical options and equal size large packets (500 octet 996 IP header and payload) 998 5. observe that the increase measured between steps 2 and 4 is 999 equivalent to the increase in ms expected due to the larger 1000 serialization time for each implementation. Most of the 1001 measurement errors in each system should cancel, if they are 1002 stationary. 1004 The common parameters used for tests in this section are: 1006 o IP header + payload = 64 octets 1008 o Periodic sampling at l packet per second 1010 o Test duration = 300 seconds total (April 12) 1012 The netem emulator was set to add constant 100ms delay. 1014 6.3.1. NetProbe and Perfas Results for Serialization 1016 When the IP header + payload size was increased from 64 octets to 500 1017 octets, there was a delay increase observed. 1019 Mean Delays in us 1020 NetProbe 1021 Payload s1 s2 sA sB 1022 500 190893 191179 190892 190971 1023 64 189642 189785 189747 189467 1024 Diff 1251 1394 1145 1505 1026 Perfas 1027 Payload p1 p2 p3 p4 1028 500 190908 190911 191126 190709 1029 64 189706 189752 189763 190220 1030 Diff 1202 1159 1363 489 1032 Serialization tests, all values in microseconds 1034 The typical delay increase when the larger packets were used was 1.1 1035 to 1.5 ms (with one outlier). The typical measurements indicate that 1036 a link with approximately 3 Mbit/s capacity is present on the path. 1038 Through investigation of the facilities involved, it was determined 1039 that the lowest speed link was approximately 45 Mbit/s, and therefore 1040 the estimated difference should be about 0.077 ms. The observed 1041 differences are much higher. 1043 The unexpected large delay difference was also the outcome when 1044 testing serialization times in a lab environment, using the NIST Net 1045 Emulator and NetProbe [ref to earlier lab tests]. 1047 6.3.2. Conclusions for Serialization 1049 Since it was not possible to confirm the estimated serialization time 1050 increases in field tests, we resort to examination of the 1051 implementations to determine compliance. 1053 NetProbe performs all time stamping above the IP-layer, accepting 1054 that some compromises must be made to achieve extreme portability and 1055 measurement scale. Therefore, the first-to-last bit convention is 1056 supported because the serialization time is included in the one-way 1057 delay measurement, enabling comparison with other implementations. 1059 Perfas is optimized for its purpose and performs all time stamping 1060 close to the interface hardware. The first-to-last bit convention is 1061 supported because the serialization time is included in the one-way 1062 delay measurement, enabling comparison with other implementations. 1064 6.4. One-way Delay, Difference Sample Metric (Lab) 1066 This test determines if implementations register the same relative 1067 increase in delay from one measurement to another under different 1068 delay conditions. This test tends to cancel the sources of error 1069 which may be present in an implementation. 1071 This test is intended to evaluate measurements in sections 3 and 4 of 1072 [RFC2679]. 1074 1. configure an L2TPv3 path between test sites, and each pair of 1075 measurement devices to operate tests in their designated pair of 1076 VLANs. 1078 2. measure (average) one-way delay with 2 or more implementations, 1079 using identical options 1081 3. configure the path with X+Y ms one-way delay 1083 4. repeat measurements 1085 5. observe that the (average) increase measured in steps 2 and 4 is 1086 ~Y ms for each implementation. Most of the measurement errors in 1087 each system should cancel, if they are stationary. 1089 In this test, X=1000ms and Y=1000ms. 1091 The common parameters used for tests in this section are: 1093 o IP header + payload = 64 octets 1095 o Poisson sampling at lambda = 1 packet per second 1097 o Test duration = 900 seconds total (March 21) 1099 The netem emulator was set to add constant delays as specified in the 1100 procedure above. 1102 6.4.1. NetProbe results for Differential Delay 1104 Average pre-increase delay, microseconds 1089868.0 1105 Average post 1s additional, microseconds 2089686.0 1106 Difference (should be ~= Y = 1s) 999818.0 1108 Average delays before/after 1 second increase 1110 The NetProbe implementation observed a 1 second increase with a 182 1111 microsecond error (assuming that the netem emulated delay difference 1112 is exact). 1114 We note that this differential delay test has been run under lab 1115 conditions and published in prior work [ref to "advance metrics" 1116 draft]. The error was 6 microseconds. 1118 6.4.2. Perfas results for Differential Delay 1120 Average pre-increase delay, microseconds 1089794.0 1121 Average post 1s additional, microseconds 2089801.0 1122 Difference (should be ~= Y = 1s) 1000007.0 1124 Average delays before/after 1 second increase 1126 The Perfas implementation observed a 1 second increase with a 7 1127 microsecond error. 1129 6.4.3. Conclusions for Differential Delay 1131 Again, the live network conditions appear to have influenced the 1132 results, but both implementations measured the same delay increase 1133 within their calibration accuracy. 1135 6.5. Implementation of Statistics for One-way Delay 1137 The ADK tests the extent to which the sample distributions of one-way 1138 delay singletons from two implementations of [RFC2679] appear to be 1139 from the same overall distribution. By testing this way, we 1140 economize on the number of comparisons, because comparing a set of 1141 individual summary statistics (as defined in Section 5 of [RFC2679]) 1142 would require another set of individual evaluations of equivalence. 1143 Instead, we can simply check which statistics were implemented, and 1144 report on those facts, noting that Section 5 of [RFC2679] does not 1145 specify the calculations exactly, and gives only some illustrative 1146 examples. 1148 NetProbe Perfas 1150 5.1. Type-P-One-way-Delay-Percentile yes no 1152 5.2. Type-P-One-way-Delay-Median yes no 1154 5.3. Type-P-One-way-Delay-Minimum yes yes 1156 5.4. Type-P-One-way-Delay-Inverse-Percentile no no 1158 Implementation of Section 5 Statistics 1160 Only the Type-P-One-way-Delay-Inverse-Percentile has been ignored in 1161 both implementations, so it is a candidate for removal in RFC2679bis. 1163 7. Security Considerations 1165 The security considerations that apply to any active measurement of 1166 live networks are relevant here as well. See [RFC4656] and 1167 [RFC5357]. 1169 8. IANA Considerations 1171 This memo makes no requests of IANA, and hopes that IANA will be as 1172 accepting of our new computer overlords as the authors intend to be. 1174 9. Acknowledgements 1176 The authors thank Lars Eggert for his continued encouragement to 1177 advance the IPPM metrics during his tenure as AD Advisor. 1179 Nicole Kowalski supplied the needed CPE router for the NetProbe side 1180 of the test set-up, and graciously managed her testing in spite of 1181 issues caused by dual-use of the router. Thanks Nicole! 1183 The "NetProbe Team" also acknowledges many useful discussions with 1184 Ganga Maguluri. 1186 10. References 1187 10.1. Normative References 1189 [I-D.ietf-ippm-metrictest] 1190 Geib, R., Morton, A., Fardid, R., and A. Steinmitz, "IPPM 1191 standard advancement testing", 1192 draft-ietf-ippm-metrictest-05 (work in progress), 1193 November 2011. 1195 [RFC2026] Bradner, S., "The Internet Standards Process -- Revision 1196 3", BCP 9, RFC 2026, October 1996. 1198 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1199 Requirement Levels", BCP 14, RFC 2119, March 1997. 1201 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 1202 "Framework for IP Performance Metrics", RFC 2330, 1203 May 1998. 1205 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1206 Delay Metric for IPPM", RFC 2679, September 1999. 1208 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1209 Packet Loss Metric for IPPM", RFC 2680, September 1999. 1211 [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network 1212 performance measurement with periodic streams", RFC 3432, 1213 November 2002. 1215 [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. 1216 Zekauskas, "A One-way Active Measurement Protocol 1217 (OWAMP)", RFC 4656, September 2006. 1219 [RFC4814] Newman, D. and T. Player, "Hash and Stuffing: Overlooked 1220 Factors in Network Device Benchmarking", RFC 4814, 1221 March 2007. 1223 [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an 1224 IANA Considerations Section in RFCs", BCP 26, RFC 5226, 1225 May 2008. 1227 [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. 1228 Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", 1229 RFC 5357, October 2008. 1231 [RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation 1232 and Implementation Reports for Advancement to Draft 1233 Standard", BCP 9, RFC 5657, September 2009. 1235 10.2. Informative References 1237 [ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling 1238 Tests of fit, for continuous and discrete cases", 1239 University of Washington, Technical Report No. 81, 1240 May 1986. 1242 [I-D.morton-ippm-advance-metrics] 1243 Morton, A., "Lab Test Results for Advancing Metrics on the 1244 Standards Track", draft-morton-ippm-advance-metrics-02 1245 (work in progress), October 2010. 1247 [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling 1248 Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. 1250 [Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and 1251 Combinations of Such Tests. R package version 1.0.", , 1252 2008. 1254 [Rtool] R Development Core Team, "R: A language and environment 1255 for statistical computing. R Foundation for Statistical 1256 Computing, Vienna, Austria. ISBN 3-900051-07-0, URL 1257 http://www.R-project.org/", , 2011. 1259 Authors' Addresses 1261 Len Ciavattone 1262 AT&T Labs 1263 200 Laurel Avenue South 1264 Middletown, NJ 07748 1265 USA 1267 Phone: +1 732 420 1239 1268 Fax: 1269 Email: lencia@att.com 1270 URI: 1272 Ruediger Geib 1273 Deutsche Telekom 1274 Heinrich Hertz Str. 3-7 1275 Darmstadt, 64295 1276 Germany 1278 Phone: +49 6151 58 12747 1279 Email: Ruediger.Geib@telekom.de 1280 Al Morton 1281 AT&T Labs 1282 200 Laurel Avenue South 1283 Middletown, NJ 07748 1284 USA 1286 Phone: +1 732 420 1571 1287 Fax: +1 732 368 1192 1288 Email: acmorton@att.com 1289 URI: http://home.comcast.net/~acmacm/ 1291 Matthias Wieser 1292 Technical University Darmstadt 1293 Darmstadt, 1294 Germany 1296 Phone: 1297 Email: matthias_michael.wieser@stud.tu-darmstadt.de