idnits 2.17.1 draft-ietf-ippm-testplan-rfc2680-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 7 instances of lines with non-RFC2606-compliant FQDNs in the document. == There are 11 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 840 has weird spacing: '...1] and pexp...' == Line 848 has weird spacing: '...1] and pexp...' == Line 856 has weird spacing: '...1] and pexp...' == Line 864 has weird spacing: '...1] and pexp...' == Line 906 has weird spacing: '...p1 and pexp...' == (5 more instances...) == The document seems to contain a disclaimer for pre-RFC5378 work, but was first submitted on or after 10 November 2008. The disclaimer is usually necessary only for documents that revise or obsolete older RFCs, and that take significant amounts of text from those RFCs. If you can contact all authors of the source material and they are willing to grant the BCP78 rights to the IETF Trust, you can and should remove the disclaimer. Otherwise, the disclaimer is needed and you can ignore this comment. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (April 3, 2014) is 3670 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'AS 7018' is mentioned on line 345, but not defined == Missing Reference: 'AS 3320' is mentioned on line 349, but not defined ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) == Outdated reference: A later version (-04) exists of draft-morton-ippm-2680-bis-02 Summary: 1 error (**), 0 flaws (~~), 13 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group L. Ciavattone 3 Internet-Draft AT&T Labs 4 Intended status: Informational R. Geib 5 Expires: October 5, 2014 Deutsche Telekom 6 A. Morton 7 AT&T Labs 8 M. Wieser 9 Technical University Darmstadt 10 April 3, 2014 12 Test Plan and Results for Advancing RFC 2680 on the Standards Track 13 draft-ietf-ippm-testplan-rfc2680-05 15 Abstract 17 This memo proposes to advance a performance metric RFC along the 18 standards track, specifically RFC 2680 on One-way Loss Metrics. 19 Observing that the metric definitions themselves should be the 20 primary focus rather than the implementations of metrics, this memo 21 describes the test procedures to evaluate specific metric requirement 22 clauses to determine if the requirement has been interpreted and 23 implemented as intended. Two completely independent implementations 24 have been tested against the key specifications of RFC 2680. 26 Requirements Language 28 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 29 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 30 document are to be interpreted as described in RFC 2119 [RFC2119]. 32 Status of This Memo 34 This Internet-Draft is submitted in full conformance with the 35 provisions of BCP 78 and BCP 79. 37 Internet-Drafts are working documents of the Internet Engineering 38 Task Force (IETF). Note that other groups may also distribute 39 working documents as Internet-Drafts. The list of current Internet- 40 Drafts is at http://datatracker.ietf.org/drafts/current/. 42 Internet-Drafts are draft documents valid for a maximum of six months 43 and may be updated, replaced, or obsoleted by other documents at any 44 time. It is inappropriate to use Internet-Drafts as reference 45 material or to cite them other than as "work in progress." 47 This Internet-Draft will expire on October 5, 2014. 49 Copyright Notice 51 Copyright (c) 2014 IETF Trust and the persons identified as the 52 document authors. All rights reserved. 54 This document is subject to BCP 78 and the IETF Trust's Legal 55 Provisions Relating to IETF Documents 56 (http://trustee.ietf.org/license-info) in effect on the date of 57 publication of this document. Please review these documents 58 carefully, as they describe your rights and restrictions with respect 59 to this document. Code Components extracted from this document must 60 include Simplified BSD License text as described in Section 4.e of 61 the Trust Legal Provisions and are provided without warranty as 62 described in the Simplified BSD License. 64 This document may contain material from IETF Documents or IETF 65 Contributions published or made publicly available before November 66 10, 2008. The person(s) controlling the copyright in some of this 67 material may not have granted the IETF Trust the right to allow 68 modifications of such material outside the IETF Standards Process. 69 Without obtaining an adequate license from the person(s) controlling 70 the copyright in such materials, this document may not be modified 71 outside the IETF Standards Process, and derivative works of it may 72 not be created outside the IETF Standards Process, except to format 73 it for publication as an RFC or to translate it into languages other 74 than English. 76 Table of Contents 78 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 79 1.1. RFC 2680 Coverage . . . . . . . . . . . . . . . . . . . . 4 80 2. A Definition-centric metric advancement process . . . . . . . 4 81 3. Test configuration . . . . . . . . . . . . . . . . . . . . . 4 82 4. Error Calibration, RFC 2680 . . . . . . . . . . . . . . . . . 9 83 4.1. Clock Synchronization Calibration . . . . . . . . . . . . 9 84 4.2. Packet Loss Determination Error . . . . . . . . . . . . . 10 85 5. Pre-determined Limits on Equivalence . . . . . . . . . . . . 10 86 6. Tests to evaluate RFC 2680 Specifications . . . . . . . . . . 11 87 6.1. One-way Loss, ADK Sample Comparison . . . . . . . . . . . 11 88 6.1.1. 340B/Periodic Cross-imp. results . . . . . . . . . . 12 89 6.1.2. 64B/Periodic Cross-imp. results . . . . . . . . . . . 13 90 6.1.3. 64B/Poisson Cross-imp. results . . . . . . . . . . . 14 91 6.1.4. Conclusions on the ADK Results for One-way Packet 92 Loss . . . . . . . . . . . . . . . . . . . . . . . . 15 93 6.2. One-way Loss, Delay threshold . . . . . . . . . . . . . . 15 94 6.2.1. NetProbe results for Loss Threshold . . . . . . . . . 16 95 6.2.2. Perfas Results for Loss Threshold . . . . . . . . . . 17 96 6.2.3. Conclusions for Loss Threshold . . . . . . . . . . . 17 98 6.3. One-way Loss with Out-of-Order Arrival . . . . . . . . . 17 99 6.4. Poisson Sending Process Evaluation . . . . . . . . . . . 18 100 6.4.1. NetProbe Results . . . . . . . . . . . . . . . . . . 19 101 6.4.2. Perfas+ Results . . . . . . . . . . . . . . . . . . . 20 102 6.4.3. Conclusions for Goodness-of-Fit . . . . . . . . . . . 22 103 6.5. Implementation of Statistics for One-way Loss . . . . . . 22 104 7. Conclusions for RFC 2680bis . . . . . . . . . . . . . . . . . 23 105 8. Security Considerations . . . . . . . . . . . . . . . . . . . 23 106 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 107 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 24 108 11. Appendix - Network Configuration and sample commands . . . . 24 109 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 27 110 12.1. Normative References . . . . . . . . . . . . . . . . . . 27 111 12.2. Informative References . . . . . . . . . . . . . . . . . 28 112 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 29 114 1. Introduction 116 The IETF (specifically the IP Performance Metrics working group, or 117 IPPM) has considered how to advance their metrics along the standards 118 track since 2001. 120 The renewed work effort sought to investigate ways in which the 121 measurement variability could be reduced and thereby simplify the 122 problem of comparison for equivalence. As a result, there is 123 consensus (captured in [RFC6576]) that equivalent results from 124 independent implementations of metric specifications are sufficient 125 evidence that the specifications themselves are clear and 126 unambiguous; it is the parallel concept of protocol interoperability 127 for metric specifications. The advancement process either produces 128 confidence that the metric definitions and supporting material are 129 clearly worded and unambiguous, OR, identifies ways in which the 130 metric definitions should be revised to achieve clarity. It is a 131 non-goal to compare the specific implementations themselves. 133 The process also permits identification of options described in the 134 metric RFC that were not implemented, so that they can be removed 135 from the advancing specification (this is an aspect more typical of 136 protocol advancement along the standards track). 138 This memo's purpose is to implement the current approach for 139 [RFC2680] and document the results. 141 In particular, this memo documents consensus on the extent of 142 tolerable errors when assessing equivalence in the results. In 143 discussions, the IPPM working group agreed that test plan and 144 procedures should include the threshold for determining equivalence, 145 and this information should be available in advance of cross- 146 implementation comparisons. This memo includes procedures for same- 147 implementation comparisons to help set the equivalence threshold. 149 Another aspect of the metric RFC advancement process is the 150 requirement to document the work and results. The procedures of 151 [RFC2026] are expanded in[RFC5657], including sample implementation 152 and interoperability reports. This memo follows the template in 153 [RFC6808] for the report that accompanies the protocol action request 154 submitted to the Area Director, including description of the test 155 set-up, procedures, results for each implementation, and conclusions. 157 The conclusion reached is that [RFC2680] should be advanced on the 158 Standards Track with modifications. The revised text of RFC 2680bis 159 is ready for review [I-D.morton-ippm-2680-bis], but awaits work-in 160 progress to update the IPPM Framework [RFC2330]. Therefore, this 161 memo documents the information to support [RFC2680] advancement, and 162 the approval of RFC2680bis is left for future action. 164 1.1. RFC 2680 Coverage 166 This plan is intended to cover all critical requirements and sections 167 of [RFC2680]. 169 Note that there are only five instances of the requirement term 170 "MUST" in [RFC2680] outside of the boilerplate and [RFC2119] 171 reference. 173 Material may be added as it is "discovered" (apparently, not all 174 requirements use requirements language). 176 2. A Definition-centric metric advancement process 178 The process described in Section 3.5 of [RFC6576] takes as a first 179 principle that the metric definitions, embodied in the text of the 180 RFCs, are the objects that require evaluation and possible revision 181 in order to advance to the next step on the standards track. This 182 memo follows that process. 184 3. Test configuration 186 One metric implementation used was NetProbe version 5.8.5 (an earlier 187 version is used in the WIPM system and deployed world-wide [WIPM]). 188 NetProbe uses UDP packets of variable size, and can produce test 189 streams with Periodic [RFC3432] or Poisson [RFC2330] sample 190 distributions. 192 The other metric implementation used was Perfas+ version 3.1, 193 developed by Deutsche Telekom [Perfas]. Perfas+ uses UDP unicast 194 packets of variable size (but also supports TCP and multicast). Test 195 streams with periodic, Poisson, or uniform sample distributions may 196 be used. 198 Figure 1 shows a view of the test path as each Implementation's test 199 flows pass through the Internet and the L2TPv3 tunnel IDs (1 and 2), 200 based on Figure 1 of [RFC6576]. 202 +------------+ +------------+ 203 | Imp 1 | ,---. | Imp 2 | 204 +------------+ / \ +-------+ +------------+ 205 | V100 ^ V200 / \ | Tunnel| | V300 ^ V400 206 | | ( ) | Head | | | 207 +--------+ +------+ | |__| Router| +----------+ 208 |Ethernet| |Tunnel| |Internet | +---B---+ |Ethernet | 209 |Switch |--|Head |-| | | |Switch | 210 +-+--+---+ |Router| | | +---+---+--+--+--+----+ 211 |__| +--A---+ ( ) |Network| |__| 212 \ / |Emulat.| 213 U-turn \ / |"netem"| U-turn 214 V300 to V400 `-+-' +-------+ V100 to V200 216 Implementations ,---. +--------+ 217 +~~~~~~~~~~~/ \~~~~~~| Remote | 218 +------->-----F2->-| / \ |->---. | 219 | +---------+ | Tunnel ( ) | | | 220 | | transmit|-F1->-| ID 1 | | |->. | | 221 | | Imp 1 | +~~~~~~~~~| |~~~~| | | | 222 | | receive |-<--+ | | | F1 F2 | 223 | +---------+ | |Internet | | | | | 224 *-------<-----+ F1 | | | | | | 225 +---------+ | | +~~~~~~~~~| |~~~~| | | | 226 | transmit|-* *-| | | |<-* | | 227 | Imp 2 | | Tunnel ( ) | | | 228 | receive |-<-F2-| ID 2 \ / |<----* | 229 +---------+ +~~~~~~~~~~~\ /~~~~~~| Switch | 230 `-+-' +--------+ 232 Illustrations of a test setup with a bi-directional tunnel. The 233 upper diagram emphasizes the VLAN connectivity and geographical 234 location (where "Imp #" is the sender and receiver of implementation 235 1 or 2, either Perfas+ and NetProbe in this test). The lower diagram 236 shows example flows traveling between two measurement 237 implementations. For simplicity only two flows are shown, and netem 238 is omitted (it would appear before or after the Internet, depending 239 on the flow). 241 Figure 1 243 The testing employs the Layer 2 Tunnel Protocol, version 3 (L2TPv3) 244 [RFC3931] tunnel between test sites on the Internet. The tunnel IP 245 and L2TPv3 headers are intended to conceal the test equipment 246 addresses and ports from hash functions that would tend to spread 247 different test streams across parallel network resources, with likely 248 variation in performance as a result. 250 At each end of the tunnel, one pair of VLANs encapsulated in the 251 tunnel are looped-back so that test traffic is returned to each test 252 site. Thus, test streams traverse the L2TP tunnel twice, but appear 253 to be one-way tests from the test equipment point of view. 255 The network emulator is a host running Fedora 14 Linux [Fedora] with 256 IP forwarding enabled and the "netem" Network emulator as part of the 257 Fedora Kernel 2.6.35.11 [netem] loaded and operating. The standard 258 kernel is "tickless" replacing the previous periodic timer (250HZ, 259 with 4ms uncertainty) interrupts with on-demand interrupts. 260 Connectivity across the netem/Fedora host was accomplished by 261 bridging Ethernet VLAN interfaces together with "brctl" commands 262 (e.g., eth1.100 <-> eth2.100). The netem emulator was activated on 263 one interface (eth1) and only operates on test streams traveling in 264 one direction. In some tests, independent netem instances operated 265 separately on each VLAN. See the Appendix for more details. 267 The links between the netem emulator host and router and switch were 268 found to be 100baseTx-HD (100Mbps half duplex) as reported by "mii- 269 tool" [mii-tool], when testing was complete. Use of half duplex was 270 not intended, but probably added a small amount of delay variation 271 that could have been avoided in full duplex mode. 273 Each individual test was run with common packet rates (1 pps, 10pps) 274 Poisson/Periodic distributions, and IP packet sizes of 64, 340, and 275 500 Bytes. 277 For these tests, a stream of at least 300 packets was sent from 278 source to destination in each implementation. Periodic streams (as 279 per [RFC3432]) with 1 second spacing were used, except as noted. 281 As required in Section 2.8.1 of [RFC2680], packet Type-P must be 282 reported. The packet Type-P for this test was IP-UDP with Best 283 Effort DSCP. These headers were encapsulated according to the L2TPv3 284 specifications [RFC3931], and thus may not influence the treatment 285 received as the packets traversed the Internet. 287 With the L2TPv3 tunnel in use, the metric name for the testing 288 configured here (with respect to the IP header exposed to Internet 289 processing) is: 291 Type-IP-protocol-115-One-way-Packet-Loss--Stream 293 With (Section 3.2. [RFC2680]) metric parameters: 295 + Src, the IP address of a host (12.3.167.16 or 193.159.144.8) 297 + Dst, the IP address of a host (193.159.144.8 or 12.3.167.16) 299 + T0, a time 301 + Tf, a time 303 + lambda, a rate in reciprocal seconds 305 + Thresh, a maximum waiting time in seconds (see Section 2.8.2 of 306 [RFC2680]) and (Section 3.8. [RFC2680]) 308 Metric Units: A sequence of pairs; the elements of each pair are: 310 + T, a time, and 312 + L, either a zero or a one 314 The values of T in the sequence are monotonically increasing. Note 315 that T would be a valid parameter of *singleton* Type-P-One-way- 316 Packet-Loss, and that L would be a valid value of Type-P-One-way- 317 Packet Loss (see Section 2 of [RFC2680]). 319 Also, Section 2.8.4 of [RFC2680] recommends that the path SHOULD be 320 reported. In this test set-up, most of the path details will be 321 concealed from the implementations by the L2TPv3 tunnels, thus a more 322 informative path trace route can be conducted by the routers at each 323 location. 325 When NetProbe is used in production, a traceroute is conducted in 326 parallel at the outset of measurements. 328 Perfas+ does not support traceroute. 330 IPLGW#traceroute 193.159.144.8 332 Type escape sequence to abort. 333 Tracing the route to 193.159.144.8 335 1 12.126.218.245 [AS 7018] 0 msec 0 msec 4 msec 336 2 cr84.n54ny.ip.att.net (12.123.2.158) [AS 7018] 4 msec 4 msec 337 cr83.n54ny.ip.att.net (12.123.2.26) [AS 7018] 4 msec 338 3 cr1.n54ny.ip.att.net (12.122.105.49) [AS 7018] 4 msec 339 cr2.n54ny.ip.att.net (12.122.115.93) [AS 7018] 0 msec 340 cr1.n54ny.ip.att.net (12.122.105.49) [AS 7018] 0 msec 341 4 n54ny02jt.ip.att.net (12.122.80.225) [AS 7018] 4 msec 0 msec 342 n54ny02jt.ip.att.net (12.122.80.237) [AS 7018] 4 msec 343 5 192.205.34.182 [AS 7018] 0 msec 344 192.205.34.150 [AS 7018] 0 msec 345 192.205.34.182 [AS 7018] 4 msec 346 6 da-rg12-i.DA.DE.NET.DTAG.DE (62.154.1.30) [AS 3320] 88 msec 88 msec 347 88 msec 348 7 217.89.29.62 [AS 3320] 88 msec 88 msec 88 msec 349 8 217.89.29.55 [AS 3320] 88 msec 88 msec 88 msec 350 9 * * * 352 NetProbe Traceroute 354 It was only possible to conduct the traceroute for the measured path 355 on one of the tunnel-head routers (the normal trace facilities of the 356 measurement systems are confounded by the L2TPv3 tunnel 357 encapsulation). 359 4. Error Calibration, RFC 2680 361 An implementation is required to report calibration results on clock 362 synchronization in Section 2.8.3 of [RFC2680] (also required in 363 Section 3.7 of [RFC2680] for sample metrics). 365 Also, it is recommended to report the probability that a packet 366 successfully arriving at the destination network interface is 367 incorrectly designated as lost due to resource exhaustion in 368 Section 2.8.3 of [RFC2680]. 370 4.1. Clock Synchronization Calibration 372 For NetProbe and Perfas+ clock synchronization test results, refer to 373 Section 4 of [RFC6808]. 375 4.2. Packet Loss Determination Error 377 Since both measurement implementations have resource limitations, it 378 is theoretically possible that these limits could be exceeded and a 379 packet that arrived at the destination successfully might be 380 discarded in error. 382 In previous test efforts [I-D.morton-ippm-advance-metrics], NetProbe 383 produced 6 multicast streams with an aggregate bit rate over 53 Mbit/ 384 s, in order to characterize the 1-way capacity of a NISTNet-based 385 emulator. Neither the emulator nor the pair of NetProbe 386 implementations used in this testing dropped any packets in these 387 streams. 389 The maximum load used here between any 2 NetProbe implementations was 390 11.5 Mbit/s divided equally among 3 unicast test streams. We 391 concluded that steady resource usage does not contribute error 392 (additional loss) to the measurements. 394 5. Pre-determined Limits on Equivalence 396 In this section, we provide the numerical limits on comparisons 397 between implementations in order to declare that the results are 398 equivalent and therefore, the tested specification is clear. 400 A key point is that the allowable errors, corrections, and confidence 401 levels only need to be sufficient to detect misinterpretation of the 402 tested specification resulting in diverging implementations. 404 Also, the allowable error must be sufficient to compensate for 405 measured path differences. It was simply not possible to measure 406 fully identical paths in the VLAN-loopback test configuration used, 407 and this practical compromise must be taken into account. 409 For Anderson-Darling K-sample (ADK) [ADK] comparisons, the required 410 confidence factor for the cross-implementation comparisons SHALL be 411 the smallest of: 413 o 0.95 confidence factor at 1 packet resolution, or 415 o the smallest confidence factor (in combination with resolution) of 416 the two same-implementation comparisons for the same test 417 conditions (if the number of streams is sufficient to allow such 418 comparisons). 420 For Anderson-Darling Goodness-of-Fit (ADGoF) [Radgof] comparisons, 421 the required level of significance for the same-implementation 422 Goodness-of-Fit (GoF) SHALL be 0.05 or 5%, as specified in 423 Section 11.4 of [RFC2330]. This is equivalent to a 95% confidence 424 factor. 426 6. Tests to evaluate RFC 2680 Specifications 428 This section describes some results from production network (cross- 429 Internet) tests with measurement devices implementing IPPM metrics 430 and a network emulator to create relevant conditions, to determine 431 whether the metric definitions were interpreted consistently by 432 implementors. 434 The procedures are similar contained in Appendix A.1 of [RFC6576] for 435 One-way Delay. 437 6.1. One-way Loss, ADK Sample Comparison 439 This test determines if implementations produce results that appear 440 to come from a common packet loss distribution, as an overall 441 evaluation of Section 3 of [RFC2680], "A Definition for Samples of 442 One-way Packet Loss". Same-implementation comparison results help to 443 set the threshold of equivalence that will be applied to cross- 444 implementation comparisons. 446 This test is intended to evaluate measurements in sections 2, 3, and 447 4 of [RFC2680]. 449 By testing the extent to which the counts of one-way packet loss 450 counts on different test streams of two [RFC2680] implementations 451 appear to be from the same loss process, we reduce comparison steps 452 because comparing the resulting summary statistics (as defined in 453 Section 4 of [RFC2680]) would require a redundant set of equivalence 454 evaluations. We can easily check whether the single statistic in 455 Section 4 of [RFC2680] was implemented, and report on that fact. 457 1. Configure an L2TPv3 path between test sites, and each pair of 458 measurement devices to operate tests in their designated pair of 459 VLANs. 461 2. Measure a sample of one-way packet loss singletons with 2 or more 462 implementations, using identical options and network emulator 463 settings (if used). 465 3. Measure a sample of one-way packet loss singletons with *four or 466 more* instances of the *same* implementations, using identical 467 options, noting that connectivity differences SHOULD be the same 468 as for cross implementation testing. 470 4. If less than ten test streams are available, skip to step 7. 472 5. Apply the ADK comparison procedures (see Appendix C of [RFC6576]) 473 and determine the resolution and confidence factor for 474 distribution equivalence of each same-implementation comparison 475 and each cross-implementation comparison. 477 6. Take the coarsest resolution and confidence factor for 478 distribution equivalence from the same-implementation pairs, or 479 the limit defined in Section 5 above, as a limit on the 480 equivalence threshold for these experimental conditions. 482 7. Compare the cross-implementation ADK performance with the 483 equivalence threshold determined in step 5 to determine if 484 equivalence can be declared. 486 The metric parameters varied for each loss test, and they are listed 487 first in each sub-section below. 489 The cross-implementation comparison uses a simple ADK analysis 490 [Rtool] [Radk], where all NetProbe loss counts are compared with all 491 Perfas+ loss results. 493 In the result analysis of this section: 495 o All comparisons used 1 packet resolution. 497 o No Correction Factors were applied. 499 o The 0.95 confidence factor (1.960 for cross-implementation 500 comparison) was used. 502 6.1.1. 340B/Periodic Cross-imp. results 504 Tests described in this section used: 506 o IP header + payload = 340 octets 508 o Periodic sampling at 1 packet per second 510 o Test duration = 1200 seconds (during April 7, 2011, EDT) 512 The netem emulator was set for 100ms constant delay, with 10% loss 513 ratio. In this experiment, the netem emulator was configured to 514 operate independently on each VLAN and thus the emulator itself is a 515 potential source of error when comparing streams that traverse the 516 test path in different directions. 518 ======================================= 520 A07bps_loss <- c(114, 175, 138, 142, 181, 105) (NetProbe) 521 A07per_loss <- c(115, 128, 136, 127, 139, 138) (Perfas+) 523 > A07bps_loss <- c(114, 175, 138, 142, 181, 105) 524 > A07per_loss <- c(115, 128, 136, 127, 139, 138) 525 > 526 > A07cross_loss_ADK <- adk.test(A07bps_loss, A07per_loss) 527 > A07cross_loss_ADK 528 Anderson-Darling k-sample test. 530 Number of samples: 2 531 Sample sizes: 6 6 532 Total number of values: 12 533 Number of unique values: 11 535 Mean of Anderson Darling Criterion: 1 536 Standard deviation of Anderson Darling Criterion: 0.6569 538 T = (Anderson Darling Criterion - mean)/sigma 540 Null Hypothesis: All samples come from a common population. 542 t.obs P-value extrapolation 543 not adj. for ties 0.52043 0.20604 0 544 adj. for ties 0.62679 0.18607 0 546 ======================================= 548 The cross-implementation comparisons pass the ADK criterion. 550 6.1.2. 64B/Periodic Cross-imp. results 552 Tests described in this section used: 554 o IP header + payload = 64 octets 556 o Periodic sampling at 1 packet per second 558 o Test duration = 300 seconds (during March 24, 2011, EDT) 560 The netem emulator was set for 0ms constant delay, with 10% loss 561 ratio. 563 ======================================= 565 > M24per_loss <- c(42,34,35,35) (Perfas+) 566 > M24apd_23BC_loss <- c(27,39,29,24) (NetProbe) 567 > M24apd_loss23BC_ADK <- adk.test(M24apd_23BC_loss,M24per_loss) 568 > M24apd_loss23BC_ADK 569 Anderson-Darling k-sample test. 571 Number of samples: 2 572 Sample sizes: 4 4 573 Total number of values: 8 574 Number of unique values: 7 576 Mean of Anderson Darling Criterion: 1 577 Standard deviation of Anderson Darling Criterion: 0.60978 579 T = (Anderson Darling Criterion - mean)/sigma 581 Null Hypothesis: All samples come from a common population. 583 t.obs P-value extrapolation 584 not adj. for ties 0.76921 0.16200 0 585 adj. for ties 0.90935 0.14113 0 587 Warning: At least one sample size is less than 5. 588 p-values may not be very accurate. 590 ======================================= 592 The cross-implementation comparisons pass the ADK criterion. 594 6.1.3. 64B/Poisson Cross-imp. results 596 Tests described in this section used: 598 o IP header + payload = 64 octets 600 o Poisson sampling at lambda = 1 packet per second 602 o Test duration = 20 minutes (during April 27, 2011, EDT) 604 The netem configuration was 0ms delay and 10% loss, but there were 605 two passes through an emulator for each stream, and loss emulation 606 was present for 18 minutes of the 20 minute test. 608 ======================================= 610 A27aps_loss <- c(91,110,113,102,111,109,112,113) (NetProbe) 611 A27per_loss <- c(95,123,126,114) (Perfas+) 613 A27cross_loss_ADK <- adk.test(A27aps_loss, A27per_loss) 615 > A27cross_loss_ADK 616 Anderson-Darling k-sample test. 618 Number of samples: 2 619 Sample sizes: 8 4 620 Total number of values: 12 621 Number of unique values: 11 623 Mean of Anderson Darling Criterion: 1 624 Standard deviation of Anderson Darling Criterion: 0.65642 626 T = (Anderson Darling Criterion - mean)/sigma 628 Null Hypothesis: All samples come from a common population. 630 t.obs P-value extrapolation 631 not adj. for ties 2.15099 0.04145 0 632 adj. for ties 1.93129 0.05125 0 634 Warning: At least one sample size is less than 5. 635 p-values may not be very accurate. 636 > 638 ======================================= 640 The cross-implementation comparisons barely pass the ADK criterion at 641 95% = 1.960 when adjusting for ties. 643 6.1.4. Conclusions on the ADK Results for One-way Packet Loss 645 We conclude that the two implementations are capable of producing 646 equivalent one-way packet loss measurements based on their 647 interpretation of [RFC2680]. 649 6.2. One-way Loss, Delay threshold 651 This test determines if implementations use the same configured 652 maximum waiting time delay from one measurement to another under 653 different delay conditions, and correctly declare packets arriving in 654 excess of the waiting time threshold as lost. 656 See Section 2.8.2 of [RFC2680]. 658 1. Configure an L2TPv3 path between test sites, and each pair of 659 measurement devices to operate tests in their designated pair of 660 VLANs. 662 2. Configure the network emulator to add 1sec one-way constant delay 663 in one direction of transmission. 665 3. Measure (average) one-way delay with 2 or more implementations, 666 using identical waiting time thresholds (Thresh) for loss set at 667 3 seconds. 669 4. Configure the network emulator to add 3 sec one-way constant 670 delay in one direction of transmission equivalent to 2 seconds of 671 additional one-way delay (or change the path delay while test is 672 in progress, when there are sufficient packets at the first delay 673 setting). 675 5. Repeat/continue measurements. 677 6. Observe that the increase measured in step 5 caused all packets 678 with 2 sec additional delay to be declared lost, and that all 679 packets that arrive successfully in step 3 are assigned a valid 680 one-way delay. 682 The common parameters used for tests in this section are: 684 o IP header + payload = 64 octets 686 o Poisson sampling at lambda = 1 packet per second 688 o Test duration = 900 seconds total (March 21, 2011 EDT) 690 The netem emulator settings added constant delays as specified in the 691 procedure above. 693 6.2.1. NetProbe results for Loss Threshold 695 In NetProbe, the Loss Threshold was implemented uniformly over all 696 packets as a post-processing routine. With the Loss Threshold set at 697 3 seconds, all packets with one-way delay >3 seconds were marked 698 "Lost" and included in the Lost Packet list with their transmission 699 time (as required in Section 3.3 of [RFC2680]). This resulted in 342 700 packets designated as lost in one of the test streams (with average 701 delay = 3.091 sec). 703 6.2.2. Perfas Results for Loss Threshold 705 Perfas+ uses a fixed Loss Threshold which was not adjustable during 706 this study. The Loss Threshold is approximately one minute, and 707 emulation of a delay of this size was not attempted. However, it is 708 possible to implement any delay threshold desired with a post- 709 processing routine and subsequent analysis. Using this method, 195 710 packets would be declared lost (with average delay = 3.091 sec). 712 6.2.3. Conclusions for Loss Threshold 714 Both implementations assume that any constant delay value desired can 715 be used as the Loss Threshold, since all delays are stored as a pair 716 as required in [RFC2680]. This is a simple way to 717 enforce the constant loss threshold envisioned in [RFC2680] (see 718 specific section reference above). We take the position that the 719 assumption of post-processing is compliant, and that the text of the 720 RFC should be revised slightly to include this point. 722 6.3. One-way Loss with Out-of-Order Arrival 724 Section 3.6 of [RFC2680] indicates that implementations need to 725 ensure that reordered packets are handled correctly using an 726 uncapitalized "must". In essence, this is an implied requirement 727 because the correct packet must be identified as lost if it fails to 728 arrive before its delay threshold under all circumstances, and 729 reordering is always a possibility on IP network paths. See 730 [RFC4737] for the definition of reordering used in IETF standard- 731 compliant measurements. 733 Using the procedure of section 6.1, the netem emulator was set to 734 introduce 10% loss, significant delay (2000 ms) and delay variation 735 (1000 ms), which was sufficient to produce packet reordering because 736 each packet's emulated delay is independent from others. 738 The tests described in this section used: 740 o IP header + payload = 64 octets 742 o Periodic sampling = 1 packet per second 744 o Test duration = 600 seconds (during May 2, 2011, EDT) 745 ======================================= 747 > Y02aps_loss <- c(53,45,67,55) (NetProbe) 748 > Y02per_loss <- c(59,62,67,69) (Perfas+) 749 > Y02cross_loss_ADK <- adk.test(Y02aps_loss, Y02per_loss) 750 > Y02cross_loss_ADK 751 Anderson-Darling k-sample test. 753 Number of samples: 2 754 Sample sizes: 4 4 755 Total number of values: 8 756 Number of unique values: 7 758 Mean of Anderson Darling Criterion: 1 759 Standard deviation of Anderson Darling Criterion: 0.60978 761 T = (Anderson Darling Criterion - mean)/sigma 763 Null Hypothesis: All samples come from a common population. 765 t.obs P-value extrapolation 766 not adj. for ties 1.11282 0.11531 0 767 adj. for ties 1.19571 0.10616 0 769 Warning: At least one sample size is less than 5. 770 p-values may not be very accurate. 771 > 773 ======================================= 775 The test results indicate that extensive reordering was present. 776 Both implementations capture the extensive delay variation between 777 adjacent packets. In NetProbe, packet arrival order is preserved in 778 the raw measurement files, so an examination of arrival packet 779 sequence numbers also indicates reordering. 781 Despite extensive continuous packet reordering present in the 782 transmission path, the distributions of loss counts from the two 783 implementations pass the ADK criterion at 95% = 1.960. 785 6.4. Poisson Sending Process Evaluation 787 Section 3.7 of [RFC2680] indicates that implementations need to 788 ensure that their sending process is reasonably close to a classic 789 Poisson distribution when used. Much more detail on sample 790 distribution generation and Goodness-of-Fit testing is specified in 791 Section 11.4 of [RFC2330] and the Appendix of [RFC2330]. 793 In this section, each implementation's Poisson distribution is 794 compared with an idealistic version of the distribution available in 795 the base functionality of the R-tool for Statistical Analysis[Rtool], 796 and performed using the Anderson-Darling Goodness-of-Fit test package 797 (ADGofTest) [Radgof]. The Goodness-of-Fit criterion derived from 798 [RFC2330] requires a test statistic value AD <= 2.492 for 5% 799 significance. The Appendix of [RFC2330] also notes that there may be 800 difficulty satisfying the ADGofTest when the sample includes many 801 packets (when 8192 were used, the test always failed, but smaller 802 sets of the stream passed). 804 Both implementations were configured to produce Poisson distributions 805 with lambda = 1 packet per second, and assign received packet 806 timestamps in the measurement application (above UDP layer, see the 807 calibration results in Section 4 of [RFC6808] for assessment of 808 error). 810 6.4.1. NetProbe Results 812 Section 11.4 of [RFC2330] suggests three possible measurement points 813 to evaluate the Poisson distribution. The NetProbe analysis uses 814 "user-level timestamps made just before or after the system call for 815 transmitting the packet". 817 The statistical summary for two NetProbe streams is below: 819 ======================================= 821 > summary(a27ms$s1[2:1152]) 822 Min. 1st Qu. Median Mean 3rd Qu. Max. 823 0.0100 0.2900 0.6600 0.9846 1.3800 8.6390 824 > summary(a27ms$s2[2:1152]) 825 Min. 1st Qu. Median Mean 3rd Qu. Max. 826 0.010 0.280 0.670 0.979 1.365 8.829 828 ======================================= 830 We see that both the Means are near the specified lambda = 1. 832 The results of ADGoF tests for these two streams is shown below: 834 ======================================= 836 > ad.test( a27ms$s1[2:101], pexp, 1) 838 Anderson-Darling GoF Test 840 data: a27ms$s1[2:101] and pexp 841 AD = 0.8908, p-value = 0.4197 842 alternative hypothesis: NA 844 > ad.test( a27ms$s1[2:1001], pexp, 1) 846 Anderson-Darling GoF Test 848 data: a27ms$s1[2:1001] and pexp 849 AD = 0.9284, p-value = 0.3971 850 alternative hypothesis: NA 852 > ad.test( a27ms$s2[2:101], pexp, 1) 854 Anderson-Darling GoF Test 856 data: a27ms$s2[2:101] and pexp 857 AD = 0.3597, p-value = 0.8873 858 alternative hypothesis: NA 860 > ad.test( a27ms$s2[2:1001], pexp, 1) 862 Anderson-Darling GoF Test 864 data: a27ms$s2[2:1001] and pexp 865 AD = 0.6913, p-value = 0.5661 866 alternative hypothesis: NA 868 ======================================= 870 We see that both 100 and 1000 packet sets from two different streams 871 (s1 and s2) all passed the AD <= 2.492 criterion. 873 6.4.2. Perfas+ Results 875 Section 11.4 of [RFC2330] suggests three possible measurement points 876 to evaluate the Poisson distribution. The Perfas+ analysis uses 877 "wire times for the packets as recorded using a packet filter". 878 However, due to limited access at the Perfas+ side of the test setup, 879 the captures were made after the Perfas+ streams traversed the 880 production network, adding a small amount of unwanted delay variation 881 to the wire times (and possibly error due to packet loss). 883 The statistical summary for two Perfas+ streams is below: 885 ======================================= 887 > summary(a27pe$p1) 888 Min. 1st Qu. Median Mean 3rd Qu. Max. 889 0.004 0.347 0.788 1.054 1.548 4.231 890 > summary(a27pe$p2) 891 Min. 1st Qu. Median Mean 3rd Qu. Max. 892 0.0010 0.2710 0.7080 0.9696 1.3740 7.1160 894 ======================================= 896 We see that both the means are near the specified lambda = 1. 898 The results of ADGoF tests for these two streams is shown below: 900 ======================================= 902 > ad.test(a27pe$p1, pexp, 1 ) 904 Anderson-Darling GoF Test 906 data: a27pe$p1 and pexp 907 AD = 1.1364, p-value = 0.2930 908 alternative hypothesis: NA 910 > ad.test(a27pe$p2, pexp, 1 ) 912 Anderson-Darling GoF Test 914 data: a27pe$p2 and pexp 915 AD = 0.5041, p-value = 0.7424 916 alternative hypothesis: NA 918 > ad.test(a27pe$p1[1:100], pexp, 1 ) 920 Anderson-Darling GoF Test 922 data: a27pe$p1[1:100] and pexp 923 AD = 0.7202, p-value = 0.5419 924 alternative hypothesis: NA 926 > ad.test(a27pe$p1[101:193], pexp, 1 ) 928 Anderson-Darling GoF Test 930 data: a27pe$p1[101:193] and pexp 931 AD = 1.4046, p-value = 0.201 932 alternative hypothesis: NA 934 > ad.test(a27pe$p2[1:100], pexp, 1 ) 936 Anderson-Darling GoF Test 938 data: a27pe$p2[1:100] and pexp 939 AD = 0.4758, p-value = 0.7712 940 alternative hypothesis: NA 942 > ad.test(a27pe$p2[101:193], pexp, 1 ) 944 Anderson-Darling GoF Test 946 data: a27pe$p2[101:193] and pexp 947 AD = 0.3381, p-value = 0.9068 948 alternative hypothesis: NA 950 > 952 ======================================= 954 We see that both 193, 100, and 93 packet sets from two different 955 streams (p1 and p2) all passed the AD <= 2.492 criterion. 957 6.4.3. Conclusions for Goodness-of-Fit 959 Both NetProbe and Perfas+ implementations produce adequate Poisson 960 distributions according to the Anderson-Darling Goodness-of-Fit at 961 the 5% significance (1-alpha = 0.05, or 95% confidence level). 963 6.5. Implementation of Statistics for One-way Loss 965 We check which statistics were implemented, and report on those 966 facts, noting that Section 4 of [RFC2680] does not specify the 967 calculations exactly, and gives only some illustrative examples. 969 NetProbe Perfas 971 4.1. Type-P-One-way-Packet-Loss-Average yes yes 972 (this is more commonly referred to as loss ratio) 974 Implementation of Section 4 Statistics 975 We note that implementations refer to this metric as a loss ratio, 976 and this is an area for likely revision of the text to make it more 977 consistent with wide-spread usage. 979 7. Conclusions for RFC 2680bis 981 This memo concludes that [RFC2680] should be advanced on the 982 standards track, and recommends the following edits to improve the 983 text (which are not deemed significant enough to affect maturity). 985 o Revise Type-P-One-way-Packet-Loss-Ave to Type-P-One-way-Delay- 986 Packet-Loss-Ratio . 988 o Regarding implementation of the loss delay threshold (section 989 6.2), the assumption of post-processing is compliant, and the text 990 of RFC 2680bis should be revised slightly to include this point. 992 o The IETF has reached consensus on guidance for reporting metrics 993 in [RFC6703], and this memo should be referenced in RFC2680bis to 994 incorporate recent experience where appropriate. 996 We note that there are at least two Errata on [RFC2680] and these 997 should be processed as part of the editing process. 999 We recognize the existence of BCP 170 [RFC6390] providing guidelines 1000 for development of drafts describing new performance metrics. 1001 However, the advancement of [RFC2680] represents fine-tuning of long- 1002 standing specifications based on experience that helped to formulate 1003 BCP 170, and material that satisfies some of the requirements of 1004 [RFC6390] can be found in other RFCs, such as the IPPM Framework 1005 [RFC2330]. Thus, no specific changes to address BCP 170 guidelines 1006 are recommended for RFC 2680bis. 1008 8. Security Considerations 1010 The security considerations that apply to any active measurement of 1011 live networks are relevant here as well. See [RFC4656] and 1012 [RFC5357]. 1014 9. IANA Considerations 1016 This memo makes no requests of IANA, and the authors hope that IANA 1017 personnel will be able to use their valuable time in other worthwhile 1018 pursuits. 1020 10. Acknowledgements 1022 The authors thank Lars Eggert for his continued encouragement to 1023 advance the IPPM metrics during his tenure as AD Advisor. 1025 Nicole Kowalski supplied the needed CPE router for the NetProbe side 1026 of the test set-up, and graciously managed her testing in spite of 1027 issues caused by dual-use of the router. Thanks Nicole! 1029 The "NetProbe Team" also acknowledges many useful discussions on 1030 statistical interpretation with Ganga Maguluri. 1032 Constructive comments and helpful reviews where also provided by Bill 1033 Cerveny, Joachim Fabini, and Ann Cerveny. 1035 11. Appendix - Network Configuration and sample commands 1037 This Appendix provides some background information on the host 1038 configuration and sample tc commands for the "netem" network 1039 emulator, as described in Section 3 and Figure 1 in the body of this 1040 memo. These details are also applicable to the test plan in 1041 [RFC6808]. 1043 The host interface and configuration is shown below: 1045 [system@dell4-4 ~]$ su 1046 Password: 1047 [root@dell4-4 system]# service iptables save 1048 iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ] 1049 [root@dell4-4 system]# service iptables stop 1050 iptables: Flushing firewall rules: [ OK ] 1051 iptables: Setting chains to policy ACCEPT: nat filter [ OK ] 1052 iptables: Unloading modules: [ OK ] 1053 [root@dell4-4 system]# brctl show 1054 bridge name bridge id STP enabled interfaces 1055 virbr0 8000.000000000000 yes 1056 [root@dell4-4 system]# ifconfig eth1.300 0.0.0.0 promisc up 1057 [root@dell4-4 system]# ifconfig eth1.400 0.0.0.0 promisc up 1058 [root@dell4-4 system]# ifconfig eth2.400 0.0.0.0 promisc up 1059 [root@dell4-4 system]# ifconfig eth2.300 0.0.0.0 promisc up 1060 [root@dell4-4 system]# brctl addbr br300 1061 [root@dell4-4 system]# brctl addif br300 eth1.300 1062 [root@dell4-4 system]# brctl addif br300 eth2.300 1063 [root@dell4-4 system]# ifconfig br300 up 1064 [root@dell4-4 system]# brctl addbr br400 1065 [root@dell4-4 system]# brctl addif br400 eth1.400 1066 [root@dell4-4 system]# brctl addif br400 eth2.400 1067 [root@dell4-4 system]# ifconfig br400 up 1069 [root@dell4-4 system]# brctl show 1070 bridge name bridge id STP enabled interfaces 1071 br300 8000.0002b3109b8a no eth1.300 1072 eth2.300 1073 br400 8000.0002b3109b8a no eth1.400 1074 eth2.400 1075 virbr0 8000.000000000000 yes 1077 [root@dell4-4 system]# brctl showmacs br300 1078 port no mac addr is local? ageing timer 1079 2 00:02:b3:10:9b:8a yes 0.00 1080 1 00:02:b3:10:9b:99 yes 0.00 1081 1 00:02:b3:c4:c9:7a no 0.52 1082 2 00:02:b3:cf:02:c6 no 0.52 1083 2 00:0b:5f:54:de:81 no 0.01 1084 [root@dell4-4 system]# brctl showmacs br400 1085 port no mac addr is local? ageing timer 1086 2 00:02:b3:10:9b:8a yes 0.00 1087 1 00:02:b3:10:9b:99 yes 0.00 1088 2 00:02:b3:c4:c9:7a no 0.60 1089 1 00:02:b3:cf:02:c6 no 0.42 1090 2 00:0b:5f:54:de:81 no 0.33 1091 [root@dell4-4 system]# tc qdisc add dev eth1.300 root netem delay 100ms 1093 [root@dell4-4 system]# ifconfig eth1.200 0.0.0.0 promisc up 1094 [root@dell4-4 system]# vconfig add eth1 100 1095 Added VLAN with VID == 100 to IF -:eth1:- 1097 [root@dell4-4 system]# ifconfig eth1.100 0.0.0.0 promisc up 1099 [root@dell4-4 system]# vconfig add eth2 100 1100 Added VLAN with VID == 100 to IF -:eth2:- 1102 [root@dell4-4 system]# ifconfig eth2.100 0.0.0.0 promisc up 1103 [root@dell4-4 system]# ifconfig eth2.200 0.0.0.0 promisc up 1104 [root@dell4-4 system]# brctl addbr br100 1105 [root@dell4-4 system]# brctl addif br100 eth1.100 1106 [root@dell4-4 system]# brctl addif br100 eth2.100 1107 [root@dell4-4 system]# ifconfig br100 up 1108 [root@dell4-4 system]# brctl addbr br200 1109 [root@dell4-4 system]# brctl addif br200 eth1.200 1110 [root@dell4-4 system]# brctl addif br200 eth2.200 1111 [root@dell4-4 system]# ifconfig br200 up 1112 [root@dell4-4 system]# brctl show 1113 bridge name bridge id STP enabled interfaces 1114 br100 8000.0002b3109b8a no eth1.100 1115 eth2.100 1116 br200 8000.0002b3109b8a no eth1.200 1117 eth2.200 1118 br300 8000.0002b3109b8a no eth1.300 1119 eth2.300 1120 br400 8000.0002b3109b8a no eth1.400 1121 eth2.400 1122 virbr0 8000.000000000000 yes 1123 [root@dell4-4 system]# brctl showmacs br100 1124 port no mac addr is local? ageing timer 1125 2 00:02:b3:10:9b:8a yes 0.00 1126 1 00:02:b3:10:9b:99 yes 0.00 1127 1 00:0a:e4:83:89:07 no 0.19 1128 2 00:0b:5f:54:de:81 no 0.91 1129 2 00:e0:ed:0f:72:86 no 1.28 1130 [root@dell4-4 system]# brctl showmacs br200 1131 port no mac addr is local? ageing timer 1132 2 00:02:b3:10:9b:8a yes 0.00 1133 1 00:02:b3:10:9b:99 yes 0.00 1134 2 00:0a:e4:83:89:07 no 1.14 1135 2 00:0b:5f:54:de:81 no 1.87 1136 1 00:e0:ed:0f:72:86 no 0.24 1137 [root@dell4-4 system]# tc qdisc add dev eth1.100 root netem delay 100ms 1138 [root@dell4-4 system]# 1140 ====================================================================== 1142 Some sample tc command lines controlling netem and its impairments 1143 are given below. 1145 tc qdisc add dev eth1.100 root netem loss 0% 1146 tc qdisc add dev eth1.200 root netem loss 0% 1147 tc qdisc add dev eth1.300 root netem loss 0% 1148 tc qdisc add dev eth1.400 root netem loss 0% 1150 Add delay and delay variation: 1151 tc qdisc change dev eth1.100 root netem delay 100ms 50ms 1152 tc qdisc change dev eth1.200 root netem delay 100ms 50ms 1153 tc qdisc change dev eth1.300 root netem delay 100ms 50ms 1154 tc qdisc change dev eth1.400 root netem delay 100ms 50ms 1156 Add delay, delay variation, and loss: 1157 tc qdisc change dev eth1 root netem delay 2000ms 1000ms loss 10% 1159 ===================================================================== 1161 12. References 1163 12.1. Normative References 1165 [RFC2026] Bradner, S., "The Internet Standards Process -- Revision 1166 3", BCP 9, RFC 2026, October 1996. 1168 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1169 Requirement Levels", BCP 14, RFC 2119, March 1997. 1171 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 1172 "Framework for IP Performance Metrics", RFC 2330, May 1173 1998. 1175 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1176 Packet Loss Metric for IPPM", RFC 2680, September 1999. 1178 [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network 1179 performance measurement with periodic streams", RFC 3432, 1180 November 2002. 1182 [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. 1183 Zekauskas, "A One-way Active Measurement Protocol 1184 (OWAMP)", RFC 4656, September 2006. 1186 [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, 1187 S., and J. Perser, "Packet Reordering Metrics", RFC 4737, 1188 November 2006. 1190 [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. 1191 Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", 1192 RFC 5357, October 2008. 1194 [RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation 1195 and Implementation Reports for Advancement to Draft 1196 Standard", BCP 9, RFC 5657, September 2009. 1198 [RFC6390] Clark, A. and B. Claise, "Guidelines for Considering New 1199 Performance Metric Development", BCP 170, RFC 6390, 1200 October 2011. 1202 [RFC6576] Geib, R., Morton, A., Fardid, R., and A. Steinmitz, "IP 1203 Performance Metrics (IPPM) Standard Advancement Testing", 1204 BCP 176, RFC 6576, March 2012. 1206 [RFC6703] Morton, A., Ramachandran, G., and G. Maguluri, "Reporting 1207 IP Network Performance Metrics: Different Points of View", 1208 RFC 6703, August 2012. 1210 [RFC6808] Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test 1211 Plan and Results Supporting Advancement of RFC 2679 on the 1212 Standards Track", RFC 6808, December 2012. 1214 12.2. Informative References 1216 [ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling 1217 Tests of Fit, for Continuous and Discrete cases", 1218 University of Washington, Technical Report No. 81, May 1219 1986. 1221 [Fedora] "http://fedoraproject.org/", . 1223 [I-D.morton-ippm-2680-bis] 1224 Almes, G., Zekauskas, M., and A. Morton, "A One-Way Loss 1225 Metric for IPPM", draft-morton-ippm-2680-bis-02 (work in 1226 progress), February 2014. 1228 [I-D.morton-ippm-advance-metrics] 1229 Morton, A., "Lab Test Results for Advancing Metrics on the 1230 Standards Track", draft-morton-ippm-advance-metrics-02 1231 (work in progress), October 2010. 1233 [Perfas] Heidemann, C., "Qualitaet in IP-Netzen Messverfahren", 1234 published by ITG Fachgruppe, 2nd meeting 5.2.3 (NGN) 1235 http://www.itg523.de/oeffentlich/01nov/ 1236 Heidemann_QOS_Messverfahren.pdf , November 2001. 1238 [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling 1239 Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. 1241 [Radgof] Bellosta, C., "ADGofTest: Anderson-Darling Goodness-of-Fit 1242 Test. R package version 0.3.", http://cran.r-project.org/ 1243 web/packages/ADGofTest/index.html, December 2011. 1245 [Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and 1246 Combinations of Such Tests. R package version 1.0.", , 1247 2008. 1249 [Rtool] R Development Core Team, , "R: A language and environment 1250 for statistical computing. R Foundation for Statistical 1251 Computing, Vienna, Austria. ISBN 3-900051-07-0, URL 1252 http://www.R-project.org/", , 2011. 1254 [WIPM] "AT&T Global IP Network", 1255 http://ipnetwork.bgtmo.ip.att.net/pws/index.html, 2012. 1257 [mii-tool] 1258 "http://man7.org/linux/man-pages/man8/mii-tool.8.html", . 1260 [netem] "http://www.linuxfoundation.org/collaborate/workgroups/ 1261 networking/netem", . 1263 Authors' Addresses 1265 Len Ciavattone 1266 AT&T Labs 1267 200 Laurel Avenue South 1268 Middletown, NJ 07748 1269 USA 1271 Phone: +1 732 420 1239 1272 Email: lencia@att.com 1274 Ruediger Geib 1275 Deutsche Telekom 1276 Heinrich Hertz Str. 3-7 1277 Darmstadt 64295 1278 Germany 1280 Phone: +49 6151 58 12747 1281 Email: Ruediger.Geib@telekom.de 1283 Al Morton 1284 AT&T Labs 1285 200 Laurel Avenue South 1286 Middletown, NJ 07748 1287 USA 1289 Phone: +1 732 420 1571 1290 Fax: +1 732 368 1192 1291 Email: acmorton@att.com 1292 URI: http://home.comcast.net/~acmacm/ 1294 Matthias Wieser 1295 Technical University Darmstadt 1296 Darmstadt 1297 Germany 1299 Email: matthias_michael.wieser@stud.tu-darmstadt.de