idnits 2.17.1 draft-ietf-ippm-testplan-rfc2680-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 1 character in excess of 72. == There are 7 instances of lines with non-RFC2606-compliant FQDNs in the document. == There are 11 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 827 has weird spacing: '...1] and pexp...' == Line 835 has weird spacing: '...1] and pexp...' == Line 843 has weird spacing: '...1] and pexp...' == Line 851 has weird spacing: '...1] and pexp...' == Line 885 has weird spacing: '...p1 and pexp...' == (5 more instances...) == The document seems to contain a disclaimer for pre-RFC5378 work, but was first submitted on or after 10 November 2008. The disclaimer is usually necessary only for documents that revise or obsolete older RFCs, and that take significant amounts of text from those RFCs. If you can contact all authors of the source material and they are willing to grant the BCP78 rights to the IETF Trust, you can and should remove the disclaimer. Otherwise, the disclaimer is needed and you can ignore this comment. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (July 8, 2013) is 3944 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'AS 7018' is mentioned on line 361, but not defined == Missing Reference: 'AS 3320' is mentioned on line 365, but not defined == Unused Reference: 'RFC2679' is defined on line 1024, but no explicit reference was found in the text == Unused Reference: 'RFC4814' is defined on line 1042, but no explicit reference was found in the text == Unused Reference: 'RFC5226' is defined on line 1046, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) ** Obsolete normative reference: RFC 5226 (Obsoleted by RFC 8126) Summary: 4 errors (**), 0 flaws (~~), 15 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group L. Ciavattone 3 Internet-Draft AT&T Labs 4 Intended status: Informational R. Geib 5 Expires: January 9, 2014 Deutsche Telekom 6 A. Morton 7 AT&T Labs 8 M. Wieser 9 Technical University Darmstadt 10 July 8, 2013 12 Test Plan and Results for Advancing RFC 2680 on the Standards Track 13 draft-ietf-ippm-testplan-rfc2680-03 15 Abstract 17 This memo proposes to advance a performance metric RFC along the 18 standards track, specifically RFC 2680 on One-way Loss Metrics. 19 Observing that the metric definitions themselves should be the 20 primary focus rather than the implementations of metrics, this memo 21 describes the test procedures to evaluate specific metric requirement 22 clauses to determine if the requirement has been interpreted and 23 implemented as intended. Two completely independent implementations 24 have been tested against the key specifications of RFC 2680. 26 Requirements Language 28 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 29 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 30 document are to be interpreted as described in RFC 2119 [RFC2119]. 32 Status of this Memo 34 This Internet-Draft is submitted in full conformance with the 35 provisions of BCP 78 and BCP 79. 37 Internet-Drafts are working documents of the Internet Engineering 38 Task Force (IETF). Note that other groups may also distribute 39 working documents as Internet-Drafts. The list of current Internet- 40 Drafts is at http://datatracker.ietf.org/drafts/current/. 42 Internet-Drafts are draft documents valid for a maximum of six months 43 and may be updated, replaced, or obsoleted by other documents at any 44 time. It is inappropriate to use Internet-Drafts as reference 45 material or to cite them other than as "work in progress." 47 This Internet-Draft will expire on January 9, 2014. 49 Copyright Notice 51 Copyright (c) 2013 IETF Trust and the persons identified as the 52 document authors. All rights reserved. 54 This document is subject to BCP 78 and the IETF Trust's Legal 55 Provisions Relating to IETF Documents 56 (http://trustee.ietf.org/license-info) in effect on the date of 57 publication of this document. Please review these documents 58 carefully, as they describe your rights and restrictions with respect 59 to this document. Code Components extracted from this document must 60 include Simplified BSD License text as described in Section 4.e of 61 the Trust Legal Provisions and are provided without warranty as 62 described in the Simplified BSD License. 64 This document may contain material from IETF Documents or IETF 65 Contributions published or made publicly available before November 66 10, 2008. The person(s) controlling the copyright in some of this 67 material may not have granted the IETF Trust the right to allow 68 modifications of such material outside the IETF Standards Process. 69 Without obtaining an adequate license from the person(s) controlling 70 the copyright in such materials, this document may not be modified 71 outside the IETF Standards Process, and derivative works of it may 72 not be created outside the IETF Standards Process, except to format 73 it for publication as an RFC or to translate it into languages other 74 than English. 76 Table of Contents 78 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 79 1.1. RFC 2680 Coverage . . . . . . . . . . . . . . . . . . . . 5 80 2. A Definition-centric metric advancement process . . . . . . . 5 81 3. Test configuration . . . . . . . . . . . . . . . . . . . . . . 6 82 4. Error Calibration, RFC 2680 . . . . . . . . . . . . . . . . . 10 83 4.1. Clock Synchronization Calibration . . . . . . . . . . . . 10 84 4.2. Packet Loss Determination Error . . . . . . . . . . . . . 10 85 5. Pre-determined Limits on Equivalence . . . . . . . . . . . . . 11 86 6. Tests to evaluate RFC 2680 Specifications . . . . . . . . . . 12 87 6.1. One-way Loss, ADK Sample Comparison . . . . . . . . . . . 12 88 6.1.1. 340B/Periodic Cross-imp. results . . . . . . . . . . . 13 89 6.1.2. 64B/Periodic Cross-imp. results . . . . . . . . . . . 14 90 6.1.3. 64B/Poisson Cross-imp. results . . . . . . . . . . . . 15 91 6.1.4. Conclusions on the ADK Results for One-way Packet 92 Loss . . . . . . . . . . . . . . . . . . . . . . . . . 16 93 6.2. One-way Loss, Delay threshold . . . . . . . . . . . . . . 16 94 6.2.1. NetProbe results for Loss Threshold . . . . . . . . . 17 95 6.2.2. Perfas Results for Loss Threshold . . . . . . . . . . 18 96 6.2.3. Conclusions for Loss Threshold . . . . . . . . . . . . 18 97 6.3. One-way Loss with Out-of-Order Arrival . . . . . . . . . . 18 98 6.4. Poisson Sending Process Evaluation . . . . . . . . . . . . 19 99 6.4.1. NetProbe Results . . . . . . . . . . . . . . . . . . . 20 100 6.4.2. Perfas+ Results . . . . . . . . . . . . . . . . . . . 21 101 6.4.3. Conclusions for Goodness-of-Fit . . . . . . . . . . . 23 102 6.5. Implementation of Statistics for One-way Loss . . . . . . 23 103 7. Conclusions for RFC 2680bis . . . . . . . . . . . . . . . . . 23 104 8. Security Considerations . . . . . . . . . . . . . . . . . . . 24 105 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 24 106 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 24 107 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 25 108 11.1. Normative References . . . . . . . . . . . . . . . . . . . 25 109 11.2. Informative References . . . . . . . . . . . . . . . . . . 26 110 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 27 112 1. Introduction 114 The IETF (IP Performance Metrics working group, IPPM) has considered 115 how to advance their metrics along the standards track since 2001. 117 A renewed work effort sought to investigate ways in which the 118 measurement variability could be reduced and thereby simplify the 119 problem of comparison for equivalence. 121 There is consensus [RFC6576] that the metric definitions should be 122 the primary focus of evaluation rather than the implementations of 123 metrics, and equivalent results are deemed to be evidence that the 124 metric specifications are clear and unambiguous. This is the metric 125 specification equivalent of protocol interoperability. The 126 advancement process either produces confidence that the metric 127 definitions and supporting material are clearly worded and 128 unambiguous, OR, identifies ways in which the metric definitions 129 should be revised to achieve clarity. 131 The process should also permit identification of options that were 132 not implemented, so that they can be removed from the advancing 133 specification (this is an aspect more typical of protocol advancement 134 along the standards track). 136 This memo's purpose is to implement the current approach for 137 [RFC2680]. 139 In particular, this memo documents consensus on the extent of 140 tolerable errors when assessing equivalence in the results. In 141 discussions, the IPPM working group agreed that test plan and 142 procedures should include the threshold for determining equivalence, 143 and this information should be available in advance of cross- 144 implementation comparisons. This memo includes procedures for same- 145 implementation comparisons to help set the equivalence threshold. 147 Another aspect of the metric RFC advancement process is the 148 requirement to document the work and results. The procedures of 149 [RFC2026] are expanded in[RFC5657], including sample implementation 150 and interoperability reports. This memo follows the template in 151 [I-D.morton-ippm-advance-metrics] for the report that accompanies the 152 protocol action request submitted to the Area Director, including 153 description of the test set-up, procedures, results for each 154 implementation and conclusions. 156 Although the conclusion reached through testing is that [RFC2680] 157 should be advanced on the Standards Track with modifications, the 158 revised text of RFC 2680bis is not yet ready for review. Therefore, 159 this memo documents the information to support [RFC2680] advancement, 160 and the approval of RFC2680bis is left for future action. 162 1.1. RFC 2680 Coverage 164 This plan is intended to cover all critical requirements and sections 165 of [RFC2680]. 167 Note that there are only five instances of the requirement term 168 "MUST" in [RFC2680] outside of the boilerplate and [RFC2119] 169 reference. 171 Material may be added as it is "discovered" (apparently, not all 172 requirements use requirements language). 174 2. A Definition-centric metric advancement process 176 The process described in Section 3.5 of [RFC6576] takes as a first 177 principle that the metric definitions, embodied in the text of the 178 RFCs, are the objects that require evaluation and possible revision 179 in order to advance to the next step on the standards track. 181 IF two implementations do not measure an equivalent singleton or 182 sample, or produce the an equivalent statistic, 184 AND sources of measurement error do not adequately explain the lack 185 of agreement, 187 THEN the details of each implementation should be audited along with 188 the exact definition text, to determine if there is a lack of clarity 189 that has caused the implementations to vary in a way that affects the 190 correspondence of the results. 192 IF there was a lack of clarity or multiple legitimate interpretations 193 of the definition text, 195 THEN the text should be modified and the resulting memo proposed for 196 consensus and advancement along the standards track. 198 Finally, all the findings MUST be documented in a report that can 199 support advancement on the standards track, similar to those 200 described in [RFC5657]. The list of measurement devices used in 201 testing satisfies the implementation requirement, while the test 202 results provide information on the quality of each specification in 203 the metric RFC (the surrogate for feature interoperability). 205 3. Test configuration 207 One metric implementation used was NetProbe version 5.8.5, (an 208 earlier version is used in the WIPM system and deployed world-wide 209 [WIPM]). NetProbe uses UDP packets of variable size, and can produce 210 test streams with Periodic [RFC3432] or Poisson [RFC2330] sample 211 distributions. 213 The other metric implementation used was Perfas+ version 3.1, 214 developed by Deutsche Telekom [Perfas]. Perfas+ uses UDP unicast 215 packets of variable size (but supports also TCP and multicast). Test 216 streams with periodic, Poisson or uniform sample distributions may be 217 used. 219 Figure 1 shows a view of the test path as each Implementation's test 220 flows pass through the Internet and the L2TPv3 tunnel IDs (1 and 2), 221 based on Figure 1 of [RFC6576]. 223 +----+ +----+ +----+ +----+ 224 |Imp1| |Imp1| ,---. |Imp2| |Imp2| 225 +----+ +----+ / \ +-------+ +----+ +----+ 226 | V100 | V200 / \ | Tunnel| | V300 | V400 227 | | ( ) | Head | | | 228 +--------+ +------+ | |__| Router| +----------+ 229 |Ethernet| |Tunnel| |Internet | +---B---+ |Ethernet | 230 |Switch |--|Head |-| | | |Switch | 231 +-+--+---+ |Router| | | +---+---+--+--+--+----+ 232 |__| +--A---+ ( ) |Network| |__| 233 \ / |Emulat.| 234 U-turn \ / |"netem"| U-turn 235 V300 to V400 `-+-' +-------+ V100 to V200 237 Implementations ,---. +--------+ 238 +~~~~~~~~~~~/ \~~~~~~| Remote | 239 +------->-----F2->-| / \ |->---. | 240 | +---------+ | Tunnel ( ) | | | 241 | | transmit|-F1->-| ID 1 ( ) |->. | | 242 | | Imp 1 | +~~~~~~~~~| |~~~~| | | | 243 | | receive |-<--+ ( ) | F1 F2 | 244 | +---------+ | |Internet | | | | | 245 *-------<-----+ F1 | | | | | | 246 +---------+ | | +~~~~~~~~~| |~~~~| | | | 247 | transmit|-* *-| | | |<-* | | 248 | Imp 2 | | Tunnel ( ) | | | 249 | receive |-<-F2-| ID 2 \ / |<----* | 250 +---------+ +~~~~~~~~~~~\ /~~~~~~| Switch | 251 `-+-' +--------+ 253 Illustrations of a test setup with a bi-directional tunnel. The 254 upper diagram emphasizes the VLAN connectivity and geographical 255 location. The lower diagram shows example flows traveling between 256 two measurement implementations (for simplicity, only two flows are 257 shown). 259 Figure 1 261 The testing employs the Layer 2 Tunnel Protocol, version 3 (L2TPv3) 262 [RFC3931] tunnel between test sites on the Internet. The tunnel IP 263 and L2TPv3 headers are intended to conceal the test equipment 264 addresses and ports from hash functions that would tend to spread 265 different test streams across parallel network resources, with likely 266 variation in performance as a result. 268 At each end of the tunnel, one pair of VLANs encapsulated in the 269 tunnel are looped-back so that test traffic is returned to each test 270 site. Thus, test streams traverse the L2TP tunnel twice, but appear 271 to be one-way tests from the test equipment point of view. 273 The network emulator is a host running Fedora 14 Linux 274 [http://fedoraproject.org/] with IP forwarding enabled and the 275 "netem" Network emulator as part of the Fedora Kernel 2.6.35.11 [http 276 ://www.linuxfoundation.org/collaborate/workgroups/networking/netem] 277 loaded and operating. Connectivity across the netem/Fedora host was 278 accomplished by bridging Ethernet VLAN interfaces together with 279 "brctl" commands (e.g., eth1.100 <-> eth2.100). The netem emulator 280 was activated on one interface (eth1) and only operates on test 281 streams traveling in one direction. In some tests, independent netem 282 instances operated separately on each VLAN. 284 The links between the netem emulator host and router and switch were 285 found to be 100baseTx-HD (100Mbps half duplex) as reported by "mii- 286 tool"when the testing was complete. Use of Half Duplex was not 287 intended, but probably added a small amount of delay variation that 288 could have been avoided in full duplex mode. 290 Each individual test was run with common packet rates (1 pps, 10pps) 291 Poisson/Periodic distributions, and IP packet sizes of 64, 340, and 292 500 Bytes. 294 For these tests, a stream of at least 300 packets were sent from 295 Source to Destination in each implementation. Periodic streams (as 296 per [RFC3432]) with 1 second spacing were used, except as noted. 298 As required in Section 2.8.1 of [RFC2680], packet Type-P must be 299 reported. The packet Type-P for this test was IP-UDP with Best 300 Effort DCSP. These headers were encapsulated according to the L2TPv3 301 specifications [RFC3931], and thus may not influence the treatment 302 received as the packets traversed the Internet. 304 With the L2TPv3 tunnel in use, the metric name for the testing 305 configured here (with respect to the IP header exposed to Internet 306 processing) is: 308 Type-IP-protocol-115-One-way-Packet-Loss--Stream 310 With (Section 3.2. [RFC2680]) Metric Parameters: 312 + Src, the IP address of a host (12.3.167.16 or 193.159.144.8) 314 + Dst, the IP address of a host (193.159.144.8 or 12.3.167.16) 316 + T0, a time 317 + Tf, a time 319 + lambda, a rate in reciprocal seconds 321 + Thresh, a maximum waiting time in seconds (see Section 2.8.2 of 322 [RFC2680]) and (Section 3.8. [RFC2680]) 324 Metric Units: A sequence of pairs; the elements of each pair are: 326 + T, a time, and 328 + L, either a zero or a one 330 The values of T in the sequence are monotonic increasing. Note that 331 T would be a valid parameter to the *singleton* Type-P-One-way- 332 Packet-Loss, and that L would be a valid value of Type-P-One-way- 333 Packet Loss (see Section 2 of [RFC2680]). 335 Also, Section 2.8.4 of [RFC2680] recommends that the path SHOULD be 336 reported. In this test set-up, most of the path details will be 337 concealed from the implementations by the L2TPv3 tunnels, thus a more 338 informative path trace route can be conducted by the routers at each 339 location. 341 When NetProbe is used in production, a traceroute is conducted in 342 parallel at the outset of measurements. 344 Perfas+ does not support traceroute. 346 IPLGW#traceroute 193.159.144.8 348 Type escape sequence to abort. 349 Tracing the route to 193.159.144.8 351 1 12.126.218.245 [AS 7018] 0 msec 0 msec 4 msec 352 2 cr84.n54ny.ip.att.net (12.123.2.158) [AS 7018] 4 msec 4 msec 353 cr83.n54ny.ip.att.net (12.123.2.26) [AS 7018] 4 msec 354 3 cr1.n54ny.ip.att.net (12.122.105.49) [AS 7018] 4 msec 355 cr2.n54ny.ip.att.net (12.122.115.93) [AS 7018] 0 msec 356 cr1.n54ny.ip.att.net (12.122.105.49) [AS 7018] 0 msec 357 4 n54ny02jt.ip.att.net (12.122.80.225) [AS 7018] 4 msec 0 msec 358 n54ny02jt.ip.att.net (12.122.80.237) [AS 7018] 4 msec 359 5 192.205.34.182 [AS 7018] 0 msec 360 192.205.34.150 [AS 7018] 0 msec 361 192.205.34.182 [AS 7018] 4 msec 362 6 da-rg12-i.DA.DE.NET.DTAG.DE (62.154.1.30) [AS 3320] 88 msec 88 msec 363 88 msec 364 7 217.89.29.62 [AS 3320] 88 msec 88 msec 88 msec 365 8 217.89.29.55 [AS 3320] 88 msec 88 msec 88 msec 366 9 * * * 368 It was only possible to conduct the traceroute for the measured path 369 on one of the tunnel-head routers (the normal trace facilities of the 370 measurement systems are confounded by the L2TPv3 tunnel 371 encapsulation). 373 4. Error Calibration, RFC 2680 375 An implementation is required to report calibration results on clock 376 synchronization in Section 2.8.3 of [RFC2680] (also required in 377 Section 3.7 of [RFC2680] for sample metrics). 379 Also, it is recommended to report the probability that a packet 380 successfully arriving at the destination network interface is 381 incorrectly designated as lost due to resource exhaustion in Section 382 2.8.3 of [RFC2680]. 384 4.1. Clock Synchronization Calibration 386 For NetProbe and Perfas+ clock synchronization test results, refer to 387 Section 4 of [RFC6808]. 389 4.2. Packet Loss Determination Error 391 Since both measurement implementations have resource limitations, it 392 is theoretically possible that these limits could be exceeded and a 393 packet that arrived at the destination successfully might be 394 discarded in error. 396 In previous test efforts [I-D.morton-ippm-advance-metrics], NetProbe 397 produced 6 multicast streams with an aggregate bit rate over 53 398 Mbit/s, in order to characterize the 1-way capacity of a NISTNet- 399 based emulator. Neither the emulator nor the pair of NetProbe 400 implementations used in this testing dropped any packets in these 401 streams. 403 The maximum load used here between any 2 NetProbe implementations was 404 be 11.5 Mbit/s divided equally among 3 unicast test streams. We 405 conclude that steady resource usage does not contribute error 406 (additional loss) to the measurements. 408 5. Pre-determined Limits on Equivalence 410 In this section, we provide the numerical limits on comparisons 411 between implementations, in order to declare that the results are 412 equivalent and therefore, the tested specification is clear. 414 A key point is that the allowable errors, corrections, and confidence 415 levels only need to be sufficient to detect mis-interpretation of the 416 tested specification resulting in diverging implementations. 418 Also, the allowable error must be sufficient to compensate for 419 measured path differences. It was simply not possible to measure 420 fully identical paths in the VLAN-loopback test configuration used, 421 and this practical compromise must be taken into account. 423 For Anderson-Darling K-sample (ADK) [ADK] comparisons, the required 424 confidence factor for the cross-implementation comparisons SHALL be 425 the smallest of: 427 o 0.95 confidence factor at 1 packet resolution, or 429 o the smallest confidence factor (in combination with resolution) of 430 the two same-implementation comparisons for the same test 431 conditions (if the number of streams is sufficient to allow such 432 comparisons). 434 For Anderson-Darling Goodness-of-Fit (ADGoF) [Radgof] comparisons, 435 the required level of significance for the same-implementation 436 Goodness-of-Fit (GoF) SHALL be 0.05 or 5%, as specified in Section 437 11.4 of [RFC2330]. This is equivalent to a 95% confidence factor. 439 6. Tests to evaluate RFC 2680 Specifications 441 This section describes some results from production network (cross- 442 Internet) tests with measurement devices implementing IPPM metrics 443 and a network emulator to create relevant conditions, to determine 444 whether the metric definitions were interpreted consistently by 445 implementors. 447 The procedures are similar contained in Appendix A.1 of [RFC6576] for 448 One-way Delay. 450 6.1. One-way Loss, ADK Sample Comparison 452 This test determines if implementations produce results that appear 453 to come from a common packet loss distribution, as an overall 454 evaluation of Section 3 of [RFC2680], "A Definition for Samples of 455 One-way Packet Loss". Same-implementation comparison results help to 456 set the threshold of equivalence that will be applied to cross- 457 implementation comparisons. 459 This test is intended to evaluate measurements in sections 2, 3, and 460 4 of [RFC2680]. 462 By testing the extent to which the counts of one-way packet loss 463 counts on different test streams of two [RFC2680] implementations 464 appear to be from the same loss process, we reduce comparison steps 465 because comparing the resulting summary statistics (as defined in 466 Section 4 of [RFC2680]) would require a redundant set of equivalence 467 evaluations. We can easily check whether the single statistic in 468 Section 4 of [RFC2680] was implemented, and report on that fact. 470 1. Configure an L2TPv3 path between test sites, and each pair of 471 measurement devices to operate tests in their designated pair of 472 VLANs. 474 2. Measure a sample of one-way packet loss singletons with 2 or more 475 implementations, using identical options and network emulator 476 settings (if used). 478 3. Measure a sample of one-way packet loss singletons with *four or 479 more* instances of the *same* implementations, using identical 480 options, noting that connectivity differences SHOULD be the same 481 as for the cross implementation testing. 483 4. If less than ten test streams are available, skip to step 7. 485 5. Apply the ADK comparison procedures (see Appendix C of [RFC6576]) 486 and determine the resolution and confidence factor for 487 distribution equivalence of each same-implementation comparison 488 and each cross-implementation comparison. 490 6. Take the coarsest resolution and confidence factor for 491 distribution equivalence from the same-implementation pairs, or 492 the limit defined in Section 5 above, as a limit on the 493 equivalence threshold for these experimental conditions. 495 7. Compare the cross-implementation ADK performance with the 496 equivalence threshold determined in step 5 to determine if 497 equivalence can be declared. 499 The common parameters used for tests in this section are: 501 The cross-implementation comparison uses a simple ADK analysis 502 [Rtool] [Radk], where all NetProbe loss counts are compared with all 503 Perfas+ loss results. 505 In the result analysis of this section: 507 o All comparisons used 1 packet resolution. 509 o No Correction Factors were applied. 511 o The 0.95 confidence factor (1.960 for cross-implementation 512 comparison) was used. 514 6.1.1. 340B/Periodic Cross-imp. results 516 Tests described in this section used: 518 o IP header + payload = 340 octets 520 o Periodic sampling at 1 packet per second 522 o Test duration = 1200 seconds (during April 7, 2011, EDT) 524 The netem emulator was set for 100ms constant delay, with 10% loss 525 ratio. In this experiment, the netem emulator was configured to 526 operate independently on each VLAN and thus the emulator itself is a 527 potential source of error when comparing streams that traverse the 528 test path in different directions. 530 A07bps_loss <- c(114, 175, 138, 142, 181, 105) (NetProbe) 531 A07per_loss <- c(115, 128, 136, 127, 139, 138) (Perfas+) 533 > A07bps_loss <- c(114, 175, 138, 142, 181, 105) 534 > A07per_loss <- c(115, 128, 136, 127, 139, 138) 535 > 536 > A07cross_loss_ADK <- adk.test(A07bps_loss, A07per_loss) 537 > A07cross_loss_ADK 538 Anderson-Darling k-sample test. 540 Number of samples: 2 541 Sample sizes: 6 6 542 Total number of values: 12 543 Number of unique values: 11 545 Mean of Anderson Darling Criterion: 1 546 Standard deviation of Anderson Darling Criterion: 0.6569 548 T = (Anderson Darling Criterion - mean)/sigma 550 Null Hypothesis: All samples come from a common population. 552 t.obs P-value extrapolation 553 not adj. for ties 0.52043 0.20604 0 554 adj. for ties 0.62679 0.18607 0 556 The cross-implementation comparisons pass the ADK criterion. 558 6.1.2. 64B/Periodic Cross-imp. results 560 Tests described in this section used: 562 o IP header + payload = 64 octets 564 o Periodic sampling at 1 packet per second 566 o Test duration = 300 seconds (during March 24, 2011, EDT) 568 The netem emulator was set for 0ms constant delay, with 10% loss 569 ratio. 571 > M24per_loss <- c(42,34,35,35) (Perfas+) 572 > M24apd_23BC_loss <- c(27,39,29,24) (NetProbe) 573 > M24apd_loss23BC_ADK <- adk.test(M24apd_23BC_loss,M24per_loss) 574 > M24apd_loss23BC_ADK 575 Anderson-Darling k-sample test. 577 Number of samples: 2 578 Sample sizes: 4 4 579 Total number of values: 8 580 Number of unique values: 7 582 Mean of Anderson Darling Criterion: 1 583 Standard deviation of Anderson Darling Criterion: 0.60978 585 T = (Anderson Darling Criterion - mean)/sigma 587 Null Hypothesis: All samples come from a common population. 589 t.obs P-value extrapolation 590 not adj. for ties 0.76921 0.16200 0 591 adj. for ties 0.90935 0.14113 0 593 Warning: At least one sample size is less than 5. 594 p-values may not be very accurate. 596 The cross-implementation comparisons pass the ADK criterion. 598 6.1.3. 64B/Poisson Cross-imp. results 600 Tests described in this section used: 602 o IP header + payload = 64 octets 604 o Poisson sampling at lambda = 1 packet per second 606 o Test duration = 20 minutes (during April 27, 2011, EDT) 608 The netem configuration was 0ms delay and 10% loss, but there were 609 two passes through an emulator for each stream, and loss emulation 610 was present for 18 minutes of the 20 minute test . 612 A27aps_loss <- c(91,110,113,102,111,109,112,113) (NetProbe) 613 A27per_loss <- c(95,123,126,114) (Perfas+) 615 A27cross_loss_ADK <- adk.test(A27aps_loss, A27per_loss) 617 > A27cross_loss_ADK 618 Anderson-Darling k-sample test. 620 Number of samples: 2 621 Sample sizes: 8 4 622 Total number of values: 12 623 Number of unique values: 11 625 Mean of Anderson Darling Criterion: 1 626 Standard deviation of Anderson Darling Criterion: 0.65642 628 T = (Anderson Darling Criterion - mean)/sigma 630 Null Hypothesis: All samples come from a common population. 632 t.obs P-value extrapolation 633 not adj. for ties 2.15099 0.04145 0 634 adj. for ties 1.93129 0.05125 0 636 Warning: At least one sample size is less than 5. 637 p-values may not be very accurate. 638 > 640 The cross-implementation comparisons barely pass the ADK criterion at 641 95% = 1.960 when adjusting for ties. 643 6.1.4. Conclusions on the ADK Results for One-way Packet Loss 645 We conclude that the two implementations are capable of producing 646 equivalent one-way packet loss measurements based on their 647 interpretation of [RFC2680] . 649 6.2. One-way Loss, Delay threshold 651 This test determines if implementations use the same configured 652 maximum waiting time delay from one measurement to another under 653 different delay conditions, and correctly declare packets arriving in 654 excess of the waiting time threshold as lost. 656 See Section 2.8.2 of [RFC2680]. 658 1. configure an L2TPv3 path between test sites, and each pair of 659 measurement devices to operate tests in their designated pair of 660 VLANs. 662 2. configure the network emulator to add 1.0 sec one-way constant 663 delay in one direction of transmission. 665 3. measure (average) one-way delay with 2 or more implementations, 666 using identical waiting time thresholds (Thresh) for loss set at 667 3 seconds. 669 4. configure the network emulator to add 3 sec one-way constant 670 delay in one direction of transmission equivalent to 2 seconds of 671 additional one-way delay (or change the path delay while test is 672 in progress, when there are sufficient packets at the first delay 673 setting) 675 5. repeat/continue measurements 677 6. observe that the increase measured in step 5 caused all packets 678 with 2 sec additional delay to be declared lost, and that all 679 packets that arrive successfully in step 3 are assigned a valid 680 one-way delay. 682 The common parameters used for tests in this section are: 684 o IP header + payload = 64 octets 686 o Poisson sampling at lambda = 1 packet per second 688 o Test duration = 900 seconds total (March 21) 690 The netem emulator was set to add constant delays as specified in the 691 procedure above. 693 6.2.1. NetProbe results for Loss Threshold 695 In NetProbe, the Loss Threshold is implemented uniformly over all 696 packets as a post-processing routine. With the Loss Threshold set at 697 3 seconds, all packets with one-way delay >3 seconds are marked 698 "Lost" and included in the Lost Packet list with their transmission 699 time (as required in Section 3.3 of [RFC2680]). This resulted in 342 700 packets designated as lost in one of the test streams (with average 701 delay = 3.091 sec). 703 6.2.2. Perfas Results for Loss Threshold 705 Perfas+ uses a fixed Loss Threshold which was not adjustable during 706 this study. The Loss Threshold is approximately one minute, and 707 emulation of a delay of this size was not attempted. However, it is 708 possible to implement any delay threshold desired with a post- 709 processing routine and subsequent analysis. Using this method, 195 710 packets would be declared lost (with average delay = 3.091 sec). 712 6.2.3. Conclusions for Loss Threshold 714 Both implementations assume that any constant delay value desired can 715 be used as the Loss Threshold, since all delays are stored as a pair 716 as required in [RFC2680]. This is a simple way to 717 enforce the constant loss threshold envisioned in [RFC2680] (see 718 specific section reference above). We take the position that the 719 assumption of post-processing is compliant, and that the text of the 720 RFC should be revised slightly to include this point. 722 6.3. One-way Loss with Out-of-Order Arrival 724 Section 3.6 of [RFC2680] indicates that implementations need to 725 ensure that reordered packets are handled correctly using an 726 uncapitalized "must". In essence, this is an implied requirement 727 because the correct packet must be identified as lost if it fails to 728 arrive before its delay threshold under all circumstances, and 729 reordering is always a possibility on IP network paths. See 730 [RFC4737] for the definition of reordering used in IETF standard- 731 compliant measurements. 733 Using the procedure of section 6.1, the netem emulator was set to 734 introduce significant delay (2000 ms) and delay variation (1000 ms), 735 which was sufficient to produce packet reordering because each 736 packet's emulated delay is independent from others, and 10% loss. 738 The tests described in this section used: 740 o IP header + payload = 64 octets 742 o Periodic sampling = 1 packet per second 744 o Test duration = 600 seconds (during May 2, 2011, EDT) 745 > Y02aps_loss <- c(53,45,67,55) (NetProbe) 746 > Y02per_loss <- c(59,62,67,69) (Perfas+) 747 > Y02cross_loss_ADK <- adk.test(Y02aps_loss, Y02per_loss) 748 > Y02cross_loss_ADK 749 Anderson-Darling k-sample test. 751 Number of samples: 2 752 Sample sizes: 4 4 753 Total number of values: 8 754 Number of unique values: 7 756 Mean of Anderson Darling Criterion: 1 757 Standard deviation of Anderson Darling Criterion: 0.60978 759 T = (Anderson Darling Criterion - mean)/sigma 761 Null Hypothesis: All samples come from a common population. 763 t.obs P-value extrapolation 764 not adj. for ties 1.11282 0.11531 0 765 adj. for ties 1.19571 0.10616 0 767 Warning: At least one sample size is less than 5. 768 p-values may not be very accurate. 769 > 771 The test results indicate that extensive reordering was present. 772 Both implementations capture the extensive delay variation between 773 adjacent packets. In NetProbe, packet arrival order is preserved in 774 the raw measurement files, so an examination of arrival packet 775 sequence numbers also indicates reordering. 777 Despite extensive continuous packet reordering present in the 778 transmission path, the distributions of loss counts from the two 779 implementations pass the ADK criterion at 95% = 1.960. 781 6.4. Poisson Sending Process Evaluation 783 Section 3.7 of [RFC2680] indicates that implementations need to 784 ensure that their sending process is reasonably close to a classic 785 Poisson distribution when used. Much more detail on sample 786 distribution generation and Goodness-of-Fit testing is specified in 787 Section 11.4 of [RFC2330] and the Appendix of [RFC2330]. 789 In this section, each implementation's Poisson distribution is 790 compared with an idealistic version of the distribution available in 791 the base functionality of the R-tool for Statistical Analysis[Rtool], 792 and performed using the Anderson-Darling Goodness-of-Fit test package 793 (ADGofTest) [Radgof]. The Goodness-of-Fit criterion derived from 794 [RFC2330] requires a test statistic value AD <= 2.492 for 5% 795 significance. The Appendix of [RFC2330] also notes that there may be 796 difficulty satisfying the ADGofTest when the sample includes many 797 packets (when 8192 were used, the test always failed, but smaller 798 sets of the stream passed). 800 Both implementations were configured to produce Poisson distributions 801 with lambda = 1 packet per second. 803 6.4.1. NetProbe Results 805 Section 11.4 of [RFC2330] suggests three possible measurement points 806 to evaluate the Poisson distribution. The NetProbe analysis uses 807 "user-level timestamps made just before or after the system call for 808 transmitting the packet". 810 The statistical summary for two NetProbe streams is below: 812 > summary(a27ms$s1[2:1152]) 813 Min. 1st Qu. Median Mean 3rd Qu. Max. 814 0.0100 0.2900 0.6600 0.9846 1.3800 8.6390 815 > summary(a27ms$s2[2:1152]) 816 Min. 1st Qu. Median Mean 3rd Qu. Max. 817 0.010 0.280 0.670 0.979 1.365 8.829 819 We see that both the Means are near the specified lambda = 1. 821 The results of ADGoF tests for these two streams is shown below: 823 > ad.test( a27ms$s1[2:101], pexp, 1) 825 Anderson-Darling GoF Test 827 data: a27ms$s1[2:101] and pexp 828 AD = 0.8908, p-value = 0.4197 829 alternative hypothesis: NA 831 > ad.test( a27ms$s1[2:1001], pexp, 1) 833 Anderson-Darling GoF Test 835 data: a27ms$s1[2:1001] and pexp 836 AD = 0.9284, p-value = 0.3971 837 alternative hypothesis: NA 839 > ad.test( a27ms$s2[2:101], pexp, 1) 841 Anderson-Darling GoF Test 843 data: a27ms$s2[2:101] and pexp 844 AD = 0.3597, p-value = 0.8873 845 alternative hypothesis: NA 847 > ad.test( a27ms$s2[2:1001], pexp, 1) 849 Anderson-Darling GoF Test 851 data: a27ms$s2[2:1001] and pexp 852 AD = 0.6913, p-value = 0.5661 853 alternative hypothesis: NA 855 We see that both 100 and 1000 packet sets from two different streams 856 (s1 and s2) all passed the AD <= 2.492 criterion. 858 6.4.2. Perfas+ Results 860 Section 11.4 of [RFC2330] suggests three possible measurement points 861 to evaluate the Poisson distribution. The Perfas+ analysis uses 862 "wire times for the packets as recorded using a packet filter". 863 However, due to limited access at the Perfas+ side of the test setup, 864 the captures were made after the Perfas+ streams traversed the 865 production network, adding a small amount of unwanted delay variation 866 to the wire times (and possibly error due to packet loss). 868 The statistical summary for two Perfas+ streams is below: 870 > summary(a27pe$p1) 871 Min. 1st Qu. Median Mean 3rd Qu. Max. 872 0.004 0.347 0.788 1.054 1.548 4.231 873 > summary(a27pe$p2) 874 Min. 1st Qu. Median Mean 3rd Qu. Max. 875 0.0010 0.2710 0.7080 0.9696 1.3740 7.1160 877 We see that both the Means are near the specified lambda = 1. 879 The results of ADGoF tests for these two streams is shown below: 881 > ad.test(a27pe$p1, pexp, 1 ) 883 Anderson-Darling GoF Test 885 data: a27pe$p1 and pexp 886 AD = 1.1364, p-value = 0.2930 887 alternative hypothesis: NA 889 > ad.test(a27pe$p2, pexp, 1 ) 891 Anderson-Darling GoF Test 893 data: a27pe$p2 and pexp 894 AD = 0.5041, p-value = 0.7424 895 alternative hypothesis: NA 897 > ad.test(a27pe$p1[1:100], pexp, 1 ) 899 Anderson-Darling GoF Test 901 data: a27pe$p1[1:100] and pexp 902 AD = 0.7202, p-value = 0.5419 903 alternative hypothesis: NA 905 > ad.test(a27pe$p1[101:193], pexp, 1 ) 907 Anderson-Darling GoF Test 909 data: a27pe$p1[101:193] and pexp 910 AD = 1.4046, p-value = 0.201 911 alternative hypothesis: NA 913 > ad.test(a27pe$p2[1:100], pexp, 1 ) 915 Anderson-Darling GoF Test 917 data: a27pe$p2[1:100] and pexp 918 AD = 0.4758, p-value = 0.7712 919 alternative hypothesis: NA 921 > ad.test(a27pe$p2[101:193], pexp, 1 ) 923 Anderson-Darling GoF Test 925 data: a27pe$p2[101:193] and pexp 926 AD = 0.3381, p-value = 0.9068 927 alternative hypothesis: NA 929 > 931 We see that both 193, 100, and 93 packet sets from two different 932 streams (p1 and p2) all passed the AD <= 2.492 criterion. 934 6.4.3. Conclusions for Goodness-of-Fit 936 Both NetProbe and Perfas+ implementations produce adequate Poisson 937 distributions when according to the Anderson-Darling Goodness-of-Fit 938 at the 5% significance (1-alpha = 0.05, or 95% confidence level). 940 6.5. Implementation of Statistics for One-way Loss 942 We check which statistics were implemented, and report on those 943 facts, noting that Section 4 of [RFC2680] does not specify the 944 calculations exactly, and gives only some illustrative examples. 946 NetProbe Perfas 948 4.1. Type-P-One-way-Packet-Loss-Average yes yes 949 (this is more commonly referred to as loss ratio) 951 Implementation of Section 4 Statistics 953 We note that implementations refer to this metric as a loss ratio, 954 and this is an area for likely revision of the text to make it more 955 consistent with wide-spread usage. 957 7. Conclusions for RFC 2680bis 959 This memo concludes that [RFC2680] should be advanced on the 960 standards track, and recommends the following edits to improve the 961 text (which are not deemed significant enough to affect maturity). 963 o Revise Type-P-One-way-Packet-Loss-Ave to Type-P-One-way-Delay- 964 Packet-Loss-Ratio 966 o Regarding implementation of the loss delay threshold (section 967 6.2), the assumption of post-processing is compliant, and the text 968 of RFC 2680bis should be revised slightly to include this point. 970 o The IETF has reached consensus on guidance for reporting metrics 971 in [RFC6703], and this memo should be referenced in RFC2680bis to 972 incorporate recent experience where appropriate. 974 We note that there are at least two Errata on [RFC2680] and these 975 should be processed as part of the editing process. 977 We recognize the existence of BCP 170 [RFC6390] providing guidelines 978 for development of drafts describing new performance metrics. 979 However, the advancement of [RFC2680] represents fine-tuning of long- 980 standing specifications based on experience that helped to formulate 981 BCP 170, and material that satisfies some of the requirements of 982 [RFC6390] can be found in other RFCs, such as the IPPM Framework 983 [RFC2330]. Thus, no specific changes to address BCP 170 guidelines 984 are recommended for RFC 2680bis. 986 8. Security Considerations 988 The security considerations that apply to any active measurement of 989 live networks are relevant here as well. See [RFC4656] and 990 [RFC5357]. 992 9. IANA Considerations 994 This memo makes no requests of IANA, and the authors hope that IANA 995 personnel will be able to use their valuable time in other worthwhile 996 pursuits. 998 10. Acknowledgements 1000 The authors thank Lars Eggert for his continued encouragement to 1001 advance the IPPM metrics during his tenure as AD Advisor. 1003 Nicole Kowalski supplied the needed CPE router for the NetProbe side 1004 of the test set-up, and graciously managed her testing in spite of 1005 issues caused by dual-use of the router. Thanks Nicole! 1007 The "NetProbe Team" also acknowledges many useful discussions on 1008 statistical interpretation with Ganga Maguluri. 1010 11. References 1012 11.1. Normative References 1014 [RFC2026] Bradner, S., "The Internet Standards Process -- Revision 1015 3", BCP 9, RFC 2026, October 1996. 1017 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1018 Requirement Levels", BCP 14, RFC 2119, March 1997. 1020 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 1021 "Framework for IP Performance Metrics", RFC 2330, 1022 May 1998. 1024 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1025 Delay Metric for IPPM", RFC 2679, September 1999. 1027 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1028 Packet Loss Metric for IPPM", RFC 2680, September 1999. 1030 [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network 1031 performance measurement with periodic streams", RFC 3432, 1032 November 2002. 1034 [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. 1035 Zekauskas, "A One-way Active Measurement Protocol 1036 (OWAMP)", RFC 4656, September 2006. 1038 [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, 1039 S., and J. Perser, "Packet Reordering Metrics", RFC 4737, 1040 November 2006. 1042 [RFC4814] Newman, D. and T. Player, "Hash and Stuffing: Overlooked 1043 Factors in Network Device Benchmarking", RFC 4814, 1044 March 2007. 1046 [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an 1047 IANA Considerations Section in RFCs", BCP 26, RFC 5226, 1048 May 2008. 1050 [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. 1051 Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", 1052 RFC 5357, October 2008. 1054 [RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation 1055 and Implementation Reports for Advancement to Draft 1056 Standard", BCP 9, RFC 5657, September 2009. 1058 [RFC6390] Clark, A. and B. Claise, "Guidelines for Considering New 1059 Performance Metric Development", BCP 170, RFC 6390, 1060 October 2011. 1062 [RFC6576] Geib, R., Morton, A., Fardid, R., and A. Steinmitz, "IP 1063 Performance Metrics (IPPM) Standard Advancement Testing", 1064 BCP 176, RFC 6576, March 2012. 1066 [RFC6703] Morton, A., Ramachandran, G., and G. Maguluri, "Reporting 1067 IP Network Performance Metrics: Different Points of View", 1068 RFC 6703, August 2012. 1070 [RFC6808] Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test 1071 Plan and Results Supporting Advancement of RFC 2679 on the 1072 Standards Track", RFC 6808, December 2012. 1074 11.2. Informative References 1076 [ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling 1077 Tests of fit, for continuous and discrete cases", 1078 University of Washington, Technical Report No. 81, 1079 May 1986. 1081 [I-D.morton-ippm-advance-metrics] 1082 Morton, A., "Lab Test Results for Advancing Metrics on the 1083 Standards Track", draft-morton-ippm-advance-metrics-02 1084 (work in progress), October 2010. 1086 [Perfas] Heidemann, C., "Qualitaet in IP-Netzen Messverfahren", 1087 published by ITG Fachgruppe, 2nd meeting 5.2.3 (NGN) http: 1088 //www.itg523.de/oeffentlich/01nov/ 1089 Heidemann_QOS_Messverfahren.pdf , November 2001. 1091 [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling 1092 Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. 1094 [Radgof] Bellosta, C., "ADGofTest: Anderson-Darling Goodness-of-Fit 1095 Test. R package version 0.3.", http://cran.r-project.org/ 1096 web/packages/ADGofTest/index.html, December 2011. 1098 [Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and 1099 Combinations of Such Tests. R package version 1.0.", , 1100 2008. 1102 [Rtool] R Development Core Team, "R: A language and environment 1103 for statistical computing. R Foundation for Statistical 1104 Computing, Vienna, Austria. ISBN 3-900051-07-0, URL 1105 http://www.R-project.org/", , 2011. 1107 [WIPM] "AT&T Global IP Network", 1108 http://ipnetwork.bgtmo.ip.att.net/pws/index.html, 2012. 1110 Authors' Addresses 1112 Len Ciavattone 1113 AT&T Labs 1114 200 Laurel Avenue South 1115 Middletown, NJ 07748 1116 USA 1118 Phone: +1 732 420 1239 1119 Fax: 1120 Email: lencia@att.com 1121 URI: 1123 Ruediger Geib 1124 Deutsche Telekom 1125 Heinrich Hertz Str. 3-7 1126 Darmstadt, 64295 1127 Germany 1129 Phone: +49 6151 58 12747 1130 Email: Ruediger.Geib@telekom.de 1132 Al Morton 1133 AT&T Labs 1134 200 Laurel Avenue South 1135 Middletown, NJ 07748 1136 USA 1138 Phone: +1 732 420 1571 1139 Fax: +1 732 368 1192 1140 Email: acmorton@att.com 1141 URI: http://home.comcast.net/~acmacm/ 1142 Matthias Wieser 1143 Technical University Darmstadt 1144 Darmstadt, 1145 Germany 1147 Phone: 1148 Email: matthias_michael.wieser@stud.tu-darmstadt.de