| < draft-ietf-ippm-metrictest-04.txt | draft-ietf-ippm-metrictest-05.txt > | |||
|---|---|---|---|---|
| Internet Engineering Task Force R. Geib, Ed. | Internet Engineering Task Force R. Geib, Ed. | |||
| Internet-Draft Deutsche Telekom | Internet-Draft Deutsche Telekom | |||
| Intended status: Standards Track A. Morton | Intended status: BCP A. Morton | |||
| Expires: April 26, 2012 AT&T Labs | Expires: June 1, 2012 AT&T Labs | |||
| R. Fardid | R. Fardid | |||
| Cariden Technologies | Cariden Technologies | |||
| A. Steinmitz | A. Steinmitz | |||
| Deutsche Telekom | Deutsche Telekom | |||
| October 24, 2011 | November 29, 2011 | |||
| IPPM standard advancement testing | IPPM standard advancement testing | |||
| draft-ietf-ippm-metrictest-04 | draft-ietf-ippm-metrictest-05 | |||
| Abstract | Abstract | |||
| This document specifies tests to determine if multiple independent | This document specifies tests to determine if multiple independent | |||
| instantiations of a performance metric RFC have implemented the | instantiations of a performance metric RFC have implemented the | |||
| specifications in the same way. This is the performance metric | specifications in the same way. This is the performance metric | |||
| equivalent of interoperability, required to advance RFCs along the | equivalent of interoperability, required to advance RFCs along the | |||
| standards track. Results from different implementations of metric | standards track. Results from different implementations of metric | |||
| RFCs will be collected under the same underlying network conditions | RFCs will be collected under the same underlying network conditions | |||
| and compared using state of the art statistical methods. The goal is | and compared using statistical methods. The goal is an evaluation of | |||
| an evaluation of the metric RFC itself, whether its definitions are | the metric RFC itself; whether its definitions are clear and | |||
| clear and unambiguous to implementors and therefore a candidate for | unambiguous to implementors and therefore a candidate for advancement | |||
| advancement on the IETF standards track. | on the IETF standards track. This document is an Internet Best | |||
| Current Practice. | ||||
| Status of this Memo | Status of this Memo | |||
| This Internet-Draft is submitted in full conformance with the | This Internet-Draft is submitted in full conformance with the | |||
| provisions of BCP 78 and BCP 79. | provisions of BCP 78 and BCP 79. | |||
| Internet-Drafts are working documents of the Internet Engineering | Internet-Drafts are working documents of the Internet Engineering | |||
| Task Force (IETF). Note that other groups may also distribute | Task Force (IETF). Note that other groups may also distribute | |||
| working documents as Internet-Drafts. The list of current Internet- | working documents as Internet-Drafts. The list of current Internet- | |||
| Drafts is at http://datatracker.ietf.org/drafts/current/. | Drafts is at http://datatracker.ietf.org/drafts/current/. | |||
| Internet-Drafts are draft documents valid for a maximum of six months | Internet-Drafts are draft documents valid for a maximum of six months | |||
| and may be updated, replaced, or obsoleted by other documents at any | and may be updated, replaced, or obsoleted by other documents at any | |||
| time. It is inappropriate to use Internet-Drafts as reference | time. It is inappropriate to use Internet-Drafts as reference | |||
| material or to cite them other than as "work in progress." | material or to cite them other than as "work in progress." | |||
| This Internet-Draft will expire on April 26, 2012. | This Internet-Draft will expire on June 1, 2012. | |||
| Copyright Notice | Copyright Notice | |||
| Copyright (c) 2011 IETF Trust and the persons identified as the | Copyright (c) 2011 IETF Trust and the persons identified as the | |||
| document authors. All rights reserved. | document authors. All rights reserved. | |||
| This document is subject to BCP 78 and the IETF Trust's Legal | This document is subject to BCP 78 and the IETF Trust's Legal | |||
| Provisions Relating to IETF Documents | Provisions Relating to IETF Documents | |||
| (http://trustee.ietf.org/license-info) in effect on the date of | (http://trustee.ietf.org/license-info) in effect on the date of | |||
| publication of this document. Please review these documents | publication of this document. Please review these documents | |||
| carefully, as they describe your rights and restrictions with respect | carefully, as they describe your rights and restrictions with respect | |||
| to this document. Code Components extracted from this document must | to this document. Code Components extracted from this document must | |||
| include Simplified BSD License text as described in Section 4.e of | include Simplified BSD License text as described in Section 4.e of | |||
| the Trust Legal Provisions and are provided without warranty as | the Trust Legal Provisions and are provided without warranty as | |||
| described in the Simplified BSD License. | described in the Simplified BSD License. | |||
| Table of Contents | Table of Contents | |||
| 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 | 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 | |||
| 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 7 | 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 5 | |||
| 2. Basic idea . . . . . . . . . . . . . . . . . . . . . . . . . . 7 | 2. Basic idea . . . . . . . . . . . . . . . . . . . . . . . . . . 5 | |||
| 3. Verification of conformance to a metric specification . . . . 8 | 3. Verification of conformance to a metric specification . . . . 7 | |||
| 3.1. Tests of an individual implementation against a metric | 3.1. Tests of an individual implementation against a metric | |||
| specification . . . . . . . . . . . . . . . . . . . . . . 9 | specification . . . . . . . . . . . . . . . . . . . . . . 8 | |||
| 3.2. Test setup resulting in identical live network testing | 3.2. Test setup resulting in identical live network testing | |||
| conditions . . . . . . . . . . . . . . . . . . . . . . . . 11 | conditions . . . . . . . . . . . . . . . . . . . . . . . . 9 | |||
| 3.3. Tests of two or more different implementations against | 3.3. Tests of two or more different implementations against | |||
| a metric specification . . . . . . . . . . . . . . . . . . 16 | a metric specification . . . . . . . . . . . . . . . . . . 15 | |||
| 3.4. Clock synchronisation . . . . . . . . . . . . . . . . . . 17 | 3.4. Clock synchronisation . . . . . . . . . . . . . . . . . . 16 | |||
| 3.5. Recommended Metric Verification Measurement Process . . . 18 | 3.5. Recommended Metric Verification Measurement Process . . . 17 | |||
| 3.6. Proposal to determine an "equivalence" threshold for | 3.6. Proposal to determine an "equivalence" threshold for | |||
| each metric evaluated . . . . . . . . . . . . . . . . . . 21 | each metric evaluated . . . . . . . . . . . . . . . . . . 20 | |||
| 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 22 | 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 21 | |||
| 5. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 22 | 5. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 21 | |||
| 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 | 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 | |||
| 7. Security Considerations . . . . . . . . . . . . . . . . . . . 23 | 7. Security Considerations . . . . . . . . . . . . . . . . . . . 21 | |||
| 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 23 | 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21 | |||
| 8.1. Normative References . . . . . . . . . . . . . . . . . . . 23 | 8.1. Normative References . . . . . . . . . . . . . . . . . . . 21 | |||
| 8.2. Informative References . . . . . . . . . . . . . . . . . . 24 | 8.2. Informative References . . . . . . . . . . . . . . . . . . 22 | |||
| Appendix A. An example on a One-way Delay metric validation . . . 25 | Appendix A. An example on a One-way Delay metric validation . . . 23 | |||
| A.1. Compliance to Metric specification requirements . . . . . 25 | A.1. Compliance to Metric specification requirements . . . . . 23 | |||
| A.2. Examples related to statistical tests for One-way Delay . 27 | A.2. Examples related to statistical tests for One-way Delay . 25 | |||
| Appendix B. Anderson-Darling K-sample Reference and 2 sample | Appendix B. Anderson-Darling K-sample Reference and 2 sample | |||
| C++ code . . . . . . . . . . . . . . . . . . . . . . 29 | C++ code . . . . . . . . . . . . . . . . . . . . . . 27 | |||
| Appendix C. Glossary . . . . . . . . . . . . . . . . . . . . . . 37 | Appendix C. Glossary . . . . . . . . . . . . . . . . . . . . . . 36 | |||
| Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 38 | Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 36 | |||
| 1. Introduction | 1. Introduction | |||
| The Internet Standards Process RFC2026 [RFC2026] requires that for a | The Internet Standards Process as updated by RFC6410 [RFC6410] | |||
| IETF specification to advance beyond the Proposed Standard level, at | specifies that widespread deployment and use is sufficient to show | |||
| least two genetically unrelated implementations must be shown to | interoperability as a condition for advancement to Internet Standard. | |||
| interoperate correctly with all features and options. This | The previous requirement of interoperability tests prior to advancing | |||
| requirement can be met by supplying: | an RFC to the Standard maturity level specified in RFC 2026 [RFC2026] | |||
| and RFC 5657 [RFC5657] has been removed. While the modified | ||||
| o evidence that (at least a sub-set of) the specification has been | requirement is applicable to protocols, wide deployment of different | |||
| implemented by multiple parties, thus indicating adoption by the | measurement systems does not prove that the implementations measure | |||
| IETF community and the extent of feature coverage. | metrics in a standard way. Section 5.3 of RFC 5657 [RFC5657] | |||
| explicitly mentions the special case of Standards that are not "on- | ||||
| o evidence that each feature of the specification is sufficiently | the-wire" protocols. While this special case is not explicitly | |||
| well-described to support interoperability, as demonstrated | mentioned by RFC6410 [RFC6410], the four criteria in Section 2.2 of | |||
| through testing and/or user experience with deployment. | RFC6410 [RFC6410] are augmented by this document for RFCs that | |||
| specify performance metrics. This document takes the position that | ||||
| flexible metric definitions can be proven to be clear and unambiguous | ||||
| through tests that compare the results from independent | ||||
| implementations. It describes tests which infer whether metric | ||||
| specifications are sufficient using a definition of metric | ||||
| "interoperability": measuring equivalent results (in a statistical | ||||
| sense) under the same network conditions. The document expands on | ||||
| this problem and its solution below. | ||||
| In the case of a protocol specification, the notion of | In the case of a protocol specification, the notion of | |||
| "interoperability" is reasonably intuitive - the implementations must | "interoperability" is reasonably intuitive - the implementations must | |||
| successfully "talk to each other", while exercising all features and | successfully "talk to each other", while exercising all features and | |||
| options. To achieve interoperability, two implementors need to | options. To achieve interoperability, two implementors need to | |||
| interpret the protocol specifications in equivalent ways. In the | interpret the protocol specifications in equivalent ways. In the | |||
| case of IP Performance Metrics (IPPM), this definition of | case of IP Performance Metrics (IPPM), this definition of | |||
| interoperability is only useful for test and control protocols like | interoperability is only useful for test and control protocols like | |||
| the One-Way Active Measurement Protocol, OWAMP [RFC4656], and the | the One-Way Active Measurement Protocol, OWAMP [RFC4656], and the | |||
| Two-Way Active Measurement Protocol, TWAMP [RFC5357]. | Two-Way Active Measurement Protocol, TWAMP [RFC5357]. | |||
| skipping to change at page 4, line 7 ¶ | skipping to change at page 4, line 15 ¶ | |||
| Since many implementations of IP metrics are embedded in measurement | Since many implementations of IP metrics are embedded in measurement | |||
| systems that do not interact with one another (they were built before | systems that do not interact with one another (they were built before | |||
| OWAMP and TWAMP), the interoperability evaluation called for in the | OWAMP and TWAMP), the interoperability evaluation called for in the | |||
| IETF standards process cannot be determined by observing that | IETF standards process cannot be determined by observing that | |||
| independent implementations interact properly for various protocol | independent implementations interact properly for various protocol | |||
| exchanges. Instead, verifying that different implementations give | exchanges. Instead, verifying that different implementations give | |||
| statistically equivalent results under controlled measurement | statistically equivalent results under controlled measurement | |||
| conditions takes the place of interoperability observations. Even | conditions takes the place of interoperability observations. Even | |||
| when evaluating OWAMP and TWAMP RFCs for standards track advancement, | when evaluating OWAMP and TWAMP RFCs for standards track advancement, | |||
| the methods described here are useful to evaluate the measurement | the methods described here are useful to evaluate the measurement | |||
| results because their validity would not be ascertained in typical | results because their validity would not be ascertained in protocol | |||
| interoperability testing. | interoperability testing. | |||
| The standards advancement process aims at producing confidence that | The standards advancement process aims at producing confidence that | |||
| the metric definitions and supporting material are clearly worded and | the metric definitions and supporting material are clearly worded and | |||
| unambiguous, or reveals ways in which the metric definitions can be | unambiguous, or reveals ways in which the metric definitions can be | |||
| revised to achieve clarity. The process also permits identification | revised to achieve clarity. The process also permits identification | |||
| of options that were not implemented, so that they can be removed | of options that were not implemented, so that they can be removed | |||
| from the advancing specification. Thus, the product of this process | from the advancing specification. Thus, the product of this process | |||
| is information about the metric specification RFC itself: | is information about the metric specification RFC itself: | |||
| determination of the specifications or definitions that are clear and | determination of the specifications or definitions that are clear and | |||
| skipping to change at page 4, line 43 ¶ | skipping to change at page 4, line 51 ¶ | |||
| Conclusions on equivalence are reached by two measures. | Conclusions on equivalence are reached by two measures. | |||
| First, implementations are compared against individual metric | First, implementations are compared against individual metric | |||
| specifications to make sure that differences in implementation are | specifications to make sure that differences in implementation are | |||
| minimised or at least known. | minimised or at least known. | |||
| Second, a test setup is proposed ensuring identical networking | Second, a test setup is proposed ensuring identical networking | |||
| conditions so that unknowns are minimized and comparisons are | conditions so that unknowns are minimized and comparisons are | |||
| simplified. The resulting separate data sets may be seen as samples | simplified. The resulting separate data sets may be seen as samples | |||
| taken from the same underlying distribution. Using state of the art | taken from the same underlying distribution. Using statistical | |||
| statistical methods, the equivalence of the results is verified. To | methods, the equivalence of the results is verified. To illustrate | |||
| illustrate application of the process and methods defined here, | application of the process and methods defined here, evaluation of | |||
| evaluation of the One-way Delay Metric [RFC2679] is provided in an | the One-way Delay Metric [RFC2679] is provided in an Appendix. While | |||
| Appendix. While test setups will vary with the metrics to be | test setups will vary with the metrics to be validated, the general | |||
| validated, the general methodology of determining equivalent results | methodology of determining equivalent results will not. Documents | |||
| will not. Documents defining test setups to evaluate other metrics | defining test setups to evaluate other metrics should be developed | |||
| should be developed once the process proposed here has been agreed | once the process proposed here has been agreed and approved. | |||
| and approved. | ||||
| The metric RFC advancement process begins with a request for protocol | The metric RFC advancement process begins with a request for protocol | |||
| action accompanied by a memo that documents the supporting tests and | action accompanied by a memo that documents the supporting tests and | |||
| results. The procedures of [RFC2026] are expanded in[RFC5657], | results. The procedures of [RFC2026] are expanded in[RFC5657], | |||
| including sample implementation and interoperability reports. | including sample implementation and interoperability reports. | |||
| Section 3 of [morton-advance-metrics-01] can serve as a template for | [morton-testplan-rfc2679] can serve as a template for a metric RFC | |||
| a metric RFC report which accompanies the protocol action request to | report which accompanies the protocol action request to the Area | |||
| the Area Director, including description of the test set-up, | Director, including description of the test set-up, procedures, | |||
| procedures, results for each implementation and conclusions. | results for each implementation and conclusions. | |||
| Changes from WG-03 to WG-04: | ||||
| o Revisions to Appendix B code and add reference to "R" in the | ||||
| Appendix and the text of section 3.6. | ||||
| Changes from WG-02 to WG-03: | ||||
| o Changes stemming from experiments that implemented this plan, in | ||||
| general. | ||||
| o Adoption of the VLAN loopback figure in the main body of the memo | ||||
| (section 3.2). | ||||
| Changes from WG-01 to WG-02: | ||||
| o Clarification of the number of test streams recommended in section | ||||
| 3.2. | ||||
| o Clarifications on testing details in sections 3.3 and 3.4. | ||||
| o Spelling corrections throughout. | ||||
| Changes from WG -00 to WG -01 draft | ||||
| o Discussion on merits and requirements of a distributed lab test | ||||
| using only local load generators. | ||||
| o Proposal of metrics suitable for tests using the proposed | ||||
| measurement configuration. | ||||
| o Hint on delay caused by software based L2TPv3 implementation. | ||||
| o Added an appendix with a test configuration allowing remote tests | ||||
| comparing different implementations across the network. | ||||
| o Proposal for maximum error of "equivalence", based on performance | ||||
| comparison of identical implementations. This may be useful for | ||||
| both ADK and non-ADK comparisons. | ||||
| Changes from prior ID -02 to WG -00 draft | ||||
| o Incorporation of aspects of reporting to support the protocol | ||||
| action request in the Introduction and section 3.5 | ||||
| o Overhaul of section 3.2 regarding tunneling: Added generic | ||||
| tunneling requirements and L2TPv3 as an example tunneling | ||||
| mechanism fulfilling the tunneling requirements. Removed and | ||||
| adapted some of the prior references to other tunneling protocols | ||||
| o Softened a requirement within section 3.4 (MUST to SHOULD on | ||||
| precision) and removed some comments of the authors. | ||||
| o Updated contact information of one author and added a new author. | ||||
| o Added example C++ code of an Anderson-Darling two sample test | ||||
| implementation. | ||||
| Changes from ID -01 to ID -02 version | ||||
| o Major editorial review, rewording and clarifications on all | ||||
| contents. | ||||
| o Additional text on parallel testing using VLANs and GRE or | ||||
| Pseudowire tunnels. | ||||
| o Additional examples and a glossary. | ||||
| Changes from ID -00 to ID -01 version | ||||
| o Addition of a comparison of individual metric implementations | ||||
| against the metric specification (trying to pick up problems and | ||||
| solutions for metric advancement [morton-advance-metrics]). | ||||
| o More emphasis on the requirement to carefully design and document | ||||
| the measurement setup of the metric comparison. | ||||
| o Proposal of testing conditions under identical WAN network | ||||
| conditions using IP in IP tunneling or Pseudo Wires and parallel | ||||
| measurement streams. | ||||
| o Proposing the requirement to document the smallest resolution at | ||||
| which an ADK test was passed by 95%. As no minimum resolution is | ||||
| specified, IPPM metric compliance is not linked to a particular | ||||
| performance of an implementation. | ||||
| o Reference to RFC 2330 and RFC 2679 for the 95% confidence interval | ||||
| as preferred criterion to decide on statistical equivalence | ||||
| o Reducing the proposed statistical test to ADK with 95% confidence. | ||||
| 1.1. Requirements Language | 1.1. Requirements Language | |||
| The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", | The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", | |||
| "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this | "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this | |||
| document are to be interpreted as described in RFC 2119 [RFC2119]. | document are to be interpreted as described in RFC 2119 [RFC2119]. | |||
| 2. Basic idea | 2. Basic idea | |||
| The implementation of a standard compliant metric is expected to meet | The implementation of a standard compliant metric is expected to meet | |||
| the requirements of the related metric specification. So before | the requirements of the related metric specification. So before | |||
| comparing two metric implementations, each metric implementation is | comparing two metric implementations, each metric implementation is | |||
| individually compared against the metric specification. | individually compared against the metric specification. | |||
| Most metric specifications leave freedom to implementors on non- | Most metric specifications leave freedom to implementors on non- | |||
| fundamental aspects of an individual metric (or options). Comparing | fundamental aspects of an individual metric (or options). Comparing | |||
| different measurement results using a statistical test with the | different measurement results using a statistical test with the | |||
| assumption of identical test path and testing conditions requires | assumption of identical test path and testing conditions requires | |||
| knowledge of all differences in the overall test setup. Metric | knowledge of all differences in the overall test setup. Metric | |||
| specification options chosen by implementors have to be documented. | specification options chosen by implementors have to be documented. | |||
| It is REQUIRED to use identical implementation options wherever | It is RECOMMENDED to use identical metric options for any test | |||
| possible for any test proposed here. Calibrations proposed by metric | proposed here (an exception would be if a variable parameter of the | |||
| standards should be performed to further identify (and possibly | metric definition is not configurable in one or more | |||
| reduce) potential sources of errors in the test setup. | implementations). Calibrations specified by metric standards SHOULD | |||
| be performed to further identify (and possibly reduce) potential | ||||
| sources of error in the test setup. | ||||
| The Framework for IP Performance Metrics [RFC2330] expects that a | The Framework for IP Performance Metrics [RFC2330] expects that a | |||
| "methodology for a metric should have the property that it is | "methodology for a metric should have the property that it is | |||
| repeatable: if the methodology is used multiple times under identical | repeatable: if the methodology is used multiple times under identical | |||
| conditions, it should result in consistent measurements." This means | conditions, it should result in consistent measurements." This means | |||
| an implementation is expected to repeatedly measure a metric with | an implementation is expected to repeatedly measure a metric with | |||
| consistent results (repeatability with the same result). Small | consistent results (repeatability with the same result). Small | |||
| deviations in the test setup are expected to lead to small deviations | deviations in the test setup are expected to lead to small deviations | |||
| in results only. To characterise statistical equivalence in the case | in results only. To characterise statistical equivalence in the case | |||
| of small deviations, RFC 2330 and [RFC2679] suggest to apply a 95% | of small deviations, RFC 2330 and [RFC2679] suggest to apply a 95% | |||
| skipping to change at page 8, line 17 ¶ | skipping to change at page 6, line 33 ¶ | |||
| measurement setup on the result, network conditions and paths MUST | measurement setup on the result, network conditions and paths MUST | |||
| be identical for the compared implementations to the largest | be identical for the compared implementations to the largest | |||
| possible degree. This includes both the stability and non- | possible degree. This includes both the stability and non- | |||
| ambiguity of routes taken by the measurement packets. See RFC | ambiguity of routes taken by the measurement packets. See RFC | |||
| 2330 for a discussion on self-consistency. | 2330 for a discussion on self-consistency. | |||
| o To minimize the influence of implementation options on the result, | o To minimize the influence of implementation options on the result, | |||
| metric implementations SHOULD use identical options and parameters | metric implementations SHOULD use identical options and parameters | |||
| for the metric under evaluation. | for the metric under evaluation. | |||
| o The error induced by the sample size must be small enough to | o The sample size must be large enough to minimize its influence on | |||
| minimize its influence on the test result. This may have to be | the consistency of the test results. This consideration may be | |||
| respected, especially if two implementations measure with | especially important if two implementations measure with different | |||
| different average probing rates. | average packet transmission rates. | |||
| o The implementation with the lowest probing frequency determines | o The implementation with the lowest average packet transmission | |||
| the smallest temporal interval for which samples can be compared. | rate determines the smallest temporal interval for which samples | |||
| can be compared. | ||||
| o Repeat comparisons with several independent metric samples to | o Repeat comparisons with several independent metric samples to | |||
| avoid random indications of compatibility (or the lack of it). | avoid random indications of compatibility (or the lack of it). | |||
| The metric specifications themselves are the primary focus of | The metric specifications themselves are the primary focus of | |||
| evaluation, rather than the implementations of metrics. The | evaluation, rather than the implementations of metrics. The | |||
| documentation produced by the advancement process should identify | documentation produced by the advancement process should identify | |||
| which metric definitions and supporting material were found to be | which metric definitions and supporting material were found to be | |||
| clearly worded and unambiguous, OR, it should identify ways in which | clearly worded and unambiguous, OR, it should identify ways in which | |||
| the metric specification text should be revised to achieve clarity | the metric specification text should be revised to achieve clarity | |||
| skipping to change at page 9, line 30 ¶ | skipping to change at page 7, line 47 ¶ | |||
| impairment generators. | impairment generators. | |||
| o Use remotely separated test labs to compare the implementations | o Use remotely separated test labs to compare the implementations | |||
| and measure across the Internet. | and measure across the Internet. | |||
| o Use remotely separated test labs to compare the implementations | o Use remotely separated test labs to compare the implementations | |||
| and measure across the Internet and include a single impairment | and measure across the Internet and include a single impairment | |||
| generator to impact all measurement flows in non discriminatory | generator to impact all measurement flows in non discriminatory | |||
| way. | way. | |||
| The first two approaches work, but cause higher expenses than the | The first two approaches work, but involve higher expenses than the | |||
| other ones (due to travel and/or shipping+installation). For the | others (due to travel and/or shipping plus installation). For the | |||
| third option, ensuring two identically configured impairment | third option, ensuring two identically configured impairment | |||
| generators requires well defined test cases and possibly identical | generators requires well defined test cases and possibly identical | |||
| hard- and software. | hardware and software. | |||
| As documented in a test report [morton-testplan-rfc2679], the last | As documented in a test report [morton-testplan-rfc2679], the last | |||
| option was required to prove compatibility of two delay metric | option was required to prove compatibility of two delay metric | |||
| implementations. An impairment generator is probably required when | implementations. An impairment generator is probably required when | |||
| testing compatibility of most other metrics and it therefore | testing compatibility of most other metrics and it is therefore | |||
| RECOMMENDED to include an impairment generator in metric test set | RECOMMENDED to include an impairment generator in metric test setups. | |||
| ups. | ||||
| 3.1. Tests of an individual implementation against a metric | 3.1. Tests of an individual implementation against a metric | |||
| specification | specification | |||
| A metric implementation MUST support the requirements classified as | A metric implementation is compliant with a metric specification if | |||
| "MUST" and "REQUIRED" of the related metric specification to be | it supports the requirements classified as "MUST" and "REQUIRED" of | |||
| compliant to the latter. | the related metric specification. An implementation that implements | |||
| all requirements is fully compliant with the specification, and the | ||||
| degree of compliance SHOULD be noted in the conclusions of the | ||||
| report. | ||||
| Further, supported options of a metric implementation SHOULD be | Further, supported options of a metric implementation SHOULD be | |||
| documented in sufficient detail. The documentation of chosen options | documented in sufficient detail to evaluate whether the specification | |||
| is RECOMMENDED to minimise (and recognise) differences in the test | was correctly interpreted. The documentation of chosen options | |||
| setup if two metric implementations are compared. Further, this | should minimise (and recognise) differences in the test setup if two | |||
| documentation is used to validate and improve the underlying metric | metric implementations are compared. Further, this documentation is | |||
| specification option, to remove options which saw no implementation | used to validate or clarify the wording of the metric specification | |||
| or which are badly specified from the metric specification to be | option, to remove options which saw no implementation or which are | |||
| promoted to a standard. This documentation SHOULD be made for all | badly specified from the metric specification. This documentation | |||
| implementation-relevant specifications of a metric picked for a | SHOULD be included for all implementation-relevant specifications of | |||
| comparison that are not explicitly marked as "MUST" or "REQUIRED" in | a metric picked for a comparison, even those that are not explicitly | |||
| the RFC text. This applies for the following sections of all metric | marked as "MUST" or "REQUIRED" in the RFC text. This applies for the | |||
| specifications: | following sections of all metric specifications: | |||
| o Singleton Definition of the Metric. | o Singleton Definition of the Metric. | |||
| o Sample Definition of the Metric. | o Sample Definition of the Metric. | |||
| o Statistics Definition of the Metric. As statistics are compared | o Statistics Definition of the Metric. As statistics are compared | |||
| by the test specified here, this documentation is required even in | by the test specified here, this documentation is required even in | |||
| the case, that the metric specification does not contain a | the case, that the metric specification does not contain a | |||
| Statistics Definition. | Statistics Definition. | |||
| o Timing and Synchronisation related specification (if relevant for | o Timing and Synchronisation related specification (if relevant for | |||
| the Metric). | the Metric). | |||
| o Any other technical part present or missing in the metric | o Any other technical part present or missing in the metric | |||
| specification, which is relevant for the implementation of the | specification, which is relevant for the implementation of the | |||
| Metric. | Metric. | |||
| RFC2330 and RFC2679 emphasise precision as an aim of IPPM metric | RFC2330 and RFC2679 emphasise precision as an aim of IPPM metric | |||
| implementations. A single IPPM conformant implementation MUST under | implementations. A single IPPM conforming implementation should | |||
| otherwise identical network conditions produce precise results for | under otherwise identical network conditions produce precise results | |||
| repeated measurements of the same metric. | for repeated measurements of the same metric. | |||
| RFC 2330 prefers the "empirical distribution function" EDF to | RFC 2330 prefers the "empirical distribution function" EDF to | |||
| describe collections of measurements. RFC 2330 determines, that | describe collections of measurements. RFC 2330 determines, that | |||
| "unless otherwise stated, IPPM goodness-of-fit tests are done using | "unless otherwise stated, IPPM goodness-of-fit tests are done using | |||
| 5% significance." The goodness of fit test determines by which | 5% significance." The goodness of fit test determines by which | |||
| precision two or more samples of a metric implementation belong to | precision two or more samples of a metric implementation belong to | |||
| the same underlying distribution (of measured network performance | the same underlying distribution (of measured network performance | |||
| events). The goodness of fit test suggested for the metric test is | events). The goodness of fit test suggested for the metric test is | |||
| the Anderson-Darling K sample test (ADK sample test, K stands for the | the Anderson-Darling K sample test (ADK sample test, K stands for the | |||
| number of samples to be compared) [ADK]. Please note that RFC 2330 | number of samples to be compared) [ADK]. Please note that RFC 2330 | |||
| skipping to change at page 11, line 9 ¶ | skipping to change at page 9, line 27 ¶ | |||
| The results of a repeated test with a single implementation MUST pass | The results of a repeated test with a single implementation MUST pass | |||
| an ADK sample test with confidence level of 95%. The conditions for | an ADK sample test with confidence level of 95%. The conditions for | |||
| which the ADK test has been passed with the specified confidence | which the ADK test has been passed with the specified confidence | |||
| level MUST be documented. To formulate this differently: The | level MUST be documented. To formulate this differently: The | |||
| requirement is to document the set of parameters with the smallest | requirement is to document the set of parameters with the smallest | |||
| deviation, at which the results of the tested metric implementation | deviation, at which the results of the tested metric implementation | |||
| pass an ADK test with a confidence level of 95%. The minimum | pass an ADK test with a confidence level of 95%. The minimum | |||
| resolution available in the reported results from each implementation | resolution available in the reported results from each implementation | |||
| MUST be taken into account in the ADK test. | MUST be taken into account in the ADK test. | |||
| The test conditions which MUST be documented for a passed metric test | The test conditions to be documented for a passed metric test | |||
| include: | include: | |||
| o The metric resolution at which a test was passed (e.g. the | o The metric resolution at which a test was passed (e.g. the | |||
| resolution of timestamps) | resolution of timestamps) | |||
| o The parameters modified by an impairment generator. | o The parameters modified by an impairment generator. | |||
| o The impairment generator parameter settings. | o The impairment generator parameter settings. | |||
| 3.2. Test setup resulting in identical live network testing conditions | 3.2. Test setup resulting in identical live network testing conditions | |||
| skipping to change at page 11, line 36 ¶ | skipping to change at page 10, line 6 ¶ | |||
| mechanisms like those that achieve load balancing (see [RFC4928]). | mechanisms like those that achieve load balancing (see [RFC4928]). | |||
| This section proposes two measures to deal with both issues. | This section proposes two measures to deal with both issues. | |||
| Tunneling mechanisms can be used to avoid parallel processing of | Tunneling mechanisms can be used to avoid parallel processing of | |||
| different flows in the network. Measuring by separate parallel probe | different flows in the network. Measuring by separate parallel probe | |||
| flows results in repeated collection of data. If both measures are | flows results in repeated collection of data. If both measures are | |||
| combined, WAN network conditions are identical for a number of | combined, WAN network conditions are identical for a number of | |||
| independent measurement flows, no matter what the network conditions | independent measurement flows, no matter what the network conditions | |||
| are in detail. | are in detail. | |||
| Any measurement setup MUST be made to avoid the probing traffic | Any measurement setup must be made to avoid the probing traffic | |||
| itself to impede the metric measurement. The created measurement | itself to impede the metric measurement. The created measurement | |||
| load MUST NOT result in congestion at the access link connecting the | load must not result in congestion at the access link connecting the | |||
| measurement implementation to the WAN. The created measurement load | measurement implementation to the WAN. The created measurement load | |||
| MUST NOT overload the measurement implementation itself, e.g., by | must not overload the measurement implementation itself, e.g., by | |||
| causing a high CPU load or by creating imprecisions due to internal | causing a high CPU load or by creating imprecisions due to internal | |||
| transmit (receive respectively) probe packet collisions. | transmit (receive respectively) probe packet collisions. | |||
| Tunneling multiple flows reaching a network element on a single | Tunneling multiple flows reaching a network element on a single | |||
| physical port may allow to transmit all packets of the tunnel via the | physical port may allow to transmit all packets of the tunnel via the | |||
| same path. Applying tunnels to avoid undesired influence of standard | same path. Applying tunnels to avoid undesired influence of standard | |||
| routing for measurement purposes is a concept known from literature, | routing for measurement purposes is a concept known from literature, | |||
| see e.g. GRE encapsulated multicast probing [GU+Duffield]. An | see e.g. GRE encapsulated multicast probing [GU-Duffield]. An | |||
| existing IP in IP tunnel protocol can be applied to avoid Equal-Cost | existing IP in IP tunnel protocol can be applied to avoid Equal-Cost | |||
| Multi-Path (ECMP) routing of different measurement streams if it | Multi-Path (ECMP) routing of different measurement streams if it | |||
| meets the following criteria: | meets the following criteria: | |||
| o Inner IP packets from different measurement implementations are | o Inner IP packets from different measurement implementations are | |||
| mapped into a single tunnel with single outer IP origin and | mapped into a single tunnel with single outer IP origin and | |||
| destination address as well as origin and destination port numbers | destination address as well as origin and destination port numbers | |||
| which are identical for all packets. | which are identical for all packets. | |||
| o An easily accessible commodity tunneling protocol allows to carry | o An easily accessible commodity tunneling protocol allows to carry | |||
| skipping to change at page 14, line 11 ¶ | skipping to change at page 12, line 34 ¶ | |||
| to B and in the reverse direction. The remote site VLANs are | to B and in the reverse direction. The remote site VLANs are | |||
| U-bridged at the local site Ethernet switch. The measurement packets | U-bridged at the local site Ethernet switch. The measurement packets | |||
| of site 1 travel tunnel A->B first, are U-bridged at site 2 and | of site 1 travel tunnel A->B first, are U-bridged at site 2 and | |||
| travel tunnel B->A second. Measurement packets of site 2 travel | travel tunnel B->A second. Measurement packets of site 2 travel | |||
| tunnel B->A first, are U-bridged at site 1 and travel tunnel A->B | tunnel B->A first, are U-bridged at site 1 and travel tunnel A->B | |||
| second. So all measurement packets pass the same tunnel segments, | second. So all measurement packets pass the same tunnel segments, | |||
| but in different segment order. | but in different segment order. | |||
| If tunneling is applied, two tunnels MUST carry all test traffic in | If tunneling is applied, two tunnels MUST carry all test traffic in | |||
| between the test site and the remote site. For example, if 802.1Q | between the test site and the remote site. For example, if 802.1Q | |||
| Ethernet Virtual LANs (VLAN) are applied and the measurement streams | Virtual LANs (VLAN) are applied and the measurement streams are | |||
| are carried in different VLANs, the IP tunnel or Pseudo Wires | carried in different VLANs, the IP tunnel or Pseudo Wires | |||
| respectively MUST be set up in physical port mode to avoid set up of | respectively are set up in physical port mode to avoid set up of | |||
| Pseudo Wires per VLAN (which may see different paths due to ECMP | Pseudo Wires per VLAN (which may see different paths due to ECMP | |||
| routing), see RFC 4448. The remote router and the Ethernet switch | routing), see RFC 4448. The remote router and the Ethernet switch | |||
| shown in figure 3 has to support 802.1Q in this set up. | shown in figure 3 has to support 802.1Q in this set up. | |||
| The IP packet size of the metric implementation SHOULD be chosen | The IP packet size of the metric implementation SHOULD be chosen | |||
| small enough to avoid fragmentation due to the added Ethernet and | small enough to avoid fragmentation due to the added Ethernet and | |||
| tunnel headers. Otherwise, the impact of tunnel overhead on | tunnel headers. Otherwise, the impact of tunnel overhead on | |||
| fragmentation and interface MTU size MUST be understood and taken | fragmentation and interface MTU size must be understood and taken | |||
| into account (see [RFC4459]). | into account (see [RFC4459]). | |||
| An Ethernet port mode IP tunnel carrying several 802.1Q VLANs each | An Ethernet port mode IP tunnel carrying several 802.1Q VLANs each | |||
| containing measurement traffic of a single measurement system was | containing measurement traffic of a single measurement system was | |||
| successfully applied when testing compatibility of two metric | successfully applied when testing compatibility of two metric | |||
| implementations [morton-testplan-rfc2679]. | implementations [morton-testplan-rfc2679]. Ethernet over L2TPv3 | |||
| [RFC4719] was picked for this test. | ||||
| The following headers may have to be accounted for when calculating | The following headers may have to be accounted for when calculating | |||
| total packet length, if VLANs and Ethernet over L2TPv3 tunnels are | total packet length, if VLANs and Ethernet over L2TPv3 tunnels are | |||
| applied: | applied: | |||
| o Ethernet 802.1Q: 22 Byte. | o Ethernet 802.1Q: 22 Byte. | |||
| o L2TPv3 Header: 4-16 Byte for L2TPv3 data messages over IP; 16-28 | o L2TPv3 Header: 4-16 Byte for L2TPv3 data messages over IP; 16-28 | |||
| Byte for L2TPv3 data messages over UDP. | Byte for L2TPv3 data messages over UDP. | |||
| skipping to change at page 16, line 4 ¶ | skipping to change at page 14, line 32 ¶ | |||
| with 4 samples even if a 2 sample test | with 4 samples even if a 2 sample test | |||
| failed[morton-testplan-rfc2679]. | failed[morton-testplan-rfc2679]. | |||
| Some additional guidelines to calculate and compare samples to | Some additional guidelines to calculate and compare samples to | |||
| perform a metric test are: | perform a metric test are: | |||
| o To compare different probes of a common underlying distribution in | o To compare different probes of a common underlying distribution in | |||
| terms of metrics characterising a communication network requires | terms of metrics characterising a communication network requires | |||
| to respect the temporal nature for which the assumption of common | to respect the temporal nature for which the assumption of common | |||
| underlying distribution may hold. Any singletons or samples to be | underlying distribution may hold. Any singletons or samples to be | |||
| compared MUST be captured within the same time interval. | compared must be captured within the same time interval. | |||
| o If statistical events like rates are used to characterise measured | o If statistical events like rates are used to characterise measured | |||
| metrics of a time-interval, its RECOMMENDED to pick as a minimum 5 | metrics of a time-interval, a minimum 5 singletons of a relevant | |||
| singletons of a relevant metric to ensure a minimum confidence | metric should be picked to ensure a minimum confidence into the | |||
| into the reported value. The error margin of the determined rate | reported value. The error margin of the determined rate depends | |||
| depends on the number singletons (refer to statistical textbooks | on the number singletons (refer to statistical textbooks on | |||
| on Student's t-test). As an example, any packet loss measurement | Student's t-test). As an example, any packet loss measurement | |||
| interval to be compared with the results of another implementation | interval to be compared with the results of another implementation | |||
| contains at least five lost packets to have some confidence that | contains at least five lost packets to have some confidence that | |||
| the observed loss rate wasn't caused by a small number of random | the observed loss rate wasn't caused by a small number of random | |||
| packet drops. | packet drops. | |||
| o The minimum number of singletons or samples to be compared by an | o The minimum number of singletons or samples to be compared by an | |||
| Anderson-Darling test SHOULD be 100 per tested metric | Anderson-Darling test should be 100 per tested metric | |||
| implementation. Note that the Anderson-Darling test detects small | implementation. Note that the Anderson-Darling test detects small | |||
| differences in distributions fairly well and will fail for high | differences in distributions fairly well and will fail for high | |||
| number of compared results (RFC2330 mentions an example with 8192 | number of compared results (RFC2330 mentions an example with 8192 | |||
| measurements where an Anderson-Darling test always failed). | measurements where an Anderson-Darling test always failed). | |||
| o Generally, the Anderson-Darling test is sensitive to differences | o Generally, the Anderson-Darling test is sensitive to differences | |||
| in the accuracy or bias associated with varying implementations or | in the accuracy or bias associated with varying implementations or | |||
| test conditions. These dissimilarities may result in differing | test conditions. These dissimilarities may result in differing | |||
| averages of samples to be compared. An example may be different | averages of samples to be compared. An example may be different | |||
| packet sizes, resulting in a constant delay difference between | packet sizes, resulting in a constant delay difference between | |||
| skipping to change at page 16, line 50 ¶ | skipping to change at page 15, line 31 ¶ | |||
| precisely, for every positive epsilon, there exists a positive delta, | precisely, for every positive epsilon, there exists a positive delta, | |||
| such that if two sets of conditions are within delta of each other, | such that if two sets of conditions are within delta of each other, | |||
| then the resulting measurements will be within epsilon of each | then the resulting measurements will be within epsilon of each | |||
| other." A small variation in conditions in the context of the metric | other." A small variation in conditions in the context of the metric | |||
| test proposed here can be seen as different implementations measuring | test proposed here can be seen as different implementations measuring | |||
| the same metric along the same path. | the same metric along the same path. | |||
| IPPM metric specifications however allow for implementor options to | IPPM metric specifications however allow for implementor options to | |||
| the largest possible degree. It cannot be expected that two | the largest possible degree. It cannot be expected that two | |||
| implementors allow 100% identical options in their implementations. | implementors allow 100% identical options in their implementations. | |||
| Testers SHOULD to the highest degree possible pick the same | Testers SHOULD pick the same metric measurement configurations for | |||
| configurations for their systems when comparing their implementations | their systems when comparing their implementations by a metric test. | |||
| by a metric test. | ||||
| In some cases, a goodness of fit test may not be possible or show | In some cases, a goodness of fit test may not be possible or show | |||
| disappointing results. To clarify the difficulties arising from | disappointing results. To clarify the difficulties arising from | |||
| different implementation options, the individual options picked for | different metric implementation options, the individual options | |||
| every compared implementation SHOULD be documented in sufficient | picked for every compared metric implementation should be documented | |||
| detail. Based on this documentation, the underlying metric | as specified in section 3.5. If the cause of the failure is a lack | |||
| specification should be improved before it is promoted to a standard. | of specification clarity or multiple legitimate interpretations of | |||
| the definition text, the text should be modified and the resulting | ||||
| memo proposed for consensus and (possible) advancement to Internet | ||||
| Standard. | ||||
| The same statistical test as applicable to quantify precision of a | The same statistical test as applicable to quantify precision of a | |||
| single metric implementation MUST be used to compare metric result | single metric implementation must be used to compare metric result | |||
| equivalence for different implementations. To document | equivalence for different implementations. To document | |||
| compatibility, the smallest measurement resolution at which the | compatibility, the smallest measurement resolution at which the | |||
| compared implementations passed the ADK sample test MUST be | compared implementations passed the ADK sample test must be | |||
| documented. | documented. | |||
| For different implementations of the same metric, "variations in | For different implementations of the same metric, "variations in | |||
| conditions" are reasonably expected. The ADK test comparing samples | conditions" are reasonably expected. The ADK test comparing samples | |||
| of the different implementations MAY result in a lower precision than | of the different implementations may result in a lower precision than | |||
| the test for precision in the same-implementation comparison. | the test for precision in the same-implementation comparison. | |||
| 3.4. Clock synchronisation | 3.4. Clock synchronisation | |||
| Clock synchronization effects require special attention. Accuracy of | Clock synchronization effects require special attention. Accuracy of | |||
| one-way active delay measurements for any metrics implementation | one-way active delay measurements for any metrics implementation | |||
| depends on clock synchronization between the source and destination | depends on clock synchronization between the source and destination | |||
| of tests. Ideally, one-way active delay measurement (RFC 2679, | of tests. Ideally, one-way active delay measurement (RFC 2679, | |||
| [RFC2679]) test endpoints either have direct access to independent | [RFC2679]) test endpoints either have direct access to independent | |||
| GPS or CDMA-based time sources or indirect access to nearby NTP | GPS or CDMA-based time sources or indirect access to nearby NTP | |||
| primary (stratum 1) time sources, equipped with GPS receivers. | primary (stratum 1) time sources, equipped with GPS receivers. | |||
| Access to these time sources may not be available at all test | Access to these time sources may not be available at all test | |||
| locations associated with different Internet paths, for a variety of | locations associated with different Internet paths, for a variety of | |||
| reasons out of scope of this document. | reasons out of scope of this document. | |||
| When secondary (stratum 2 and above) time sources are used with NTP | When secondary (stratum 2 and above) time sources are used with NTP | |||
| running across the same network, whose metrics are subject to | running across the same network, whose metrics are subject to | |||
| comparative implementation tests, network impairments can affect | comparative implementation tests, network impairments can affect | |||
| clock synchronization, distort sample one-way values and their | clock synchronization, distort sample one-way values and their | |||
| interval statistics. It is RECOMMENDED to discard sample one-way | interval statistics. It is recommended to discard sample one-way | |||
| delay values for any implementation, when one of the following | delay values for any implementation when one of the following | |||
| reliability conditions is met: | reliability conditions is met: | |||
| o Delay is measured and is finite in one direction, but not the | o Delay is measured and is finite in one direction, but not the | |||
| other. | other. | |||
| o Absolute value of the difference between the sum of one-way | o Absolute value of the difference between the sum of one-way | |||
| measurements in both directions and round-trip measurement is | measurements in both directions and round-trip measurement is | |||
| greater than X% of the latter value. | greater than X% of the latter value. | |||
| Examination of the second condition requires RTT measurement for | Examination of the second condition requires RTT measurement for | |||
| skipping to change at page 18, line 20 ¶ | skipping to change at page 16, line 50 ¶ | |||
| unreliable one-way delay samples and misidentification of reliable | unreliable one-way delay samples and misidentification of reliable | |||
| samples under a wide range of Internet path RTTs probably requires | samples under a wide range of Internet path RTTs probably requires | |||
| further study. | further study. | |||
| An IPPM compliant metric implementation of an RFC that requires | An IPPM compliant metric implementation of an RFC that requires | |||
| synchronized clocks is expected to provide precise measurement | synchronized clocks is expected to provide precise measurement | |||
| results. | results. | |||
| IF an implementation publishes a specification of its precision, such | IF an implementation publishes a specification of its precision, such | |||
| as "a precision of 1 ms (+/- 500 us) with a confidence of 95%", then | as "a precision of 1 ms (+/- 500 us) with a confidence of 95%", then | |||
| the specification SHOULD be met over a useful measurement duration. | the specification should be met over a useful measurement duration. | |||
| For example, if the metric is measured along an Internet path which | For example, if the metric is measured along an Internet path which | |||
| is stable and not congested, then the precision specification SHOULD | is stable and not congested, then the precision specification should | |||
| be met over durations of an hour or more. | be met over durations of an hour or more. | |||
| 3.5. Recommended Metric Verification Measurement Process | 3.5. Recommended Metric Verification Measurement Process | |||
| In order to meet their obligations under the IETF Standards Process | In order to meet their obligations under the IETF Standards Process | |||
| the IESG must be convinced that each metric specification advanced to | the IESG must be convinced that each metric specification advanced to | |||
| Draft Standard or Internet Standard status is clearly written, that | Internet Standard status is clearly written, that there are a | |||
| there are a sufficient number of verified equivalent implementations, | sufficient number of verified equivalent implementations, and that | |||
| and that options that have been implemented are documented. | options that have been implemented are documented. | |||
| In the context of this document, metrics are designed to measure some | In the context of this document, metrics are designed to measure some | |||
| characteristic of a data network. An aim of any metric definition | characteristic of a data network. An aim of any metric definition | |||
| should be that it should be specified in a way that can reliably | should be that it should be specified in a way that can reliably | |||
| measure the specific characteristic in a repeatable way across | measure the specific characteristic in a repeatable way across | |||
| multiple independent implementations. | multiple independent implementations. | |||
| Each metric, statistic or option of those to be validated MUST be | Each metric, statistic or option of those to be validated MUST be | |||
| compared against a reference measurement or another implementation by | compared against a reference measurement or another implementation as | |||
| as specified by this document. | specified in this document. | |||
| Finally, the metric definitions, embodied in the text of the RFCs, | Finally, the metric definitions, embodied in the text of the RFCs, | |||
| are the objects that require evaluation and possible revision in | are the objects that require evaluation and possible revision in | |||
| order to advance to the next step on the standards track. | order to advance to Internet Standard. | |||
| IF two (or more) implementations do not measure an equivalent metric | IF two (or more) implementations do not measure an equivalent metric | |||
| as specified by this document, | as specified by this document, | |||
| AND sources of measurement error do not adequately explain the lack | AND sources of measurement error do not adequately explain the lack | |||
| of agreement, | of agreement, | |||
| THEN the details of each implementation should be audited along with | THEN the details of each implementation should be audited along with | |||
| the exact definition text, to determine if there is a lack of clarity | the exact definition text, to determine if there is a lack of clarity | |||
| that has caused the implementations to vary in a way that affects the | that has caused the implementations to vary in a way that affects the | |||
| correspondence of the results. | correspondence of the results. | |||
| IF there was a lack of clarity or multiple legitimate interpretations | IF there was a lack of clarity or multiple legitimate interpretations | |||
| of the definition text, | of the definition text, | |||
| THEN the text should be modified and the resulting memo proposed for | THEN the text should be modified and the resulting memo proposed for | |||
| consensus and (possible) advancement along the standards track. | consensus and (possible) advancement along the standards track. | |||
| Finally, all the findings MUST be documented in a report that can | Finally, all the findings MUST be documented in a report that can | |||
| support advancement on the standards track, similar to those | support advancement to Internet Standard, as described here (similar | |||
| described in [RFC5657]. The list of measurement devices used in | to those described in [RFC5657]). The list of measurement devices | |||
| testing satisfies the implementation requirement, while the test | used in testing satisfies the implementation requirement, while the | |||
| results provide information on the quality of each specification in | test results provide information on the quality of each specification | |||
| the metric RFC (the surrogate for feature interoperability). | in the metric RFC (the surrogate for feature interoperability). | |||
| The complete process of advancing a metric specification to a | The complete process of advancing a metric specification to a | |||
| standard as defined by this document is illustrated in Figure 4. | standard as defined by this document is illustrated in Figure 4. | |||
| ,---. | ,---. | |||
| / \ | / \ | |||
| ( Start ) | ( Start ) | |||
| \ / Implementations | \ / Implementations | |||
| `-+-' +-------+ | `-+-' +-------+ | |||
| | /| 1 `. | | /| 1 `. | |||
| +---+----+ / +-------+ `.-----------+ ,-------. | +---+----+ / +-------+ `.-----------+ ,-------. | |||
| | RFC | / |Check for | ,' was RFC `. YES | | RFC | / |Check for | ,' was RFC `. YES | |||
| | | / |Equivalence.... clause x ------+ | | | / |Equivalence.... clause x ------+ | |||
| | |/ +-------+ |under | `. clear? ,' | | | |/ +-------+ |under | `. clear? ,' | | |||
| | Metric \.....| 2 ....relevant | `---+---' +----+-----+ | | Metric \.....| 2 ....relevant | `---+---' +----+-----+ | |||
| | Metric |\ +-------+ |identical | No | |Report | | | Metric |\ +-------+ |identical | No | |Report | | |||
| | Metric | \ |network | +--+----+ |results + | | | Metric | \ |network | +--+----+ |results + | | |||
| | ... | \ |conditions | |Modify | |Advance | | | ... | \ |conditions | |Modify | |Advance | | |||
| | | \ +-------+ | | |Spec +--+RFC | | | | \ +-------+ | | |Spec +--+RFC | | |||
| +--------+ \| n |.'+-----------+ +-------+ |request(?)| | +--------+ \| n |.'+-----------+ +-------+ |request | | |||
| +-------+ +----------+ | +-------+ +----------+ | |||
| Illustration of the metric standardisation process | Illustration of the metric standardisation process | |||
| Figure 4 | Figure 4 | |||
| Any recommendation for the advancement of a metric specification MUST | Any recommendation for the advancement of a metric specification MUST | |||
| be accompanied by an implementation report, as is the case with all | be accompanied by an implementation report. The implementation | |||
| requests for the advancement of IETF specifications. The | report needs to include the tests performed, the applied test setup, | |||
| implementation report needs to include the tests performed, the | the specific metrics in the RFC and reports of the tests performed | |||
| applied test setup, the specific metrics in the RFC and reports of | with two or more implementations. The test plan needs to specify the | |||
| the tests performed with two or more implementations. The test plan | precision reached for each measured metric and thus define the | |||
| needs to specify the precision reached for each measured metric and | meaning of "statistically equivalent" for the specific metrics being | |||
| thus define the meaning of "statistically equivalent" for the | tested. | |||
| specific metrics being tested. | ||||
| Ideally, the test plan would co-evolve with the development of the | Ideally, the test plan would co-evolve with the development of the | |||
| metric, since that's when people have the most context in their | metric, since that's when participants have the clearest context in | |||
| thinking regarding the different subtleties that can arise. | their minds regarding the different subtleties that can arise. | |||
| In particular, the implementation report MUST as a minimum document: | In particular, the implementation report MUST as a minimum document: | |||
| o The metric compared and the RFC specifying it. This includes | o The metric compared and the RFC specifying it. This includes | |||
| statements as required by the section "Tests of an individual | statements as required by the section "Tests of an individual | |||
| implementation against a metric specification" of this document. | implementation against a metric specification" of this document. | |||
| o The measurement configuration and setup. | o The measurement configuration and setup. | |||
| o A complete specification of the measurement stream (mean rate, | o A complete specification of the measurement stream (mean rate, | |||
| skipping to change at page 21, line 9 ¶ | skipping to change at page 19, line 38 ¶ | |||
| and network conditions allowing reproduction of these laboratory | and network conditions allowing reproduction of these laboratory | |||
| conditions for readers of the implementation report. | conditions for readers of the implementation report. | |||
| o The documentation helping to improve metric specifications defined | o The documentation helping to improve metric specifications defined | |||
| by this section. | by this section. | |||
| All of the tests for each set SHOULD be run in a test setup as | All of the tests for each set SHOULD be run in a test setup as | |||
| specified in the section "Test setup resulting in identical live | specified in the section "Test setup resulting in identical live | |||
| network testing conditions." | network testing conditions." | |||
| If a different test set up is chosen, it is RECOMMENDED to avoid | If a different test setup is chosen, it is recommended to avoid | |||
| effects falsifying results of validation measurements caused by real | effects falsifying results of validation measurements caused by real | |||
| data networks (like parallelism in devices and networks). Data | data networks (like parallelism in devices and networks). Data | |||
| networks may forward packets differently in the case of: | networks may forward packets differently in the case of: | |||
| o Different packet sizes chosen for different metric | o Different packet sizes chosen for different metric | |||
| implementations. A proposed countermeasure is selecting the same | implementations. A proposed countermeasure is selecting the same | |||
| packet size when validating results of two samples or a sample | packet size when validating results of two samples or a sample | |||
| against an original distribution. | against an original distribution. | |||
| o Selection of differing IP addresses and ports used by different | o Selection of differing IP addresses and ports used by different | |||
| skipping to change at page 23, line 16 ¶ | skipping to change at page 21, line 42 ¶ | |||
| This memo does not raise any specific security issues. | This memo does not raise any specific security issues. | |||
| 8. References | 8. References | |||
| 8.1. Normative References | 8.1. Normative References | |||
| [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, | [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, | |||
| October 1996. | October 1996. | |||
| [RFC2026] Bradner, S., "The Internet Standards Process -- Revision | ||||
| 3", BCP 9, RFC 2026, October 1996. | ||||
| [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate | [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate | |||
| Requirement Levels", BCP 14, RFC 2119, March 1997. | Requirement Levels", BCP 14, RFC 2119, March 1997. | |||
| [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, | [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, | |||
| "Framework for IP Performance Metrics", RFC 2330, | "Framework for IP Performance Metrics", RFC 2330, | |||
| May 1998. | May 1998. | |||
| [RFC2661] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, | [RFC2661] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, | |||
| G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"", | G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"", | |||
| RFC 2661, August 1999. | RFC 2661, August 1999. | |||
| [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way | [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way | |||
| Delay Metric for IPPM", RFC 2679, September 1999. | Delay Metric for IPPM", RFC 2679, September 1999. | |||
| [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way | ||||
| Packet Loss Metric for IPPM", RFC 2680, September 1999. | ||||
| [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip | ||||
| Delay Metric for IPPM", RFC 2681, September 1999. | ||||
| [RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. | [RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. | |||
| Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, | Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, | |||
| March 2000. | March 2000. | |||
| [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling | [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling | |||
| Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. | Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. | |||
| [RFC4448] Martini, L., Rosen, E., El-Aawar, N., and G. Heron, | [RFC4448] Martini, L., Rosen, E., El-Aawar, N., and G. Heron, | |||
| "Encapsulation Methods for Transport of Ethernet over MPLS | "Encapsulation Methods for Transport of Ethernet over MPLS | |||
| Networks", RFC 4448, April 2006. | Networks", RFC 4448, April 2006. | |||
| [RFC4459] Savola, P., "MTU and Fragmentation Issues with In-the- | ||||
| Network Tunneling", RFC 4459, April 2006. | ||||
| [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. | [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. | |||
| Zekauskas, "A One-way Active Measurement Protocol | Zekauskas, "A One-way Active Measurement Protocol | |||
| (OWAMP)", RFC 4656, September 2006. | (OWAMP)", RFC 4656, September 2006. | |||
| [RFC4719] Aggarwal, R., Townsley, M., and M. Dos Santos, "Transport | [RFC4719] Aggarwal, R., Townsley, M., and M. Dos Santos, "Transport | |||
| of Ethernet Frames over Layer 2 Tunneling Protocol Version | of Ethernet Frames over Layer 2 Tunneling Protocol Version | |||
| 3 (L2TPv3)", RFC 4719, November 2006. | 3 (L2TPv3)", RFC 4719, November 2006. | |||
| [RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal | [RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal | |||
| Cost Multipath Treatment in MPLS Networks", BCP 128, | Cost Multipath Treatment in MPLS Networks", BCP 128, | |||
| RFC 4928, June 2007. | RFC 4928, June 2007. | |||
| [RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation | [RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation | |||
| and Implementation Reports for Advancement to Draft | and Implementation Reports for Advancement to Draft | |||
| Standard", BCP 9, RFC 5657, September 2009. | Standard", BCP 9, RFC 5657, September 2009. | |||
| [RFC6410] Housley, R., Crocker, D., and E. Burger, "Reducing the | ||||
| Standards Track to Two Maturity Levels", BCP 9, RFC 6410, | ||||
| October 2011. | ||||
| 8.2. Informative References | 8.2. Informative References | |||
| [ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling | [ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling | |||
| Tests of fit, for continuous and discrete cases", | Tests of fit, for continuous and discrete cases", | |||
| University of Washington, Technical Report No. 81, | University of Washington, Technical Report No. 81, | |||
| May 1986. | May 1986. | |||
| [GU+Duffield] | [GU-Duffield] | |||
| Gu, Y., Duffield, N., Breslau, L., and S. Sen, "GRE | Gu, Y., Duffield, N., Breslau, L., and S. Sen, "GRE | |||
| Encapsulated Multicast Probing: A Scalable Technique for | Encapsulated Multicast Probing: A Scalable Technique for | |||
| Measuring One-Way Loss", SIGMETRICS'07 San Diego, | Measuring One-Way Loss", SIGMETRICS'07 San Diego, | |||
| California, USA, June 2007. | California, USA, June 2007. | |||
| [RFC2026] Bradner, S., "The Internet Standards Process -- Revision | ||||
| 3", BCP 9, RFC 2026, October 1996. | ||||
| [RFC4459] Savola, P., "MTU and Fragmentation Issues with In-the- | ||||
| Network Tunneling", RFC 4459, April 2006. | ||||
| [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. | [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. | |||
| Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", | Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", | |||
| RFC 5357, October 2008. | RFC 5357, October 2008. | |||
| [Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and | [Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and | |||
| Combinations of Such Tests. R package version 1.0.", , | Combinations of Such Tests. R package version 1.0.", , | |||
| 2008. | 2008. | |||
| [Rtool] R Development Core Team, "R: A language and environment | [Rtool] R Development Core Team, "R: A language and environment | |||
| for statistical computing. R Foundation for Statistical | for statistical computing. R Foundation for Statistical | |||
| Computing, Vienna, Austria. ISBN 3-900051-07-0, URL | Computing, Vienna, Austria. ISBN 3-900051-07-0, URL | |||
| http://www.R-project.org/", , 2011. | http://www.R-project.org/", , 2011. | |||
| [Rule of thumb] | ||||
| Hardy, M., "Confidence interval", March 2010. | ||||
| [bradner-metrictest] | [bradner-metrictest] | |||
| Bradner, S., Mankin, A., and V. Paxson, "Advancement of | Bradner, S., Mankin, A., and V. Paxson, "Advancement of | |||
| metrics specifications on the IETF Standards Track", | metrics specifications on the IETF Standards Track", | |||
| draft -bradner-metricstest-03, (work in progress), | draft -bradner-metricstest-03, (work in progress), | |||
| July 2007. | July 2007. | |||
| [morton-advance-metrics] | ||||
| Morton, A., "Problems and Possible Solutions for Advancing | ||||
| Metrics on the Standards Track", draft -morton-ippm- | ||||
| advance-metrics-00, (work in progress), July 2009. | ||||
| [morton-advance-metrics-01] | ||||
| Morton, A., "Lab Test Results for Advancing Metrics on the | ||||
| Standards Track", draft -morton-ippm-advance-metrics-01, | ||||
| (work in progress), June 2010. | ||||
| [morton-testplan-rfc2679] | [morton-testplan-rfc2679] | |||
| Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test | Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test | |||
| Plan and Results for Advancing RFC 2679 on the Standards | Plan and Results for Advancing RFC 2679 on the Standards | |||
| Track", draft -morton-ippm-testplan-rfc2679-01, (work in | Track", draft -morton-ippm-testplan-rfc2679-01, (work in | |||
| progress), June 2011. | progress), June 2011. | |||
| Appendix A. An example on a One-way Delay metric validation | Appendix A. An example on a One-way Delay metric validation | |||
| The text of this appendix is not binding. It is an example how parts | The text of this appendix is not binding. It is an example how parts | |||
| of a One-way Delay metric test could look like. | of a One-way Delay metric test could look like. | |||
| http://xml.resource.org/public/rfc/bibxml/ | ||||
| A.1. Compliance to Metric specification requirements | A.1. Compliance to Metric specification requirements | |||
| One-way Delay, Loss threshold, RFC 2679 | One-way Delay, Loss threshold, RFC 2679 | |||
| This test determines if implementations use the same configured | This test determines if implementations use the same configured | |||
| maximum waiting time delay from one measurement to another under | maximum waiting time delay from one measurement to another under | |||
| different delay conditions, and correctly declare packets arriving in | different delay conditions, and correctly declare packets arriving in | |||
| excess of the waiting time threshold as lost. See Section 3.5 of | excess of the waiting time threshold as lost. See Section 3.5 of | |||
| RFC2679, 3rd bullet point and also Section 3.8.2 of RFC2679. | RFC2679, 3rd bullet point and also Section 3.8.2 of RFC2679. | |||
| skipping to change at page 29, line 27 ¶ | skipping to change at page 27, line 30 ¶ | |||
| but this is as it should be. | but this is as it should be. | |||
| The C++ code below will perform a 2-sample AD comparison when | The C++ code below will perform a 2-sample AD comparison when | |||
| compiled and presented with two column vectors in a file (using white | compiled and presented with two column vectors in a file (using white | |||
| space as separation). This version contains modifications to use the | space as separation). This version contains modifications to use the | |||
| vectors and run as a stand-alone module by Wes Eddy, Sept 2011. The | vectors and run as a stand-alone module by Wes Eddy, Sept 2011. The | |||
| status of the comparison can be checked on the command line with "$ | status of the comparison can be checked on the command line with "$ | |||
| echo $?" or the last line can be replaced with a printf statement for | echo $?" or the last line can be replaced with a printf statement for | |||
| adk_result instead. | adk_result instead. | |||
| /* Routines for computing the Anderson-Darling 2 sample | /* | |||
| * test statistic. | ||||
| * | ||||
| * Implemented based on the description in | ||||
| * "Anderson-Darling K Sample Test" Heckert, Alan and | ||||
| * Filliben, James, editors, Dataplot Reference Manual, | ||||
| * Chapter 15 Auxiliary, NIST, 2004. | ||||
| * Official Reference by 2010 | ||||
| * Heckert, N. A. (2001). Dataplot website at the | ||||
| * National Institute of Standards and Technology: | ||||
| * http://www.itl.nist.gov/div898/software/dataplot.html/ | ||||
| * June 2001. | ||||
| */ | ||||
| #include <iostream> | Copyright (c) 2011 IETF Trust and the persons identified | |||
| #include <fstream> | as authors of the code. All rights reserved. | |||
| #include <vector> | ||||
| #include <sstream> | ||||
| using namespace std; | Redistribution and use in source and binary forms, with | |||
| or without modification, is permitted pursuant to, and subject | ||||
| to the license terms contained in, the Simplified BSD License | ||||
| set forth in Section 4.c of the IETF Trust's Legal Provisions | ||||
| Relating to IETF Documents (http://trustee.ietf.org/license-info). | ||||
| int main() { | */ | |||
| vector<double> vec1, vec2; | ||||
| double adk_result; | ||||
| static int k, val_st_z_samp1, val_st_z_samp2, | ||||
| val_eq_z_samp1, val_eq_z_samp2, | ||||
| j, n_total, n_sample1, n_sample2, L, | ||||
| max_number_samples, line, maxnumber_z; | ||||
| static int column_1, column_2; | ||||
| static double adk, n_value, z, sum_adk_samp1, | ||||
| sum_adk_samp2, z_aux; | ||||
| static double H_j, F1j, hj, F2j, denom_1_aux, denom_2_aux; | ||||
| static bool next_z_sample2, equal_z_both_samples; | ||||
| static int stop_loop1, stop_loop2, stop_loop3,old_eq_line2, | ||||
| old_eq_line1; | ||||
| static double adk_criterium = 1.993; | /* Routines for computing the Anderson-Darling 2 sample | |||
| * test statistic. | ||||
| * | ||||
| * Implemented based on the description in | ||||
| * "Anderson-Darling K Sample Test" Heckert, Alan and | ||||
| * Filliben, James, editors, Dataplot Reference Manual, | ||||
| * Chapter 15 Auxiliary, NIST, 2004. | ||||
| * Official Reference by 2010 | ||||
| * Heckert, N. A. (2001). Dataplot website at the | ||||
| * National Institute of Standards and Technology: | ||||
| * http://www.itl.nist.gov/div898/software/dataplot.html/ | ||||
| * June 2001. | ||||
| */ | ||||
| /* vec1 and vec2 to be initialised with sample 1 and | #include <iostream> | |||
| * sample 2 values in ascending order */ | #include <fstream> | |||
| while (!cin.eof()) { | #include <vector> | |||
| double f1, f2; | #include <sstream> | |||
| cin >> f1; | ||||
| cin >> f2; | ||||
| vec1.push_back(f1); | ||||
| vec2.push_back(f2); | ||||
| } | ||||
| k = 2; | using namespace std; | |||
| n_sample1 = vec1.size() - 1; | ||||
| n_sample2 = vec2.size() - 1; | ||||
| // -1 because vec[0] is a dummy value | int main() { | |||
| n_total = n_sample1 + n_sample2; | vector<double> vec1, vec2; | |||
| double adk_result; | ||||
| static int k, val_st_z_samp1, val_st_z_samp2, | ||||
| val_eq_z_samp1, val_eq_z_samp2, | ||||
| j, n_total, n_sample1, n_sample2, L, | ||||
| max_number_samples, line, maxnumber_z; | ||||
| static int column_1, column_2; | ||||
| static double adk, n_value, z, sum_adk_samp1, | ||||
| sum_adk_samp2, z_aux; | ||||
| static double H_j, F1j, hj, F2j, denom_1_aux, denom_2_aux; | ||||
| static bool next_z_sample2, equal_z_both_samples; | ||||
| static int stop_loop1, stop_loop2, stop_loop3,old_eq_line2, | ||||
| old_eq_line1; | ||||
| /* value equal to the line with a value = zj in sample 1. | static double adk_criterium = 1.993; | |||
| * Here j=1, so the line is 1. | ||||
| */ | ||||
| val_eq_z_samp1 = 1; | ||||
| /* value equal to the line with a value = zj in sample 2. | /* vec1 and vec2 to be initialised with sample 1 and | |||
| * Here j=1, so the line is 1. | * sample 2 values in ascending order */ | |||
| */ | while (!cin.eof()) { | |||
| val_eq_z_samp2 = 1; | double f1, f2; | |||
| cin >> f1; | ||||
| cin >> f2; | ||||
| vec1.push_back(f1); | ||||
| vec2.push_back(f2); | ||||
| } | ||||
| /* value equal to the last line with a value < zj | k = 2; | |||
| * in sample 1. Here j=1, so the line is 0. | n_sample1 = vec1.size() - 1; | |||
| */ | n_sample2 = vec2.size() - 1; | |||
| val_st_z_samp1 = 0; | ||||
| /* value equal to the last line with a value < zj | // -1 because vec[0] is a dummy value | |||
| * in sample 1. Here j=1, so the line is 0. | n_total = n_sample1 + n_sample2; | |||
| */ | /* value equal to the line with a value = zj in sample 1. | |||
| val_st_z_samp2 = 0; | * Here j=1, so the line is 1. | |||
| */ | ||||
| val_eq_z_samp1 = 1; | ||||
| sum_adk_samp1 = 0; | /* value equal to the line with a value = zj in sample 2. | |||
| sum_adk_samp2 = 0; | * Here j=1, so the line is 1. | |||
| j = 1; | */ | |||
| val_eq_z_samp2 = 1; | ||||
| // as mentioned above, j=1 | /* value equal to the last line with a value < zj | |||
| equal_z_both_samples = false; | * in sample 1. Here j=1, so the line is 0. | |||
| */ | ||||
| val_st_z_samp1 = 0; | ||||
| next_z_sample2 = false; | /* value equal to the last line with a value < zj | |||
| * in sample 1. Here j=1, so the line is 0. | ||||
| */ | ||||
| val_st_z_samp2 = 0; | ||||
| //assuming the next z to be of sample 1 | sum_adk_samp1 = 0; | |||
| stop_loop1 = n_sample1 + 1; | sum_adk_samp2 = 0; | |||
| j = 1; | ||||
| // + 1 because vec[0] is a dummy, see n_sample1 declaration | // as mentioned above, j=1 | |||
| stop_loop2 = n_sample2 + 1; | equal_z_both_samples = false; | |||
| stop_loop3 = n_total + 1; | ||||
| /* The required z values are calculated until all values | next_z_sample2 = false; | |||
| * of both samples have been taken into account. See the | ||||
| * lines above for the stoploop values. Construct required | ||||
| * to avoid a mathematical operation in the While condition | ||||
| */ | ||||
| while (((stop_loop1 > val_eq_z_samp1) | ||||
| || (stop_loop2 > val_eq_z_samp2)) && stop_loop3 > j) | ||||
| { | ||||
| if(val_eq_z_samp1 < n_sample1+1) | ||||
| { | ||||
| /* here, a preliminary zj value is set. | ||||
| * See below how to calculate the actual zj. | ||||
| */ | ||||
| z = vec1[val_eq_z_samp1]; | ||||
| /* this while sequence calculates the number of values | //assuming the next z to be of sample 1 | |||
| * equal to z. | stop_loop1 = n_sample1 + 1; | |||
| */ | ||||
| while ((val_eq_z_samp1+1 < n_sample1) | ||||
| && z == vec1[val_eq_z_samp1+1] ) | ||||
| { | ||||
| val_eq_z_samp1++; | ||||
| } | ||||
| } | ||||
| else | ||||
| { | ||||
| val_eq_z_samp1 = 0; | ||||
| val_st_z_samp1 = n_sample1; | ||||
| // this should be val_eq_z_samp1 - 1 = n_sample1 | // + 1 because vec[0] is a dummy, see n_sample1 declaration | |||
| } | stop_loop2 = n_sample2 + 1; | |||
| stop_loop3 = n_total + 1; | ||||
| if(val_eq_z_samp2 < n_sample2+1) | /* The required z values are calculated until all values | |||
| { | * of both samples have been taken into account. See the | |||
| z_aux = vec2[val_eq_z_samp2];; | * lines above for the stoploop values. Construct required | |||
| * to avoid a mathematical operation in the While condition | ||||
| */ | ||||
| while (((stop_loop1 > val_eq_z_samp1) | ||||
| || (stop_loop2 > val_eq_z_samp2)) && stop_loop3 > j) | ||||
| { | ||||
| if(val_eq_z_samp1 < n_sample1+1) | ||||
| { | ||||
| /* here, a preliminary zj value is set. | ||||
| * See below how to calculate the actual zj. | ||||
| */ | ||||
| z = vec1[val_eq_z_samp1]; | ||||
| /* this while sequence calculates the number of values | /* this while sequence calculates the number of values | |||
| * equal to z_aux | * equal to z. | |||
| */ | */ | |||
| while ((val_eq_z_samp1+1 < n_sample1) | ||||
| && z == vec1[val_eq_z_samp1+1] ) | ||||
| { | ||||
| val_eq_z_samp1++; | ||||
| } | ||||
| } | ||||
| else | ||||
| { | ||||
| val_eq_z_samp1 = 0; | ||||
| val_st_z_samp1 = n_sample1; | ||||
| while ((val_eq_z_samp2+1 < n_sample2) | // this should be val_eq_z_samp1 - 1 = n_sample1 | |||
| && z_aux == vec2[val_eq_z_samp2+1] ) | } | |||
| { | ||||
| val_eq_z_samp2++; | ||||
| } | ||||
| /* the smaller of the two actual data values is picked | if(val_eq_z_samp2 < n_sample2+1) | |||
| * as the next zj. | { | |||
| */ | z_aux = vec2[val_eq_z_samp2];; | |||
| if(z > z_aux) | /* this while sequence calculates the number of values | |||
| { | * equal to z_aux | |||
| z = z_aux; | */ | |||
| next_z_sample2 = true; | ||||
| } | ||||
| else | ||||
| { | ||||
| if (z == z_aux) | ||||
| { | ||||
| equal_z_both_samples = true; | ||||
| } | ||||
| /* This is the case, if the last value of column1 is | while ((val_eq_z_samp2+1 < n_sample2) | |||
| * smaller than the remaining values of column2. | && z_aux == vec2[val_eq_z_samp2+1] ) | |||
| */ | { | |||
| if (val_eq_z_samp1 == 0) | val_eq_z_samp2++; | |||
| { | } | |||
| z = z_aux; | ||||
| next_z_sample2 = true; | ||||
| } | ||||
| } | ||||
| } | ||||
| else | ||||
| { | ||||
| val_eq_z_samp2 = 0; | ||||
| val_st_z_samp2 = n_sample2; | ||||
| // this should be val_eq_z_samp2 - 1 = n_sample2 | /* the smaller of the two actual data values is picked | |||
| * as the next zj. | ||||
| */ | ||||
| } | if(z > z_aux) | |||
| { | ||||
| z = z_aux; | ||||
| next_z_sample2 = true; | ||||
| } | ||||
| else | ||||
| { | ||||
| if (z == z_aux) | ||||
| { | ||||
| equal_z_both_samples = true; | ||||
| } | ||||
| /* in the following, sum j = 1 to L is calculated for | /* This is the case, if the last value of column1 is | |||
| * sample 1 and sample 2. | * smaller than the remaining values of column2. | |||
| */ | */ | |||
| if (equal_z_both_samples) | if (val_eq_z_samp1 == 0) | |||
| { | { | |||
| z = z_aux; | ||||
| next_z_sample2 = true; | ||||
| } | ||||
| } | ||||
| } | ||||
| else | ||||
| { | ||||
| val_eq_z_samp2 = 0; | ||||
| val_st_z_samp2 = n_sample2; | ||||
| /* hj is the number of values in the combined sample | // this should be val_eq_z_samp2 - 1 = n_sample2 | |||
| * equal to zj | ||||
| */ | ||||
| hj = val_eq_z_samp1 - val_st_z_samp1 | ||||
| + val_eq_z_samp2 - val_st_z_samp2; | ||||
| /* H_j is the number of values in the combined sample | } | |||
| * smaller than zj plus one half the the number of | ||||
| * values in the combined sample equal to zj | ||||
| * (that's hj/2). | ||||
| */ | ||||
| H_j = val_st_z_samp1 + val_st_z_samp2 | ||||
| + hj / 2; | ||||
| /* F1j is the number of values in the 1st sample | /* in the following, sum j = 1 to L is calculated for | |||
| * which are less than zj plus one half the number | * sample 1 and sample 2. | |||
| * of values in this sample which are equal to zj. | */ | |||
| */ | if (equal_z_both_samples) | |||
| { | ||||
| F1j = val_st_z_samp1 + (double) | /* hj is the number of values in the combined sample | |||
| (val_eq_z_samp1 - val_st_z_samp1) / 2; | * equal to zj | |||
| */ | ||||
| hj = val_eq_z_samp1 - val_st_z_samp1 | ||||
| + val_eq_z_samp2 - val_st_z_samp2; | ||||
| /* F2j is the number of values in the 1st sample | /* H_j is the number of values in the combined sample | |||
| * which are less than zj plus one half the number | * smaller than zj plus one half the the number of | |||
| * of values in this sample which are equal to zj. | * values in the combined sample equal to zj | |||
| */ | * (that's hj/2). | |||
| F2j = val_st_z_samp2 + (double) | */ | |||
| (val_eq_z_samp2 - val_st_z_samp2) / 2; | H_j = val_st_z_samp1 + val_st_z_samp2 | |||
| + hj / 2; | ||||
| /* set the line of values equal to zj to the | /* F1j is the number of values in the 1st sample | |||
| * actual line of the last value picked for zj. | * which are less than zj plus one half the number | |||
| */ | * of values in this sample which are equal to zj. | |||
| val_st_z_samp1 = val_eq_z_samp1; | */ | |||
| /* Set the line of values equal to zj to the actual | F1j = val_st_z_samp1 + (double) | |||
| * line of the last value picked for zjof each | (val_eq_z_samp1 - val_st_z_samp1) / 2; | |||
| * sample. This is required as data smaller than zj | ||||
| * is accounted differently than values equal to zj. | ||||
| */ | /* F2j is the number of values in the 1st sample | |||
| val_st_z_samp2 = val_eq_z_samp2; | * which are less than zj plus one half the number | |||
| * of values in this sample which are equal to zj. | ||||
| */ | ||||
| F2j = val_st_z_samp2 + (double) | ||||
| (val_eq_z_samp2 - val_st_z_samp2) / 2; | ||||
| /* next the lines of the next values z, ie. zj+1 | /* set the line of values equal to zj to the | |||
| * are addressed. | * actual line of the last value picked for zj. | |||
| */ | */ | |||
| val_eq_z_samp1++; | val_st_z_samp1 = val_eq_z_samp1; | |||
| /* next the lines of the next values z, ie. | /* Set the line of values equal to zj to the actual | |||
| * zj+1 are addressed | * line of the last value picked for zjof each | |||
| */ | * sample. This is required as data smaller than zj | |||
| val_eq_z_samp2++; | * is accounted differently than values equal to zj. | |||
| } | */ | |||
| else | val_st_z_samp2 = val_eq_z_samp2; | |||
| { | ||||
| /* the smaller z value was contained in sample 2, | /* next the lines of the next values z, ie. zj+1 | |||
| * hence this value is the zj to base the following | * are addressed. | |||
| * calculations on. | */ | |||
| */ | val_eq_z_samp1++; | |||
| if (next_z_sample2) | ||||
| { | ||||
| /* hj is the number of values in the combined | /* next the lines of the next values z, ie. | |||
| * sample equal to zj, in this case these are | * zj+1 are addressed | |||
| * within sample 2 only. | */ | |||
| */ | val_eq_z_samp2++; | |||
| hj = val_eq_z_samp2 - val_st_z_samp2; | } | |||
| else | ||||
| { | ||||
| /* H_j is the number of values in the combined sample | /* the smaller z value was contained in sample 2, | |||
| * smaller than zj plus one half the the number of | * hence this value is the zj to base the following | |||
| * values in the combined sample equal to zj | * calculations on. | |||
| * (that's hj/2). | */ | |||
| */ | if (next_z_sample2) | |||
| { | ||||
| H_j = val_st_z_samp1 + val_st_z_samp2 | /* hj is the number of values in the combined | |||
| + hj / 2; | * sample equal to zj, in this case these are | |||
| * within sample 2 only. | ||||
| */ | ||||
| hj = val_eq_z_samp2 - val_st_z_samp2; | ||||
| /* F1j is the number of values in the 1st sample which | /* H_j is the number of values in the combined sample | |||
| * are less than zj plus one half the number of values in | * smaller than zj plus one half the the number of | |||
| * this sample which are equal to zj. | * values in the combined sample equal to zj | |||
| * As val_eq_z_samp2 < val_eq_z_samp1, these are the | * (that's hj/2). | |||
| * val_st_z_samp1 only. | */ | |||
| */ | ||||
| F1j = val_st_z_samp1; | ||||
| /* F2j is the number of values in the 1st sample which | H_j = val_st_z_samp1 + val_st_z_samp2 | |||
| * are less than zj plus one half the number of values in | + hj / 2; | |||
| * this sample which are equal to zj. The latter are from | ||||
| * sample 2 only in this case. | ||||
| */ | ||||
| F2j = val_st_z_samp2 + (double) | /* F1j is the number of values in the 1st sample which | |||
| (val_eq_z_samp2 - val_st_z_samp2) / 2; | * are less than zj plus one half the number of values in | |||
| * this sample which are equal to zj. | ||||
| * As val_eq_z_samp2 < val_eq_z_samp1, these are the | ||||
| * val_st_z_samp1 only. | ||||
| */ | ||||
| F1j = val_st_z_samp1; | ||||
| /* Set the line of values equal to zj to the actual line | /* F2j is the number of values in the 1st sample which | |||
| * of the last value picked for zj of sample 2 only in | * are less than zj plus one half the number of values in | |||
| * this case. | * this sample which are equal to zj. The latter are from | |||
| */ | * sample 2 only in this case. | |||
| val_st_z_samp2 = val_eq_z_samp2; | */ | |||
| /* next the line of the next value z, ie. zj+1 is | F2j = val_st_z_samp2 + (double) | |||
| * addressed. Here, only sample 2 must be addressed. | (val_eq_z_samp2 - val_st_z_samp2) / 2; | |||
| */ | ||||
| val_eq_z_samp2++; | /* Set the line of values equal to zj to the actual line | |||
| if (val_eq_z_samp1 == 0) | * of the last value picked for zj of sample 2 only in | |||
| { | * this case. | |||
| val_eq_z_samp1 = stop_loop1; | */ | |||
| } | val_st_z_samp2 = val_eq_z_samp2; | |||
| } | ||||
| /* the smaller z value was contained in sample 2, | /* next the line of the next value z, ie. zj+1 is | |||
| * hence this value is the zj to base the following | * addressed. Here, only sample 2 must be addressed. | |||
| * calculations on. | */ | |||
| */ | ||||
| else | val_eq_z_samp2++; | |||
| { | if (val_eq_z_samp1 == 0) | |||
| { | ||||
| val_eq_z_samp1 = stop_loop1; | ||||
| } | ||||
| } | ||||
| /* hj is the number of values in the combined | /* the smaller z value was contained in sample 2, | |||
| * sample equal to zj, in this case these are | * hence this value is the zj to base the following | |||
| * within sample 1 only. | * calculations on. | |||
| */ | */ | |||
| hj = val_eq_z_samp1 - val_st_z_samp1; | ||||
| /* H_j is the number of values in the combined | else | |||
| * sample smaller than zj plus one half the the number | { | |||
| * of values in the combined sample equal to zj | ||||
| * (that's hj/2). | ||||
| */ | ||||
| H_j = val_st_z_samp1 + val_st_z_samp2 | /* hj is the number of values in the combined | |||
| + hj / 2; | * sample equal to zj, in this case these are | |||
| * within sample 1 only. | ||||
| */ | ||||
| hj = val_eq_z_samp1 - val_st_z_samp1; | ||||
| /* F1j is the number of values in the 1st sample which | /* H_j is the number of values in the combined | |||
| * are less than zj plus, in this case these are within | * sample smaller than zj plus one half the the number | |||
| * sample 1 only one half the number of values in this | * of values in the combined sample equal to zj | |||
| * sample which are equal to zj. The latter are from | * (that's hj/2). | |||
| * sample 1 only in this case. | */ | |||
| */ | ||||
| F1j = val_st_z_samp1 + (double) | H_j = val_st_z_samp1 + val_st_z_samp2 | |||
| (val_eq_z_samp1 - val_st_z_samp1) / 2; | + hj / 2; | |||
| /* F2j is the number of values in the 1st sample which | /* F1j is the number of values in the 1st sample which | |||
| * are less than zj plus one half the number of values | * are less than zj plus, in this case these are within | |||
| * in this sample which are equal to zj. As | * sample 1 only one half the number of values in this | |||
| * val_eq_z_samp1 < val_eq_z_samp2, these are the | * sample which are equal to zj. The latter are from | |||
| * val_st_z_samp2 only. | * sample 1 only in this case. | |||
| */ | */ | |||
| F2j = val_st_z_samp2; | F1j = val_st_z_samp1 + (double) | |||
| (val_eq_z_samp1 - val_st_z_samp1) / 2; | ||||
| /* Set the line of values equal to zj to the actual line | /* F2j is the number of values in the 1st sample which | |||
| * of the last value picked for zj of sample 1 only in | * are less than zj plus one half the number of values | |||
| * this case | * in this sample which are equal to zj. As | |||
| */ | * val_eq_z_samp1 < val_eq_z_samp2, these are the | |||
| * val_st_z_samp2 only. | ||||
| */ | ||||
| val_st_z_samp1 = val_eq_z_samp1; | F2j = val_st_z_samp2; | |||
| /* next the line of the next value z, ie. zj+1 is | /* Set the line of values equal to zj to the actual line | |||
| * addressed. Here, only sample 1 must be addressed. | * of the last value picked for zj of sample 1 only in | |||
| */ | * this case | |||
| val_eq_z_samp1++; | */ | |||
| if (val_eq_z_samp2 == 0) | val_st_z_samp1 = val_eq_z_samp1; | |||
| { | ||||
| val_eq_z_samp2 = stop_loop2; | ||||
| } | ||||
| } | ||||
| } | ||||
| denom_1_aux = n_total * F1j - n_sample1 * H_j; | /* next the line of the next value z, ie. zj+1 is | |||
| denom_2_aux = n_total * F2j - n_sample2 * H_j; | * addressed. Here, only sample 1 must be addressed. | |||
| */ | ||||
| val_eq_z_samp1++; | ||||
| sum_adk_samp1 = sum_adk_samp1 + hj | if (val_eq_z_samp2 == 0) | |||
| * (denom_1_aux * denom_1_aux) / | { | |||
| (H_j * (n_total - H_j) | val_eq_z_samp2 = stop_loop2; | |||
| - n_total * hj / 4); | } | |||
| sum_adk_samp2 = sum_adk_samp2 + hj | } | |||
| * (denom_2_aux * denom_2_aux) / | } | |||
| (H_j * (n_total - H_j) | ||||
| - n_total * hj / 4); | ||||
| next_z_sample2 = false; | ||||
| equal_z_both_samples = false; | ||||
| /* index to count the z. It is only required to prevent | denom_1_aux = n_total * F1j - n_sample1 * H_j; | |||
| * the while slope to execute endless | denom_2_aux = n_total * F2j - n_sample2 * H_j; | |||
| */ | ||||
| j++; | ||||
| } | ||||
| // calculating the adk value is the final step. | sum_adk_samp1 = sum_adk_samp1 + hj | |||
| adk_result = (double) (n_total - 1) / (n_total | * (denom_1_aux * denom_1_aux) / | |||
| * n_total * (k - 1)) | (H_j * (n_total - H_j) | |||
| * (sum_adk_samp1 / n_sample1 | - n_total * hj / 4); | |||
| + sum_adk_samp2 / n_sample2); | sum_adk_samp2 = sum_adk_samp2 + hj | |||
| * (denom_2_aux * denom_2_aux) / | ||||
| (H_j * (n_total - H_j) | ||||
| - n_total * hj / 4); | ||||
| /* if(adk_result <= adk_criterium) | next_z_sample2 = false; | |||
| * adk_2_sample test is passed | equal_z_both_samples = false; | |||
| */ | ||||
| return adk_result <= adk_criterium; | /* index to count the z. It is only required to prevent | |||
| } | * the while slope to execute endless | |||
| */ | ||||
| j++; | ||||
| } | ||||
| // calculating the adk value is the final step. | ||||
| adk_result = (double) (n_total - 1) / (n_total | ||||
| * n_total * (k - 1)) | ||||
| * (sum_adk_samp1 / n_sample1 | ||||
| + sum_adk_samp2 / n_sample2); | ||||
| /* if(adk_result <= adk_criterium) | ||||
| * adk_2_sample test is passed | ||||
| */ | ||||
| return adk_result <= adk_criterium; | ||||
| } | ||||
| Figure 5 | Figure 5 | |||
| Appendix C. Glossary | Appendix C. Glossary | |||
| +-------------+-----------------------------------------------------+ | +-------------+-----------------------------------------------------+ | |||
| | ADK | Anderson-Darling K-Sample test, a test used to | | | ADK | Anderson-Darling K-Sample test, a test used to | | |||
| | | check whether two samples have the same statistical | | | | check whether two samples have the same statistical | | |||
| | | distribution. | | | | distribution. | | |||
| | ECMP | Equal Cost Multipath, a load balancing mechanism | | | ECMP | Equal Cost Multipath, a load balancing mechanism | | |||
| End of changes. 129 change blocks. | ||||
| 575 lines changed or deleted | 498 lines changed or added | |||
This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/ | ||||