idnits 2.17.1 draft-bradner-metricstest-01.txt: ** The Abstract section seems to be numbered Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 19. -- Found old boilerplate from RFC 3978, Section 5.5, updated by RFC 4748 on line 262. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 273. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 280. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 286. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. ** There are 2 instances of too long lines in the document, the longest one being 70 characters in excess of 72. ** The abstract seems to contain references ([RFC2026]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 130: '...ware. The tests MAY be sequential on ...' RFC 2119 keyword, line 132: '...inishes, or they MAY be run in paralle...' RFC 2119 keyword, line 134: '... RECOMMENDED that the tests be run n...' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust Copyright Line does not match the current year == Line 56 has weird spacing: '...tically unrel...' == Line 57 has weird spacing: '...rrectly with ...' -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (April 2007) is 6220 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'RFC2438' is defined on line 224, but no explicit reference was found in the text Summary: 8 errors (**), 0 flaws (~~), 5 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group Scott Bradner 3 Internet-Draft Harvard University 4 Allison Mankin 5 USC/ISI 6 Vern Paxson 7 ACIRI 8 April 2007 10 Advancement of metrics specifications on the IETF Standards Track 12 14 1. Status of this Memo 16 By submitting this Internet-Draft, each author represents that any 17 applicable patent or other IPR claims of which he or she is aware 18 have been or will be disclosed, and any of which he or she becomes 19 aware will be disclosed, in accordance with Section 6 of BCP 79. 21 Internet-Drafts are working documents of the Internet Engineering 22 Task Force (IETF), its areas, and its working groups. Note that 23 other groups may also distribute working documents as Internet- 24 Drafts. 26 Internet-Drafts are draft documents valid for a maximum of six months 27 and may be updated, replaced, or obsoleted by other documents at any 28 time. It is inappropriate to use Internet-Drafts as reference 29 material or to cite them other than as "work in progress." 31 The list of current Internet-Drafts can be accessed at 32 http://www.ietf.org/ietf/1id-abstracts.txt 34 The list of Internet-Draft Shadow Directories can be accessed at 35 http://www.ietf.org/shadow.html 37 This Internet-Draft will expire on October 20, 2007. 39 Copyright Notice 40 Copyright (C) The IETF Trust (2007). 42 2. Abstract 44 The Internet Standards Process [RFC2026] requires that all IETF 45 Standards Track specifications must have "multiple, independent, and 46 interoperable implementations" before they can be advanced beyond 47 Proposed Standard status. This document specifies the test which the 48 IESG will use to determine if a metrics specification document meets 49 these requirements. It also discusses the rationale for this 50 process. 52 3. The Nature of the Problem 54 The Internet Standards Process [RFC2026] requires that for a IETF 55 specification to advance beyond the Proposed Standard level, at least 56 two genetically unrelated implementations must be shown to 57 interoperate correctly with all features and options. There are two 58 distinct reasons for this requirement. 60 The first reason is to verify that the text of the specification is 61 adequately clear and accurate. This is demonstrated by showing that 62 multiple implementation efforts have used the specification to 63 achieved interoperable implementations. 65 The second reason is to discourage excessive options and "feature 66 creep". This is accomplished by requiring interoperable 67 implementation of all features, including options. If an option is 68 not included in at least two different interoperable implementations, 69 it is safe to assume that it has not been deemed useful and must be 70 removed before the specification can advance. 72 In the case of a protocol specification which specifies the "bits on 73 the wire" exchanged by executing state machines, the notion of 74 "interoperability" is reasonably intuitive - the implementations must 75 successfully "talk to each other", exchanging "bits on the wire", 76 while exercising all features and options. 78 In the case of a specification for a performance metric, network 79 latency for example, exactly what constitutes "interoperation" is 80 less obvious. This document specifies how the IESG has decided to 81 judge "metric specification interoperability" in the context of the 82 IETF Standards Process. 84 The aim is to ensure that the dual goals of specification clarity and 85 feature evaluation have been met using an interpretation of the 86 concept of metric specification interoperability that strikes a 87 balance between testing complexity and practicality. 89 4. On The Nature of metric specifications 91 Compared to "state machine" protocols which focus on procedural 92 specifications, a metric specification describes a method of testing 93 and a way to report the results of this testing. One example of such 94 a metric would be a way to test and report the latency that data 95 packets would incur while being sent from one network location to 96 another. 98 Since implementations of testing metrics are by their nature stand- 99 alone and do not interact with each other, the level of the 100 interoperability called for in the IETF standards process cannot be 101 simply determined by seeing that the implementations interact 102 properly. 104 5. Discussion and Recommended Process 106 In order to meet their obligations under the IETF Standards Process 107 the IESG must be convinced that each metric specification advanced to 108 Draft Standard or Internet Standard status is clearly written, that 109 there are the required multiple interoperable implementations, and 110 that all options have been implemented. There may be multiple ways 111 to achieve this goal; this memo documents the way that the IESG will 112 use to determine if the requirements have been met. 114 In the context of this memo, metrics are designed to measure some 115 characteristic of a data network. An aim of any metric definition 116 should be that it should be specified in a way that can reliably 117 measure the specific characteristic in a repeatable way. Thus 118 running the test a number of times on a stable network should produce 119 essentially the same results. In the same way, sequentially running 120 different implementations of software that perform the tests 121 described in the metric document on a stable network, or 122 simultaneously on a network that may or may not be stable should 123 produce essentially the same results. 125 Following these assumptions any recommendation for the advancement of 126 a metric specification must be accompanied by an implementation 127 report, as is the case with all requests for the advancement of IETF 128 specifications. The implementation report must include reports of 129 tests performed between sets of points on a network with two or more 130 implementations of the software. The tests MAY be sequential on the 131 same or different hardware, with each implementation being run after 132 the previous one finishes, or they MAY be run in parallel with 133 multiple implementations each on different hardware. It is 134 RECOMMENDED that the tests be run not in deterministic order, but 135 instead by executing a randomizing algorithm on the schedule, and 136 interspersing tests from the several implementations at randomly 137 selected times until all tests of all implementations are complete. 138 This is a way of avoiding a biased result in relation to the network 139 conditions and other factors while making the comparative tests. 141 All of the tests for each set should be run in the same direction 142 between the same two points on the same network. The tests should be 143 run simultaneously unless the network is stable enough to ensure that 144 the path the data takes through the network will not change between 145 tests. The tests should be run multiple times and the results 146 compared and the mean and variance determined. The results of the 147 tests using different implementations of the testing software must be 148 within the same variance as the results obtained from multiple 149 executions of the same software. 151 In particular, if the variance using Method A is sigma^2, then the 152 value yielded by Method B should fall within 2*sigma of the mean 153 value reported by Method A at least ### % of the time (The percentage 154 value will be filled in during a discussion of statistics and metrics 155 in the upcoming IPPM meeting). Note that if Method A and Method B 156 are identical, and the value they are estimating is distributed 157 according to a Gaussian distribution, then we would expect that 158 Method A's estimates would fall within 2*sigma of its mean value ### 159 % of the time. We allow more deviation in recognition of the many 160 pragmatic (and sometimes systemic) difficulties in realizing 161 consistent network measurement, and the fact that many quantities 162 related to networking have distributions quite different from 163 Gaussian. In addition, we explicitly recognize the IESG's 164 prerogative to relax the restriction of ### % within 2*sigma in light 165 of the particular measurement factors and difficulties, providing 166 these are expressed (in a communication with the relevant working 167 group or document author) by the IESG. 169 If the metric has options, all of the options must be tested in the 170 same way. For example, if the metric includes a list of packet sizes 171 that can be used, all of the listed sizes should be tested. If some 172 of the options are not supported or tested they must be removed from 173 the specification before the specification can be advanced on the 174 standards track. 176 As stated above, the measurements are made over a set of points in a 177 network. The Area Director(s), in consultation with the working 178 group chairs (if applicable), will recommend to the IESG their 179 judgment as to the adequacy of the set of measurement points in 180 covering the range of relevant network conditions. 182 An implementation report is required for both the advancement from 183 Proposed Standard to Draft Standard and from Draft Standard to 184 Internet Standard. The implementation report for advancement from 185 Draft Standard to Internet Standard can be an updated version of the 186 one used for the advancement from Proposed Standard to Draft 187 Standard. 189 The prime concern of the IESG will be that the underlying reasons for 190 the interoperable implementations are met, i.e. that the text of the 191 specification is clear and unambiguous, and that features of the 192 specification which have not garnered support have been removed. 194 The implementation report will be placed on the IETF web page along 195 with the other pre-advancement implementation reports and will be 196 specifically referred to in the IETF Last-Call. As with all such 197 implementation reports, the determination of adequacy is made by the 198 IESG upon recommendation by the Area Director(s) of the relevant IETF 199 Area. This determination of adequacy can be challenged during the 200 Last-Call period. 202 6. Security Considerations 204 Some may view this policy as possibly leading to a reduction in the 205 level of confidence people can have in metrics specifications, but 206 the IESG feels that it will adequately ensure a reasonable evaluation 207 of the level of clarity and ensure that unused options can be 208 identified and removed before the advancement of a specification. 210 Good, clearly written metric specifications can be of great 211 assistance in the measurement and management of the Internet and thus 212 assist in the reduction of some types of security threats. 214 8. Acknowledgements 215 The basic format and some of the text for this memo came from 216 RFC2438], "Advancement of MIB specifications on the IETF Standards 217 Track", which provides similar guidance for the advancement of MIBs. 219 8. References 221 [RFC2026] "The Internet Standards Process -- Revision 3", Bradner, 222 October 1996 224 [RFC2438] "Advancement of MIB specifications on the IETF Standards 225 Track", O'Dell, Alvestrand, Wijnen, & Bradner, October 1998 227 9. Author's Addresses 229 Scott Bradner 230 Harvard University 231 29 Oxford St. 232 Cambridge MA 02138 234 Email: sob@harvard.edu 235 Phone: +1-617-495-3864 237 Allison Mankin 238 NSF 239 Email: mankin@psg.com 241 Vern Paxson 242 ICRI 243 1947 Center St., Suite 600, 244 Berkeley, CA 94704-1198 245 USA 247 Email: vern@icir.org 249 Full Copyright Statement 251 Copyright (C) The IETF Trust (2007). 253 This document is subject to the rights, licenses and restrictions 254 contained in BCP 78, and except as set forth therein, the authors 255 retain all their rights. 257 This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 258 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND 259 THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS 260 OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF 261 THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 262 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 264 Intellectual Property 266 The IETF takes no position regarding the validity or scope of any 267 Intellectual Property Rights or other rights that might be claimed to 268 pertain to the implementation or use of the technology described in 269 this document or the extent to which any license under such rights 270 might or might not be available; nor does it represent that it has 271 made any independent effort to identify any such rights. Information 272 on the procedures with respect to rights in RFC documents can be 273 found in BCP 78 and BCP 79. 275 Copies of IPR disclosures made to the IETF Secretariat and any 276 assurances of licenses to be made available, or the result of an 277 attempt made to obtain a general license or permission for the use of 278 such proprietary rights by implementers or users of this 279 specification can be obtained from the IETF on-line IPR repository at 280 http://www.ietf.org/ipr. 282 The IETF invites any interested party to bring to its attention any 283 copyrights, patents or patent applications, or other proprietary 284 rights that may cover technology that may be required to implement 285 this standard. Please address the information to the IETF at 286 ietf-ipr@ietf.org. 288 Acknowledgment 290 Funding for the RFC Editor function is provided by the IETF Administrative Support Activity (IASA).