idnits 2.17.1 draft-iesg-odell-mibtest-01.txt: ** The Abstract section seems to be numbered Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Cannot find the required boilerplate sections (Copyright, IPR, etc.) in this document. Expected boilerplate is as follows today (2024-04-25) according to https://trustee.ietf.org/license-info : IETF Trust Legal Provisions of 28-dec-2009, Section 6.a: This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. IETF Trust Legal Provisions of 28-dec-2009, Section 6.b(i), paragraph 2: Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved. IETF Trust Legal Provisions of 28-dec-2009, Section 6.b(i), paragraph 3: This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** Missing expiration date. The document expiration date should appear on the first and last page. ** The document seems to lack a 1id_guidelines paragraph about Internet-Drafts being working documents. ** The document seems to lack a 1id_guidelines paragraph about 6 months document validity -- however, there's a paragraph with a matching beginning. Boilerplate error? ** The document seems to lack a 1id_guidelines paragraph about the list of current Internet-Drafts. ** The document seems to lack a 1id_guidelines paragraph about the list of Shadow Directories. == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. ** The abstract seems to contain references ([1]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (August 1998) is 9385 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '1' on line 51 == Unused Reference: 'RFC2026' is defined on line 178, but no explicit reference was found in the text Summary: 11 errors (**), 0 flaws (~~), 2 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group Mike O'Dell 3 Internet-Draft UUNET Technologies 4 Harald T. Alvestrand 5 Maxware 6 Bert Wijnen 7 IBM T. J. Watson Research 8 Scott Bradner 9 Harvard University 10 August 1998 12 Advancement of MIB specifications on the IETF Standards Track 14 16 1. Status of this Memo 18 This document is an Internet-Draft. Internet-Drafts are working 19 documents of the Internet Engineering Task Force (IETF), its areas, 20 and its working groups. Note that other groups may also distribute 21 working documents as Internet-Drafts. 23 Internet-Drafts are draft documents valid for a maximum of six months 24 and may be updated, replaced, or obsoleted by other documents at any 25 time. It is inappropriate to use Internet- Drafts as reference 26 material or to cite them other than as "work in progress." 28 To view the entire list of current Internet-Drafts, please check the 29 "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow 30 Directories on ftp.is.co.za (Africa), ftp.nordu.net (Northern 31 Europe), ftp.nis.garr.it (Southern Europe), munnari.oz.au (Pacific 32 Rim), ftp.ietf.org (US East Coast), or ftp.isi.edu (US West Coast). 34 A revised version of this draft document will be submitted to the RFC 35 editor as a BCP (Best Current Practice) documenting an IESG procedure 36 for the Internet Community. Discussion and suggestions for 37 improvement are requested. This document will expire before January, 38 1999. Distribution of this draft is unlimited. 40 2. Abstract 42 The Internet Standards Process [1] requires that all IETF Standards 43 Track specifications must have "multiple, independent, and 44 interoperable implementations" before they can be advanced beyond 45 Proposed Standard status. This document specifies the process which 46 the IESG will use to determine if a MIB document meets these 47 requirements. It also discusses the rationale for this process. 49 3. The Nature of the Problem 51 The Internet Standards Process [1] requires that for a IETF 52 specification to advance beyond the Proposed Standard level, at least 53 two genetically unrelated implementations must be shown to 54 interoperate correctly with all features and options. There are two 55 distinct reasons for this requirement. 57 The first reason is to verify that the text of the specification is 58 adequately clear and accurate. This is demonstrated by showing that 59 multiple implementation efforts have used the specification to 60 achieved interoperable implementations. 62 The second reason is to discourage excessive options and "feature 63 creep". This is accomplished by requiring interoperable 64 implementation of all features, including options. If an option is 65 not included in at least two different interoperable implementations, 66 it is safe to assume that it has not been deemed useful and must be 67 removed before the specification can advance. 69 In the case of a protocol specification which specifies the "bits on 70 the wire" exchanged by executing state machines, the notion of 71 "interoperability" is reasonably intuitive - the implementations must 72 successfully "talk to each other", exchanging "bits on the wire", 73 while exercising all features and options. 75 In the case of an SNMP Management Information Base (MIB) 76 specification, exactly what constitutes "interoperation" is less 77 obvious. This document specifies how the IESG has decided to judge 78 "MIB interoperability" in the context of the IETF Standards Process. 80 There are a number of plausible interpretations of MIB 81 interoperability, many of which have merit but which have very 82 different costs and difficulties in realization. 84 The aim is to ensure that the dual goals of specification clarity and 85 feature evaluation have been met using an interpretation of the 86 concept of MIB interoperability that strikes a balance between 87 testing complexity and practicality. 89 4. On The Nature of MIBs 91 Compared to "state machine" protocols which focus on procedural 92 specifications, a MIB is much more data oriented. To over- 93 generalize, in a typical MIB the collection of data type and instance 94 specifications outnumbers inter-object procedural or causal semantics 95 by a significant amount. 97 A central issue is that a MIB does not stand alone; it forms the 98 access interface to the instrumentation underneath it. Without the 99 instrumentation, a MIB has form but no values. Coupled with the large 100 number of objects even in a simple MIB, a MIB tends to have more of 101 the look and feel of an API or a dictionary than a state machine 102 protocol. 104 It is important to distinguish between assessing the interoperability 105 of applications which may use or interact with MIBs, and the MIBs 106 themselves. It is fairly obvious that "black-box testing" can be 107 applied to such applications and that the approach enjoys a certain 108 maturity in the software engineering arts. A MIB, on the other hand 109 is not readily amenable to black box test plans. 111 5. Discussion and Recommended Process 113 In order to meet their obligations under the IETF Standards Process 114 the Operations and Management Area Directors and the IESG must be 115 convinced that each MIB specification advanced to Draft Standard or 116 Internet Standard status is clearly written, that there are the 117 required multiple interoperable implementations, and that all options 118 have been implemented. There are multiple ways to achieve this goal. 119 Appendix A lists some testing approaches that could be used when 120 attempting to document multiple implementations. 122 The Full Coverage or Stimulus-Response approaches are very through, 123 and would increase confidence that the requirement has been met, if 124 applied. However, experience in real-world software engineering 125 makes it clear that such confidence comes at an extremely high price; 126 even with the most exhaustive testing, it is often not clear what 127 precisely has been demonstrated by such testing. We believe that 128 both of those standards of evidence are materially beyond what can be 129 reasonably accomplished in an operational sense, and achieving the 130 requisite semantic specifications are even more unlikely. 132 Therefore, the Operations and Management Area and the IESG have 133 adopted a more pragmatic approach to determining the suitability of a 134 MIB specification for advancement on the standards track beyond 135 Proposed Standard status. Each MIB specification suggested for 136 advancement must have one or more advocates who can make a convincing 137 argument that the MIB specification meets the multiple implementation 138 and feature support requirements of the IETF Standards Process. The 139 specific way to make the argument is left to the advocate, but will 140 normally include reports that basic object comparison testing has 141 been done. 143 Thus any recommendation for the advancement of a MIB specification 144 must be accompanied by an implementation report, as is the case with 145 all requests for the advancement of IETF specifications. The 146 implementation report must include the reasons why the IESG should 147 believe that there are multiple implementations of the MIB in 148 question and that the all of the MIB objects in the specification to 149 be advanced are supported in more than one implementation. But note 150 that the prime concern of the IESG will be that the underlying 151 reasons for the interoperable implementations are met, i.e. that the 152 text of the specification is clear and unambiguous, and that features 153 of the specification which have not garnered support have been 154 removed. 156 The implementation report will be placed on the IETF web page along 157 with the other pre-advancement implementation reports and will be 158 specifically referred to in the IETF Last-Call. As with all such 159 implementation reports, the determination of adequacy is made by the 160 Area Director(s) of the relevant IETF Area. This determination of 161 adequacy can be challenged during the Last-Call period. 163 6. Security Considerations 165 Some may view this policy as possibly leading to a reduction in the 166 level of confidence people can have in MIBs but the O&M Area 167 Directors and the IESG feel that it will adequately ensure a 168 reasonable evaluation of the level of clarity of MIB specifications 169 and to ensure that unused options can be identified and removed 170 before the advancement of a specification. 172 Good, clearly written MIBs can be of great assistance in the 173 management of the Internet and other networks and thus assist in the 174 reduction of some types of security threats. 176 8. References 178 [RFC2026] "The Internet Standards Process -- Revision 3", Bradner, 179 October 1996 181 9. Author's Addresses 183 Michael D. O'Dell 184 UUNET Technologies, Inc. 185 3060 Williams Drive 186 Fairfax, VA 22031 188 Email: mo@uu.net 189 Phone: +1-703-206-5890 190 Harald T. Alvestrand 191 Maxware 192 Pirsenteret 193 N-7005 Trondheim, Norway 195 EMail: Harald.Alvestrand@maxware.no 196 Phone: +47-73-54-57-94 198 Bert Wijnen 199 IBM T. J. Watson Research 200 Schagen 33 201 3461 GL Linschoten 202 Netherlands 204 EMail: wijnen@vnet.ibm.com 205 phone: +31-348-432-794 207 Scott Bradner 208 Harvard University 209 1350 Mass. Ave. 210 Cambridge MA 02138 212 Email: sob@harvard.edu 213 Phone: +1-617-495-3864 215 Appendix A 217 A. Some Testing Alternatives 219 The IESG debated a number of interoperability and testing 220 models in formulating this specification. The following 221 list is not an exhaustive enumeration of the alternatives, 222 but it does capture the major plausible models which were 223 examined in the course of the discussion. 225 A.1 Basic Object Comparison 227 Assume the requisite two genetically unrelated implementations of 228 the MIB in an SNMP agent and an SNMP management station which can 229 do a "MIB Dump" (extract the complete set of MIB object types and 230 values from the agent implementation). Extract a MIB Dump from 231 each implementation and compare the two dumps to verify that both 232 provide the complete set of mandatory and optional objects and 233 that the individual objects are of the correct types. 235 A.2 Stimulus/Response Testing 236 Proceed as in A.1, but in addition, comprehensively exercise the 237 two (network) elements containing the agent implementations to 238 verify that all the MIB objects reflect plausible values in 239 operational conditions. An even stricter interpretation would 240 require that the MIB objects in the two network elements track 241 identically given the identical stimulus. While this would 242 test "read-only" or "monitoring" information obtained from the 243 underlying instrumentation, it is important to observe that 244 such instrumentation is actually an *application* which uses 245 the MIB and is not part of the MIB itself. 247 A.3 Full Coverage Testing 249 This model extends the notion of Stimulus/Response Testing to its 250 logical extreme. The MIB is viewed as an API and the 251 software engineering notion of full coverage testing is 252 applied to a MIB. This involves exercising all paths through the 253 causal semantics and verifying that all objects change state 254 correctly in all cases. Again, note that much more than the 255 MIB definition is being exercised and evaluated.