Network Working Group L. Masinter Internet-Draft Adobe Systems Expires: April 14, 2006 October 11, 2005 Formalizing IETF Interoperability Reporting draft-ietf-newtrk-interop-reports-00.txt Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on April 14, 2006. Copyright Notice Copyright (C) The Internet Society (2005). Abstract This document suggests another way of reforming IETF standards process by formalizing the mechanism for interoperability reporting, as a way of facilitating standards development. It establishes two kinds of reports: a 'Protocol Feature Set', which lays out the set of features from IETF specifications that constitute a protocol, and a 'Protocol Implementation Report', which is submitted by an individual or group to report on implementation and interoperability testing. Masinter Expires April 14, 2006 [Page 1] Internet-Draft Interoperability Reports October 2005 1. Introduction The basic idea is to create formal structures for o ("Protocol Feature Set") Describing the set of specifications, and the features within them, that constitute a single "protocol", from the point of view of testing interoperability. (See below for format & publication process.) o ("Protocol Implementation Report") Creating a standard way that individuals can report on implementation and interoperability testing of a protocol. (See below for format & publication process.) These structures can be used to enhance the IETF standards process in the following ways: o Working groups (or individuals) preparing specifications for new protocols may also prepare the initial Protocol Feature Set. The IETF should publish these if they represent rough consensus. o Working groups preparing specifications for updating existing protocols or adding new options of features to an existing protocol may prepare a proposed extension to an existing published Protocol Feature Set. Again, updated Protocol Feature Sets that represent community (rough) consensus should be published. o Individuals or groups who have an implementation of a protocol, and those who have tested interoperability between independent implementations may prepare implementation reports (which may include reports of successful interoperability). o Implementation reports may contain comments about existing specifications. Groups interested in updating existing specifications to facilitate their advancement in standards status may use comments within implementation reports to give weight to "running code"; they may use the lack of implementation of particular features as motiviation for removal of those features in subsequent updates. o The IETF may use the existence of reports of successful interoperability by multiple independent implementations of every feature within a specification as evidence for advancing that specification. Note that specifications may require updates in order to make them suitable for advancement, as in current practice. o Implementation reports may also include assertions about widespread deployment of the implementations, and the IETF may use these reports as part of the basis for judgement of widespread deployment of protocol implementations as a basis for advancement of specifications. 2. Format of Protocol Feature Set Requirements: Masinter Expires April 14, 2006 [Page 2] Internet-Draft Interoperability Reports October 2005 o List of referenced technical specifications. o List of features, where a feature is a specification with chapter, section, paragraph, or quoted text. o A feature description may contain additional explanatory text, to clarify or otherwise elaborate the feature. o A feature description should indicate whether implementation is REQUIRED or optional for the protocol. o Protocols may define multiple roles (e.g., client/server/proxy). Protocol Feature Set can include sets of roles, and feature specifications can identify the roles for which the feature is appropriate. o May include references to other Protocol Feature Sets which are REQUIRED or OPTIONAL o Could be specified as an XML-based format, with text format automatically derived, and both XML and text published. 3. Scope and Granularity of Protocol Feature Set There is a great deal of judgement needed about how details to get in the protocol feature set. At the coarsest granularity, a feature set could have a single feature, which listed a single specification, at least for protocols with no options. How difficult it is to create the Protocol Feature Set depends a great deal on the quality of the original technical specifications. Protocol Feature Sets require rough consensus before they are published. However, rough consensus may be judged by the willingness of implementors to prepare Protocol Implementation Reports using the Protocol Feature Set framework. 4. Format of Protocol Implementation Report Requirements: o May not cover entire PFS. o Identity of implementation. Relationship to other previously reported implementations, if any. o If any IPR noted for any technical specification referenced in PFS, relationship of source of implementation to owner of IPR and/or other exercises of license process. o Optionally, assertions about deployment o Version history, for implementation report updates o Identity of author, relationship to implementation, IPR. o If implementation report has been reviewed by someone else (working group chair, interoperability event host), identity of reviewer. o For each feature: Masinter Expires April 14, 2006 [Page 3] Internet-Draft Interoperability Reports October 2005 * Was feature implemented? * If so, has feature been sucessfully tested as interoperable with at least one independent implementation? * Optionally, the identity of the other implementations against which interoperability was successfully tested * For asymmetric protocols (e.g., client/server, or different roles), repeat for each role played. * Optionally, a short comment about the way in which the feature had to be interpreted to be interoperable. This shouldn't be a place to publish a long article, however. * Could be specified as an XML representation, with text format automatically derived, to facilitate tools for automatic merging and summarizing implementation reports. o If Protocol Feature Set contains references to other Protocol Feature Sets, the Protocol Implementation Report may also reference corresponding Protocol Implementation Reports. o QUESTION: Is it a requirement to allow for anonomous implementation reports, where the implementation is not specifically identified? In some cases, interop events allow for this because product managers don't want competitors to use their implemetation reports in negative marketing. 5. Process for publication of Protocol Feature Set Authored against template. Should be reviewed by working group (if active) or IESG. Perhaps IETF last call not necessary? After all, proof is in whether there are actually any implementations willing to report on it. Updates to a Protocol Feature Set could be proposed by listing the proposed delta. In general, if specifications change, feature sets should be extended, not updated, unless there was some mistake. That is, the "feature" corresponds to the documented feature. 6. Process for publication of Protocol Implementation Report Preferably produced by someone responsible for the implementation. Perhaps could be reported by someone else, as long as actual implementor can update. May be updated at any time, old reports are still available. Updates can include new information or correction to old information. Perhaps there could be a mechanism for publishing comments on implementation reports. 7. Tools for viewing Protocol Feature Sets and Protocol Implementation Reports Masinter Expires April 14, 2006 [Page 4] Internet-Draft Interoperability Reports October 2005 If the format for submission of both kinds of reports are in XML, there could be tools for generating HTML and plain text versions of these reports. 8. Tools for combining information for combined reports To facilitate seeing the "whole picture", it would be useful to have some tools that would take the information in the published Protocol Feature Sets and Protocol Implementation Reports and generate implementation reports that could summarize, for each feature of a given protocol, o Whether it was not implemented o How many implementations there were. o How many implementations reported interoperability with an independent implementation. o The list of all comments about the feature. 9. Updating IETF processes Once we have provided a way of formalized interoperability reporting, we could consider ways in which IETF RFC 2026 standards process could be updated to make use of these. For example, we could consider automating progression of specifications from Proposed Standard to Draft Standard if sufficient combined interoperability reports existed. We would need to be clear about the minimum requirement for implementation reports. Alternatively, we could consider removing "Draft Standard" as a formal approval step; and instead (automatically) report which Standards Track documents had adequate interoperability reports. Since the IESG does not currently evaluate the accuracy of interoperability reporting, it would make it clearer that the judgment about the maturity of a protocol specification and its interoperable implementation is left to the reader of the specification and its interoperability reports. This would also simplify the decisions about "downreference", since references from widely implemented specifications to those with mixed implementation would not result in confusion. Finally, we could change the judgment of "full standard" from a judgement about the protocol specification to a judgement about what constitutes "widespread deployment" and whether the implementations reported had reached that status. 10. Comparison with ISD and SRD Note that this section will be removed if this proposal advances. The idea for formalizing interoperability reporting was based on the Masinter Expires April 14, 2006 [Page 5] Internet-Draft Interoperability Reports October 2005 ideas from ISDs and SRDs that we should have a single document that pulls together all of the specifications of a single "protocol". However, basing the full description of what constitutes a single "protocol" on the operational need to test interoperability creates a better justification for putting energy into the task, motivates a different category of individuals to work on it, and gives it an operational criteria for judging success. I imagine that a PFS wouldn't take much more work to author than an ISD. 11. Acknowledgements Thanks to Sam and others who helped flesh out the idea. 12. Informative References Masinter Expires April 14, 2006 [Page 6] Internet-Draft Interoperability Reports October 2005 Author's Address Larry Masinter Adobe Systems 345 Park Ave San Jose, CA 95110 US Phone: +1 408 536 3024 Email: LMM@acm.org URI: http://larry.masinter.net Masinter Expires April 14, 2006 [Page 7] Internet-Draft Interoperability Reports October 2005 Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2005). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. Masinter Expires April 14, 2006 [Page 8]