ICAR D. Partain Internet-Draft Ericsson Expires: November 30, 2004 June 2004 An Experiment in Early Cross-Area Review for the IETF draft-ietf-icar-experiment-early-review-00.txt Status of this Memo By submitting this Internet-Draft, I certify that any applicable patent or other IPR claims of which I am aware have been disclosed, and any of which I become aware will be disclosed, in accordance with RFC 3668. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http:// www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on November 30, 2004. Copyright Notice Copyright (C) The Internet Society (2004). All Rights Reserved. Abstract This memo captures current working group rough consensus on procedures for improved cross-area early review in the IETF. It is a product of the ICAR working group. This memo describes an experiment to improve early cross-area review in the IETF. Early cross-area review aims to improve quality of IETF work by identifying problems early in document development. Improved quality may, in turn, mean that documents can be processed faster in the IESG. Partain Expires November 30, 2004 [Page 1] Internet-Draft ICAR Experiment June 2004 Table of Contents 1. Introduction to ICAR . . . . . . . . . . . . . . . . . . . . . 3 2. Overview of this memo . . . . . . . . . . . . . . . . . . . . 3 3. Prerequisites to the ICAR early review process . . . . . . . . 4 4. ICAR review pool . . . . . . . . . . . . . . . . . . . . . . . 4 4.1 Makeup of the review pool . . . . . . . . . . . . . . . . 4 4.1.1 Seeding the initial reviewer pool . . . . . . . . . . 5 4.1.2 Objective criteria for entering the review pool . . . 5 4.1.3 Subjective criteria for entering the review pool . . . 5 5. Requesting a review . . . . . . . . . . . . . . . . . . . . . 5 5.1 Who requests a review . . . . . . . . . . . . . . . . . . 5 5.2 Available information about reviewers . . . . . . . . . . 6 5.3 How the review is requested . . . . . . . . . . . . . . . 6 5.4 Review announcements . . . . . . . . . . . . . . . . . . . 7 6. Results of a review . . . . . . . . . . . . . . . . . . . . . 7 6.1 Openly published and archived . . . . . . . . . . . . . . 7 6.2 Review is not binding on the WG . . . . . . . . . . . . . 7 6.3 WG's response to the review . . . . . . . . . . . . . . . 7 7. An initial experiment . . . . . . . . . . . . . . . . . . . . 8 8. Open issues . . . . . . . . . . . . . . . . . . . . . . . . . 8 9. Security Considerations . . . . . . . . . . . . . . . . . . . 9 Author's Address . . . . . . . . . . . . . . . . . . . . . . . 9 Intellectual Property and Copyright Statements . . . . . . . . 10 Partain Expires November 30, 2004 [Page 2] Internet-Draft ICAR Experiment June 2004 1. Introduction to ICAR Improved cross-area review (ICAR) aims to improve the overall quality of the work produced by the IETF's working groups and to increase the speed of that work's progression through the standardization process. This memo is an attempt to capture in a coherent form the rough consensus from the working group as well as the state of the on-going debate. Cross-area review is not a new idea in the IETF. Most particularly, the IESG has carried out this kind of review when documents have been put on their table. The IETF as a whole could function as a large review body and exactly this kind of review is theoretically solicited during a document's IETF Last Call. However, current review practices appear to have significant problems. Little or no feedback is the norm during an IETF Last Call. Waiting until the IESG review is deemed to be too late to correct significant problems with a document. ICAR proposes mechanisms by which early cross-area review is solicited, the review is carried out within an IETF-wide pool of reviewers, and the review itself as well as the working group's response to the review are publicly documented. The goal of the ICAR early review process is to ensure that reviews actually happen and the results are adequately considered by the working group. 2. Overview of this memo This document begins by outlining the necessary prerequisites for a successful ICAR early review process in Section 3. Thereafter, Section 4 is a discussion of the makeup of the ICAR review pool, how it is constituted, how reviewers are added, and how reviewers are removed from the pool. Section 5 then discusses who requests a review and how they do so. Section 6 discusses how the results are handled and how the working group requesting the review is expected to respond to the review. In order to prove the value of ICAR reviews, the ICAR working group aims to start a small-scale experiment, which is outlined in Section 7. Partain Expires November 30, 2004 [Page 3] Internet-Draft ICAR Experiment June 2004 Finally, Section 8 points out known open issues (to the editor...) that have yet to be resolved. 3. Prerequisites to the ICAR early review process In order for the ICAR early review process to be effective, the following need to be in place. 1. An open mailing list will be created on which requests for ICAR reviews will be posted. This will not be the mechanism by which the reviews are requested but is instead the way that interested parties and members of the IETF community who are not official reviewers can find out what reviews are needed. Quite apart from the ICAR process, any member of the IETF community may, of course, review a WG document at any time s/he wishes and provide that input to the WG. 2. A pool of volunteers needs to be available for reviews. This is discussed in Section 4. 3. The processes for handling review requests and results need to be ironed out (the goal of this memo). 4. There need to be tools available for managing the review flow. This includes information about reviewers (rough availability, review history, areas of expertise, etc.). The current tools at publication time are available at http://www.machshav.com/~icar/ reviews/ but are rough and are expected to evolve with the experiment. 5. The open issues listed in Section 8 should be resolved. 4. ICAR review pool A pool of volunteers who can review documents is needed. The WG considered the alternatives of one pool covering all IETF work and multiple pools, each covering an area of the IETF. The WG consensus is to have an IETF-wide review pool. 4.1 Makeup of the review pool The initial pool of reviewers when starting the ICAR early review process will be constructed as detailed in the following two subsections. The consensus of the ICAR working group is that there needs to be some form of admission control to ensure that those in the review pool have sufficient technical expertise to provide high-quality reviews as well as an adequate understanding of the way that the IETF works. The criteria listed below are tools to ensure that both of these are true. Partain Expires November 30, 2004 [Page 4] Internet-Draft ICAR Experiment June 2004 4.1.1 Seeding the initial reviewer pool In order to seed the review pool, ADs will suggest 5-6 people from their area whom the ADs will "sponsor" and who will commit to engaging in this activity. This would form the minimum pool that is needed to get ICAR reviews underway. In addition, there will be an open call for reviewers to join the pool. Those who volunteer will need to meet one of the objective or subjective criteria listed in the following subsections. 4.1.2 Objective criteria for entering the review pool The following criteria will be used as one way to determine eligibility for the review pool (and is based on the SIRS experiment - editor: add reference). The criteria listed in the bullets below will apply both when the initial review pool is constructed and on a continuing basis for those who wish to be added later. o people who have authored 2 IETF RFCs approved for publication by the IESG, which includes non-WG documents submitted directly to the IESG, but not documents submitted directly to the RFC editor o current/former IAB members o former IESG members o former WG chairs o current WG chairs whose WG has produced 2 RFCs o MIB doctors o directorate members 4.1.3 Subjective criteria for entering the review pool There is some push within the ICAR working group to include a more subjective method of getting into the review pool. However, there is not yet consensus as to how this would happen in practice. This section is simply a placeholder for the practice that the WG agrees to. 5. Requesting a review 5.1 Who requests a review In general, it is up to the working group in question to request an ICAR review. However, an AD may also request a review. ICAR reviews for working group documents are requested by a WG chair / secretary. If a document falls into two or more working group's charters, the working group chairs are expected to coordinate their calls for an ICAR review. Partain Expires November 30, 2004 [Page 5] Internet-Draft ICAR Experiment June 2004 A working group may request a review for any document it deems relevant to its work. This may be official working group drafts, but it may also include documents named as personal drafts but which are functioning as working group drafts or are under consideration to be WG items. An AD may request an ICAR review, which would typically be when a document does not clearly fall into a specific WG's domain. Those requesting a review are expected to request reviews from people with different areas of expertise. 5.2 Available information about reviewers The following information is available about reviewers in the pool: o Contact information o Area(s) of expertise o Some approximation of availability o Archive of previous reviews o Total number of pending requests to reviewer 5.3 How the review is requested It is up to the WG chair(s) / secretary to choose the reviewers who review the document. In order to do this, reviewers in the pool must be clearly labeled with their area(s) of expertise they feel qualified to provide . The ICAR WG does not believe that a broadcast request for reviews works. As such, when the requester decides to request an ICAR review, s/he will look through the list of reviewers and the information about them (see Section 5.2) and determine which reviewers are available and have the appropriate technical expertise. It is then the responsibility of the requester to make contact with these individuals to request a review. If the reviewer cannot do the review, the requester will have to find an alternative reviewer. When a requester and a reviewer agree on doing a review then the "ICAR infrastructure" should be notified (see Section 3 on tools for the ICAR experiment) so that information about the review and the review itself can be updated. Editor: Some in the WG have suggested that a management function is needed to shepherd the early review process. There is not yet any consensus in that regard. Partain Expires November 30, 2004 [Page 6] Internet-Draft ICAR Experiment June 2004 5.4 Review announcements In parallel to the closed ICAR reviewer pool, requests for reviews will be sent to an announce list, allowing the community to track which documents are actively being reviewed and encouraging people outside the ICAR reviewer pool to submit their own reviews of those documents. In addition, the web infrastructure set up to aid the ICAR early review process will include information about which documents are currently under review. 6. Results of a review 6.1 Openly published and archived All ICAR reviews should be openly published (on the WG list at a minimum). All ICAR reviews should be archived. Reviews by individuals in response to ICAR review request announcements are not archived in the ICAR tools infrastructure. They are assumed to be archived in the normal WG mailing list archives. 6.2 Review is not binding on the WG The ICAR reviewers do not have any kind of veto power over the working groups that request reviews. Reviewers can in no way force working groups to follow their advice. That is, the reviews are not binding on WGs. 6.3 WG's response to the review Although reviews are not binding, WGs are nonetheless obligated to consider and discuss each suggestion from the review. Not only must it be discussed, but there must also be adequate discussion documented to demonstrate why a working group chose not to implement a particular change requested by the reviewer. Like the review, the WG's response to the review should be archived. This is important, because the reasons for a WG's decision to disagree with a recommendation from a reviewer can be important input to the IESG when the document is submitted to the IESG for consideration. Partain Expires November 30, 2004 [Page 7] Internet-Draft ICAR Experiment June 2004 At least during the initial ICAR experiment, it would be useful for the WG to note explicitly in documents that they have undergone an ICAR review. For example, this could be noted in the "Status of this Memo" section or in the abstract. The ICAR process has no way of influencing how non-ICAR reviews are handled by WGs and cannot force WGs to respond in a similar fashion to these reviews. However, the IETF spirit is that all comments should be addressed in an appropriate manner. Documentation of the WG's response to such reviews is of course desirable. 7. An initial experiment The ICAR WG intends to recommend that an "experiment" be carried out with a group of willing participating reviewers and working groups. This section outlines the steps in carrying out that initial experiment. o Ask the ADs to choose one or two WGs per area whose chairs are amenable to using an ICAR review pool and to helping the ICAR working group gauge how the experiment is working. o Have the WGs in the WG pool solicit reviews on their current documents, any new documents, etc. Of course, there will be a mechanism by which we ensure that the initial burst will not be overwhelming. Editor: two per week max? o As we gain experience we can try to add more groups. o WGs from outside the initial WG pool may request reviews if they are so inclined. However, these WGs will not be expected to make such requests, whereas the WGs participating in the experiment will be expected to use the ICAR review-pool. 8. Open issues Editor: This list is composed of issues that seem to still be open in the working group. This list needs to be confirmed within the working group, and, if necessary, addressed in this document. 1. What guidelines should we set, if any, for the form of the review? Can we say that reviewers should provide something like a numbered list of concise comments and not a rambling stream? This facilitates solid responses from the WG and nicely separates issues that will inevitably be dealt with differently. 2. Do we need addition information available about potential reviewers as discussed in Section 5.2? For example, do we want history of number of requests to this reviewer? 3. What are the subjective criteria to be used for admittance to the pool (to be put into Section 4.1.3)? Partain Expires November 30, 2004 [Page 8] Internet-Draft ICAR Experiment June 2004 4. How are people removed from the pool or is such a process needed at all? 5. How does one measure the success/failure of ICAR experiments? 6. What level of involvement in the ICAR early review process can we expect / hope for from the IESG? In what way should the reviewers, prospective reviewers, and working groups interact with the IESG? 7. Do we want to stipulate the form in which a review and the WG's response to it are documented? 9. Security Considerations None. Author's Address David Partain Ericsson AB P.O. Box 1248 SE-581 12 Linkoping, Sweden EMail: david.partain@ericsson.com Partain Expires November 30, 2004 [Page 9] Internet-Draft ICAR Experiment June 2004 Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the IETF's procedures with respect to rights in IETF Documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2004). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. Partain Expires November 30, 2004 [Page 10]