idnits 2.17.1 draft-ietf-icar-experiment-early-review-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3667, Section 5.1 on line 14. -- Found old boilerplate from RFC 3978, Section 5.5 on line 419. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 403. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 409. ** Found boilerplate matching RFC 3978, Section 5.4, paragraph 1 (on line 425), which is fine, but *also* found old RFC 2026, Section 10.4C, paragraph 1 text on line 35. ** The document seems to lack an RFC 3978 Section 5.1 IPR Disclosure Acknowledgement -- however, there's a paragraph with a matching beginning. Boilerplate error? ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. ** The document seems to lack an RFC 3979 Section 5, para. 1 IPR Disclosure Acknowledgement -- however, there's a paragraph with a matching beginning. Boilerplate error? ** The document uses RFC 3667 boilerplate or RFC 3978-like boilerplate instead of verbatim RFC 3978 boilerplate. After 6 May 2005, submission of drafts without verbatim RFC 3978 boilerplate is not accepted. The following non-3978 patterns matched text found in the document. That text should be removed or replaced: By submitting this Internet-Draft, I certify that any applicable patent or other IPR claims of which I am aware have been disclosed, or will be disclosed, and any of which I become aware will be disclosed, in accordance with RFC 3668. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** There is 1 instance of lines with control characters in the document. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (June 2004) is 7256 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) No issues found here. Summary: 9 errors (**), 0 flaws (~~), 2 warnings (==), 6 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 ICAR D. Partain 3 Internet-Draft Ericsson 4 Expires: November 30, 2004 June 2004 6 An Experiment in Early Cross-Area Review for the IETF 7 draft-ietf-icar-experiment-early-review-00.txt 9 Status of this Memo 11 By submitting this Internet-Draft, I certify that any applicable 12 patent or other IPR claims of which I am aware have been disclosed, 13 and any of which I become aware will be disclosed, in accordance with 14 RFC 3668. 16 Internet-Drafts are working documents of the Internet Engineering 17 Task Force (IETF), its areas, and its working groups. Note that other 18 groups may also distribute working documents as Internet-Drafts. 20 Internet-Drafts are draft documents valid for a maximum of six months 21 and may be updated, replaced, or obsoleted by other documents at any 22 time. It is inappropriate to use Internet-Drafts as reference 23 material or to cite them other than as "work in progress." 25 The list of current Internet-Drafts can be accessed at http:// 26 www.ietf.org/ietf/1id-abstracts.txt. 28 The list of Internet-Draft Shadow Directories can be accessed at 29 http://www.ietf.org/shadow.html. 31 This Internet-Draft will expire on November 30, 2004. 33 Copyright Notice 35 Copyright (C) The Internet Society (2004). All Rights Reserved. 37 Abstract 39 This memo captures current working group rough consensus on 40 procedures for improved cross-area early review in the IETF. It is a 41 product of the ICAR working group. 43 This memo describes an experiment to improve early cross-area review 44 in the IETF. Early cross-area review aims to improve quality of IETF 45 work by identifying problems early in document development. Improved 46 quality may, in turn, mean that documents can be processed faster in 47 the IESG. 49 Table of Contents 51 1. Introduction to ICAR . . . . . . . . . . . . . . . . . . . . . 3 52 2. Overview of this memo . . . . . . . . . . . . . . . . . . . . 3 53 3. Prerequisites to the ICAR early review process . . . . . . . . 4 54 4. ICAR review pool . . . . . . . . . . . . . . . . . . . . . . . 4 55 4.1 Makeup of the review pool . . . . . . . . . . . . . . . . 4 56 4.1.1 Seeding the initial reviewer pool . . . . . . . . . . 5 57 4.1.2 Objective criteria for entering the review pool . . . 5 58 4.1.3 Subjective criteria for entering the review pool . . . 5 59 5. Requesting a review . . . . . . . . . . . . . . . . . . . . . 5 60 5.1 Who requests a review . . . . . . . . . . . . . . . . . . 5 61 5.2 Available information about reviewers . . . . . . . . . . 6 62 5.3 How the review is requested . . . . . . . . . . . . . . . 6 63 5.4 Review announcements . . . . . . . . . . . . . . . . . . . 7 64 6. Results of a review . . . . . . . . . . . . . . . . . . . . . 7 65 6.1 Openly published and archived . . . . . . . . . . . . . . 7 66 6.2 Review is not binding on the WG . . . . . . . . . . . . . 7 67 6.3 WG's response to the review . . . . . . . . . . . . . . . 7 68 7. An initial experiment . . . . . . . . . . . . . . . . . . . . 8 69 8. Open issues . . . . . . . . . . . . . . . . . . . . . . . . . 8 70 9. Security Considerations . . . . . . . . . . . . . . . . . . . 9 71 Author's Address . . . . . . . . . . . . . . . . . . . . . . . 9 72 Intellectual Property and Copyright Statements . . . . . . . . 10 74 1. Introduction to ICAR 76 Improved cross-area review (ICAR) aims to improve the overall quality 77 of the work produced by the IETF's working groups and to increase the 78 speed of that work's progression through the standardization process. 80 This memo is an attempt to capture in a coherent form the rough 81 consensus from the working group as well as the state of the on-going 82 debate. 84 Cross-area review is not a new idea in the IETF. Most particularly, 85 the IESG has carried out this kind of review when documents have been 86 put on their table. The IETF as a whole could function as a large 87 review body and exactly this kind of review is theoretically 88 solicited during a document's IETF Last Call. 90 However, current review practices appear to have significant 91 problems. Little or no feedback is the norm during an IETF Last 92 Call. Waiting until the IESG review is deemed to be too late to 93 correct significant problems with a document. 95 ICAR proposes mechanisms by which early cross-area review is 96 solicited, the review is carried out within an IETF-wide pool of 97 reviewers, and the review itself as well as the working group's 98 response to the review are publicly documented. 100 The goal of the ICAR early review process is to ensure that reviews 101 actually happen and the results are adequately considered by the 102 working group. 104 2. Overview of this memo 106 This document begins by outlining the necessary prerequisites for a 107 successful ICAR early review process in Section 3. 109 Thereafter, Section 4 is a discussion of the makeup of the ICAR 110 review pool, how it is constituted, how reviewers are added, and how 111 reviewers are removed from the pool. 113 Section 5 then discusses who requests a review and how they do so. 115 Section 6 discusses how the results are handled and how the working 116 group requesting the review is expected to respond to the review. 118 In order to prove the value of ICAR reviews, the ICAR working group 119 aims to start a small-scale experiment, which is outlined in Section 120 7. 122 Finally, Section 8 points out known open issues (to the editor...) 123 that have yet to be resolved. 125 3. Prerequisites to the ICAR early review process 127 In order for the ICAR early review process to be effective, the 128 following need to be in place. 130 1. An open mailing list will be created on which requests for ICAR 131 reviews will be posted. This will not be the mechanism by which 132 the reviews are requested but is instead the way that interested 133 parties and members of the IETF community who are not official 134 reviewers can find out what reviews are needed. Quite apart from 135 the ICAR process, any member of the IETF community may, of 136 course, review a WG document at any time s/he wishes and provide 137 that input to the WG. 138 2. A pool of volunteers needs to be available for reviews. This is 139 discussed in Section 4. 140 3. The processes for handling review requests and results need to be 141 ironed out (the goal of this memo). 142 4. There need to be tools available for managing the review flow. 143 This includes information about reviewers (rough availability, 144 review history, areas of expertise, etc.). The current tools at 145 publication time are available at http://www.machshav.com/~icar/ 146 reviews/ but are rough and are expected to evolve with the 147 experiment. 148 5. The open issues listed in Section 8 should be resolved. 150 4. ICAR review pool 152 A pool of volunteers who can review documents is needed. The WG 153 considered the alternatives of one pool covering all IETF work and 154 multiple pools, each covering an area of the IETF. The WG consensus 155 is to have an IETF-wide review pool. 157 4.1 Makeup of the review pool 159 The initial pool of reviewers when starting the ICAR early review 160 process will be constructed as detailed in the following two 161 subsections. 163 The consensus of the ICAR working group is that there needs to be 164 some form of admission control to ensure that those in the review 165 pool have sufficient technical expertise to provide high-quality 166 reviews as well as an adequate understanding of the way that the IETF 167 works. The criteria listed below are tools to ensure that both of 168 these are true. 170 4.1.1 Seeding the initial reviewer pool 172 In order to seed the review pool, ADs will suggest 5-6 people from 173 their area whom the ADs will "sponsor" and who will commit to 174 engaging in this activity. This would form the minimum pool that is 175 needed to get ICAR reviews underway. 177 In addition, there will be an open call for reviewers to join the 178 pool. Those who volunteer will need to meet one of the objective or 179 subjective criteria listed in the following subsections. 181 4.1.2 Objective criteria for entering the review pool 183 The following criteria will be used as one way to determine 184 eligibility for the review pool (and is based on the SIRS experiment 185 - editor: add reference). The criteria listed in the bullets below 186 will apply both when the initial review pool is constructed and on a 187 continuing basis for those who wish to be added later. 189 o people who have authored 2 IETF RFCs approved for publication by 190 the IESG, which includes non-WG documents submitted directly to 191 the IESG, but not documents submitted directly to the RFC editor 192 o current/former IAB members 193 o former IESG members 194 o former WG chairs 195 o current WG chairs whose WG has produced 2 RFCs 196 o MIB doctors 197 o directorate members 199 4.1.3 Subjective criteria for entering the review pool 201 There is some push within the ICAR working group to include a more 202 subjective method of getting into the review pool. However, there is 203 not yet consensus as to how this would happen in practice. This 204 section is simply a placeholder for the practice that the WG agrees 205 to. 207 5. Requesting a review 209 5.1 Who requests a review 211 In general, it is up to the working group in question to request an 212 ICAR review. However, an AD may also request a review. 214 ICAR reviews for working group documents are requested by a WG chair 215 / secretary. If a document falls into two or more working group's 216 charters, the working group chairs are expected to coordinate their 217 calls for an ICAR review. 219 A working group may request a review for any document it deems 220 relevant to its work. This may be official working group drafts, but 221 it may also include documents named as personal drafts but which are 222 functioning as working group drafts or are under consideration to be 223 WG items. 225 An AD may request an ICAR review, which would typically be when a 226 document does not clearly fall into a specific WG's domain. 228 Those requesting a review are expected to request reviews from people 229 with different areas of expertise. 231 5.2 Available information about reviewers 233 The following information is available about reviewers in the pool: 235 o Contact information 236 o Area(s) of expertise 237 o Some approximation of availability 238 o Archive of previous reviews 239 o Total number of pending requests to reviewer 241 5.3 How the review is requested 243 It is up to the WG chair(s) / secretary to choose the reviewers who 244 review the document. In order to do this, reviewers in the pool must 245 be clearly labeled with their area(s) of expertise they feel 246 qualified to provide . 248 The ICAR WG does not believe that a broadcast request for reviews 249 works. As such, when the requester decides to request an ICAR 250 review, s/he will look through the list of reviewers and the 251 information about them (see Section 5.2) and determine which 252 reviewers are available and have the appropriate technical expertise. 254 It is then the responsibility of the requester to make contact with 255 these individuals to request a review. If the reviewer cannot do the 256 review, the requester will have to find an alternative reviewer. 258 When a requester and a reviewer agree on doing a review then the 259 "ICAR infrastructure" should be notified (see Section 3 on tools for 260 the ICAR experiment) so that information about the review and the 261 review itself can be updated. 263 Editor: Some in the WG have suggested that a management function is 264 needed to shepherd the early review process. There is not yet any 265 consensus in that regard. 267 5.4 Review announcements 269 In parallel to the closed ICAR reviewer pool, requests for reviews 270 will be sent to an announce list, allowing the community to track 271 which documents are actively being reviewed and encouraging people 272 outside the ICAR reviewer pool to submit their own reviews of those 273 documents. 275 In addition, the web infrastructure set up to aid the ICAR early 276 review process will include information about which documents are 277 currently under review. 279 6. Results of a review 281 6.1 Openly published and archived 283 All ICAR reviews should be openly published (on the WG list at a 284 minimum). 286 All ICAR reviews should be archived. 288 Reviews by individuals in response to ICAR review request 289 announcements are not archived in the ICAR tools infrastructure. 290 They are assumed to be archived in the normal WG mailing list 291 archives. 293 6.2 Review is not binding on the WG 295 The ICAR reviewers do not have any kind of veto power over the 296 working groups that request reviews. Reviewers can in no way force 297 working groups to follow their advice. That is, the reviews are not 298 binding on WGs. 300 6.3 WG's response to the review 302 Although reviews are not binding, WGs are nonetheless obligated to 303 consider and discuss each suggestion from the review. Not only must 304 it be discussed, but there must also be adequate discussion 305 documented to demonstrate why a working group chose not to implement 306 a particular change requested by the reviewer. 308 Like the review, the WG's response to the review should be archived. 310 This is important, because the reasons for a WG's decision to 311 disagree with a recommendation from a reviewer can be important input 312 to the IESG when the document is submitted to the IESG for 313 consideration. 315 At least during the initial ICAR experiment, it would be useful for 316 the WG to note explicitly in documents that they have undergone an 317 ICAR review. For example, this could be noted in the "Status of this 318 Memo" section or in the abstract. 320 The ICAR process has no way of influencing how non-ICAR reviews are 321 handled by WGs and cannot force WGs to respond in a similar fashion 322 to these reviews. However, the IETF spirit is that all comments 323 should be addressed in an appropriate manner. Documentation of the 324 WG's response to such reviews is of course desirable. 326 7. An initial experiment 328 The ICAR WG intends to recommend that an "experiment" be carried out 329 with a group of willing participating reviewers and working groups. 330 This section outlines the steps in carrying out that initial 331 experiment. 333 o Ask the ADs to choose one or two WGs per area whose chairs are 334 amenable to using an ICAR review pool and to helping the ICAR 335 working group gauge how the experiment is working. 336 o Have the WGs in the WG pool solicit reviews on their current 337 documents, any new documents, etc. Of course, there will be a 338 mechanism by which we ensure that the initial burst will not be 339 overwhelming. Editor: two per week max? 340 o As we gain experience we can try to add more groups. 341 o WGs from outside the initial WG pool may request reviews if they 342 are so inclined. However, these WGs will not be expected to make 343 such requests, whereas the WGs participating in the experiment 344 will be expected to use the ICAR review-pool. 346 8. Open issues 348 Editor: This list is composed of issues that seem to still be open 349 in the working group. This list needs to be confirmed within the 350 working group, and, if necessary, addressed in this document. 352 1. What guidelines should we set, if any, for the form of the 353 review? Can we say that reviewers should provide something like 354 a numbered list of concise comments and not a rambling stream? 355 This facilitates solid responses from the WG and nicely separates 356 issues that will inevitably be dealt with differently. 357 2. Do we need addition information available about potential 358 reviewers as discussed in Section 5.2? For example, do we want 359 history of number of requests to this reviewer? 360 3. What are the subjective criteria to be used for admittance to the 361 pool (to be put into Section 4.1.3)? 363 4. How are people removed from the pool or is such a process needed 364 at all? 365 5. How does one measure the success/failure of ICAR experiments? 366 6. What level of involvement in the ICAR early review process can we 367 expect / hope for from the IESG? In what way should the 368 reviewers, prospective reviewers, and working groups interact 369 with the IESG? 370 7. Do we want to stipulate the form in which a review and the WG's 371 response to it are documented? 373 9. Security Considerations 375 None. 377 Author's Address 379 David Partain 380 Ericsson AB 381 P.O. Box 1248 382 SE-581 12 Linkoping, 383 Sweden 385 EMail: david.partain@ericsson.com 387 Intellectual Property Statement 389 The IETF takes no position regarding the validity or scope of any 390 Intellectual Property Rights or other rights that might be claimed to 391 pertain to the implementation or use of the technology described in 392 this document or the extent to which any license under such rights 393 might or might not be available; nor does it represent that it has 394 made any independent effort to identify any such rights. Information 395 on the IETF's procedures with respect to rights in IETF Documents can 396 be found in BCP 78 and BCP 79. 398 Copies of IPR disclosures made to the IETF Secretariat and any 399 assurances of licenses to be made available, or the result of an 400 attempt made to obtain a general license or permission for the use of 401 such proprietary rights by implementers or users of this 402 specification can be obtained from the IETF on-line IPR repository at 403 http://www.ietf.org/ipr. 405 The IETF invites any interested party to bring to its attention any 406 copyrights, patents or patent applications, or other proprietary 407 rights that may cover technology that may be required to implement 408 this standard. Please address the information to the IETF at 409 ietf-ipr@ietf.org. 411 Disclaimer of Validity 413 This document and the information contained herein are provided on an 414 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 415 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET 416 ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, 417 INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE 418 INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 419 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 421 Copyright Statement 423 Copyright (C) The Internet Society (2004). This document is subject 424 to the rights, licenses and restrictions contained in BCP 78, and 425 except as set forth therein, the authors retain all their rights. 427 Acknowledgment 429 Funding for the RFC Editor function is currently provided by the 430 Internet Society.