idnits 2.17.1
draft-zinin-early-review-01.txt:
Checking boilerplate required by RFC 5378 and the IETF Trust (see
https://trustee.ietf.org/license-info):
----------------------------------------------------------------------------
** It looks like you're using RFC 3978 boilerplate. You should update this
to the boilerplate described in the IETF Trust License Policy document
(see https://trustee.ietf.org/license-info), which is required now.
-- Found old boilerplate from RFC 3978, Section 5.1 on line 17.
** This document has an original RFC 3978 Section 5.4 Copyright Line,
instead of the newer IETF Trust Copyright according to RFC 4748.
** The document seems to lack an RFC 3978 Section 5.4 Reference to BCP 78
-- however, there's a paragraph with a matching beginning. Boilerplate
error?
** The document seems to lack an RFC 3978 Section 5.5 (updated by RFC 4748)
Disclaimer -- however, there's a paragraph with a matching beginning.
Boilerplate error?
** The document seems to lack an RFC 3979 Section 5, para. 1 IPR Disclosure
Acknowledgement.
** The document seems to lack an RFC 3979 Section 5, para. 2 IPR Disclosure
Acknowledgement.
** The document seems to lack an RFC 3979 Section 5, para. 3 IPR Disclosure
Invitation.
Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt:
----------------------------------------------------------------------------
** The document seems to lack a 1id_guidelines paragraph about
Internet-Drafts being working documents.
** The document seems to lack a 1id_guidelines paragraph about 6 months
document validity.
== No 'Intended status' indicated for this document; assuming Proposed
Standard
== The page length should not exceed 58 lines per page, but there was 11
longer pages, the longest (page 2) being 60 lines
== It seems as if not all pages are separated by form feeds - found 0 form
feeds but 12 pages
Checking nits according to https://www.ietf.org/id-info/checklist :
----------------------------------------------------------------------------
** The document seems to lack an IANA Considerations section. (See Section
2.2 of https://www.ietf.org/id-info/checklist for how to handle the case
when there are no actions for IANA.)
** The document seems to lack separate sections for Informative/Normative
References. All references will be assumed normative when checking for
downward references.
Miscellaneous warnings:
----------------------------------------------------------------------------
== The copyright year in the RFC 3978 Section 5.4 Copyright Line does not
match the current year
-- The document seems to lack a disclaimer for pre-RFC5378 work, but may
have content which was first submitted before 10 November 2008. If you
have contacted all the original authors and they are all willing to grant
the BCP78 rights to the IETF Trust, then this is fine, and you can ignore
this comment. If not, you may need to add the pre-RFC5378 disclaimer.
(See the Legal Provisions document at
https://trustee.ietf.org/license-info for more information.)
-- The document date (May 2005) is 6921 days in the past. Is this
intentional?
Checking references for intended status: Proposed Standard
----------------------------------------------------------------------------
(See RFCs 3967 and 4897 for information about using normative references
to lower-maturity documents in RFCs)
No issues found here.
Summary: 11 errors (**), 0 flaws (~~), 4 warnings (==), 3 comments (--).
Run idnits with the --verbose option for more detailed information about
the items above.
--------------------------------------------------------------------------------
2 Network Working Group Alex Zinin
3 Internet Draft Alcatel
4 Expiration Date: November 2005 May 2005
5 File name: draft-zinin-early-review-01.txt
7 Area Review Teams for
8 Early Cross-functional Reviews
10 draft-zinin-early-review-00.txt
12 Status of this Memo
14 By submitting this Internet-Draft, each author represents that any
15 applicable patent or other IPR claims of which he or she is aware
16 have been or will be disclosed, and any of which he or she becomes
17 aware will be disclosed, in accordance with Section 6 of BCP 79.
19 Internet Drafts are working documents of the Internet Engineering
20 Task Force (IETF), its Areas, and its Working Groups. Note that other
21 groups may also distribute working documents as Internet Drafts.
23 Internet Drafts are draft documents valid for a maximum of six
24 months. Internet Drafts may be updated, replaced, or obsoleted by
25 other documents at any time. It is not appropriate to use Internet
26 Drafts as reference material or to cite them other than as a "working
27 draft" or "work in progress".
29 The list of current Internet-Drafts can be accessed at
30 http://www.ietf.org/ietf/1id-abstracts.txt
32 The list of Internet-Draft Shadow Directories can be accessed at
33 http://www.ietf.org/shadow.html.
35 Abstract
37 This document contains a proposal for cross-functional IETF review
38 process that can be initiated at early stages of a document life
39 cycle. The approach is based on existing experience with area
40 directorates and other expert groups within the IETF.
42 Please send any comments on this document to the IESG (iesg@ietf.org)
44 1. Introduction
45 Cross-functional (intra-area and cross-area) review are the
46 properties of the IETF that are supposed to ensure high quality of
47 the produced technologies, their scalability and safety for the
48 Internet as a whole. It has been widely acknowledged that these
49 properties are among the core values of the IETF and have to be
50 preserved in order for the IETF to continue being a successful
51 engineering and standards organization.
53 Currently, the AD review and IESG review processes are the only
54 formal ways within the IETF to ensure cross-functional, and in
55 particular cross-area reviews. Since these reviews happen only when
56 the documents are submitted for final approval, issues are brought up
57 late in the process, often when contributors have invested a lot of
58 cycles into the document, the technology has possibly been
59 implemented, and changes in direction or certain technical details
60 are painful and frustrating. If interim cross-functional or cross-
61 area review is needed, it is currently done informally, and this
62 process is not necessarily coordinated with the following review by
63 the IESG members.
65 Creating a more formal and better described mechanism for cross-area
66 review that would be coordinated with the IESG review process, and
67 could be involved at any point within a life cycle of a document
68 should substantially improve the quality of the documents submitted
69 to the IESG and minimize the number of substantial issues identified
70 late in the process. A straight-forward way to do this could be to
71 request interim IESG review before the document is completed and
72 submitted for final approval. However, if applied to a considerable
73 number of documents (and we do want more documents to benefit from
74 cross-area review), this would put additional load on the IESG and
75 could impact document processing time.
77 This document proposes a cross-area review process that should have
78 better scaling characteristics. The method is based on delegation of
79 the document review function from ADs to the area review teams,
80 currently known as "area directorates" in some areas, or "doctor
81 groups" in others. Note that the described process does not obviate
82 the need for the cross-functional peer review performed by regular
83 IETF participants (members of the review teams or not). On the other
84 hand, the peer review process can be improved by directly (and
85 informally) soliciting comments from the review teams.
87 The proposed method is based on the experience several IETF Area
88 Directors have with involving groups of experts in the process of
89 early document review and during the IESG review cycle.
91 More specifically:
93 o the Routing Area Directors have been using the Routing Area
94 Directorate group (rtg-dir) for review of the documents coming
95 out of the WGs within the Routing Area before they are submit-
96 ted for final approval to the IESG. The directorate is also
97 asked to review certain documents appearing on the IESG agenda
98 from time to time.
100 o Operations directorate (ops-dir) has also been used for early
101 document review and during the IESG review period.
103 o The MIB-Doctors group is consistently involved in preparation
104 of MIB documents before they are brought to the IESG
106 Should add info on the Transport-Doctors group here
107
109 At this point the process of document review by these expert groups
110 is not described and has no formal standing with the IETF. Not all
111 areas have similar expert groups and/or they are not publicly known
112 to the wide community. There is also no described way for WG chairs
113 to request review by experts groups in other areas and expect a guar-
114 anteed follow-up.
116 This proposal formalizes the notion of such expert review teams,
117 describes how they are used for cross-functional review, the relation
118 of this process to the final IESG review process, and how the cross-
119 functional review can be requested and should be followed up on.
121 2. Proposal
123 2.1 Overview
125 Briefly, the cross-functional review process may be described as fol-
126 lows.
128 Each area has an area review team (ART) which ADs delegate the
129 interim document review function to. When necessary (early in the
130 process, or during the WG Last call, or both), the WG chairs request
131 the review for a document by sending an e-mail to all required ARTs
132 (at a minimum the ART of the area the WG belongs to). The IETF-wide
133 Last Call announcement is cc'ed to all ARTs. Provided feedback is
134 taken to the WG for discussion. Consistency with the later IESG
135 review process is ensured through training of the reviewers by the
136 ADs and reviewers communicating the recommendations wrt the documents
137 to the ADs.
139 The following sections provide more details regarding the proposed
140 approach.
142 2.2 Assumptions
144 The described proposal is predicated on certain assumptions that are
145 worth spelling out:
147 1. The IESG remains the body responsible for final approval of
148 the IETF documents.
150 The implication of this assumption is that the documents will
151 finally have to go through the IESG review process with indi-
152 vidual IESG members checking the document. This implies that
153 interim reviews performed before that stage should be coordi-
154 nated with later IESG reviews. Otherwise it is possible that
155 the issues discussed previously will be brought up again. The
156 coordination is also need to usefully off-load (part of) the
157 document review function from individual ADs.
159 2. The ADs remain to be trusted by the community (through the
160 NomCom process) to function as final document reviewers and
161 are responsible for the document quality.
163 The implication here is that since the ADs are held responsi-
164 ble for the results of the document review, they need to feel
165 comfortable delegating this review function to the ART mem-
166 bers, which will have bearings on the method of ART member
167 selection.
169 The reasoning behind using these assumptions is to use the existing
170 mechanisms and tools (known running code) as much as possible and
171 avoid extreme changes to the document approval process that may
172 result in a DoS on it.
174 2.3 Area Review Teams
176 Each area creates an Area Review Team (ART) that is addressable
177 through a well-known mailing list address (e.g. "-review@ietf.org"). At a minimum, the group includes the ADs,
179 however it is expected that it will include technical experts, will-
180 ing to contribute their time to reviewing the IETF documents from the
181 same and other areas and providing consultation to the area direc-
182 tors. The ADs will delegate the review function for some (or all)
183 documents to ART members.
185 Selection of the ART members is done personally by the ADs. Possible
186 variations, however, may include open call for nominations, followed
187 up by ADs interviewing the candidates and personally approving them.
189
190 See assumption 2 in section 2.2 for the reasons
192
194 When a document needs to be reviewed by ART, the AD assigns two ART
195 members as "token holders". All ART members are encouraged to review
196 the document, however, the token holders are held responsible for
197 providing comments within a 2-week time frame and following up on
198 them with the document authors and/or the hosting WG. The token hold-
199 ers will also provide the ADs with their recommendation including the
200 summary of the discussion, the list of issues and how they have been
201 addressed.
203 2.4 Cross-Area Review Process
205 This section describes the actual cross-functional review process
206 that can be initiated at any point within the life cycle of a docu-
207 ment. This process is automatically initiated whenever an IETF- wide
208 Last Call is started for a document.
210 1. The set of area review teams engaged is determined by the ini-
211 tiator of the review process in consultation with the respon-
212 sible AD. At a minimum, this set includes the ART of the host-
213 ing area. For the IETF Last Call, this set includes all ARTs.
215
217 the stimulus behind choosing the right set of review teams is
218 to get the review comments that are likely to be brought up
219 during the final IESG review earlier in the process
221
223 The set of review teams may also include the IAB review team
224 (assuming this is a subset of IAB members, or the IAB itself
225 otherwise).
227 2. An e-mail message with the review request is sent to the mail-
228 ing lists of all ARTs that need to be engaged.
230 3. ADs for each engaged ART have the choice of either "signing
231 off" on the referred document if they believe that the docu-
232 ment does not need a detailed review from the perspective of
233 that area, or assigning the token holders (2 persons) that
234 will conduct the document review within a 2-week time frame.
235 Other members of the review teams are strongly encouraged to
236 provide feedback, but will not be held responsible for this by
237 the ADs.
239 4. Document reviewers bring their concerns to the attention of
240 the document authors and/or the WG via e-mail communication on
241 the WG mailing list and (if necessary) the mailing list of the
242 ART they are members of.
244 5. Based on the discussion with the authors/within the WG, the
245 reviewers provide their responsible AD with a recommendation
246 regarding the document. The recommendation includes the sum-
247 mary of the review, as well as the list of issues and their
248 resolution.
250 6. The authors/WG treat the feedback as part of the WG or IETF-
251 wide review process
253
255 An important point here is that the discussion initiated by
256 this review process is integrated within the normal WG
257 process, rather than treated as a pronouncement from a higher
258 authority.
260
262 7. If a document returns to an ART (e.g., the document is under
263 the IETF Last Call and was reviewed during the WG Last Call),
264 the same token holders will "own" the document whenever possi-
265 ble. The token holders check that the new revision of the doc-
266 ument reflects the previous discussion correctly. If no addi-
267 tional concerns arise, the recommendation to the ADs of the
268 ART remain the same.
270 8. The ADs have the power to bring up during the IESG review
271 process the reviewers' comments that were not addressed by the
272 authors/WG if they believe this is appropriate. The ADs also
273 have the power to override the comments from their correspond-
274 ing review teams.
276
278 The above gives the ADs the ability to insist on fixing cer-
279 tain comments that they believe represent serious issues if
280 they were discarded while processing the cross-area review
281 feedback during the WG process as described above. This also
282 gives them the right to withdraw certain points from the con-
283 sideration or change them if they believe this is appropriate
284 for the progress of a document. This is consistent with the
285 concept that the ADs are responsible for the final document
286 approval and that the review teams provide their expert
287 recommendations based on the discussions within the WG.
289
291 2.5 Role of Cross-Area Review within the Standards Process
293 The cross-function review process is integrated within the IETF Stan-
294 dards Process as described below:
296 1. WG process:
298 When initiated early within the life cycle of a document, the
299 feedback from the cross-area review process is considered part
300 of the WG discussion and consensus forming process.
302 2. WG Last Calls:
304 It is expected that the cross-area review will also be
305 requested during the WG LC period. Procedurally, this will
306 have the same value as during the WG process, but would ensure
307 that final cross-area checks are performed before the docu-
308 ments comes to the IESG.
310 3. AD-review process
312 It is expected that the ADs will use the review teams to dele-
313 gate the review function and thus off-load a considerable part
314 of this function when and as deemed appropriate. The ADs are
315 expected to coordinate with the ART members to ensure that
316 consistent review criteria is applied to documents so that
317 most issues that would otherwise be brought up during the AD-
318 review process are resolved earlier in the life cycle of the
319 document.
321
323 Note that ADs are given a tool they can use to off-load docu-
324 ment review to the extent they believe is necessary, but they
325 are not required to do so. It is then left to the ADs to make
326 sure they are using this tool appropriately and sufficiently.
328
330 4. IETF Last Call
332 As all ARTs are informed about IETF Last Calls, it is expected
333 that by the time the documents is on the IESG agenda, it will
334 have received adequate cross-functional review and all ADs
335 will have some recommendation on the document from their ARTs.
337 5. IESG review process
339 It is expected that individual ADs will organize the review
340 process in the review teams in such a way that a positive rec-
341 ommendation from the review team should be sufficient for the
342 ADs to feel comfortable that most of the possible technical
343 issues have been identified, followed up on, and resolved, and
344 that the ADs only need to look for very high-level, architec-
345 tural issues. This, in turn, should a) decrease the amount of
346 time the ADs need to spend on the document review and follow
347 up, and b) minimize the number of "late surprises" arising
348 during the IESG review process.
350 2.6 Initiation of Review Process
352 The cross-functional review process can be initiated either by an AD
353 or by a WG chair after consultation with the ADs.
355
357 Consultation with the AD is a sanity check to make sure the set of
358 engage ARTs is chosen right
360
362 The review process may be initiated at an early stage of a document
363 (e.g. when the WG is starting to consider an approach) to ensure
364 architectural validity and correctness of the general direction
366 The review process should be initiated as part of the WG LC to mini-
367 mize late surprises during the IESG review process
369 The review process must be initiated for all documents going through
370 the IETF Last Call.
372 The same process may be used by the ADs and the RFC-Editor to request
373 cross-functional review for individual submissions they are shepherd-
374 ing or checking for conflicts.
376 A simplified version of this process (no token holders, no issue
377 tracking within ARTs, etc.) can be used to inform ART members about
378 technical discussions and solicit their comments.
380 2.7 Documenting Results of Review
381 It is important that the feedback from ARTs and the results of the
382 discussion with the authors/WG are documented for later reference
383 during the IESG review process.
385 For WG documents this is ensures by the WG chairs who keep track of
386 the issues (as part of the improved WG process) and summarize them
387 when submitting the document to their ADs for IESG processing.
389 For individual submissions this function is either performed by the
390 review initiator (an AD) or delegated to a member of the review team
391 with a consequent report to the ADs
393 For ARTs in the other (non-hosting) area, token holders within the
394 review team assigned to the documents are responsible for tracking
395 the issues on their side and summarizing them in their recommendation
396 to the ADs
398 2.8 Trust, Responsibility, and Accountability
400 An important aspect of the proposal documented here is that the mod-
401 els of trust, responsibility, and accountability currently used and
402 practiced within the IETF are not changed.
404 More specifically, the ADs, selected by the NomCom process as indi-
405 viduals "trusted" by the community to perform the ultimate technical
406 review and document approval functions, are still held responsible
407 and accountable for these functions. In other words, the tool of del-
408 egating the document review function to the review teams does not
409 remove the responsibility of the ADs for the results of such review.
410 The ADs are expected to personally ensure that the individuals
411 selected by them as members of the review teams have appropriate
412 qualification (through required training if needed) to perform this
413 function. It is also the AD's responsibility to adjust the list of
414 members (hire more members or fire ill-performing ones) to maintain
415 adequacy of the review process and require level of off-loading.
417 2.10 Motivation and Credit
419 The following methods are proposed to make the role of an ART member
420 attractive and keep ART members motivated and acknowledged:
422 1. Formalization of the review team role.
424 Formal introduction of ARTs within the IETF standard process
425 should result in recognition of this role individual members
426 and their employers.
428 2. Open area meetings (plenary)
430 It is possible to give ART members public exposure by holding
431 ART plenary during the open area meetings
433 3. Acknowledging reviewers
435 Reviewers engaged in ARTs are acknowledged in the published
436 RFCs.
438 4. Dots on the badges
440 ART members are identified at the face-to-face meetings with
441 dots on their badges
443 3. Problems being addressed
445 The proposal described here addresses the following problems experi-
446 enced within the IESG and IETF in general:
448 1. Low document quality
450 It is expected that additional review of the documents should
451 substantially improve the quality of the IETF documents.
453 2. Individual AD load
455 It is expected that improved quality of the documents submit-
456 ted to the ADs for AD-review, should decrease the amount of
457 time spent on document review and follow-up on the issues.
458 Delegation of the document review function should also result
459 in considerable off-loading.
461 3. Overall IESG load
463 Is is expected that improved quality of the documents submit-
464 ted to the IESG should decrease the document review load on
465 the IESG. Reports from the ARTs on previously reviewed docu-
466 ments should also make it easier for individual IESG members
467 to assess the quality of incoming documents.
469 4. Late surprises
471 Earlier cross-functional review coordinated with later IESG
472 review should minimize the number of unexpected issues identi-
473 fied at the later stages of a document life cycle.
475 5. Lack of cross-area review
477 This proposal directly encourages cross-area review
479 6. Lack of cross-functional expertise
481 This proposal should encourage learning of technologies in
482 different areas of IETF and should help growing cross-area
483 expertise.
485 7. Growing future management
487 Involvement of more IETF participants in the Standards Process
488 should help increase the number of individuals capable of per-
489 forming the tasks of WG chairs and IESG members.
491 4. Security Considerations
493 This type of non-protocol document does not directly affect the secu-
494 rity of the Internet. However, the cross-functional review process
495 described here should improve the security aspects of specific
496 approaches being reviewed.
498 5. References
500 6. Acknowledgements
502 The author would like to thank Harald Alvestrand and Ted Hardie for
503 their comments on the document.
505 7. Author's address
507 Alex Zinin
508 Alcatel
509 E-mail: zinin@psg.com
511 Copyright (C) The Internet Society (2005). This document is subject
512 to the rights, licenses and restrictions contained in BCP 78, and
513 except as set forth therein, the authors retain all their rights."
515 This document and the information contained herein are provided on
516 an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRE-
517 SENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE
518 INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR
519 IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
520 THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
521 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.