idnits 2.17.1 draft-zinin-icar-arts-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** The document seems to lack a 1id_guidelines paragraph about Internet-Drafts being working documents. ** The document seems to lack a 1id_guidelines paragraph about 6 months document validity. == No 'Intended status' indicated for this document; assuming Proposed Standard == The page length should not exceed 58 lines per page, but there was 10 longer pages, the longest (page 2) being 60 lines == It seems as if not all pages are separated by form feeds - found 0 form feeds but 11 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. Miscellaneous warnings: ---------------------------------------------------------------------------- -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (March 2004) is 7339 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) No issues found here. Summary: 5 errors (**), 0 flaws (~~), 3 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group Alex Zinin 3 Internet Draft Alcatel 4 Expiration Date: September 2004 March 2004 5 File name: draft-zinin-icar-arts-00.txt 7 Area Review Teams for 8 Early Cross-functional Reviews 10 draft-zinin-icar-arts-00.txt 12 Status of this Memo 14 This document is an Internet-Draft and is in full conformance with 15 all provisions of Section 10 of RFC2026. 17 Internet Drafts are working documents of the Internet Engineering 18 Task Force (IETF), its Areas, and its Working Groups. Note that other 19 groups may also distribute working documents as Internet Drafts. 21 Internet Drafts are draft documents valid for a maximum of six 22 months. Internet Drafts may be updated, replaced, or obsoleted by 23 other documents at any time. It is not appropriate to use Internet 24 Drafts as reference material or to cite them other than as a "working 25 draft" or "work in progress". 27 The list of current Internet-Drafts can be accessed at 28 http://www.ietf.org/ietf/1id-abstracts.txt 30 The list of Internet-Draft Shadow Directories can be accessed at 31 http://www.ietf.org/shadow.html. 33 Abstract 35 This document contains a proposal for cross-functional IETF review 36 process that can be initiated at early stages of a document life 37 cycle. The approach is based on existing experience with area 38 directorates and other expert groups within the IETF. 40 Please send any comments on this document to the IESG (iesg@ietf.org) 42 1. Introduction 44 Cross-functional (intra-area and cross-area) review are the 45 properties of the IETF that are supposed to ensure high quality of 46 the produced technologies, their scalability and safety for the 47 Internet as a whole. It has been widely acknowledged that these 48 properties are among the core values of the IETF and have to be 49 preserved in order for the IETF to continue being a successful 50 engineering and standards organization. 52 Currently, the AD review and IESG review processes are the only 53 formal ways within the IETF to ensure cross-functional, and in 54 particular cross-area reviews. Since these reviews happen only when 55 the documents are submitted for final approval, issues are brought up 56 late in the process, often when contributors have invested a lot of 57 cycles into the document, the technology has possibly been 58 implemented, and changes in direction or certain technical details 59 are painful and frustrating. If interim cross-functional or cross- 60 area review is needed, it is currently done informally, and this 61 process is not necessarily coordinated with the following review by 62 the IESG members. 64 Creating a more formal and better described mechanism for cross-area 65 review that would be coordinated with the IESG review process, and 66 could be involved at any point within a life cycle of a document 67 should substantially improve the quality of the documents submitted 68 to the IESG and minimize the number of substantial issues identified 69 late in the process. A straight-forward way to do this could be to 70 request interim IESG review before the document is completed and 71 submitted for final approval. However, if applied to a considerable 72 number of documents (and we do want more documents to benefit from 73 cross-area review), this would put additional load on the IESG and 74 could impact document processing time. 76 This document proposes a cross-area review process that should have 77 better scaling characteristics. The method is based on delegation of 78 the document review function from ADs to the area review teams, 79 currently known as "area directorates" in some areas, or "doctor 80 groups" in others. Note that the described process does not obviate 81 the need for the cross-functional peer review performed by regular 82 IETF participants (members of the review teams or not). On the other 83 hand, the peer review process can be improved by directly (and 84 informally) soliciting comments from the review teams. 86 The proposed method is based on the experience several IETF Area 87 Directors have with involving groups of experts in the process of 88 early document review and during the IESG review cycle. 90 More specifically: 92 o the Routing Area Directors have been using the Routing Area 93 Directorate group (rtg-dir) for review of the documents coming 94 out of the WGs within the Routing Area before they are submit- 95 ted for final approval to the IESG. The directorate is also 96 asked to review certain documents appearing on the IESG agenda 97 from time to time. 99 o Operations directorate (ops-dir) has also been used for early 100 document review and during the IESG review period. 102 o The MIB-Doctors group is consistently involved in preparation 103 of MIB documents before they are brought to the IESG 105 Should add info on the Transport-Doctors group here 106 108 At this point the process of document review by these expert groups 109 is not described and has no formal standing with the IETF. Not all 110 areas have similar expert groups and/or they are not publicly known 111 to the wide community. There is also no described way for WG chairs 112 to request review by experts groups in other areas and expect a guar- 113 anteed follow-up. 115 This proposal formalizes the notion of such expert review teams, 116 describes how they are used for cross-functional review, the relation 117 of this process to the final IESG review process, and how the cross- 118 functional review can be requested and should be followed up on. 120 2. Proposal 122 2.1 Overview 124 Briefly, the cross-functional review process may be described as fol- 125 lows. 127 Each area has an area review team (ART) which ADs delegate the 128 interim document review function to. When necessary (early in the 129 process, or during the WG Last call, or both), the WG chairs request 130 the review for a document by sending an e-mail to all required ARTs 131 (at a minimum the ART of the area the WG belongs to). The IETF-wide 132 Last Call announcement is cc'ed to all ARTs. Provided feedback is 133 taken to the WG for discussion. Consistency with the later IESG 134 review process is ensured through training of the reviewers by the 135 ADs and reviewers communicating the recommendations wrt the documents 136 to the ADs. 138 The following sections provide more details regarding the proposed 139 approach. 141 2.2 Assumptions 142 The described proposal is predicated on certain assumptions that are 143 worth spelling out: 145 1. The IESG remains the body responsible for final approval of 146 the IETF documents. 148 The implication of this assumption is that the documents will 149 finally have to go through the IESG review process with indi- 150 vidual IESG members checking the document. This implies that 151 interim reviews performed before that stage should be coordi- 152 nated with later IESG reviews. Otherwise it is possible that 153 the issues discussed previously will be brought up again. The 154 coordination is also need to usefully off-load (part of) the 155 document review function from individual ADs. 157 2. The ADs remain to be trusted by the community (through the 158 NomCom process) to function as final document reviewers and 159 are responsible for the document quality. 161 The implication here is that since the ADs are held responsi- 162 ble for the results of the document review, they need to feel 163 comfortable delegating this review function to the ART mem- 164 bers, which will have bearings on the method of ART member 165 selection. 167 The reasoning behind using these assumptions is to use the existing 168 mechanisms and tools (known running code) as much as possible and 169 avoid extreme changes to the document approval process that may 170 result in a DoS on it. 172 2.3 Area Review Teams 174 Each area creates an Area Review Team (ART) that is addressable 175 through a well-known mailing list address (e.g. "-review@ietf.org"). At a minimum, the group includes the ADs, 177 however it is expected that it will include technical experts, will- 178 ing to contribute their time to reviewing the IETF documents from the 179 same and other areas and providing consultation to the area direc- 180 tors. The ADs will delegate the review function for some (or all) 181 documents to ART members. 183 Selection of the ART members is done personally by the ADs. Possible 184 variations, however, may include open call for nominations, followed 185 up by ADs interviewing the candidates and personally approving them. 187 189 See assumption 2 in section 2.2 for the reasons 190 192 When a document needs to be reviewed by ART, the AD assigns two ART 193 members as "token holders". All ART members are encouraged to review 194 the document, however, the token holders are held responsible for 195 providing comments within a 2-week time frame and following up on 196 them with the document authors and/or the hosting WG. The token hold- 197 ers will also provide the ADs with their recommendation including the 198 summary of the discussion, the list of issues and how they have been 199 addressed. 201 2.4 Cross-Area Review Process 203 This section describes the actual cross-functional review process 204 that can be initiated at any point within the life cycle of a docu- 205 ment. This process is automatically initiated whenever an IETF- wide 206 Last Call is started for a document. 208 1. The set of area review teams engaged is determined by the ini- 209 tiator of the review process in consultation with the respon- 210 sible AD. At a minimum, this set includes the ART of the host- 211 ing area. For the IETF Last Call, this set includes all ARTs. 213 215 the stimulus behind choosing the right set of review teams is 216 to get the review comments that are likely to be brought up 217 during the final IESG review earlier in the process 219 221 The set of review teams may also include the IAB review team 222 (assuming this is a subset of IAB members, or the IAB itself 223 otherwise). 225 2. An e-mail message with the review request is sent to the mail- 226 ing lists of all ARTs that need to be engaged. 228 3. ADs for each engaged ART have the choice of either "signing 229 off" on the referred document if they believe that the docu- 230 ment does not need a detailed review from the perspective of 231 that area, or assigning the token holders (2 persons) that 232 will conduct the document review within a 2-week time frame. 233 Other members of the review teams are strongly encouraged to 234 provide feedback, but will not be held responsible for this by 235 the ADs. 237 4. Document reviewers bring their concerns to the attention of 238 the document authors and/or the WG via e-mail communication on 239 the WG mailing list and (if necessary) the mailing list of the 240 ART they are members of. 242 5. Based on the discussion with the authors/within the WG, the 243 reviewers provide their responsible AD with a recommendation 244 regarding the document. The recommendation includes the sum- 245 mary of the review, as well as the list of issues and their 246 resolution. 248 6. The authors/WG treat the feedback as part of the WG or IETF- 249 wide review process 251 253 An important point here is that the discussion initiated by 254 this review process is integrated within the normal WG 255 process, rather than treated as a pronouncement from a higher 256 authority. 258 260 7. If a document returns to an ART (e.g., the document is under 261 the IETF Last Call and was reviewed during the WG Last Call), 262 the same token holders will "own" the document whenever possi- 263 ble. The token holders check that the new revision of the doc- 264 ument reflects the previous discussion correctly. If no addi- 265 tional concerns arise, the recommendation to the ADs of the 266 ART remain the same. 268 8. The ADs have the power to bring up during the IESG review 269 process the reviewers' comments that were not addressed by the 270 authors/WG if they believe this is appropriate. The ADs also 271 have the power to override the comments from their correspond- 272 ing review teams. 274 276 The above gives the ADs the ability to insist on fixing cer- 277 tain comments that they believe represent serious issues if 278 they were discarded while processing the cross-area review 279 feedback during the WG process as described above. This also 280 gives them the right to withdraw certain points from the con- 281 sideration or change them if they believe this is appropriate 282 for the progress of a document. This is consistent with the 283 concept that the ADs are responsible for the final document 284 approval and that the review teams provide their expert 285 recommendations based on the discussions within the WG. 287 289 2.5 Role of Cross-Area Review within the Standards Process 291 The cross-function review process is integrated within the IETF Stan- 292 dards Process as described below: 294 1. WG process: 296 When initiated early within the life cycle of a document, the 297 feedback from the cross-area review process is considered part 298 of the WG discussion and consensus forming process. 300 2. WG Last Calls: 302 It is expected that the cross-area review will also be 303 requested during the WG LC period. Procedurally, this will 304 have the same value as during the WG process, but would ensure 305 that final cross-area checks are performed before the docu- 306 ments comes to the IESG. 308 3. AD-review process 310 It is expected that the ADs will use the review teams to dele- 311 gate the review function and thus off-load a considerable part 312 of this function when and as deemed appropriate. The ADs are 313 expected to coordinate with the ART members to ensure that 314 consistent review criteria is applied to documents so that 315 most issues that would otherwise be brought up during the AD- 316 review process are resolved earlier in the life cycle of the 317 document. 319 321 Note that ADs are given a tool they can use to off-load docu- 322 ment review to the extent they believe is necessary, but they 323 are not required to do so. It is then left to the ADs to make 324 sure they are using this tool appropriately and sufficiently. 326 328 4. IETF Last Call 330 As all ARTs are informed about IETF Last Calls, it is expected 331 that by the time the documents is on the IESG agenda, it will 332 have received adequate cross-functional review and all ADs 333 will have some recommendation on the document from their ARTs. 335 5. IESG review process 337 It is expected that individual ADs will organize the review 338 process in the review teams in such a way that a positive rec- 339 ommendation from the review team should be sufficient for the 340 ADs to feel comfortable that most of the possible technical 341 issues have been identified, followed up on, and resolved, and 342 that the ADs only need to look for very high-level, architec- 343 tural issues. This, in turn, should a) decrease the amount of 344 time the ADs need to spend on the document review and follow 345 up, and b) minimize the number of "late surprises" arising 346 during the IESG review process. 348 2.6 Initiation of Review Process 350 The cross-functional review process can be initiated either by an AD 351 or by a WG chair after consultation with the ADs. 353 355 Consultation with the AD is a sanity check to make sure the set of 356 engage ARTs is chosen right 358 360 The review process may be initiated at an early stage of a document 361 (e.g. when the WG is starting to consider an approach) to ensure 362 architectural validity and correctness of the general direction 364 The review process should be initiated as part of the WG LC to mini- 365 mize late surprises during the IESG review process 367 The review process must be initiated for all documents going through 368 the IETF Last Call. 370 The same process may be used by the ADs and the RFC-Editor to request 371 cross-functional review for individual submissions they are shepherd- 372 ing or checking for conflicts. 374 A simplified version of this process (no token holders, no issue 375 tracking within ARTs, etc.) can be used to inform ART members about 376 technical discussions and solicit their comments. 378 2.7 Documenting Results of Review 379 It is important that the feedback from ARTs and the results of the 380 discussion with the authors/WG are documented for later reference 381 during the IESG review process. 383 For WG documents this is ensures by the WG chairs who keep track of 384 the issues (as part of the improved WG process) and summarize them 385 when submitting the document to their ADs for IESG processing. 387 For individual submissions this function is either performed by the 388 review initiator (an AD) or delegated to a member of the review team 389 with a consequent report to the ADs 391 For ARTs in the other (non-hosting) area, token holders within the 392 review team assigned to the documents are responsible for tracking 393 the issues on their side and summarizing them in their recommendation 394 to the ADs 396 2.8 Trust, Responsibility, and Accountability 398 An important aspect of the proposal documented here is that the mod- 399 els of trust, responsibility, and accountability currently used and 400 practiced within the IETF are not changed. 402 More specifically, the ADs, selected by the NomCom process as indi- 403 viduals "trusted" by the community to perform the ultimate technical 404 review and document approval functions, are still held responsible 405 and accountable for these functions. In other words, the tool of del- 406 egating the document review function to the review teams does not 407 remove the responsibility of the ADs for the results of such review. 408 The ADs are expected to personally ensure that the individuals 409 selected by them as members of the review teams have appropriate 410 qualification (through required training if needed) to perform this 411 function. It is also the AD's responsibility to adjust the list of 412 members (hire more members or fire ill-performing ones) to maintain 413 adequacy of the review process and require level of off-loading. 415 2.10 Motivation and Credit 417 The following methods are proposed to make the role of an ART member 418 attractive and keep ART members motivated and acknowledged: 420 1. Formalization of the review team role. 422 Formal introduction of ARTs within the IETF standard process 423 should result in recognition of this role individual members 424 and their employers. 426 2. Open area meetings (plenary) 428 It is possible to give ART members public exposure by holding 429 ART plenary during the open area meetings 431 3. Acknowledging reviewers 433 Reviewers engaged in ARTs are acknowledged in the published 434 RFCs. 436 4. Dots on the badges 438 ART members are identified at the face-to-face meetings with 439 dots on their badges 441 3. Problems being addressed 443 The proposal described here addresses the following problems experi- 444 enced within the IESG and IETF in general: 446 1. Low document quality 448 It is expected that additional review of the documents should 449 substantially improve the quality of the IETF documents. 451 2. Individual AD load 453 It is expected that improved quality of the documents submit- 454 ted to the ADs for AD-review, should decrease the amount of 455 time spent on document review and follow-up on the issues. 456 Delegation of the document review function should also result 457 in considerable off-loading. 459 3. Overall IESG load 461 Is is expected that improved quality of the documents submit- 462 ted to the IESG should decrease the document review load on 463 the IESG. Reports from the ARTs on previously reviewed docu- 464 ments should also make it easier for individual IESG members 465 to assess the quality of incoming documents. 467 4. Late surprises 469 Earlier cross-functional review coordinated with later IESG 470 review should minimize the number of unexpected issues identi- 471 fied at the later stages of a document life cycle. 473 5. Lack of cross-area review 475 This proposal directly encourages cross-area review 477 6. Lack of cross-functional expertise 479 This proposal should encourage learning of technologies in 480 different areas of IETF and should help growing cross-area 481 expertise. 483 7. Growing future management 485 Involvement of more IETF participants in the Standards Process 486 should help increase the number of individuals capable of per- 487 forming the tasks of WG chairs and IESG members. 489 4. Security Considerations 491 This type of non-protocol document does not directly affect the secu- 492 rity of the Internet. However, the cross-functional review process 493 described here should improve the security aspects of specific 494 approaches being reviewed. 496 5. References 498 6. Acknowledgements 500 The author would like to thank Harald Alvestrand and Ted Hardie for 501 their comments on the document. 503 7. Author's address 505 Alex Zinin 506 Alcatel 507 E-mail: zinin@psg.com