2.2.2 Improved Cross-Area Review (icar)

In addition to this official charter maintained by the IETF Secretariat, there is additional information about this working group on the Web at:

       http://www.machshav.com/~icar/ -- Additional ICAR Web Page
NOTE: This charter is a snapshot of the 60th IETF Meeting in San Diego, CA USA. It may now be out-of-date.

Last Modified: 2004-06-17

Joel Halpern <joel@stevecrocker.com>
Mark Allman <mallman@icir.org>
General Area Director(s):
Harald Alvestrand <harald@alvestrand.no>
General Area Advisor:
Harald Alvestrand <harald@alvestrand.no>
Mailing Lists:
General Discussion: icar@ietf.org
To Subscribe: icar-request@ietf.org
Archive: http://www.ietf.org/mail-archive/web/icar/index.html
Description of Working Group:
The WG will work out mechanisms for improved cross-functional review within the IETF, addressing the issues identified by the PROBLEM WG. This includes a better community review, as well as more structured pre-IESG review that may be used to improve scalability of the IESG review function.


o Cross-functional review: document review covering different aspects of Internet technology, including those represented by WGs within different IETF areas.

o Community review: document review performed by individual IETF participants and not caused by their responsibilities within the IETF management structure.

o Structured review: more formal and role-based document review performed by individuals assuming certain responsibilities (such as by WG chairs, directorate and IESG members.)

It is an explicit goal of the WG to come up with mechanisms encouraging early review of the documents while ideas are still in the formation stage. An early review is best for catching architectural problems while they're still relatively easy to solve. In particular, many cross-functional interactions can be spotted and dealt with, thus avoiding many "late surprises". A final review (currently done by the IESG) can catch remaining cross-functional interactions, as well as deal with overall quality issues.

The WG will cooperate with others in starting and evaluating experiments with early community and structured reviews. The evaluation of such experiments may be published as Informational RFCs if the group so desires.

The WG will also coordinate with other WGs involved in the IETF reform process on proposed changes to the IETF Working Group operations and Standards process if those are considered necessary.

Goals and Milestones:
Feb 04  Publish Internet Drafts on improved community and structured reviews
Sep 04  Submit draft on improved community review to the IESG for publication as BCP
Sep 04  Submit draft on improved structured review to the IESG for publication as BCP
Jan 05  Evaluate WG progress and potential; close or recharter
  • - draft-ietf-icar-experiment-early-review-00.txt
  • No Request For Comments

    Current Meeting Report

    Improved Cross-Area Review WG (icar)

    Wednesday, August 4 at 1530-1730
    CHAIRS: Mark Allman <mallman@icir.org>
    Joel Halpern <joel@stevecrocker.com>
    NOTE-TAKER: Spencer Dawkins <spencer@mcsr-labs.org>


    1. intro/agenda bashing - co-chairs

    2. discussion of draft-ietf-icar-experiment-early-review-00.txt - David Partain

    Draft to capture the state of WG discussions - supposed to capture WG mindset

    Open issues:

    - guidelines, if any, for the form of the review?

    From TSV - ask for important/crucial issues at the top of the review, as a summary of the review

    From ITU - major technical, minor technical, editorial categories

    This isn't about binding, it's about demonstrating to ADs that "major technical" items have been considered

    Working groups have to consider comments, can't just ignore them

    What is "must respond"? Working group is free to say "we're right", not free to say "we've never heard this comment before"

    - information about potential reviewers? How many requests, how many reviews ...

    Number of reviews is also a recognition item for reviewers

    Average time to complete a review per page of document

    Need to be collecting raw information as we go

    Need to avoid "top 10" lists of reviewers, etc.

    Don't set the threshold to become a reviewer too high - need to be bringing in new reviewers (see later discussion)

    Reviewer specializations? Mark will show what we have so far

    What kind of review are we asking for? Very specific topics, or more general? Need to accommodate specialist reviewers and generalist reviewers

    - Subjective criteria for admittance to the reviewer pool?

    (This discussion is in the context of the WG having agreed to a set of objective criteria, by which folks are eligible for the pool without additional approval.)

    Start by doing real work before you're admitted? AD can just decide?

    Requests are published publicly, so one could establish a track record

    Sponsorship by an AD? Definition of "good" for an AD is "what I would have figured out anyway"?

    Volunteering for reviewer pool and accepting/rejecting by an AD should be in secret

    If "n" (reviews) is sufficiently large, that flushes out a lot of the idiots

    "Sorry, but your reviews don't seem useful" - as a teaching tool - but need to keep from turning ADs into fulltime mentors - are reviewers expected to be willing to serve as mentors? Could this be this an area directorate responsibility (defined broadly)?

    EDU team should (obviously) be willing to help here

    Could have a separate pool of mentors, set up independently

    Are we doing early or late reviews? This document is focused on early review - the hope is that this helps with late reviews as well

    WG needs some assurance that they are likely to get a good review from anyone in the pool

    Nothing but early review can reduce late surprises - this needs to be part of the success criteria for ICAR

    Some set of reviews plus AD sponsorship?

    ADs can sponsor anyone, but recommend looking at previous reviews? We trust ADs to do a lot more than make choices like this without adult supervision

    (Objective - it's "two or more" RFCs, not "two")

    - How do we measure success/failure of ICAR experiments?

    Some types of failure are easy to detect, but success is harder - look for absence of failures and keep going?

    Objective criteria for failure, subjective criteria for success

    Helped or hindered in RFC production? Experiment will be over long before we get an RFC out through this mechanism

    Ask the chairs, not the document editors, for success, but ask the document editors for feedback as well

    - How are people removed from the pool, or is this process needed at all?

    At least remove yourself!

    Let bad reviewers rot on the vine?

    Need a minimal "periodic survey" for self-removal? People should "time-out" after six months, or something

    Removal on the basis of quality - one person's crank is another person's helpful reviewer

    If you qualify and time out, you still qualify when you ask to be added back

    ADs need a removal-for-cause mechanism - we do remove WG chairs from time to time

    If reviewers don't provide reviews - performance criteria forremaining in the pool?

    This is a voluntary process anyway - why do we need to throw people out of the pool? What's the worst case? Is it as bad as what a WG chair can do? We can throw them out now...

    Are we talking one percent bad reviewers or ten percent bad reviewers? We don't need to create an expulsion mechanism for the experiments

    We pretty much agree we need a timeout

    What about a reviewer with an ax to grind? Is "serve at the pleasure" language good enough? Is "the grapevine" good enough? Bad reviewers are worse for new working groups with new chairs.

    Why is this worse than a bozo appearing at IETF last call? The pool is supposed to have more clue, and the working group is supposed to respond

    What about reviewers that don't review? Just timing out is good enough

    Underburden bad reviewers, ignore them, or overburden them? We already have these mechanisms

    There is a difference between a process that's seldom used and one that doesn't exist

    We don't have consensus here - take this question to the list

    How does AD removal map to pre-qualified participants? To thelist...

    - What can we hope for from the IESG? How do reviewers, prospective reviewers, and working groups interact with IESG?

    IESG has promised Harald they will provide sacrificial working groups for the experiment, and are concerned about providing names of people who don't have time to review documents for ADs now

    "Perhaps your working group should get an ICAR review"

    We're asking "help us find working groups, help us find reviewers, sponsor reviewers" - IESG is mumbling on the middle answer

    - Other comments?

    Non-performance will be a problem - need backup reviewers - yes, and WGs can handle this however they want to - we could learn this in an experiment, instead of noodling about it

    We are depending on volunteers - dates and plans to follow

    3. the ICAR web page and soliciting for volunteers - Mark Allman

    Web page available from ICAR charter page

    Describes the proposal for an experiment

    ADs can request reviews for non-WG documents, too

    Pool members send reviews to icar mailing address

    WG considers comments and suggestions included in the reviews

    Reviewer attributes - expertise, reviews, ideal queue depth, current queue depth, plus arbitrary bio

    Reviews and responses available from reviewer page

    Could be sliced by WG, by draft, etc.

    Logging time when information about requests and reviews are entered into the system

    Feedback? Codify expertise? At least sort by expertise? Grade expertise? What about subjective grade inflation? What about "wireless", "XML", etc.

    4. experience with the general area review team - Harald Alvestrand

    Gen-ART is late review - one week before IESG processing - and general review, including readability

    IETF chair unable to get responsible and timely reviews, and other people had directorates and review teams - so asked for help

    Called an experiment, getting experience with fast reviews, identify what turns out to be problems, and actually publish reviews

    One-area pilot, so Harald didn't have to ask for agreement, and Harald already had a web server and mail server

    Review guidelines stolen from SIR, driven by IESG mechanics

    Reviewers assigned round-robin, reviews and names are public, started January 22

    Reviewers recruited word-of-mouth, "known to the AD", but also asked for volunteers on ICAR, no reviewers have crumbled yet

    One-person, one-document? Don't double up, because we have 20 documents per telechat and less than 40 reviewers

    10 reviewers, 135 reviews, lots of concerns, 5 DISCUSSes (at least)

    How did this compare to SIR experiment? This is review at a different level

    Load on AD: down. Community involvement: Up

    Lessons learned - one week is a short time, reviewers need a discussion forum, ten people isn't enough, it needs to be a team, procedures are necessary, IT support is optional but nice.

    Trying to move standards-track review to IETF Last Call time

    Would a single discussion forum scale to 200 reviewers?

    Unresolved issues: Follow-up is important. Procedures for pointing at information is important. Getting the WG's attention for non-critical issues isn't a solved problem.

    Not sure who to send reviews to (ADs, WG chairs, editors, or some combination).

    Reviews need to be more visible to the community (you can find them now, if you're psychic).

    5. General Discussion

    What sort of expectations for reviewer responsiveness on committing to a review?

    Hard deadlines help us focus...

    Short timeframes are good, when we're asking for a commitment - Dave will suggest a number on the mailing list


    None received.