Last Modified: 2004-06-17
o Cross-functional review: document review covering different aspects of Internet technology, including those represented by WGs within different IETF areas.
o Community review: document review performed by individual IETF participants and not caused by their responsibilities within the IETF management structure.
o Structured review: more formal and role-based document review performed by individuals assuming certain responsibilities (such as by WG chairs, directorate and IESG members.)
It is an explicit goal of the WG to come up with mechanisms encouraging early review of the documents while ideas are still in the formation stage. An early review is best for catching architectural problems while they're still relatively easy to solve. In particular, many cross-functional interactions can be spotted and dealt with, thus avoiding many "late surprises". A final review (currently done by the IESG) can catch remaining cross-functional interactions, as well as deal with overall quality issues.
The WG will cooperate with others in starting and evaluating experiments with early community and structured reviews. The evaluation of such experiments may be published as Informational RFCs if the group so desires.
The WG will also coordinate with other WGs involved in the IETF reform process on proposed changes to the IETF Working Group operations and Standards process if those are considered necessary.
|Feb 04||Publish Internet Drafts on improved community and structured reviews|
|Sep 04||Submit draft on improved community review to the IESG for publication as BCP|
|Sep 04||Submit draft on improved structured review to the IESG for publication as BCP|
|Jan 05||Evaluate WG progress and potential; close or recharter|
Improved Cross-Area Review WG (icar)
Wednesday, August 4 at 1530-1730
CHAIRS: Mark Allman <firstname.lastname@example.org>
Joel Halpern <email@example.com>
NOTE-TAKER: Spencer Dawkins <firstname.lastname@example.org>
1. intro/agenda bashing - co-chairs
2. discussion of draft-ietf-icar-experiment-early-review-00.txt - David Partain
Draft to capture the state of WG discussions - supposed to capture WG mindset
- guidelines, if any, for the form of the review?
From TSV - ask for important/crucial issues at the top of the review, as a summary of the review
From ITU - major technical, minor technical, editorial categories
This isn't about binding, it's about demonstrating to ADs that "major technical" items have been considered
Working groups have to consider comments, can't just ignore them
What is "must respond"? Working group is free to say "we're right", not free to say "we've never heard this comment before"
- information about potential reviewers? How many requests, how many reviews ...
Number of reviews is also a recognition item for reviewers
Average time to complete a review per page of document
Need to be collecting raw information as we go
Need to avoid "top 10" lists of reviewers, etc.
Don't set the threshold to become a reviewer too high - need to be bringing in new reviewers (see later discussion)
Reviewer specializations? Mark will show what we have so far
What kind of review are we asking for? Very specific topics, or more general? Need to accommodate specialist reviewers and generalist reviewers
- Subjective criteria for admittance to the reviewer pool?
(This discussion is in the context of the WG having agreed to a set of objective criteria, by which folks are eligible for the pool without additional approval.)
Start by doing real work before you're admitted? AD can just decide?
Requests are published publicly, so one could establish a track record
Sponsorship by an AD? Definition of "good" for an AD is "what I would have figured out anyway"?
Volunteering for reviewer pool and accepting/rejecting by an AD should be in secret
If "n" (reviews) is sufficiently large, that flushes out a lot of the idiots
"Sorry, but your reviews don't seem useful" - as a teaching tool - but need to keep from turning ADs into fulltime mentors - are reviewers expected to be willing to serve as mentors? Could this be this an area directorate responsibility (defined broadly)?
EDU team should (obviously) be willing to help here
Could have a separate pool of mentors, set up independently
Are we doing early or late reviews? This document is focused on early review - the hope is that this helps with late reviews as well
WG needs some assurance that they are likely to get a good review from anyone in the pool
Nothing but early review can reduce late surprises - this needs to be part of the success criteria for ICAR
Some set of reviews plus AD sponsorship?
ADs can sponsor anyone, but recommend looking at previous reviews? We trust ADs to do a lot more than make choices like this without adult supervision
(Objective - it's "two or more" RFCs, not "two")
- How do we measure success/failure of ICAR experiments?
Some types of failure are easy to detect, but success is harder - look for absence of failures and keep going?
Objective criteria for failure, subjective criteria for success
Helped or hindered in RFC production? Experiment will be over long before we get an RFC out through this mechanism
Ask the chairs, not the document editors, for success, but ask the document editors for feedback as well
- How are people removed from the pool, or is this process needed at all?
At least remove yourself!
Let bad reviewers rot on the vine?
Need a minimal "periodic survey" for self-removal? People should "time-out" after six months, or something
Removal on the basis of quality - one person's crank is another person's helpful reviewer
If you qualify and time out, you still qualify when you ask to be added back
ADs need a removal-for-cause mechanism - we do remove WG chairs from time to time
If reviewers don't provide reviews - performance criteria forremaining in the pool?
This is a voluntary process anyway - why do we need to throw people out of the pool? What's the worst case? Is it as bad as what a WG chair can do? We can throw them out now...
Are we talking one percent bad reviewers or ten percent bad reviewers? We don't need to create an expulsion mechanism for the experiments
We pretty much agree we need a timeout
What about a reviewer with an ax to grind? Is "serve at the pleasure" language good enough? Is "the grapevine" good enough? Bad reviewers are worse for new working groups with new chairs.
Why is this worse than a bozo appearing at IETF last call? The pool is supposed to have more clue, and the working group is supposed to respond
What about reviewers that don't review? Just timing out is good enough
Underburden bad reviewers, ignore them, or overburden them? We already have these mechanisms
There is a difference between a process that's seldom used and one that doesn't exist
We don't have consensus here - take this question to the list
How does AD removal map to pre-qualified participants? To thelist...
- What can we hope for from the IESG? How do reviewers, prospective reviewers, and working groups interact with IESG?
IESG has promised Harald they will provide sacrificial working groups for the experiment, and are concerned about providing names of people who don't have time to review documents for ADs now
"Perhaps your working group should get an ICAR review"
We're asking "help us find working groups, help us find reviewers, sponsor reviewers" - IESG is mumbling on the middle answer
- Other comments?
Non-performance will be a problem - need backup reviewers - yes, and WGs can handle this however they want to - we could learn this in an experiment, instead of noodling about it
We are depending on volunteers - dates and plans to follow
3. the ICAR web page and soliciting for volunteers - Mark Allman
Web page available from ICAR charter page
Describes the proposal for an experiment
ADs can request reviews for non-WG documents, too
Pool members send reviews to icar mailing address
WG considers comments and suggestions included in the reviews
Reviewer attributes - expertise, reviews, ideal queue depth, current queue depth, plus arbitrary bio
Reviews and responses available from reviewer page
Could be sliced by WG, by draft, etc.
Logging time when information about requests and reviews are entered into the system
Feedback? Codify expertise? At least sort by expertise? Grade expertise? What about subjective grade inflation? What about "wireless", "XML", etc.
4. experience with the general area review team - Harald Alvestrand
Gen-ART is late review - one week before IESG processing - and general review, including readability
IETF chair unable to get responsible and timely reviews, and other people had directorates and review teams - so asked for help
Called an experiment, getting experience with fast reviews, identify what turns out to be problems, and actually publish reviews
One-area pilot, so Harald didn't have to ask for agreement, and Harald already had a web server and mail server
Review guidelines stolen from SIR, driven by IESG mechanics
Reviewers assigned round-robin, reviews and names are public, started January 22
Reviewers recruited word-of-mouth, "known to the AD", but also asked for volunteers on ICAR, no reviewers have crumbled yet
One-person, one-document? Don't double up, because we have 20 documents per telechat and less than 40 reviewers
10 reviewers, 135 reviews, lots of concerns, 5 DISCUSSes (at least)
How did this compare to SIR experiment? This is review at a different level
Load on AD: down. Community involvement: Up
Lessons learned - one week is a short time, reviewers need a discussion forum, ten people isn't enough, it needs to be a team, procedures are necessary, IT support is optional but nice.
Trying to move standards-track review to IETF Last Call time
Would a single discussion forum scale to 200 reviewers?
Unresolved issues: Follow-up is important. Procedures for pointing at information is important. Getting the WG's attention for non-critical issues isn't a solved problem.
Not sure who to send reviews to (ADs, WG chairs, editors, or some combination).
Reviews need to be more visible to the community (you can find them now, if you're psychic).
5. General Discussion
What sort of expectations for reviewer responsiveness on committing to a review?
Hard deadlines help us focus...
Short timeframes are good, when we're asking for a commitment - Dave will suggest a number on the mailing list