 |
 |
 |
 |
|
 |
|
 |
|
 |
|
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
These are my notes from our session today (and I apologize in advance for HTML),
Spencr
Improved Cross-Area Review WG
(icar) Wednesday, August 4 at
1530-1730 ================================ CHAIRS: Mark Allman
<mallman at icir.org>
Joel Halpern
<joel at stevecrocker.com> AGENDA: 1. intro/agenda bashing -
co-chairs 2. discussion of
draft-ietf-icar-experiment-early-review-00.txt - David Partain Draft to capture the state of WG discussions ? supposed to capture WG mindset Open issues: - guidelines, if any, for the form of the review? From TSV ? ask for important/crucial issues at the top of the review, as a summary of the review From ITU ? major technical, minor technical, editorial categories This isn?t about binding, it?s about demonstrating to ADs that ?major technical? items have been considered Working groups have to consider comments, can?t just ignore them What is ?must respond?? Working group is free to say ?we?re right?, not free to say ?we?ve never heard this comment before? - information about potential reviewers? How many requests, how many reviews ... Number of reviews is also a recognition item for reviewers Average time to complete a review per page of document Need to be collecting raw information as we go Need to avoid ?top 10? lists of reviewers, etc. Don?t set the threshold to become a reviewer too high ? need to be bringing in new reviewers (see later discussion) Reviewer specializations? Mark will show what we have so far What kind of review are we asking for? Very specific topics, or more general? Need to accommodate specialist reviewers and generalist reviewers A little history is a good thing... - Subjective criteria for admittance to the reviewer pool? Start by doing real work before you?re admitted? AD can just decide? Requests are published publicly, so one could establish a track record Sponsorship by an AD? Definition of ?good? for an AD is ?what I would have figured out anyway?? Volunteering and accepting/rejecting should be in secret If ?n? (reviews) is sufficiently large, that flushes out a lot of the idiots ?Sorry, but your reviews don?t seem useful? ? as a teaching tool ? but need to keep from turning ADs into fulltime mentors ? are reviewers expected to be willing to serve as mentors? Could this be this an area directorate responsibility (defined broadly)? EDU team should (obviously) be willing to help here Could have a separate pool of mentors, set up independently Are we doing early or late reviews? ICAR is focused on early review ? the hope is that this helps with late reviews as well WG needs some assurance that they are likely to get a good review from anyone in the pool Nothing but early review can reduce late surprises ? this needs to be part of the success criteria for ICAR Some set of reviews plus AD sponsorship? ADs can sponsor anyone, but recommend looking at previous reviews? We trust ADs to do a lot more than make choices like this without adult supervision (Objective ? it?s ?two or more? RFCs, not ?two?) - How do we measure success/failure of ICAR experiments? Some types of failure are easy to detect, but success is harder ? look for absence of failures and keep going? Objective criteria for failure, subjective criteria for success Helped or hindered in RFC production? Experiment will be over long before we get an RFC out through this mechanism Ask the chairs, not the document editors, for success, but ask the document editors for feedback as well - How are people removed from the pool, or is this process needed at all? At least remove yourself! Let bad reviewers rot on the vine? Need a minimal ?periodic survey? for self-removal? People should ?time-out? after six months, or something Removal on the basis of quality ? one person?s crank is another person?s helpful reviewer If you qualify and time out, you still qualify when you ask to be added back ADs need a removal-for-cause mechanism ? we do remove WG chairs from time to time If reviewers don?t provide reviews ? performance criteria for remaining in the pool? This is a voluntary process anyway ? why do we need to throw people out of the pool? What?s the worst case? Is it as bad as what a WG chair can do? We can throw them out now... Are we talking one percent bad reviewers or ten percent bad reviewers? We don?t need to create an expulsion mechanism for the experiments We pretty much agree we need a timeout What about a reviewer with an ax to grind? Is ?serve at the pleasure? language good enough? Is ?the grapevine? good enough? Bad reviewers are worse for new working groups with new chairs. Why is this worse than a bozo appearing at IETF last call? The pool is supposed to have more clue, and the working group is supposed to respond What about reviewers that don?t review? Just timing out is good enough Underburden bad reviewers, ignore them, or overburden them? We already have three mechanisms There is a difference between a process that?s seldom used and one that doesn?t exist We don?t have consensus here ? take this question to the list How does AD removal map to pre-qualified participants? To the list... - What can we hope for from the IESG? How do reviewers, prospective reviewers, and working groups interact with IESG? IESG has promised Harald they will provide sacrificial working groups for the experiment, and are concerned about providing names of people who don?t have time to review documents for ADs now ?Perhaps your working group should get an ICAR review? We?re asking ?help us find working groups, help us find reviewers, sponsor reviewers? ? IESG is mumbling on the middle answer - Other comments? Non-performance will be a problem ? need backup reviewers ? yes, and WGs can handle this however they want to ? we could learn this in an experiment, instead of noodling about it We are depending on volunteers ? dates and plans to follow 3. the ICAR web page and soliciting for volunteers - Mark Allman Web page available from ICAR charter page Describes the proposal for an experiment ADs can request reviews for non-WG documents, too Pool members send reviews to icar mailing address WG considers comments and suggestions included in the reviews Reviewer attributes ? expertise, reviews, ideal queue depth, current queue depth, plus arbitrary bio Reviews and responses available from reviewer page Could be sliced by WG, by draft, etc. Logging time when information about requests and reviews are entered into the system Feedback? Codify expertise? At least sort by expertise? Grade expertise? What about subjective grade inflation? What about ?wireless?, ?XML?, etc. 4. experience with the general area review
team - Harald Alvestrand Gen-ART is late review ? one week before IESG processing ? and general review, including readability IETF chair unable to get responsible and timely reviews, and other people had directorates and review teams ? so asked for help Called an experiment, getting experience with fast reviews, identify what turns out to be problems, and actually publish reviews One-area pilot, so Harald didn?t have to ask for agreement, and Harald already had a web server and mail server Review guidelines stolen from SIR, driven by IESG mechanics Reviewers assigned round-robin, reviews and names are public, started January 22 Reviewers recruited word-of-mouth, ?known to the AD?, but also asked for volunteers on ICAR, no reviewers have crumbled yet One-person, one-document? Don?t double up, because we have 20 documents per telechat and less than 40 reviewers 10 reviewers, 135 reviews, lots of concerns, 5 DISCUSSes (at least) How did this compare to SIR experiment? This is review at a different level Load on AD: down. Community involvement: Up Lessons learned ? one week is a short time, reviewers need a discussion forum, ten people isn?t enough, it needs to be a team, procedures are necessary, IT support is optional but nice. Trying to move standards-track review to IETF Last Call time Would a single discussion forum scale to 200 reviewers? Unresolved issues: Follow-up is important. Procedures for pointing at information is important. Getting the WG?s attention for non-critical issues isn?t a solved problem. Not sure who to send reviews to (ADs, WG chairs, editors, or some combination). Reviews need to be more visible to the community (you can find them now, if you?re psychic). 5. General Discussion What sort of expectations for reviewer responsiveness on committing to a review? Hard deadlines help us focus... Short timeframes are good, when we?re asking for a commitment ? Dave will suggest a number on the mailing list |
_______________________________________________ Icar mailing list Icar at ietf.org https://www1.ietf.org/mailman/listinfo/icar