[Icar] draft minutes from San Diego

[Mark Allman <mallman at icir.org>]

The following are draft minutes from the ICAR meeting in San Diego.
Thanks to Spencer for taking the minutes.  Please send corrections.

allman






Improved Cross-Area Review WG (icar)
 
Wednesday, August 4 at 1530-1730
================================
CHAIRS:     Mark Allman <mallman at icir.org>
            Joel Halpern <joel at stevecrocker.com>
NOTE-TAKER: Spencer Dawkins <spencer at mcsr-labs.org>


AGENDA:

1.  intro/agenda bashing - co-chairs

2.  discussion of draft-ietf-icar-experiment-early-review-00.txt -
    David Partain

    Draft to capture the state of WG discussions - supposed to
    capture WG mindset

    Open issues:

    - guidelines, if any, for the form of the review?

      From TSV - ask for important/crucial issues at the top of the
      review, as a summary of the review 

      From ITU - major technical, minor technical, editorial
      categories 
 
      This isn't about binding, it's about demonstrating to ADs that
      "major technical" items have been considered 

      Working groups have to consider comments, can't just ignore
      them 

      What is "must respond"? Working group is free to say "we're
      right", not free to say "we've never heard this comment
      before"

    - information about potential reviewers? How many requests, how
      many reviews ... 

      Number of reviews is also a recognition item for reviewers

      Average time to complete a review per page of document

      Need to be collecting raw information as we go

      Need to avoid "top 10" lists of reviewers, etc.

      Don't set the threshold to become a reviewer too high - need
      to be bringing in new reviewers (see later discussion)

      Reviewer specializations? Mark will show what we have so far 

      What kind of review are we asking for? Very specific topics,
      or more general?  Need to accommodate specialist reviewers and
      generalist reviewers

    - Subjective criteria for admittance to the reviewer pool?

      (This discussion is in the context of the WG having agreed to
      a set of objective criteria, by which folks are eligible for
      the pool without additional approval.)

      Start by doing real work before you're admitted? AD can just
      decide? 

      Requests are published publicly, so one could establish a
      track record

      Sponsorship by an AD? Definition of "good" for an AD is "what
      I would have figured out anyway"? 

      Volunteering for reviewer pool and accepting/rejecting by an
      AD should be in secret

      If "n" (reviews) is sufficiently large, that flushes out a lot
      of the idiots 

      "Sorry, but your reviews don't seem useful" - as a teaching
      tool - but need to keep from turning ADs into fulltime mentors
      - are reviewers expected to be willing to serve as mentors?
      Could this be this an area directorate responsibility (defined
      broadly)? 

      EDU team should (obviously) be willing to help here

      Could have a separate pool of mentors, set up independently

      Are we doing early or late reviews? This document is focused
      on early review - the hope is that this helps with late
      reviews as well
      
      WG needs some assurance that they are likely to get a good
      review from anyone in the pool 

      Nothing but early review can reduce late surprises - this
      needs to be part of the success criteria for ICAR 

      Some set of reviews plus AD sponsorship? 

      ADs can sponsor anyone, but recommend looking at previous
      reviews? We trust ADs to do a lot more than make choices like
      this without adult supervision 

      (Objective - it's "two or more" RFCs, not "two")

    - How do we measure success/failure of ICAR experiments?

      Some types of failure are easy to detect, but success is
      harder - look for absence of failures and keep going? 

      Objective criteria for failure, subjective criteria for
      success 

      Helped or hindered in RFC production? Experiment will be over
      long before we get an RFC out through this mechanism 

      Ask the chairs, not the document editors, for success, but ask
      the document editors for feedback as well 

    - How are people removed from the pool, or is this process needed at all?

      At least remove yourself!

      Let bad reviewers rot on the vine?

      Need a minimal "periodic survey" for self-removal? People
      should "time-out" after six months, or something 

      Removal on the basis of quality - one person's crank is
      another person's helpful reviewer 

      If you qualify and time out, you still qualify when you ask to
      be added back 

      ADs need a removal-for-cause mechanism - we do remove WG
      chairs from time to time 

      If reviewers don't provide reviews - performance criteria for
      remaining in the pool? 

      This is a voluntary process anyway - why do we need to throw
      people out of the pool? What's the worst case? Is it as bad as
      what a WG chair can do? We can throw them out now... 

      Are we talking one percent bad reviewers or ten percent bad
      reviewers? We don't need to create an expulsion mechanism for
      the experiments 

      We pretty much agree we need a timeout

      What about a reviewer with an ax to grind? Is "serve at the
      pleasure" language good enough? Is "the grapevine" good
      enough? Bad reviewers are worse for new working groups with
      new chairs.  

      Why is this worse than a bozo appearing at IETF last call? The
      pool is supposed to have more clue, and the working group is
      supposed to respond 

      What about reviewers that don't review? Just timing out is
      good enough 

      Underburden bad reviewers, ignore them, or overburden them? We
      already have these mechanisms 

      There is a difference between a process that's seldom used and
      one that doesn't exist 

      We don't have consensus here - take this question to the list

      How does AD removal map to pre-qualified participants? To the
      list... 

    - What can we hope for from the IESG? How do reviewers,
      prospective reviewers, and working groups interact with IESG? 

      IESG has promised Harald they will provide sacrificial working
      groups for the experiment, and are concerned about providing
      names of people who don't have time to review documents for
      ADs now 

      "Perhaps your working group should get an ICAR review"

      We're asking "help us find working groups, help us find
      reviewers, sponsor reviewers" - IESG is mumbling on the middle
      answer 

    - Other comments?

      Non-performance will be a problem - need backup reviewers -
      yes, and WGs can handle this however they want to - we could
      learn this in an experiment, instead of noodling about it 

      We are depending on volunteers - dates and plans to follow

3.  the ICAR web page and soliciting for volunteers - Mark Allman

    Web page available from ICAR charter page

    Describes the proposal for an experiment

    ADs can request reviews for non-WG documents, too

    Pool members send reviews to icar mailing address

    WG considers comments and suggestions included in the reviews

    Reviewer attributes - expertise, reviews, ideal queue depth,
    current queue depth, plus arbitrary bio 

    Reviews and responses available from reviewer page

    Could be sliced by WG, by draft, etc.

    Logging time when information about requests and reviews are
    entered into the system 

    Feedback? Codify expertise? At least sort by expertise? Grade
    expertise? What about subjective grade inflation? What about
    "wireless", "XML", etc. 

4.  experience with the general area review team - Harald Alvestrand

    Gen-ART is late review - one week before IESG processing - and
    general review, including readability 

    IETF chair unable to get responsible and timely reviews, and
    other people had directorates and review teams - so asked for
    help 

    Called an experiment, getting experience with fast reviews,
    identify what turns out to be problems, and actually publish
    reviews 

    One-area pilot, so Harald didn't have to ask for agreement, and
    Harald already had a web server and mail server 

    Review guidelines stolen from SIR, driven by IESG mechanics

    Reviewers assigned round-robin, reviews and names are public,
    started January 22 

    Reviewers recruited word-of-mouth, "known to the AD", but also
    asked for volunteers on ICAR, no reviewers have crumbled yet 

    One-person, one-document? Don't double up, because we have 20
    documents per telechat and less than 40 reviewers 

    10 reviewers, 135 reviews, lots of concerns, 5 DISCUSSes (at
    least) 

    How did this compare to SIR experiment? This is review at a
    different level 

    Load on AD: down. Community involvement: Up

    Lessons learned - one week is a short time, reviewers need a
    discussion forum, ten people isn't enough, it needs to be a
    team, procedures are necessary, IT support is optional but
    nice. 

    Trying to move standards-track review to IETF Last Call time 

    Would a single discussion forum scale to 200 reviewers?

    Unresolved issues: Follow-up is important. Procedures for
    pointing at information is important. Getting the WG's attention
    for non-critical issues isn't a solved problem. 

    Not sure who to send reviews to (ADs, WG chairs, editors, or
    some combination). 

    Reviews need to be more visible to the community (you can find
    them now, if you're psychic). 

5.  General Discussion

    What sort of expectations for reviewer responsiveness on
    committing to a review? 

    Hard deadlines help us focus...

    Short timeframes are good, when we're asking for a commitment -
    Dave will suggest a number on the mailing list

Attachment: pgpsmsJjXR71Y.pgp
Description: PGP signature

_______________________________________________
Icar mailing list
Icar at ietf.org
https://www1.ietf.org/mailman/listinfo/icar