2.2.3 Comprehensive apprOACH to quality (coach) Bof

Current Meeting Report

COACH: A Comprehensive Approach to Quality


Scott Bradner, Introduction:
Existing RFC Editorial guidelines,
(no viewgraphs)

Questions he was asked to address:
What is the aim of an IETF document?
Are IETF documents any good?
There is a general view outside the IETF that IETF documents aren't that 
good, based in part on the assumptions of other groups (IEEE, ITU) about 
standards documents.
What is the trend over time, in terms of document quality?  Not 
What are the problems of bad documents?
RFC 2360: Guide for Internet Standards Writers.


Bernard Aboba:
A Comprehensive Approach to Quality (10 minutes) 

Poor quality is the result of insufficient resources.
What is a WG Quality Plan: a public document, at most 5 pages.
Potential components of the WQ Quality Plan: 
  challenge assessment; 
    Are the goals achievable?  What are the risks?
  tools plan; 
    Mailing list; web site; document production; revision control;
    issue tracking and reporting review plan.
    Review of charter; work item review; midterm assessment; late review
    Cross-area review is critical.
The Post Mortem: at most three pages.

John Loughney: Many WGs do this already informally.

Dave Crocker: We don't always pay attention to the operations impact. 

Rodney Hass: Question about the slide on challenge assessment. The 
challenge assessment might be a very difficult document to write, 
because different WG members have different views. On poor writing: How 
does this address poor writing skills?

Eric Gottman:  I like where this is going.  One problem is that we lose the 
history of what we have done.

David Black:  A few cautionary notes.  The challenge assessment needs to be 
viewed as a living document.  Be careful about the challenge 
assessment being turned into a weapon to kill something.

Jonathan Rosenburg:  About poor writing, he is not sure that the plan will 
help.  It takes a lot of experience to write good protocol 
specifications.  He is not sure that each working group needs 
completely new plans - different working groups will probably have 
similar plans.


Margaret Wasserman:
IETF Problem Resolution Processes 

Problem Resolution is being addressed in the Problem working group.
She is reporting on the problem resolution process 
Near-term activities:
  COACH (on working group processes); 
  Internal education; 
  Tools improvement;
  Inter-working-group communication
Long-term activities:
Her suggestion for improving process:
  Identify promising proposals; define metrics; measure 
performance; institute change; measure the results.
  This is classic process improvement methodology.


Leslie Daigle:
The BoF Process: A Critique
Personal observations.
Pathological BoF cases:
  * BoF proponents are told no, come back when you have figured us out.
  * BoF proponents pester ADs until then get a BOF, and then a WG
The BoF can be about gaming to get approval to be a WG.
Proposal for a different approach (called the oven process):
  * First get agreement on a problem statement within a small group.
  * Then have a BoF.

Scott Bradner: 
  As Area Director, he used to have a similar approach, of requiring 
several things before the BoF. BoF chairs are not necessarily good WG 

?: He assumes that the oven process was already in play. But there is a 
problem that people will feel locked out, so the BoF is useful to 
address that.

Aaron Falk: The process of baking in the oven is subtle.  Success 
depends a lot on the right people being added to that process.


John Klensin:
(no viewgraphs)
IESG Overload and Quality of WGs 

(This is a report on an old proposal, from a conversation with Mike 

We have an incentive problem:  
  Someone is very interested in getting something done. There is nothing 
fundamentally wrong with it, so we end up just doing it. Some of the WGs 
last forever and take a lot of time.
We start too many WGs, and we don't kill them off fast enough.
To change the incentive model:
  Put a ceiling on the number of working groups, to give an incentive to the 
IESG to be more strict, and make judgements.

James Polk: Is it about the number of WGs, or the number of efforts 
(e.g., SIP and SIPPING)?

John: It is easier to limit WGs than limiting efforts.
Margaret Wasserman: I have a number of concerns with this proposal. Any 
time that you set up arbitrary rules, you invite people to game the 
system some other way. I think we want the IETF to grow and do more 

John:  We are in complete agreement except for two things. The 
community tells the IESG that we are not interested in solving this 
problem, so the alternative is to change the incentives.

Dave Crocker:  You are saying that a community that is highly expert in 
resource contention problems doesn't know how to apply it to 
ourselves.  The really hard part is that the set of incentives that will 
work the right way, would be incentives that people would apply to 
themselves.  We need incentives to maintain quality and control 

Jonathan Rosenburg: The definition of what is important is hard to nail 
down, because there are many communities for which different things are 


Margaret Wasserman:
Decision points/milestones in the WG process 

Purpose of the document:
* Provide terminology and a framework.
* Improve WG chairs training.
Steps in the WG process:
Initial submission; Author refinement; WG acceptance; Editor 
selection; WG refinement; WG last call;

Scott Bradner: If there is not much support to include a document in the 
working group, silence does not mean consent.

David Black: On the importance of design teams, and the proper 
management of design teams.  Closed design teams should open up after a 
document becomes a WG item.

?: This is interesting.  Is it consensus when four people are 
enthusiastic, and everyone else in the WG passes?

Margaret: We are in no danger of saying No too firmly or too often.


Brian Carpenter:
Careful Additional Review of Documents (CARD) (10 minutes) - Brian 
Was: SIR: Senior IETF Reviewer
New: AIR: Agreed IETF Reviewer
This proposal is targetting many of the problems cited in the problems 
Basic proposal: 
* Create a team of reviewers
* Have all drafts reviews early, mid-term, and late.
* Build trust in this process.
The standards process doesn't change.
Creating the team: create a set of 200 reviewers, for 1800 

Bernard: Do we have 200 people to do this?  He doesn't think so.

Increasing and maintaining the team.
Triggering reviews - this comes from the WG.
What's in a review?  These need to be public reviews.

James Polk: Of the 200 reviewers, how do you find the right reviewer?

Brian: We need to do more work on this.

Mechanics: web site, http://graybeards.net/sirs/ web tools.

Ross: I am worried about the publicness of the comments.  Politics about 
company A/company B?

Margaret Wasserman:  There are a lot of costs in trying to do this.

Harald: There are two things I would like to see:  a few samples of 
reviews, listed on a web page.  

Allison: I think these reviews could be very useful, but the perfect can be 
the enemy of the good.  I don't think that *all* documents needs to be 
reviewed, *three* times.  It might be sufficient for some documents to be 
reviewed well, once.

Henning: This is not a new mechanism.  Scientific publications have been 
doing peer reviewing for ages.  Timeliness is an issue, because we don't 
want the reviewing to add delay.

Aaron Falk and Allison Mankin:
The Review Process in Action: The DCCP WG 

IESG reviews sometimes surface late surprises.
Early Review Proposal from Leslie and Allison at an IAB/IESG retreat: 
Get broad inter-area review well before WG and IESG last call.

The DCCP Design Review:
DCCP, planning for WG Last Call this fall.
The expert written review was for depth.
The design review in the WG is for breadth.  Perhaps this one should have 
happened earlier.
Out of scope of the meeting: WG charter, problem statement.

Eric Rescorla: Two observations: being a designated reviewer makes a big 
difference, in terms of the review actually getting done. It took him two 
days to do his review.

Spencer Dawkins:  How much does it make a difference to have IESG 
collaboration in getting reviewers?

Brian Carpenter: What if DCCP was an individual submission?  If this level of 
review, then could the RFC editor review process be not needed?

Aaron:  The RFC Editor gives a very different kind of review.

: Do you have sense of the time it takes the RFC editor to do a review?

Aaron: Sorry, no.


The next steps will be mailing list discussion.

General comments from the mike?

Henning:  These all sound like proposals that can be done well on an 
experimental basis.  We need to try experiments, and judge them 
successful or not.

Dave Crocker:  If we don't have enough reviewers, that can be a 
feature. If one can't get competent reviews, then the work shouldn't 
progress to the next stage.

Bernard: I would agree that if you can't come up with the resources to 
finish the work, then the work should not be started.

Dave C.:  We're having trouble putting quality and timeliness in the work 
that we do now, do lets do more?


Problem Resolution Process
Steps in the WG Process
SIRs, or AIRs, or something