COACH: A Comprehensive Approach to Quality Agenda: Scott Bradner, Introduction: Existing RFC Editorial guidelines, (no viewgraphs) Questions he was asked to address: What is the aim of an IETF document? Are IETF documents any good? There is a general view outside the IETF that IETF documents aren't that good, based in part on the assumptions of other groups (IEEE, ITU) about standards documents. What is the trend over time, in terms of document quality? Not increasing. What are the problems of bad documents? RFC 2360: Guide for Internet Standards Writers. ---------- Bernard Aboba: A Comprehensive Approach to Quality (10 minutes) draft-loughney-coach-00.txt Poor quality is the result of insufficient resources. What is a WG Quality Plan: a public document, at most 5 pages. Potential components of the WQ Quality Plan: charter; challenge assessment; Are the goals achievable? What are the risks? tools plan; Mailing list; web site; document production; revision control; issue tracking and reporting review plan. Review of charter; work item review; midterm assessment; late review Cross-area review is critical. The Post Mortem: at most three pages. John Loughney: Many WGs do this already informally. Dave Crocker: We don't always pay attention to the operations impact. Rodney Hass: Question about the slide on challenge assessment. The challenge assessment might be a very difficult document to write, because different WG members have different views. On poor writing: How does this address poor writing skills? Eric Gottman: I like where this is going. One problem is that we lose the history of what we have done. David Black: A few cautionary notes. The challenge assessment needs to be viewed as a living document. Be careful about the challenge assessment being turned into a weapon to kill something. Jonathan Rosenburg: About poor writing, he is not sure that the plan will help. It takes a lot of experience to write good protocol specifications. He is not sure that each working group needs completely new plans - different working groups will probably have similar plans. ---------- Margaret Wasserman: IETF Problem Resolution Processes draft-ietf-problem-process-00.txt Problem Resolution is being addressed in the Problem working group. She is reporting on the problem resolution process recommendations. Near-term activities: COACH (on working group processes); Internal education; Tools improvement; Inter-working-group communication Long-term activities: The IMPROVE WG. Her suggestion for improving process: Identify promising proposals; define metrics; measure performance; institute change; measure the results. This is classic process improvement methodology. ---------- Leslie Daigle: The BoF Process: A Critique Personal observations. Pathological BoF cases: * BoF proponents are told no, come back when you have figured us out. * BoF proponents pester ADs until then get a BOF, and then a WG The BoF can be about gaming to get approval to be a WG. Proposal for a different approach (called the oven process): * First get agreement on a problem statement within a small group. * Then have a BoF. Scott Bradner: As Area Director, he used to have a similar approach, of requiring several things before the BoF. BoF chairs are not necessarily good WG chairs. ?: He assumes that the oven process was already in play. But there is a problem that people will feel locked out, so the BoF is useful to address that. Aaron Falk: The process of baking in the oven is subtle. Success depends a lot on the right people being added to that process. ---------- John Klensin: (no viewgraphs) IESG Overload and Quality of WGs draft-klensin-overload-00.txt (This is a report on an old proposal, from a conversation with Mike O'Dell.) We have an incentive problem: Someone is very interested in getting something done. There is nothing fundamentally wrong with it, so we end up just doing it. Some of the WGs last forever and take a lot of time. We start too many WGs, and we don't kill them off fast enough. To change the incentive model: Put a ceiling on the number of working groups, to give an incentive to the IESG to be more strict, and make judgements. James Polk: Is it about the number of WGs, or the number of efforts (e.g., SIP and SIPPING)? John: It is easier to limit WGs than limiting efforts. Margaret Wasserman: I have a number of concerns with this proposal. Any time that you set up arbitrary rules, you invite people to game the system some other way. I think we want the IETF to grow and do more things. John: We are in complete agreement except for two things. The community tells the IESG that we are not interested in solving this problem, so the alternative is to change the incentives. Dave Crocker: You are saying that a community that is highly expert in resource contention problems doesn't know how to apply it to ourselves. The really hard part is that the set of incentives that will work the right way, would be incentives that people would apply to themselves. We need incentives to maintain quality and control quantity. Jonathan Rosenburg: The definition of what is important is hard to nail down, because there are many communities for which different things are important. ---------- Margaret Wasserman: Decision points/milestones in the WG process draft-wasserman-wg-process-00.txt Purpose of the document: * Provide terminology and a framework. * Improve WG chairs training. Steps in the WG process: Initial submission; Author refinement; WG acceptance; Editor selection; WG refinement; WG last call; Scott Bradner: If there is not much support to include a document in the working group, silence does not mean consent. David Black: On the importance of design teams, and the proper management of design teams. Closed design teams should open up after a document becomes a WG item. ?: This is interesting. Is it consensus when four people are enthusiastic, and everyone else in the WG passes? Margaret: We are in no danger of saying No too firmly or too often. ---------- Brian Carpenter: Careful Additional Review of Documents (CARD) (10 minutes) - Brian Carpenter http://www.ietf.org/internet-drafts /draft-carpenter-solution-sirs-01.txt Was: SIR: Senior IETF Reviewer New: AIR: Agreed IETF Reviewer This proposal is targetting many of the problems cited in the problems document. Basic proposal: * Create a team of reviewers * Have all drafts reviews early, mid-term, and late. * Build trust in this process. The standards process doesn't change. Creating the team: create a set of 200 reviewers, for 1800 reviews/year. Bernard: Do we have 200 people to do this? He doesn't think so. Increasing and maintaining the team. Triggering reviews - this comes from the WG. What's in a review? These need to be public reviews. James Polk: Of the 200 reviewers, how do you find the right reviewer? Brian: We need to do more work on this. Mechanics: web site, http://graybeards.net/sirs/ web tools. Ross: I am worried about the publicness of the comments. Politics about company A/company B? Margaret Wasserman: There are a lot of costs in trying to do this. Harald: There are two things I would like to see: a few samples of reviews, listed on a web page. Allison: I think these reviews could be very useful, but the perfect can be the enemy of the good. I don't think that *all* documents needs to be reviewed, *three* times. It might be sufficient for some documents to be reviewed well, once. Henning: This is not a new mechanism. Scientific publications have been doing peer reviewing for ages. Timeliness is an issue, because we don't want the reviewing to add delay. ---------- Aaron Falk and Allison Mankin: The Review Process in Action: The DCCP WG IESG reviews sometimes surface late surprises. Early Review Proposal from Leslie and Allison at an IAB/IESG retreat: Get broad inter-area review well before WG and IESG last call. The DCCP Design Review: DCCP, planning for WG Last Call this fall. The expert written review was for depth. The design review in the WG is for breadth. Perhaps this one should have happened earlier. Out of scope of the meeting: WG charter, problem statement. Eric Rescorla: Two observations: being a designated reviewer makes a big difference, in terms of the review actually getting done. It took him two days to do his review. Spencer Dawkins: How much does it make a difference to have IESG collaboration in getting reviewers? Brian Carpenter: What if DCCP was an individual submission? If this level of review, then could the RFC editor review process be not needed? Aaron: The RFC Editor gives a very different kind of review. : Do you have sense of the time it takes the RFC editor to do a review? Aaron: Sorry, no. ---------- The next steps will be mailing list discussion. General comments from the mike? Henning: These all sound like proposals that can be done well on an experimental basis. We need to try experiments, and judge them successful or not. Dave Crocker: If we don't have enough reviewers, that can be a feature. If one can't get competent reviews, then the work shouldn't progress to the next stage. Bernard: I would agree that if you can't come up with the resources to finish the work, then the work should not be started. Dave C.: We're having trouble putting quality and timeliness in the work that we do now, do lets do more? |