Last Modified: 2003-03-10
This group is charged with producing the document describing these problems. The analysis of the problem should seek out the root causes of the problems as well as the perceived derivative problems.
The intent is that the group will discuss issues on its mailing list, and that there will be an editing team to produce a clear concise problem statement on which the group has come to consensus and present to the IETF as a basis for an IETF consensus.
As a second work item, the group will also produce a proposal for a process to develop solutions to the problems identified by this working group.
It is not a part of this group's charter to propose solutions to the problems.
The work items will be reviewed in IESG plenary at the IETF.
|Done||First I-D of problem statement issued|
|MAR 03||Problem statement reviewed at the IESG Plenary|
|MAR 03||First I-D of draft document describing the process by which the ietf will change its processes issued|
|MAY 03||Problem statement submitted for IESG review|
|JUL 03||Draft document describing the process by which the ietf will change its processes reviewed at the IESG Plenary|
|AUG 03||Draft document describing the process by which the ietf will change its processes submitted for IESG review|
|OCT 03||Re-charter or close working group|
Problem Statement BOF (problem) Friday, March 21 at 0900-1130 ============================== CHAIR: Melinda Shore <firstname.lastname@example.org> Avri Doria <email@example.com> Notes: Spencer Dawkins <firstname.lastname@example.org>, Chris Allen <ChristopherA@AlacrityManagement.com> Charter discussion - Chairs Charter went through quickly with little controversy work is to be analytic and descriptive, with emphasis on analytic not pointing fingers aggressive schedule - done in October comments on document structure and contents both welcome Current Status Mailing list but no separate web page - do we need one? 1st draft of problem description document has been released Have editors and editor team for both chartered documents Chris Allen: suggested addition to charter - document existing procedures to keep what's effective Document Process Performance - Henning Schulzrinne Identified "IETF Goodput Measures": size, delay, throughput. This was a first attempt to figure out if this is worth looking at. Methodology: matched RFC names with draft-00 titles, looking for "end-to-end latency." Our data are extremely poor - no other mechanism works. The Internet Monthly Report doesn't contain all I-D names and the IETF Announcement list only goes back to 1998 so history has been lost. Help is welcome on this. Average number of pages spiked a lot earlier than number of RFCs per year. It hasn't changed since early 80s -- about 200 RFCs per year, about 25-30 pages per RFC. Internet draft bandwidth has been on steady linear increase since 1991. 2001 was down, 2002 was back up. 00-to-RFC delay has grown from 6 months to about two years, with a steady increase from 1991 to 1998, and constant since then. Harald Alvestrand asked if Henning had tried plotting the graph using the year the work was started instead of the year the work was ended. Henning said he hadn't. In response to a question from Chris Allen Henning said that many measures actually fell off somewhat in 2001, not just these. Several people suggested additional metrics and Henning responded that other statistics are also interesting The RFC Delay Histogram clusters at two years, but has a long tail - four, even five years. Aaron Falk asked about individual submissions that become working group document, and Henning responded that the stats were based on title matching, not I-D name matching. The data are VERY noisy - formats highly variable, Internet Monthly Report is incomplete, some RFCs were never announced. The estimates are likely low due to I-D renaming or high due to "blocking." Some suggestions: sample, don't census Look at questions like "one rev per IETF meeting" Waiting for other documents? author fixes? IESG queues? Look at reasons, not at data Does this help agreement on root causes? Initial Analysis of IESG Review - Eric Rescola Overlaps Henning's analysis. He also experienced low signal-noise ratio, but felt his findings were basically correct. The question being addressed was how long do things take from last call to document approval. What he found was: mean = 147 days, median = 93 days - standard deviation is 149 days 300 days isn't unusual Documents that are published as is are still slow Areas differ a bunch in speed: internet, Ops and transport are the fast ones applications and security are the slow ones and differ a bunch in output volume Churning takes time (60 days, volume of changes doesn't matter, length doesn't matter. The approval time is constant over study period. Dave Crocker said this work was similar to work Marshall and he did and that there were some methodological problems with the treatment of the data. he said that he was not saying the math was wrong, but use of those statistics in that environment is risky because the community of professionals use normal data, but don't know how to process log-normal data. There is a difference between the math world and the social research world. Eric disagreed about the appropriateness of the tests and said that the methodology was fine. Bill Sommerfield said that as a WG chair he's frustrated. In his working group they get one bug report, they fix it, submit, get one more bug report, fix it, etc. Are there data on small changes? Eric answered from 0 to 4 review cycles typically. Aaron Falk said that AD review isn't well documented and can be highly variable, and Eric replied that it can be highly variable between ADs, as well. Architectural and Process Inputs - James Kempf James's discussion was titled "Helping the Internet to age gracefully." Heard around IETF: "We don't do any architecture," with architectural work hidden as "frameworks", etc. Heard around 3GPP - "IETF doesn't do architecture, they do philosophy." End-to-end is an example of traditional approach and it's helpful, but we need more. We must integrate new components with existing components. There are interdependencies across working groups, or even across areas. Catching them at IESG review time is too late to affect base design. Is this really system design, and not architecture? IETF work is a stove pipe, and assumes minimal interdependencies - but the number isn't zero. Can we ignore the problem? We're sending more drafts back due to conflicts, but missed interdepencies still happen. There's an increasing lack of transparency and understandability in Internet Design. It's becoming more like the telephony world - baroque. Bob Hinden raised the issue of proposed standards and said that there are risks in implementing PS, but they implement PS in products all the time. James replied but the problem might be big - not a minor tweak, and Bob pointed out that full standards are really rare. Bill Sommerfeld: too difficult to move past PS - there's no energy after doc is stable. Henning said that he doen't disagree with slides, but needs case studies for background, with concrete examples of last-minute/too-late fixes. James: this would be useful. Working Group Participation and the "Stuckee Problem" - Ted Hardie The discussion is abstracted from and I-D that's solutions-based. The IETF working style of an Open working group on mailing lists has produced successful results, but making a comment doesn't mean you're taking responsibility for doing work. It's difficult to make anyone other than WG chairs and current authors accountable. The problem is how to track commitment without breaking openness and shutting out outside review. Dave Crocker said that we have people who consume resources but don't help make progress. Ted said that may be true but that isn't the problem he's looking at. Dave replied that the working group isn't real if people don't show up and do work, and that what Ted is saying is reasonable but represents a different philosophical or strategic difference from what the IETF ought to be. Ted answered that working groups do vary over time, but stuff does fall out the other end. There may not be a constituency when it does, but that's not what we are discussing here. The problem is that we can't do anything that has dependencies unless someone is actually responsible for those dependencies. Spencer Dawkins said that dependencies are worse for other SDOs that depend on the IETF to produce a standard. How can working group chairs responsibly say "you can count on us? Bob Hinden said that if there's no energy, the work shouldn't be done, and Ted replied that if another working group is depending on the work that's not being done, the second working group either fails or takes on all the work. Problem Resolution status - Margaret Wasserman This is the second document in working group charter. There are clearly issues about changing the IETF. Where do we go from here? The IETF is unique, but these aren't unique problems. Good times hide problems, bad times hide successes, and we need a consistent focus on operational improvement. Margaret listed Harald's core values from IETF 55 plenary. Things that aren't core values are all on the table for changes. The current environment is highly emotional but we're committed to improvement. The IETF is a fast-moving target - we can't stand still while we measure and decide how to move forward. Major process options: Evolution vs revolution ("nothing left to lose"?) Granular vs monolithic (can problems be separated?) Immediate versus ongoing Grassroots vs top-down What are the right mechanisms to use, moving forward? Should we update existing process documents? Do we need new roles documents? We need a initial draft in the next couple of weeks and haven't gotten much input yet. James Kempf comment: People are assuming that there is one solution, and that the entire IETF will adopt it. We should try a bunch of different approaches and see what ones work. This means that we have to be willing to accept that some will not work. Margaret pointed out that that is one of the listed solutions. Harald: Fixing our short-term problems is harder than fundamental changes to IETF and may be harmful to the Internet. If we focus on short-term problems it takes the heat off for five years, but if we don't fix short-term solutions, we're also dead. Dave Crocker: and what will be the experimental group? Everything we've done for 13 years has been an experiment revolutionary changes have big and unexpected impacts Bob Hinden: Harald is right - we can't fix our problems by fine-tuning. Harald: if we get to the point where we can't declare consensus, I'll go home. Problem Statement draft - Elwyn Davies This is a very provisional version (00 draft). The problems aren't new and aren't (all) IETF-specific, and many are the consequence of growth. The draft was distilled from 750 pieces of email plus 3 drafts. The topics were extracted and filed, and in the case of duplicates they were attributed to the person who mentioned them first. The next steps are to create a visible issue list and publish a second draft by late April, WG last call and IESG review by May. Aaron: should be commended for your altruism and suspected for your judgement in taking this role! However, the draft really bothered me - perceptions start to sound like facts. [Keith Moore objected to the presentation of material that's already available in the document and that was presented at the plenary, and asked that we focus on discussion rather than presentation.] Dave Crocker responded to Aaron that it doesn't matter that perceptions sound like facts - we can use this as is. Graham Travers pointed out that people holding perceptions is a fact, whether the perception is right or wrong. Chris Allen: there's a cultural element in section 2.1, and mission and vision go hand in hand. Bob Hinden said that two things were mixed in section 2.2 - WG process and protocol design engineering. Henning raised the issue of a lack of effective contract with document editors, to which Elwyn added that this leads to lack of delegation. Margaret: effective engineering processes isn't the goal, it's a solution, and we need to focus on identifying the problem. Eric Hickman: over time we apply ourselves less and less, especially towards achieving consensus in adversarial working groups. Keith: lots of 2.2 emphasis on tools and process, and problem is really more fundamental - groups that can't explain their problem clearly. Don't compare 2.3 to "good old days" - we're just not sufficiently engaged. Christian: can't stay engaged in SIP at 400 msg/day. Randy Bush: more people are engaged, but there's more people who aren't Nathan: Money has gone out of this industry, and more money to be made implementing than inventing Steve Trowbridge: Spectators aren't the problem, people frightened away is the problem Eric Guttman: people leave mailing lists because they don't want to be insulted James: physical violence threatened on one mailing list Steve: ADs/WG chairs wait too long to wade in Spencer: working group code of conduct? Keith: mailing list and meeting methods are very different - should the document split them out? Keith totally disagreed with 2.4 as it was written. It is clear that there is a perception of this, it is clear that a perceptual differences are part of the problem, but he doesn't think it is fact. Steve Trowbridge said that there is nothing in our procedures that causes this to be the case, it exists in practice. There are backroom deals and initial windows on work available only to insiders. Henning: in the good old days, ADs were a LOT more invisible, now anything of any significance needs AD involvement. Margaret said that this section is where the scaling issues of the IETF are showing up. Jonne Soinenen: no single person is the problem, but the process has problems and the perception of problems is equally important as the problems themselves. Keith commented that no one will disagree with perception that IESG hasn't scaled, but qualifying it as 'authority' is inappropriate. He was in IESG 4 years and they really can't block things. This isn't an accurate statement. He does agree that it's possible for an AD to be capricious and we should fix that. Marshall answered that there are some former ADs that say that this has happened, in spite of what Keith said. Eric Guttman: if openness is our goal, why is the IESG is deciding WG formation, document advance, etc? Eve Schooler: the IETF isn't more broken than it was ten years ago, that's good news. People do want to help, and we need to be grooming more potential leaders/insiders. Margaret said that there are other problems besides scaling and asked to what extent do we want rely on WG consensus vs AD agreement. Are our selection processes open and fair? Jonne: we should see documented discussion of decisions so we can see why they are made. People just don't know what's going on. We shouldn't require people to be on the IESG to understand the IESG. Chris Allen: in groupware field ten years ago, social scientists were looking at mailing lists. we don't teach people what's already well-known and undersood about our work media. Pete Resnick: the perception that there are upper class/lower class differences is real. Distrust is real and can increase if we're not careful. Margaret: WG chair role isn't well-defined and consistently understood in the IETF. We need to do role definition before training can be effective. Chris Allen added that the mentoring process isn't mentioned and is broken. We have document editors and WG chairs who are brand new. We should have "shadow" roles who works under old person for 6 months/one year for training. Bob Hinden: There are problems that come with getting new people and readying them to be come the new leaders. I have the feeling that it has become more exclusive. There is a perception of cycling from IAB to IESG, IESG to WG, etc. Joel Halperin: ADs CAN block things, and sometimes the things we're complaining about are the reasons why we succeed. Don't create a new problem while solving another one. Bob Hinden: "blocking" needs to happen at the beginning of the process, not at the end. Ted Hardie: Part of the problem is the metaphor "Send up to the IESG" which colors the perceptions. There are many things in the document that have a series of metaphors of consensus, and we use the words of consensus, but in fact, sometimes we use disputation. When we use disputation with consensus it can cause acrimony. We need metaphors to match our recommendations. Dave Crocker said that we may not have structural problems, but rather management problems. For instance we sometimes choose between two alternatives when we don't have to, and sometimes we don't choose when we should. Christian: "Consensus" wasn't in original IETF, but was added in 1992 as response to top-down decision making. The way we used to solve problems was by building something. Now we need agreement before anyone builds anything. Eve Schooler: We can say "here's a good Internet Draft", but "what makes a good WG chair or AD" isn't written down anywhere. Ted: we've narrowed the point of WGs until it's hard to see the whole picture, then we're surprised when we get problematic point solutions. Wrapup - chairs We're serious about hitting our milestones. The problem description is to be in WG last call in May, and the process options document is to be in WG last call in August. The latter document should be available within the next few weeks.