[00:38:20] --- dcrocker has left: Disconnected
[17:23:16] --- chopps has joined
[17:23:20] --- chopps has left
[17:50:39] --- Melinda has joined
[17:53:34] --- dcrocker has joined
[17:54:48] --- hartmans has joined
[17:58:51] --- Ralph has joined
[18:03:09] <dcrocker> i'm going to have to leave right after presenting, but want to know when we return to consideration of the drafts.
[18:03:53] --- jhutz has joined
[18:04:45] --- hta has joined
[18:06:12] --- BP has joined
[18:07:07] <jhutz> dcrocker on "SIRS"
[18:07:35] <jhutz> - team of reviewers; senior IETF participants doing "external" reviews; reveiw requests from WG or elsewhere
[18:07:43] <jhutz> - established experiment
[18:07:53] <jhutz> web page: http://graybeards.net/sirs
[18:08:01] <jhutz> mailing list: ietf-sirs@yahoogroups.com
[18:08:09] <jhutz> got reviewers - a nice start
[18:08:17] <jhutz> got some review requests and did reviews - not many
[18:08:23] <jhutz> request/review process kept very informal
[18:08:35] <jhutz> - after 3 (summer) months; rumors that SIRS had failed
[18:09:19] --- galvinjamesm has joined
[18:09:19] <jhutz> dcrocker: the fact that we got early signups at a birsk pace was a good indication we had community interest.
[18:09:45] <jhutz> [having trouble keeping up...]
[18:09:54] <jhutz> - low number of review requests
[18:11:29] --- sakai has joined
[18:12:03] <jhutz> ... the impetus for getting reviews was strictly the wG's -no going out and getting business, dictating to WG's that reviews were needed, etc.
[18:12:19] <jhutz> ... purely a consulting service - no attempt to market, awareness not what it needs to be
[18:12:37] <jhutz> ... have started getting some feedback about what sort of reviews need to happen
[18:13:04] <jhutz> ... cross-area, early review were just beginning to be discussed; now there is more attention.
[18:13:23] <jhutz> ... A group plike this (icar) will be useful in giving guidance in what kinds of reviews are needed, when, etc
[18:14:03] <jhutz> next: bert wijnen, Ops AD, MIB doctors experience
[18:14:26] <jhutz> Slides: psg.com://~bwijnen/iCarMib59.ppt
[18:14:44] <jhutz> [and I will concentrate on what berj says; read the slides if you want to see them]
[18:17:26] <jhutz> bwijnen: how does mib review work...
[18:17:46] <jhutz> ... currently too much at the end of the process.
[18:18:35] <jhutz> ... people think near the end of the process their document will proceed quickly, but review takes time due to large number of documents to review
[18:18:52] <jhutz> ... usually does happen before IETF last call, but sometimes AD issues last call before MIB doctor review is done, and then if there are major problems, the document has to go back
[18:19:15] <jhutz> ... try for consistent review process. used to be that the sort of review you got was highly dependent on the mib doctor you got
[18:19:27] <jhutz> ... mib doctors have developed a consistent review process, hope to turn into a bcp soon
[18:19:49] <jhutz> ... feedback generated by mib doctors to authors, wg mailing list, with a cc to the responsible AD and bwijnen
[18:19:59] <jhutz> ... last 2 bullets pretty important:
[18:20:41] <jhutz> ... focus on MIB syntax and structure/reuse -- mib doctors do not necessarily understand all of the technologies in use - mib-doctors won't necessarily know what you left out,
[18:21:09] <jhutz> ... only look at syntax/structure, network management issues, etc - not the information model on which you developed the mib, which is dependent on the technology and is the responsibility of the WG.
[18:21:33] <jhutz> [next slide]
[18:21:35] <dcrocker> can someone put a jabber note up when things switch back to discussion of reviewing proposals? tnx.
[18:22:08] <jhutz> ... in the beginning, we had a mailing list; I would send a message and say "who wants to review this doc". The good thing is that someone interested/with relevant knowledge would volunteer. but things didn't get picked up always
[18:22:39] <jhutz> now AD has to go to individuals and request specific people to do reviews. works better, but puts the onus on the AD to find a mib doctor for each doc
[18:22:44] <jhutz> So this is somewhat a scaling issue
[18:22:46] --- lha has joined
[18:23:16] <jhutz> ... not mandatory to go through mib-doctors, but IESG will ask "bert, is this OK wet MIB syntax, etc", and mib-doctor review is the mechanism for that
[18:23:26] <jhutz> next: steve bellovin, security AD, on security directorate
[18:24:12] <jhutz> ... Not sure when secdir was created, but in early days of Jeff Schiller's days as AD, we'd meet for lunch. good social event but not much work done
[18:24:32] <jhutz> ... jeff moved to a mailing list model, where he'd send mail to list asking for reviews. It was a resounding failure; nothing got reviewed.
[18:24:39] --- Bill has joined
[18:24:40] <jhutz> ... eventually jeff gave up
[18:24:49] <jhutz> ... I (smb) became security AD, revived secdir.
[18:25:02] <jhutz> ... what we've been doing differently is that Russ (Housley) and I would ask a specific person to review a document
[18:25:23] <jhutz> ... mention in meeting or on mailing list, but also ask a specific person; this has worked well
[18:25:52] <jhutz> ... this also means we can track activity and round trip time for reviews; "edit" membership of secdir based on responsiveness
[18:26:24] <jhutz> russ housley: the pointing to a specific person instead of just yelling for help has been the key. Since doing that, we've gotten something like 80% of the reviews we asked for, vs 5% before - a significant improvement
[18:27:07] <jhutz> CHAIR: sunce bert and steve have to go to other WG's, comments/questions about mib-doctors or secdir now...
[18:27:28] <jhutz> [XXX]: Since the mib-doctors haven't done much about _early_ review, which this WG is about,...
[18:27:51] <jhutz> ... I once started to look at MIB documents as the -00 came out and gave some feedback. This has been useful, but the
[18:28:11] <jhutz> ... mib-doctors have some tools used to check things, so if you want early review, the -00 needs to be in a parseable form so the tools can be used
[18:28:34] <jhutz> [I think XXX was bwijnen again]
[18:28:54] <jhutz> Next: alex zinin, routing AD
[18:29:07] <jhutz> ... Experiences with RTG-DIR.
[18:29:17] <jhutz> ... right now, about 20 members; selected arbitrarily by AD's.
[18:29:35] <jhutz> ... basically people we think should be on the directorate, and who agree to be on the directorate, spend time on reviews, etc
[18:29:54] <jhutz> ... official status is an advisory body to the AD's; the AD's take responsibility for reviews, etc.
[18:30:05] <jhutz> - use an internal closed-subscription mailing list for internal communication
[18:30:21] <jhutz> .... mainly use this to review documents coming out of WG's to the iesg.
[18:30:42] <jhutz> ... lessons we've learned:
[18:30:49] <jhutz> ... general appeals "someone please review" don't work
[18:31:06] <jhutz> ... need to select 1-2 people and ask them explicitly to review this document; when you see an ack, you know what's happening
[18:31:17] <jhutz> ... assign reviews depending on topic and expertise
[18:31:29] <jhutz> ... set a deadline, not open-ended; usually 1-2 weeks
[18:31:44] <jhutz> ... usually ask the directorate members to the wg mailing list, or at least to the authors
[18:32:00] <jhutz> ... in some cases where that might generate some unhappiness, we discuss comments on the (rtg-dir) list first
[18:32:21] <jhutz> ... sometimes one of the AD's needs to proxy comments back - the person commenting does not feel comfortable engaging in open discussion there.
[18:32:41] --- Milkshake has joined
[18:32:44] <jhutz> ... in this case, we are more careful about whether we as the AD's really agree with the comments, are they showstoppers; is it the right time to bring them back to the WG or not
[18:32:54] <jhutz> ... educational process inside the directorate
[18:33:13] <jhutz> ... when I do AD reviews, I CC the rtg-dir, so the directorate can see my comments; learn the sort of things we're looking for
[18:33:13] --- bert has joined
[18:33:22] <jhutz> ... comments get recorded in the tracker, so they're not lost.
[18:33:39] <jhutz> ... if it's a long list of comments, I ask the reviewers or WG chairs to summarize how the comments have been addressed
[18:33:59] <jhutz> ... Up until recently, review happened during the AD review cycle.
[18:34:06] <bert> So here is Bert (OPS AD) and I can try to follow this and anser MIB Doctor specific questions if needed
[18:34:19] <jhutz> ... recent optimization is to ask WG chairs to CC the rtg-dir on the WG last call, so they can be reviewed during those 2 weeks, for less delay
[18:34:44] <jhutz> ... sometimes AD forwards document part of iesg agenda to directorate list; sometimes with comments "please look at this doc"
[18:35:09] <jhutz> ... since this is late review, we ask reviewers to look mostly for showstoppers, and of course editorial things that have to be fixed before publicatioin
[18:35:29] <jhutz> ... ask reviews to separate technical comments from editorial, to allow reader to understand which is which; not look like nitpicking
[18:35:41] <jhutz> CHAIR: questions/comments about rtg-dir?
[18:36:39] <jhutz> hta: just a comment that one of the things I see is that we have a thread that measuring success or failure can be difficult if you don't know what you're aiming for
[18:36:57] <jhutz> ... rtg, sec dir's seem to be helping AD's make decision
[18:37:14] <jhutz> ... might be better to know more what you want
[18:37:28] <jhutz> CHAIR: Please say your name when you come to the mic
[18:37:35] <jhutz> hta: "Harald Alvestrand"
[18:37:46] <jhutz> [chuckle]
[18:38:27] <jhutz> CHAIR: what constitutes good review; what are the criteria for _early_ review?
[18:38:32] <jhutz> - is number significant?
[18:38:41] <jhutz> - is it important to get a range of skill sets among the reviewers?
[18:38:47] <jhutz> - is there a measure of the depth of review?
[18:38:57] <jhutz> - is there a measuore of breadth of cross-area coverage?
[18:39:21] <jhutz> [XXX]: Something I don't see there is idea that review contains not only comments, but suggestions for reviews.
[18:39:43] <jhutz> ... for example, MIB doctors know more about mib syntax than others... "this is broken" is not a good review without comments on how to fix
[18:39:55] <jhutz> ... especiallyt important early in process, when changes can be made
[18:40:01] <bert> multiple reviews WILL increase quality. Two MIB doctors will find more than One. I have that experience
[18:40:10] <dcrocker> if someone will speak this for me: THe kinds of things Harald is suggesting (or, at least, asking about) is essential for the long term. However,
[18:40:24] <jhutz> [YYY]: I think regarding earliy reviews, that is important ... ....
[18:40:36] --- dthaler has joined
[18:40:36] <jhutz> ... also important that it's broad review, so you get not only expert comments
[18:40:39] <bert> can sonmeone in the room say that? I am in another WG now
[18:41:09] <dcrocker> At this stage, I suggest that we focus more on concepts and less on numbers. What topics or issues or ...(?) should a review cover? What are the _qualitative_ issues about getting reviews? /end of speech
[18:41:12] <jhutz> kathleen mariarty(sp?): don't get thrown in wrong direction; reviewers should understand the problem so they don't throw suggestions in the wrong direction
[18:41:30] <jhutz> [someone want to relay bert and dcrcoker's comments?]
[18:41:43] <dcrocker> (the "asl" symbol was unintended. no idea why it happened.)
[18:42:28] <dthaler> I can relay bert's, I didn't completely follow dcrockers
[18:43:04] <jhutz> [I can, if needed]
[18:43:10] <dcrocker> I'll try to say my point more simply. sorry for the confusion.
[18:43:48] <jhutz> [bert's comment being relayed now]
[18:44:02] <dcrocker> Comment on Harald's comments: In the long term, yes we need to understand the "numbers" and statistics of review use and review utility. In the near term, we need to focus on the qualitiative issues. For example
[18:44:18] <jhutz> dmitri papadmitrio(sp?): question concerning area review...
[18:44:38] <dcrocker> What topics and issues do we need to have reviews cover? What kinds of comments from reviewers are more helpful. What kinds are less? Perhaps
[18:44:40] <jhutz> ... people reviewing document also proposing areas that might need additional review, might have future issues?
[18:44:55] <jhutz> CHAIR: sounds like a useful thing for a reviewer to say
[18:45:04] <dcrocker> this should be treated as a requirement for a BCP that gives guidance to reviewers. / end of speech. tnx.
[18:45:26] <jhutz> dp: there might be things that the people working on a document can't see, because they're concentrating on their item, and don't see the larger context
[18:45:48] <jhutz> ... I would like to see that as one of the major targets of these early reviews
[18:46:24] <jhutz> [XXX]: short comment on the scope of review: if the reviewer has to fully understand the problem, you probably limit the number of reviewers to a group who are probably closely in cooperation with those workin on the draft
[18:46:39] <jhutz> ... we might want to have expertise from different areas, and need to understand the scope of the review you're doing.
[18:46:59] <jhutz> ... if you're doing MPLS struff and want a security review, must understand that the reviewer is not an MPLS expert
[18:47:22] <jhutz> hta: I've been running an experiment in the general area. One of the things I've asked people to do is read the document and see if it's written in english
[18:47:59] <jhutz> ... we producing documents which are technicallly sound, butr we also have a responsibility to produce a document that is comprehensible by people who don't fully understand the technology
[18:48:07] <jhutz> ... that's a very different kind of review
[18:49:50] <jhutz> [I relayed dcrocker's comment]
[18:50:15] <jhutz> [XXX]: supporting harald, that's why I think broad review is important - non-experts provide useful review
[18:50:41] <jhutz> hta: ... one thing that we discovered on the iesg is that when people start revieing, the don't understand what to look for, and after having reviewed 20 documents, it's much more obvious
[18:50:43] --- memo56 has joined
[18:50:50] <jhutz> ... we need to make it clear that reviewers are made, not born
[18:51:28] <jhutz> [YYY]: I have a comment on the "name"... You're calling yourselves "graybeard" and "SIRS"; I have to wonder if you are excluding some people
[18:51:59] <jhutz> chair: Name "SIRS" was not intended to exclude people; it was something of an issue;
[18:52:06] <jhutz> [YYY]: Will you change the name?
[18:52:31] <jhutz> CHAIR: wrt SIRS? I don't know. But we'll make sure the name is not an issue for whatever this WG recommends, which may or may not be the same as SIRS
[18:53:05] <jhutz> [ZZZ]: I'm young enough remember reading my first RFC. I approached wtih some trepidation, then found it was just like a unix man page, and was happy. but some people may not like unix man pages.
[18:53:24] <jhutz> ... we need to be sure our work is accessible to technical professionals in the field; not necessarily brought up on CLI and nroff
[18:53:31] <dthaler> (that's Greg Daley)
[18:53:44] <jhutz> CHAIR: fair enough, but I'll make it clear that things like ASCII vs other and formatting are completely out of scope
[18:54:21] <jhutz> [AAA]: One issue that we want to avoid is WG's dismissing/ignoring comments because the wg agreed on that half a year ago. we need to balance between progress and cross-area comments
[18:54:31] <jhutz> [ZZZ]: density of text is more relevant than format/formatting
[18:54:51] <jhutz> CHAIR: does anyone want to comment on whether any of the issues we've raised are particularly more/less appropriate for _late_ reviews
[18:55:07] <jhutz> dmitri: I beliuve the set of people who review the document might not necessarily be the same for early vs late
[18:55:29] <jhutz> .... [couldn't follow that]
[18:55:57] <jhutz> alex zinin: In early review, we want to bring as many eyes as possible to the doc
[18:56:18] <jhutz> ... in later review, very important is consistency with IESG review. one problem is when directorate members bring something up, and
[18:56:27] <jhutz> ... then the IESG makes a completely different decision
[18:56:39] <jhutz> ... or there is an issue they think was addressed, and then the IESG says it wasn't.
[18:56:51] <jhutz> ... this is an issue with responsibility of AD's and how they delegate review tasks
[18:57:20] <jhutz> [BBB]: I think there is a conceptual difference between early/late review. Early there haven't been as many eyes, there are more likely to be substantive changes
[18:57:49] <jhutz> .... if this is done, then later comments are likely to be more clarification, changes in text, etc -- more editing than technical cahnges
[18:57:58] --- Milkshake has left
[18:57:58] <jhutz> ... might be useful for (late) reviewers to bring editing skills as well
[18:58:33] <jhutz> ... what might happen is you might find that reviews for early stages haven't caught somethihg. then it turns into an "early" review, instead of late; doesn't make much sense to treat document as complete and make editorial changes
[18:58:54] <jhutz> [CCC]: I learned here there might be early vs late review. Think early review should be done in the WG.
[18:59:06] <dcrocker> please channel: SIRS name. It's been proving an effective distraction. Objections have been roundly and persistently lodged. making acronyms is difficult. bullet-proofing a name against everyone's sensitivities is really difficult. That does not mean it should not be done, but it means that the answer is simply to have alternatives presented and eventually adopted.
[18:59:12] <jhutz> ... [???]
[18:59:49] <jhutz> [DDD]: I don't completely agree with that - I think it's important to have early review outside the WG
[19:00:02] <jhutz> ... important thing is to have others take a look at the document early, who are not involved in the work
[19:00:03] <jhutz> .
[19:00:06] <bert> We recently had a case where the MIB Doctors said MIB X is OK, but yet another IESG member found issues
[19:00:15] <dcrocker> The decision by those who signed up for SIRS was to wait on the name change until the activity became official IETF work. My own comment is that this will also be useful for making clear that it is a fresh effort.
[19:00:29] <jhutz> hta: did you mean that the review needs to take place by the wG members, or that discussion of the review need to take place in the wG.
[19:00:29] <jhutz> .
[19:00:35] <bert> in that the MIB did not represent all variations of the modeled technology.
[19:00:50] <bert> So the result was: MIB doctors say OK, and yet the doc cannot pass
[19:01:18] <bert> Point being (again): MIB doctors only review ONE or a FEW aspect(s) of the document
[19:02:35] <jhutz> [comments relayed]
[19:02:51] <jhutz> CHAIR: now to discuss documents that do exist
[19:03:07] <jhutz> [chair suggests maybe this is the time for dave to come back]
[19:03:25] <jhutz> draft-zinin-early-review-00
[19:03:37] <jhutz> draft-iesg-alvestrand-twolevel-00
[19:04:11] <jhutz> [EEE]: If you go to a WG, there's lots of discussion going on. Managed by WG chairs. Might not be enough when you get to releasing a document. extremely early review is done in the wg.
[19:04:35] <jhutz> ... "early" review as discussed here is where are we going with WG's in general, [???]
[19:04:45] <jhutz> ... when you accept as a wG document is when you have early reivew
[19:05:02] <jhutz> chair: The point is early _cross-area_ review. Maybe -00, maybe not;.
[19:05:04] <dcrocker> sorry. in the middle of our multi6 presentation. avri doria is presenting but I _really_ have to be here. I'm on jabber, tho.
[19:05:22] <jhutz> ... need to figure out how to tell what should be reviewed, how to find reviewers, etc
[19:05:41] <jhutz> ... when we talk about _early_ review, we mean early _cross-area_ review, beyond the folks actieuly participating in the WG
[19:06:01] --- hta has left: Replaced by new connection
[19:06:01] --- hta has joined
[19:06:01] --- hta has left
[19:06:02] <jhutz> ... doing cross-area review causes the "late surprise" problem where people outside think you've made a serious error
[19:06:11] <dcrocker> As I mentioned above on jabber, I think we should formulate a Guide for Reviewers, and maybe even add a Sunday tutorial.
[19:06:13] --- galvinjamesm has left: Disconnected
[19:06:15] <jhutz> ... early review helps reduce that, make changes when it is still easier to do so
[19:06:42] <jhutz> { that sounds like a good idea, dave]
[19:07:13] <dcrocker> separate from recommending reviewer group management structures, the BCP would be a separate "inherent" good.
[19:07:15] --- hta has joined
[19:07:22] <dcrocker> err, good output from this group.
[19:07:25] <jhutz> james polk: forgive my ignorance; is there a repository where if I write a draft, I can put it to have it be offered for review; a tracking mechanism to let people know it's being reviewed
[19:07:51] --- galvinjamesm has joined
[19:07:59] <jhutz> CHAIR: there is the SIRS list, which is a way of requesting a review; you can send a note to the SIRS list and ask for a review.
[19:08:10] <jhutz> ... informal process; don't have the sort of tools this WG will likely recommend
[19:09:08] <dcrocker> i gave url and mailto in my slides.
[19:09:14] <jhutz> polk: if that functionality is in SIRS, great.. If not, I think this WG should consider that as we
[19:09:40] <jhutz> hta: In the tools area... one problem we have is that in the limited-resource environment, we'de like to have some experimentation done before committing resources
[19:09:53] <jhutz> ... I sometimes wonder if we should handle reviews using weblog for instance
[19:10:14] <jhutz> ... I'd like to encourage people to run experiments; I think that the experience of 10 people doing 10 reviews using any mechanism is valuable; we should try
[19:10:32] <dcrocker> the sirs web and mailing list effort did in fact suffer from the lack of better tools. sirs folks discussed this fact. i don't know how to deal with that, short of comitting resources formally.
[19:10:38] <jhutz> ... I'm currently running my XXX experiment; visible on my (GEN area?) web site, with the simplest possible mechanism; it's a mailing list
[19:10:54] <jhutz> [dcrokcer; want me to relay any of this?]
[19:10:59] <dcrocker> yes, ploease
[19:11:14] <jhutz> dimitri?: What is the relationship between early review and area advisors for a WG
[19:11:39] <jhutz> CHAIR: fair question; the answer is probably up to us. we can suggest leaning more heavily on area advisors, make it orthogonal, or whatever
[19:11:44] <hta> http://www.alvestrand.no/ietf/gen/review-guidelines.html
[19:11:59] <jhutz> russ housley: getting security advisors for WG's is really hard, because you're asking for a time commitment much more than a single review
[19:12:10] <jhutz> ... often the ones we ask are not the ones most interested in making the WG progress
[19:12:28] * dcrocker says: i suggest that reviewers are for document content and advisors are for facilitating overall working group technical productivity.
[19:12:40] <jhutz> CHAIR: if we want to get reviews from people with sec expertise, maybe we want to lean on them less than sec area advisors
[19:13:02] --- Ted Faber has joined
[19:14:21] <jhutz> CHAIR: one of the things I noticed from SIRS activity was a lot of WG's didn't ask for reviews
[19:14:46] <jhutz> .. some dizcusion on the icar list focused on making sure wg's pay attention to the early review they get
[19:15:03] <jhutz> ... are there incentives/motivations we should have for encouraging WG's to get early reviews and pay attention to them?
[19:15:05] * dcrocker says: I strongly suggest that the review function be independent of the ietf administrative management team. The reviewing stuff is not about helping an "area" or dealing with any aspect of working group process. It is strictly for technical commentary.
[19:15:17] <jhutz> [FFF]: One incentive would be this could speed up later review.
[19:15:35] <jhutz> alex zinin: the experience is that the more review early in the process, the easier the document goes through the IESG. that's a real effect
[19:15:46] <jhutz> [thanks to alex for saying his name EVERY TIME he gets up; that helps a lot]
[19:16:01] <jhutz> [FFF]: need publicity; maybe a reminder to the wgchairs list
[19:16:18] --- Ted Faber has left
[19:16:40] <jhutz> greg davin{?}: I think the issue is if people don't pay attention to a review, and they believe they should be using the process, there's going to _be_ a lot more review; things will get send back again.
[19:16:43] * dcrocker says: getting review requests: a) the BCP will help motivate wg's I suspect, because it formalizes the expectation; and 2) ietf management culture _encouraging_ early reviews clearly will help. I think there is one more possibility that will help -
[19:16:58] <jhutz> ... this would cause discontent among reviewers, which might disincline them to do future reviews
[19:17:12] <dcrocker> Early work should have a design document that is in the realm of a detailed technical proposal. More than "idea" and less than "all the detail".
[19:17:36] <dcrocker> (early work comment meant for speaking)
[19:18:59] <jhutz> hta: everytime dave says something, I have to say the opposite
[19:19:20] <jhutz> ... if you want to create a body with power and responsibility outside the IETF management strucutre with power and responsibility, be careful what you ask for
[19:19:31] <jhutz> ... I think it would take longer than the other things proposed
[19:20:07] <jhutz> dmitri: if I have a review for a specific technology, but also involves an external standards body, how can this be integrated within the review process?
[19:20:36] <jhutz> CHAIR: for work which has impact in other standards bodies, should that be considered early? how? rasonable to consider
[19:21:05] <jhutz> CHAIR: there was a comment similar to that during charter discussion. suggestion was to put explicitly on the charter to improve cross-standards-body coordination/review
[19:21:15] * dcrocker says: what "power" does a review group have and what "benefit" is there in making such a purely technical effort subject to the varied goals of ietf management?
[19:21:15] <jhutz> ... I think we should be careful about that; there's a lot of "underwater stones" there
[19:21:45] <jhutz> ... I think we should concentrate on our internal review; how we can use our resources, before we get to our interactions with other standards bodies
[19:22:19] <jhutz> [HHH]: there are already wg's having this problem. what happens in that case?
[19:22:31] --- BP has left
[19:22:48] <jhutz> chair (alex): I'm not saying that's not important; just let's deal with our internal stuff first, then do the second step
[19:23:06] <jhutz> CHAIR (?): That may mean it's not this WG's job
[19:23:28] <jhutz> [Sorry; I'm missing a lot of names. Hope someone can fill them in]
[19:23:50] * dcrocker says: the other problem with placing the activity under the IESG is a kind of conflict of interest. Independent review needs to be _really_ indepent. Yet the IESG is a decision chain. If it is involved in the reviewing, then it must remove itself from any later conflict resolution.
[19:24:48] <jhutz> hta: if review is to make any sense, it has to influence the quality, relevance, timeliness of ourput. the managemt exists to influence the quality, relevance, timeliness of ietf output
[19:24:51] <hartmans> Why. Of course I don't see the value of independence in the review other than to get people who will see clarity problems.
[19:25:03] <jhutz> [JJJ]: Are you saying we can already ignore reviewers?
[19:25:04] <hartmans> (responding to David)
[19:25:12] <jhutz> hta: If you have reviewers who make a difference, they have power
[19:25:27] <jhutz> chair: asks that hta, dcrocker take this offline
[19:25:50] <jhutz> [now we really are going on to alex and harald's drafts]
[19:26:11] * dcrocker says: working group / iesg conflict is a fact of life. an independent review process provides _independent_ input to such conflict.
[19:26:17] <jhutz> Next: Alex Zinin on Area Review Teams draft-zinin-early-review-00.txt
[19:26:48] <jhutz> zinin: the motivation is to improve cross-area review of docs before they reach the IESG.
[19:27:03] <jhutz> ... here we're primarily talking about late review; some of the same mechanisms can be used for early review as well
[19:27:13] * dcrocker says: thanks for the channeling.
[19:27:24] <jhutz> ... review should improve quality of documents enough that time to go through IESG is decreased
[19:27:43] <jhutz> .... things go much smoother because ad's have received recommendations from expert groups and have already made up their mind
[19:27:47] <jhutz> ...
[19:28:05] <jhutz> ... basis of the proposal is experience with the existing directorate structures that have been described here
[19:28:16] <jhutz> ... work within the existing document approval structure
[19:28:33] <jhutz> ... assumption is that IESG remains _the_ body that reviews documents; AD's remain personally responsible for quality of docs they review
[19:28:48] <jhutz> ... each area creates an area review team "ART", with a well-known mailing list.
[19:28:52] --- Ralph has left
[19:28:58] <jhutz> ... many people don't know that directorates exist at all
[19:29:17] <jhutz> ... it is important that each area has a mailing list where a group of reviewers who provide recommendations to AD's can be reached
[19:29:29] --- droms has joined
[19:29:29] <jhutz> ... group should include at least AD's, but is expected to include other experts that the AD's select
[19:29:39] <jhutz> ... AD's are expected to delegate review functions to members
[19:30:07] <jhutz> ... when the document is referred ot the ART, two members are assigned to the document; others are encouraged to review but will not be held responsible for not reviewing it
[19:30:19] <jhutz> ... review process is initiated by WG chairs or by AD's for individual submission
[19:30:27] <jhutz> ... initation requires AD approval
[19:30:36] <jhutz> .... review process is automatically started for IETF last call
[19:30:58] <jhutz> ... at this point we send request to all AD review teams, so all AD's have recommendations from their ART's by the time the document reaches the iesg
[19:31:19] <jhutz> .... set of review teams in other cases is identified by WG chairs and AD's.
[19:31:27] <jhutz> ... initated through an email message
[19:31:43] <jhutz> ... AD either signs off on document; "I've looked at this before; we don't need more review" or assignes reviewers
[19:31:59] <jhutz> ... reviewers provide comments directly to the WG or authors; WG looks at them as part of normal WG process
[19:32:13] <jhutz> ... reviewrs summarize their recommendations.
[19:32:40] <jhutz> ... if the document is returning to the ART (having been there before), then the same reviewers should be assigned, to ensure consistency of review
[19:33:00] <jhutz> .. during IESG review, AD's have the right to either bring up comments that haven't been addressed, or override reviewers comments
[19:33:15] <jhutz> ... this makes it consistent with the AD's personal responsibility for review
[19:34:02] <jhutz> ... initiation of review process can happen before WG last call (feedbakck handled as part of normal WG process), during WG last call (recommended), or as part of AD review process
[19:34:10] <jhutz> ... or automatically as part of IETF last call
[19:34:23] --- leslie has joined
[19:34:27] --- lha has left
[19:34:29] <jhutz> ... it helps AD's to receive comments from ART's by the time the document is on the table for IESG discussion
[19:34:47] <jhutz> ... documenting review results is an important point here
[19:35:18] --- leslie has left
[19:41:08] <jhutz> ... wg charis would be held responsible (though can delegate) for tracking issues that have been brought up
[19:41:08] <jhutz> ... they would also summarize the resolutioin of issues when submitting to AD's
[19:41:08] <jhutz> ... for individual submissions, that would be the AD or an assigned ART member
[19:41:08] <jhutz> ... for other areas [missed this]
[19:41:08] <jhutz> ... Trust, Responsibility, Accountability:
[19:41:08] <jhutz> ... model has not changed - AD's still responsible. can delegate review functioin, but that doesn't remove responsibility
[19:41:08] <jhutz> ... AD's responsible for qualification of ART members.
[19:41:08] <jhutz> ... expectted to keep size and membership adequate to get quality reviews and keep load not too high for AD's and ART members
[19:41:08] <jhutz> ... Motivation and Credit:
[19:41:08] <jhutz> ... how do we acknowledge people who serve?
[19:41:08] <jhutz> ... recognition though formalization - since proposal is to establish ART teams as a formal structure, more people might be interested
[19:41:08] <jhutz> ... similarly, give dots to ART members, so they are identifyable
[19:41:08] <jhutz> ... acknowledgement in documents
[19:41:08] <jhutz> .... recognition (?) through exposure (plenary meetings)
[19:41:24] <jhutz> <test>
[19:41:32] <jhutz> [sorry; I missed most of the discussion, due to lost network]
[19:41:34] <dthaler> I think Alex clarified "plenary meetings" as open area meetings
[19:41:52] <jhutz> He did
[19:42:24] <jhutz> One question was about whether review groups would have to be dismissed and reselected whenever an AD changed; alex said he didn't think that would necessarily be required; it would be up to AD's.
[19:43:01] <jhutz> [KKK]: I'm concerned about selection of reviewers; maybe they should not just be appointed from above
[19:43:52] <jhutz> zinin: I'm interested in seeing people with a variety of opiniions in rtg-dir, after which ART is modelled, but there's also the issue of personal responsibility of AD's.
[19:44:07] <jhutz> ... hopefully AD's will _want_ to have a variety of opinions
[19:44:41] <jhutz> ... there was a suggestion that ART members be selected by the nomcom
[19:44:52] <jhutz> ... some folks who have served on nomcom may have comments about how well that scales
[19:44:53] * dcrocker says: KKK raises an important point. We can view reviews as a way of satisfying ADs or we can view them as part of a working group's internal effort to do good work. We say we want wg's to take more responsibility for their work and the quality of it. So _they_ should be responsibile for acquiring the reviews they need to produce good work.
[19:46:12] <jhutz> [I don't think that's his point; I'm doing a bad job of getting this across. He's concerned about selection of review teams being only by AD; I think he wants input "from below" into who's in ART's.]\
[19:46:26] <jhutz> [ I will of course forward your comment; you raise a good point]
[19:46:48] <jhutz> [I know I'm missing a lot of the exchange between KKK and alex on this issue]
[19:47:07] <hta> KKK = Olle Viktorson
[19:47:28] <jhutz> [thank you, harald. he's about half of the other 3-letter people in this transcript, too]
[19:47:46] <hta> he's fairly high up in Ericsson's development arm.
[19:48:13] <dcrocker> (no need to forward this: the debate between me and harald often is about the source of responsibility. I believe that core responsibility needs to be with the working group. harald believes it is with the iesg. grass roots vs. top-down)
[19:48:43] <jhutz> greg daily(?): sounds like there's concern that art teams will become a small personal army for the AD's. I don't think that's a problem; when folks do a review, they use their own technical expertise,
[19:49:05] <jhutz> ... which may not perfectly align with those of the AD. Review results are not guaranteed.
[19:49:21] <jhutz> ... choosing people may not be based on whether they agree with the AD's, but on qualifications
[19:49:33] <hta> dave: "the responsibility" is not a well defined term. The WG has responsibility for making the WG produce. The AD has responsibility for making the IETF produce. Scopes are different, but review is a tool that can be used in both cases.
[19:49:59] <jhutz> zinin: I hope there will be a bidirectional process. Reviews when they comment on documents influence the AD
[19:50:21] <jhutz> ... but also there is education; the AD educates the ART members, and the ART members each other, about what's important, what to look for, etc.
[19:50:32] <hta> (that was my comment to dave. orthogonal to the icar wg, but a point illustrating our disagreements)
[19:50:33] <jhutz> ... so ART members learn what the AD expects and how to review documents
[19:50:49] <dcrocker> harald - yes, responsibility is a fuzzy term. and in fact i mean both more and different from only that. but this is interactive, so it's tough to be that precise, absent significant discussion.
[19:51:08] <jhutz> [folks, please use '-' to address people; reserve ':' for indicating who is speaking]
[19:51:30] <jhutz> Next: harald alvestrand on draft-ietf-alvestrand-twolevel-00.txt (if I got this right)
[19:51:56] <jhutz> hta: mainly about noting the way the iesg works when it reviews documents, and noting this does not have to be the same as [xxx]
[19:52:12] <jhutz> ... put together teams, and try to make the collective judgement of that team cover all the aspects that need to be covered
[19:52:37] <jhutz> ... this is for _late_ review. WG thinks they've covered everythtng; want a group to say from the global persepctive that everything is in fact recovered
[19:52:39] <dcrocker> (if I violated typing protocol, i'm still not understanding how. was my "harald -" a syntactic error?
[19:52:45] <dcrocker> )
[19:53:17] <jhutz> ... noted that people tend to adjust to each other's reviews -- "I don't have to look for X, so-and-so will find it"
[19:53:23] <jhutz> [no; but harald did]
[19:53:49] <jhutz> ... asking you to look whether this is a worhtwhile direction to experiment in
[19:54:12] <jhutz> ... note document proposes giving review groups ability to approve documents [did I get this right]; that is out of scope for here
[19:54:27] <jhutz> [LLL]: how is it determined which team a document is reviewed by?
[19:54:50] <jhutz> hta: throw dice, except that when a group of docs have strong internal refrerences, they should be treated by the same review team
[19:55:29] <jhutz> [LLL]: concern I have with this is that not all review teams will be equal, not just in terms of quality, but in terms of what direction
[19:55:42] <jhutz> ... one team may say "this is a good idea" while another wants to go another direction
[19:55:54] <jhutz> ... who has ultimate responsibility? central management important
[19:56:16] <jhutz> ... can't "shop around" for a team that is friendly to what you want to do
[19:56:21] <jhutz> ... need some way to counter that
[19:57:01] <jhutz> hta: if you're going to have more than one group, you need to expect and deal with variations and judgement.
[19:57:12] * dcrocker says: shopping around is fine, if the reviews must be credible on their own right, rather than by virtue of some organizational affiliation. If a wg gets only friendly reviews, they have not done their job.
[19:57:21] <jhutz> ... need to have someplace in the system that can take a review team and say "you're out of line"
[19:57:35] <jhutz> ... i believe in central authoritty (we have no kings, but we have central authority)
[19:57:40] <jhutz> ... you had another comment?
[19:58:06] <jhutz> [LLL]: if the team is chosen randomly, and there are differences between teams, then WG may not know how to construct a document that will pass
[19:58:14] <jhutz> ... if it is not random, there may be forum shopping
[19:58:25] <jhutz> hta: if we have multiple teams, we may have more guidelines
[19:58:59] <jhutz> greg daily(?): when doc goes to ietf last call, there can be a dynamic selection of reviewers ...
[19:59:04] <dthaler> LLL = me (Dave Thaler)
[19:59:49] <jhutz> [argh; missed that comment]
[19:59:55] <jhutz> ]thanks, dthaler]
[20:00:08] <dthaler> (btw, it's Greg "Daley")
[20:00:20] <jhutz> []: question about whether group has authority to review a document
[20:00:32] --- bert has left: Disconnected
[20:00:58] <jhutz> [MMM]: you said you weren't sure about a group having the authority to review a document being in scope. what's the point if they don't have authority? you should decide here
[20:01:25] <jhutz> hta: we might be able to do it in two steps...
[20:01:38] <jhutz> ... we have three possible outcomes - 1 is that it doesn't work at all
[20:01:57] <dcrocker> differences between sets of reviews - this is goodness, not badness. the job of a wg is to resolve differences and considerations. if reviews do not differ, they probably are not adding much benefit.
[20:02:06] --- sob has joined
[20:02:13] <jhutz> ... another is that we get documents reviewed, and in some large percentage the IESG concludes the review wasn't thorough enough and there are things that neede to be dealt with.
[20:02:59] <jhutz> ... but if we run the experiment and find that in ~95% of cases, the review team finds the problems, and the IESG looks at it and review team has looked at it., comments dealt with, review team happy, then why not move the authority
[20:03:21] * dcrocker says: has anyone noticed that even a question of technical reviews has now been reduced to a focus on "authority"?
[20:03:28] <jhutz> [ I think questioner is confused. what hta said earlier was that the question of whether review teams would have the authority to _approve_ documents is out of scope for icar]
[20:03:37] <jhutz> [oops; meeting just adjourned]
[20:03:41] <jhutz> [nearly]
[20:03:46] <dcrocker> heh. heh.
[20:04:55] --- sakai has left
[20:05:39] <jhutz> {dave - I think dthaler's concern was about having multiple review teams that can approve documents, and WG's "forum shopping" to select a review team that is likely to approve their document. Whereas today you don't get to pick which IESG reviews your document (unless you delay it a long time!:-) }
[20:10:18] --- Bill has left
[20:10:53] <dcrocker> it's clear there is a _very_ different view of reviews and their "power".
[20:15:10] --- droms has left
[20:15:56] --- galvinjamesm has left
[20:16:35] --- sob has left
[20:17:12] --- Melinda has left
[20:19:02] --- memo56 has left
[20:27:40] --- dthaler has left: Disconnected
[20:31:29] --- hartmans has left
[20:33:55] --- hta has left: Disconnected
[20:45:33] --- jhutz has left
[21:05:52] --- dcrocker has left: Disconnected
[21:58:05] --- bert has joined
[21:59:43] --- hta has joined
[22:02:26] --- bert has left
[22:11:36] --- hta has left: Disconnected
[22:14:04] --- hta has joined
[22:17:56] --- dthaler has joined
[22:19:29] --- dthaler has left
[22:20:11] --- hta has left: Replaced by new connection
[22:20:11] --- hta has joined
[22:20:11] --- hta has left