Report on the RFC 8989 experiment

1 Sep 2021

RFC 8989 tasks the IESG with publishing a report on the experiment of defining additional criteria for NomCom eligibility.

Section 2 specifically calls for:

                                   Points to be considered are
whether the experiment has produced a sufficiently large and diverse
pool of individuals, whether enough of those individuals have
volunteered to produce a representative NomCom with good knowledge of
the IETF, and whether all the goals in Section 3 have been met.  If
possible, a comparison with results from the previous procedure
(i.e., RFC 8713) should be made.

This document constitutes this report. It was authored by the IESG in collaboration with the 2020-21 NomCom Chair, Barbara Stark, and the 2021-22 NomCom Chair, Gabriel Montenegro. All text in italics in this report is quoted from RFC 8989.

RFC 8989 eligibility paths

RFC 8989 defines an experiment that temporarily – for the 2021-22 NomCom – extends the NomCom eligibility criteria in RFC 8713 by adding two more “paths” towards eligibility, resulting in the following three:

• Path 1: attendance A person has attended three out of the last five IETF meetings, including meetings held entirely online. (This is the conventional path under RFC 8713, extended to also count remote participation.)
• Path 2: WG leadership A person has been a Working Group Chair or Secretary within the three years prior to the day the first call for NomCom volunteers was sent to the community.
• Path 3: authorship A person has been a listed author or editor (on the front page) of at least two IETF Stream RFCs (or IESG-approved Internet-Drafts in the RFC Editor queue) within the last five years prior to the day the call for NomCom volunteers was sent to the community.

NomCom eligibility and pool size analysis

The 2021-22 pool of NomCom volunteers consists of 116 IETF participants. For comparison, the 2020-21 pool consisted of 143 participants, and over the last ten years, the NomCom pools had sizes from 93-203 participants, with an average size of 136 at a standard deviation of 37.

This seems to show that the RFC 8989 experiment has not immediately increased the number of IETF participants that volunteered for the 2021-22 NomCom.

Path 1 alone qualifies 922 volunteers, but path 3 extends NomCom eligibility to 371 additional participants that are not eligible under path 1 alone. However, only three of them volunteered for the NomCom pool.

This seems to show that although path 3 results in a significant increase in NomCom eligibility, it does not seem to result in a corresponding increase to the NomCom volunteer pool. It remains unclear why the fraction of participants eligible via path 3 that volunteered for the NomCom pool is so much lower than the fraction of path 1 participants that did so.

RFC 8989 Appendix A shows that in 2020, there were 806 participants eligible via path 1, which path 2 and 3 together extended by 1107 participants. In 2021, 922 were eligible via path 1, but paths 2 and 3 together only led to an increase of another 394. The cause of this drop from 2020 is also unclear.

NomCom pool diversity

Given that the 2021-22 NomCom volunteer pool composition is mostly unaltered compared to earlier years where eligibility was determined under RFC 8713, and due to the limited effectiveness of path 2 and 3 eligibility qualifications, it seems unlikely that this year’s pool has a significantly different diversity than past pools.

No detailed analysis was carried out, however, due to a lack of data and of a definition of “diversity” for the purposes of the RFC 8989 experiment (other than diversity through qualifying by three different paths). For participants eligible via path 1, some limited information is available via their meeting registration records, but even that is very minimal (e.g., due to GDPR obligations). For participants only eligible via paths 2 or 3, even that information is unavailable.

The community has discussed one aspect of diversity, diversity in volunteer affiliations, in a thread on the IETF discussion list.

Goal accomplishment

The goals behind the expanded NomCom eligibility in Section 3 of RFC 8989 are quoted below, followed by a discussion as to which degree they were achieved:

* Mitigate the issue of active remote (or, rarely, in-person) participants being disenfranchised in the NomCom and recall processes.

It appears that the provision in path 1 to count attendance in fully online meetings towards NomCom eligibility met this goal.

No recall processes have been started, so no statement can be made as to RFC 8989’s effect on them.

* Enable the selection of a 2021-2022 NomCom, and possibly a 2022-2023 NomCom, when it is impossible for anyone to have attended 3 out of the last 5 IETF meetings in person.

A 2021-22 NomCom was selected, out of a pool of reasonable size. That was at least partially enabled by the RFC 8989 addition to path 1, i.e., to count attendance at fully online meetings towards NomCom eligibility. This change should be carried forward at least to the 2022-2023 NomCom, because eligibility for that NomCom will (under path 1) be determined by attendance to IETFs 109-113, where at least IETFs 109-112 will have happened fully online.

* Prepare for an era in which face-to-face plenary meetings are less frequent (thus extending the issue to many, perhaps a majority, of participants).

The addition to path 1 to count participation in fully online meetings towards NomCom eligibility seems to enable flexibility towards having fewer meetings with an in-person component.

* Ensure that those eligible have enough current understanding of IETF practices and people to make informed decisions.

The 2021-22 NomCom Chair reports that there has been no indication that any of the NomCom members have demonstrated any lack of the desired understanding and that their combined IETF expertise and experience is significant. Since the members are a random sample of the volunteer pool, it could hence be argued that the pool of volunteers seems to have had the desired characteristics. Nothing can be said regarding the “current understanding” of the entire set of eligible participants.

* Provide algorithmic criteria, so that the Secretariat can check them mechanically against available data.

The secretariat was able to check the criteria (“paths”) defined in RFC 8989 mechanically, so this part of the experiment was a success.

Discussion

Per RFC 8989, the IESG should begin a community discussion of whether to:

1. Amend [RFC8713] in time for the 2022-2023 NomCom cycle; or
2. Prolong the current experiment for a second and final year with additional clarifications specific to the 2022-2023 cycle; or
3. Run a different experiment for the next nominating cycle; or
4. Revert to [RFC8713].

As stated above, the provision in RFC 8989 to count attendance to such meetings towards NomCom eligibility is important to carry forward at least to the 2022-23 NomCom cycle. This seems to rule out option (4).

The options are hence to (1) amend RFC 8713 by permanently counting attendance to fully online meeting towards NomCom eligibility, (2) extend the RFC 8989 experiment to the 2022-23 NomCom cycle, or (3) define a different experiment.

The IESG’s suggestion to the community, as a starting point for discussion, would be to go with option (2), i.e., to prolong the RFC 8989 experiment as-is for the 2022-23 NomCom cycle. This will not only achieve the immediate goal of retaining the viability of path 1 for eligibility in the face of upcoming fully online IETF meetings, but will also gather additional data on whether the other paths gain importance as more meetings are held online.

It would be useful to understand whether initial consensus exists in IETF that remote attendance should count towards NomCom eligibility going forward. If the RFC 8989 experiment is prolonged for the 2022-23 NomCom, this question would need to be asked again next year; but it would be useful to have a sense of where the community is on this question at this time.

If the RFC 8989 experiment is continued for another year, it may be useful to have the 2022-23 NomCom Chair experiment with how the Call for Volunteers is disseminated. Some potential changes compared to 2021 could be to:

• Send a personal invitation to all participants eligible under path 2, in case some of them would otherwise remain unaware of their eligibility. (Or, send a personal invitation to anyone eligible.)
• Change the wording in the announcement to reflect the expected time commitment during November 2022 more accurately. It is possible that some of those eligible under the various paths did not plan on attending IETF 112 (November 2021) either remotely or in person, and hence did not volunteer for the pool.
• Enable a person registering for IETF 114 (July 2022) to volunteer for NomCom at the time of registration. The ability to volunteer for NomCom during registration was present for several years prior to 2021, but was absent from the IETF 111 registration form. This may have been a factor in the decrease of volunteers. NomCom 2020-21 received 61 volunteers based on the IETF 108 registration form.

It may also be useful to collect data on how many of the eligible participants and eventual volunteers are subscribed to any of the the mailing lists (ietf-announce, etc.) the Call for Volunteers is usually disseminated to, to see if there is any correlation between the eligible pool, volunteers, and list subscription. This could drive additional experimentation on how the call for volunteers is advertised.

To send questions or comments on this report, please use the eligibility-discuss discussion list