IETF 112 Plenary Experiment Evaluation

When we announced the experiment, we announced five evaluation criteria. All of those criteria were imperfect, and there is no target number of improved metrics to declare the experiment a success. Nevertheless, the results are below.

1. An improvement in survey responses reporting session conflicts compared to previous IETF online meetings.

The survey data don’t see any clear change in conflicts.

It is possible that the large 2-5 and 6-10 bins are masking some improvement, but the IESG concludes that there is no change in the survey responses as a result of this experiment.

2. Positive response to a new survey question about subjective satisfaction with the format change.

The survey indicated a mean score of 3.7 on satisfaction with "Holding the plenary the week before the IETF meeting", with a roughly 3:1 satisfied:dissatisfied ratio.

The IESG concludes that there is strong community sentiment in favor of the change, although it is possible that the positive responses are concentrated among those with no intention of attending the plenary, and the negatives among the minority who usually do attend.

3. Elimination of a ninth track, and a reduction in formal conflicts in the final agenda compared to previous online meetings.

There was no ninth track. The scheduling tool, which has not remained completely stable over time, tracks several different types of conflicts based on information provided in session requests.

Other efforts to reduce conflicts over this time, including a variable number of session requests, further complicate any conclusion. Nevertheless, the IESG concludes that the plenary experiment did coincide with fewer conflicts reported by the software.

4. Little or no reduction in plenary attendance (< 15%) compared to other online plenaries in European time zones (i.e. IETF 108 and 110) 

There were 299 attendees at the IETF 112 plenary, a drop in absolute terms of 23% from IETF 110 and 32% from 108. In relative terms to the number of attendees:

The IESG concludes that the experiment did have a significant negative impact on plenary attendance, although future attempts might reduce the extent that participants are surprised by the change and unintentionally miss the plenary.

5. The subjective experience of the IESG and Secretariat in attempting to minimize conflicts during IETF 112.

Several members of the IESG expressed that the IETF 112 scheduling process was considerably easier and required fewer tradeoffs. Some Area Directors think that this is mostly due not to the experiment, but instead because of soft limitations on session requests instituted before IETF 112.

Summary

The IESG’s consensus is that the plenary experiment did coincide with a reduction in scheduling conflicts, and most IESG members believe the experiment was an important cause. However, we also believe that the attendance drop at the IETF 112 plenary was unacceptable, although it might not be a permanent feature of this innovation.

We do not believe that the experiment results justify changing standard procedure for fully remote meetings, though we may conduct further experiments to reduce the disparity in attendance.

Regardless of this experiment outcome, the IESG fully intends for hybrid meetings, including IETF 113, to hold plenaries during the meeting week.

Acknowledgements

Thanks to Liz Flynn, Amy Vezza, and Jay Daley for helping to collect the data.