IETF 112 post-meeting survey

The survey results for the IETF 112 post-meeting survey are now available on a web-based interactive dashboard. As always, we are very grateful for the detailed feedback that we have received and will continue to process over the next few months. The commentary below highlights where changes we have made based on feedback have been a success, and areas we still need to work on.

Analysis

In total 160 responses were received. Of those 157 participated in IETF 112 from a population of 1175 giving a margin of error of +/- 7.28%. The total numbers of meeting participants was lower than for IETF 111 (1329 for IETF 111, 1196 for IETF 110, 1282 for IETF 109) while the survey response rate is at the same level as for IETF 111, which was significantly down on previous meetings (166 for IETF 111, 299 for IETF 110, 258 for IETF 109).

The results for satisfaction questions include a mean and standard deviation using a five point scale scoring system of Very satisfied = 5, Satisfied = 4, Neither satisfied nor dissatisfied = 3, Dissatisfied = 2, Very dissatisfied = 1. While there’s no hard and fast rule, a mean of above 4.50 is sometimes considered excellent, 4.00 to 4.49 is good, 3.50 to 3.99 is acceptable and below 3.50 is either poor or very poor if below 3.00. The satisfaction score tables also include a top box, the total of satisfied and very satisfied, and a bottom box, the total of dissatisfied and very dissatisfied, both in percentages.

In this commentary a comparison is made with the IETF 111 Meeting Survey results using a comparison of means that assumes the two samples are independent even though they’re not but neither are they fully dependent. A dependent means calculation may give a different result. Some comparisons may be made using a comparison of proportions.

Overall satisfaction

The mean satisfaction score for IETF 112 (Q10) was 4.15 with 87.5% either ‘Satisfied’ or ‘Very satisfied’. This is a statistically insignificant increase from the 4.13 mean satisfaction score for IETF 111.

Demographics

The geographics spread of participants (Q1) was similar to IETF 111, with proportionately many more particpants from Africa and the Middle East. Fewer newcomers particpated in IETF 112 (Q4), and that was reflected in an uptick in the number of participants who reported some form of active engagement in the IETF (e.g. posting to a mailing list) (Q2). As noted above, overall numbers were down.

Preparedness

Satisfaction with the guides for chairs and participants (Q6) remains good at 4.41 (4.42 for IETF 111) as was satisfaction with the new "New Participants Agenda" at 4.36 (Q7a). Preparedness (Q8) was also much the same for IETF 112. A recurring suggestion in the comments is for a permanentley open Meetecho session to be used for practice.

Satisfaction with the agenda

Overall satisfaction with the IETF 112 agenda was at 4.11 (Q12), a statistically significant increase from 3.91 for IETF 111 and now rated as good, up from acceptable. Looking at the individual parts of the agenda (Q11), none of the satisfaction scores had a statistically significant change with all the session parts rating good and all the peripheral parts rating as acceptable, except side meeting which rate as poor 3.46 and opportunities for social interaction remained at very poor 2.79.

Satisfaction with the structure of the meeting

Overall satisfaction with the structure of the IETF 112 meeting (Q14) was 4.23 up from 4.08 for IETF 111 though not a statistically significant increase. However, looking at the individual parts (Q13) there were statistically significant increases in satisfaction for the policy of scheduling online meetings in the timezone of the in-person meeting that they replace, for the overall length of the day, for the number of parallel tracks and for starting at 12pm "local time". This is almost the exact reverse of the changes between IETF 110 and 111 were the same measures, except that of parallel tracks, had statistically significant drops. While this has not been formally tested, it does appear that meetings with 9 parallel tracks rather than 8, rate lower in multiple aspects.

Further analysis is required to see if satisfaction with the policy of rotating timezones is linked to the timezone of the meeting.

Impact of being fully online

This was the second time that this question (Q15) was asked and the answers were much the same. 6% of respondents (9 people) answered that they would have participated less had this been an in-person meeting and 2% answered the opposite.

Sessions

37% experienced no session conflicts (Q18) and 21% just one conflict, in comparison to IETF 111 with 41% and 16%. However, satisfaction at conflict avoidance was up to 4.00 (Q20), a statistically significant increase from 3.76 for IETF 111. It is likely this was due to the moving of the plenary to before the meeting, but there is no data to support that assumption.

Moving the plenary

For IETF 112 the plenary session was moved to the week before. Satisfaction with this move was only acceptable at 3.70 (Q13a) but satisfaction with the agenda comparable at 3.96 to IETF 111 at 3.91.

Participation mechanisms

Satisfaction with Meetecho remains good at 4.36, while satisfaction with the audio streams anbd YouTube jumped to 4.41 from 3.84 and 4.09 respectively (Q22). Satisfaction with Gather fell to a poor 3.40, from an acceptable 3.77. Jabber remains in the acceptable range at 3.75.

A recurring issue raised in the comments was people coming unstuck with the process for uploading presentations. Meetecho have already implemented automatic preload of slides with the manual option remaining for late processing.

Problem reporting

Satisfaction with our response to problem reporting jumped to 4.43 from 4.00 for IETF 111 (Q25) though the number of people responding to this question remains small.

Final feedback

High satisfaction with the meeting was reflected in the positive comments. Despite this, many of us yearn to return to in-person meetings.