Skip to main content
  • IETF Meetings recording playback system now open source

    The source code of the playback system for the recordings of IETF meeting sessions was recently released by Meetecho under an open source license, and the IETF has now deployed its own instance of the system.

    2 Oct 2024
  • Workshop on the Next Era of NEtwork Management OPerationS (NEMOPS)

    A workshop organized by the Internet Architecture Board (IAB) aims to chart a path for the development of future network management protocols and techniques. The Next Era of Network Management Operations (NEMOPS) workshop will begin by assessing the impacts of the previous IAB workshop on both network operations and protocol development.

    20 Sep 2024
  • RFC data visualizations accessibility review

    RFCs are freely available to download, copy, publish, display and distribute. One benefit of providing free access to RFCs is that they can be used by the largest number of people possible to build a better Internet for all. To truly include as many people as we can, the IETF community has consistently tried to uphold accessibility standards for the publication of RFCs.

    10 Sep 2024
  • Consultation on the Second IASA2 Retrospective

    The IETF Administration LLC is soliciting community feedback on the second retrospective on the IETF Administrative Support Activity (IASA 2.0). This follows our first retrospective from 2021.

    4 Sep 2024
  • IETF email service outage deferred

    The scheduled outage of the IETF email processing system has been deferred again to allow time to complete further work discovered to be required for a successful transition.

    3 Sep 2024

Filter by topic and date

Filter by topic and date

IETF 118 post-meeting survey

30 Nov 2023

IETF 118 Prague was held 4-10 November 2023

The results of the IETF 118 Prague post-meeting survey are now available on a web-based interactive dashboard. Thank you to all of you who responded to this survey as we use your views to continually adjust the meeting experience.

Analysis

We received 336 responses, all of whom participated in IETF 118, 291 onsite and 45 remote. As no the respondents did not participate in IETF 118 the specific questions for them are not shown in the dashboard. With 1806 registered participants, this gives the survey a maximum margin of error of +/- 4.82% (4.9% for onsite and 14.17% for remote).

The results for satisfaction questions include a mean and standard deviation using a five point scale scoring system of Very satisfied = 5, Satisfied = 4, Neither satisfied nor dissatisfied = 3, Dissatisfied = 2, Very dissatisfied = 1. While there’s no hard and fast rule, a mean of above 4.50 is sometimes considered excellent, 4.00 to 4.49 is good, 3.50 to 3.99 is acceptable and below 3.50 is either poor or very poor if below 3.00. The satisfaction score tables also include a top box, the total of satisfied and very satisfied, and a bottom box, the total of dissatisfied and very dissatisfied, both in percentages. Please note that a small number of questions are on a four point scale.

Question changes since the last survey

For this survey we added some new questions about side meetings as this remains an ongoing area of concern. Some questions were removed (e.g. satisfaction with 'new' WGs) to keep the survey of a reasonable length. Not asking about the venue overall was an inadvertent omission.

Actions taken following the last survey

For this meeting, we made the following changes, prompted by survey feedback:

  • Extended the meeting to a full day on Friday to provide for more sessions, partly to ease pressure on sessions and partly to give more options to avoid conflicts.
  • Added a new Farewell Reception after the meeting closed on Friday to provide an opportunity for people to unwind and say goodbye.
  • Added an alcohol-free event in the form of a movie night.

Unfortunately, this meeting did not have a meeting host and so there were no free t-shirts and no social event.

Satisfaction

Overall satisfaction is 4.42 which is a very good result. With some key exceptions, the satisfaction scores remain high, reflecting the various improvements made since we returned to onsite meetings.

The table below shows the satisfaction scores for the last six meetings, along with colour coded indicators for the five point scale above: excellent (🔵), good (🟢), acceptable (🟡), poor (🔴), very poor (⚫️)

Satisfaction scores for the last six meetings
IETF 118 Prague IETF 117 San Francisco IETF 116 Yokohama IETF 115 London IETF 114 Phila. IETF 113 Vienna
Overall satisfaction 4.42 🟢 4.30 🟢 4.30 🟢 4.28 🟢 4.19 🟢 4.36 🟢
AGENDA
Overall agenda 4.20 🟢 4.16 🟢 4.18 🟢 4.22 🟢 4.06 🟢 4.16 🟢
Sessions for new WGs - 4.19 🟢 4.17 🟢 4.12 🟢 4.15 🟢 4.18 🟢
Sessions for existing WGs - 4.22 🟢 4.22 🟢 4.22 🟢 4.10 🟢 4.24 🟢
Sessions for WGs 4.32 🟢 - - - - -
BOFs 4.12 🟢 3.95 🟡 4.11 🟢 4.10 🟢 4.09 🟢 4.04 🟢
Sessions for existing RGs - 4.12 🟢 4.14 🟢 4.10 🟢 3.95 🟡 4.13 🟢
Sessions for RGs 4.09 🟢 - - - - -
Plenary 3.94 🟡 3.99 🟡 3.98 🟡 3.98 🟡 3.98 🟡 3.94 🟡
Side meetings 3.79 🟡 3.75 🟡 3.73 🟡 3.81 🟡 3.73 🟡 3.52 🟡
Hackathon 4.27 🟢 4.25 🟢 4.34 🟢 4.35 🟢 4.30 🟢 4.09 🟢
HotRFC 3.90 🟡 3.89 🟡 3.84 🟡 4.21 🟢 3.94 🟡 4.17 🟢
Pecha Kucha 4.00 🟢 4.15 🟢 - - - -
Office hours 4.07 🟢 3.98 🟡 4.23 🟢 4.00 🟢 4.09 🟢 3.96 🟡
Opportunities for social interaction 4.06 🟢 4.11 🟢 3.72 🟡 3.98 🟡 3.89 🟡 3.51 🟡
STRUCTURE
Overall meeting structure 4.34 🟢 4.28 🟢 4.28 🟢 4.28 🟢 4.19 🟢 4.26 🟢
Start time 4.49 🟢 (9:30am) 4.28 🟢 (9:30am) 4.16 🟢 (9:30am) 4.28 🟢 (9:30am) 4.20 🟢 (10:00am) 4.12 🟢 (10:00am)
Finish at 5pm Friday 3.65 🟡 - - - - -
Length of day 4.20 🟢 4.30 🟢 4.30 🟢 4.32 🟢 4.10 🟢 4.20 🟢
Number of days 4.18 🟢 (5+2) 4.27 🟢 (5+2) 4.30 🟢 (5+2) 4.32 🟢 (5+2) 4.30 🟢 (5+2) 4.23 🟢 (5+2)
Session lengths 4.38 🟢 (60 / 90 / 120) 4.41 🟢 (60 / 90 / 120) 4.36 🟢 (60 / 90 / 120) 4.32 🟢 (60 / 90 / 120) 4.25 🟢 (60/120) 4.31 🟢 (60/120)
Break lengths 4.38 🟢 (30/90) 4.32 🟢 (30/90) 4.38 🟢 (30/90) 4.36 🟢 (30/90) 4.25 🟢 (30/90) 4.16 🟢 (30/60)
Number of parallel tracks 3.94 🟡 (8) 4.08 🟢 (8) 4.01 🟢 (8) 3.90 🟡 (8) 3.86 🟡 (8) 3.92 🟡 (8)
PARTICIPATION MECHANISMS
Meetecho 4.47 🟢 4.35 🟢 4.45 🟢 4.45 🟢 4.23 🟢 4.36 🟢
Gather - 3.52 🟡 3.46 🔴 3.37 🔴 3.06 🔴 3.04 🔴
Zulip 3.94 🟡 3.66 🟡 3.77 🟡 3.73 🟡 3.56 🟡 2.91 ⚫️
Jabber - - - - - 3.80 🟡
Audio streams 4.27 🟢 4.02 🟢 4.21 🟢 4.04 🟢 4.05 🟢 4.14 🟢
YouTube streams 4.36 🟢 4.32 🟢 4.36 🟢 4.25 🟢 4.22 🟢 4.25 🟢
CONFLICTS
Conflict avoidance 4.00 🟢 3.90 🟡 3.94 🟡 3.91 🟡 3.78 🟡 3.89 🟡
VENUE & ACCOMM
Overall accommodation 4.34 🟢 4.07 🟢 4.09 🟢 - - -
Overall venue - 3.90 🟡 - - - -
Location 4.52 🔵 3.60 🟡 - - - -
Venue facilities 4.43 🟢 4.07 🟢 - - - -
Cost of rooms 3.54 🟡 2.87 ⚫️ - - - -
Availability of rooms 4.32 🟢 4.07 🟢 - - - -
ONSITE
Overall 4.50 🔵 4.29 🟢 - - - -
Badge collection 4.70 🔵 4.69 🔵 - - - -
WiFi 4.17 🟢 3.98 🟡 4.06 🟢 4.10 🟢 3.82 🟡 -
QR Codes 4.18 🟢 4.11 🟢 - - - -
Break F&B 4.05 🟢 4.44 🟢 - - - -
Breakout seating 3.80 🟡 4.08 🟢 - - - -
Signage 4.15 🟢 4.22 🟢 - - - -
Coffee carts 4.64 🔵 4.56 🔵 - - - -
Childcare 4.12 🟢 4.06 🟢 - - - -
Welcome reception 4.08 🟢 - - - - -
Farewell reception 4.34 🟢 - - - - -

Areas for improvement

Cost of rooms

We have only recently started to track this and so this stage we can only note that satisfaction is on the low side, particularly as we book meetings some time in advance.

Side meetings

While there is a slight uptick in satisfaction, though not significant, this remains an issue. For this survey we asked some satisfaction questions to help understand the problem better:

Satisfaction scores about aspects of side meetings
Subject Satisfaction score
The number of side meetings 3.71
The scheduling of side meetings 3.51
The length of side meetings 3.93
The remote participation technology in side meetings 3.41
The content of side meetings 3.95

These results suggest that the problem is around the organisation and support of side meetings rather than the content. There are lots of comments about side meetings that seem to indicate that the current piecemeal approach to improvement may not be enough and more comprehensive review may be needed.

Conflict avoidance

We again follow the new process to reduce conflicts as described in our blog post, and extended the meeting on Friday afternoons. This did not lead to a statistically significant increase in satisfaction with conflict avoidance. This may be something that simply cannot be improved without adversely affecting some other aspect of the meeting, such as length of the day.

Zulip

We have seen a statistically significant uptick in satisfaction for Zulip, as we hoped we would see as people became more familiar with it. However the satisfaction score is still not high enough. The comments on Zulip vary from those with views on the integration of chat into Meetecho, and those who use Zulip directly and would prefer a different product. It is really not clear if it possible to choose another product that would get a higher satisfaction score given that there are so many different products out there that people use and prefer.

Breakout seating

We thought we had this one sorted but apparently not. Back to the drawing board to understand exactly what people want, which appears to be seats within ~10 meters of the meetings rooms.

Extending to Friday afternoon

This meeting experimented with a full day on Friday and the satisfaction score for that is only acceptable not good. Given that these sorts of changes take some time to bed in, we will monitor that for the next couple of meetings. There are a number of comments that this was not advertised enough, but it is mentioned clearly in all of our emails about the meeting organisation.

Individual comments

The individual comments cover a number of key themes, some of which have already been covered above:

  • The location of the venue. Almost all of the comments here we positive, both about the city and the ease of getting there. Unfortunately, we will not be returning for several years at least as the venue now charges too much for our budget.
  • Cost of registration, accommodation and travel. There are multiple comments that the cost of participating onsite is too high, particularly from those in the academic community. The various recommendations include: a more general bursary program, lower registration fees, and lower cost/quality venues.
  • Food and beverages provided. While there is a notable amount of positive feedback, there is also a clear theme that the provision for vegans and vegetarians was insufficient and that more care needs to be taken. There is a specific request for batch brew coffee that will be considered and a request that the baristas provide tea, which we almost fell for.
  • Crowding. There are multiple concerns expressed about the density of some events, in particular the welcome reception. The background here is that this was planned to be in a larger room but this was swapped with the Hackathon room because of the very large number of people who signed up for the Hackathon. The actual number who turned up for the Hackathon was significantly lower than the number that signed up, but even then that room was crowded. The underlying issue here is how to estimate in advance the number of people coming to the large events, which is something we have to do many years in advance in order to book the space. Adding new space nearer to the time is almost impossible because it has normally been sold by the venue to other clients.

Why people participated remotely

For the second time, we asked people who participated remotely (Q5), why they did and if they would have preferred to participate onsite (Q5a). Once again, the major factor, cited by 27 of the 45, was the lack funding to travel, all of whom would have participated onsite if they could have. This means that there are several hundred who would have participated onsite if funding was available, which should be taken into consideration when considering any possibly bursary program.

And finally

Thank you everyone who responded to this survey, your feedback is much appreciated.


Share this page