Skip to main content
  • New IETF Area focuses on web and transport technologies

    The new Web and Internet Transport (WIT) area covers protocols that provide the functions of the transport layer of the Internet, including congestion control and queue management, real-time communication, as well as protocols that implement the World Wide Web and adjacent technologies.

    8 Oct 2024
  • IETF Meetings recording playback system now open source

    The source code of the playback system for the recordings of IETF meeting sessions was recently released by Meetecho under an open source license, and the IETF has now deployed its own instance of the system.

    2 Oct 2024
  • Workshop on the Next Era of NEtwork Management OPerationS (NEMOPS)

    A workshop organized by the Internet Architecture Board (IAB) aims to chart a path for the development of future network management protocols and techniques. The Next Era of Network Management Operations (NEMOPS) workshop will begin by assessing the impacts of the previous IAB workshop on both network operations and protocol development.

    20 Sep 2024
  • RFC data visualizations accessibility review

    RFCs are freely available to download, copy, publish, display and distribute. One benefit of providing free access to RFCs is that they can be used by the largest number of people possible to build a better Internet for all. To truly include as many people as we can, the IETF community has consistently tried to uphold accessibility standards for the publication of RFCs.

    10 Sep 2024
  • Consultation on the Second IASA2 Retrospective

    The IETF Administration LLC is soliciting community feedback on the second retrospective on the IETF Administrative Support Activity (IASA 2.0). This follows our first retrospective from 2021.

    4 Sep 2024

Filter by topic and date

Filter by topic and date

IETF 120 post-meeting survey

20 Aug 2024

IETF 120 Vancouver was held 20-26 July 2024

The results of the IETF 120 Vancouver post-meeting survey are now available on a interactive dashboard. Thank you to all of you who responded to this survey as we use your views to continually adjust the meeting experience.

Analysis

We received 255 responses, 200 onsite and 55 remote. With 1537 registered participants, this gives the survey a maximum margin of error of +/- 5.61%.

The results for satisfaction questions include a mean and standard deviation using a five point scale scoring system of Very satisfied = 5, Satisfied = 4, Neither satisfied nor dissatisfied = 3, Dissatisfied = 2, Very dissatisfied = 1. While there’s no hard and fast rule, a mean of above 4.50 is sometimes considered excellent, 4.00 to 4.49 is good, 3.50 to 3.99 is acceptable and below 3.50 is either poor or very poor if below 3.00. The satisfaction score tables also include a top box, the total of satisfied and very satisfied, and a bottom box, the total of dissatisfied and very dissatisfied, both in percentages. Please note that a small number of questions are on a four point scale.

Question changes since the last survey

For this survey we removed the questions specific to the previous meeting, and the one-off question about the block on submitting I-Ds two-weeks before a meeting as the results from last time were sufficient (no strong movement to change). We dropped the detailed questions about side meetings for reasons explained below. We also dropped the question about different online participation mechanism in favour of a more detailed question about Meetecho.

We kept in the question about where people get their funding from and added back the matrix questions about each individual part of the agenda and structure of the meeting and side meetings that were dropped in the last survey to make space.

We did not add back in the matrix question about venue and accommodation as some aspects are covered elsewhere is new questions. This is likely to require further tidying up in future surveys.

Actions taken following the last survey

For this meeting, we made the following changes, prompted by survey feedback:

  • We no longer made any mention of COVID but continued to provide masks and tests for those that wanted them.

Satisfaction

Overall satisfaction is 4.37, a good result. With some key exceptions, the satisfaction scores remain high, reflecting the various improvements made since we returned to onsite meetings.

The table below shows the satisfaction scores for the last six meetings, along with colour coded indicators for the five point scale above: excellent (🔵), good (🟢), acceptable (🟡), poor (🔴), very poor (⚫️)

Satisfaction scores for the last six meetings
IETF 120 Vancouver IETF 119 Brisbane IETF 118 Prague IETF 117 San Francisco IETF 116 Yokohama IETF 115 London
Overall satisfaction 4.37 🟢 4.24 🟢 4.42 🟢 4.30 🟢 4.30 🟢 4.28 🟢
AGENDA
Overall agenda 4.19 🟢 4.05 🟢 4.20 🟢 4.16 🟢 4.18 🟢 4.22 🟢
Sessions for WGs 4.29 🟢 - 4.32 🟢 4.22 🟢 4.22 🟢 4.22 🟢
BOFs 4.12 🟢 - 4.12 🟢 3.95 🟡 4.11 🟢 4.10 🟢
Sessions for RGs 4.20 🟢 - 4.09 🟢 4.12 🟢 4.14 🟢 4.10 🟢
Plenary 3.84 🟡 - 3.94 🟡 3.99 🟡 3.98 🟡 3.98 🟡
Side meetings 3.87 🟡 - 3.79 🟡 3.75 🟡 3.73 🟡 3.81 🟡
Hackathon 4.30 🟢 - 4.27 🟢 4.25 🟢 4.34 🟢 4.35 🟢
HotRFC 3.94 🟡 - 3.90 🟡 3.89 🟡 3.84 🟡 4.21 🟢
Pecha Kucha - - 4.00 🟢 4.15 🟢 - -
Office hours 3.97 🟡 - 4.07 🟢 3.98 🟡 4.23 🟢 4.00 🟢
Opportunities for social interaction - - 4.06 🟢 4.11 🟢 3.72 🟡 3.98 🟡
STRUCTURE
Overall meeting structure 4.28 🟢 4.15 🟢 4.34 🟢 4.28 🟢 4.28 🟢 4.28 🟢
Start time 4.40 🟢 - 4.49 🟢 (9:30am) 4.28 🟢 (9:30am) 4.16 🟢 (9:30am) 4.28 🟢 (9:30am)
Finish at 5pm Friday 3.67 🟡 - 3.65 🟡 - - -
Length of day 4.18 🟢 - 4.20 🟢 4.30 🟢 4.30 🟢 4.32 🟢
Number of days 4.15 🟢 - 4.18 🟢 (5+2) 4.27 🟢 (5+2) 4.30 🟢 (5+2) 4.32 🟢 (5+2)
Session lengths 4.36 🟢 - 4.38 🟢 (60 / 90 / 120) 4.41 🟢 (60 / 90 / 120) 4.36 🟢 (60 / 90 / 120) 4.32 🟢 (60 / 90 / 120)
Break lengths 4.31 🟢 - 4.38 🟢 (30/90) 4.32 🟢 (30/90) 4.38 🟢 (30/90) 4.36 🟢 (30/90)
Number of parallel tracks 3.94 🟡 (8) - 3.94 🟡 (8) 4.08 🟢 (8) 4.01 🟢 (8) 3.90 🟡 (8)
PARTICIPATION MECHANISMS
Meetecho 4.38 🟢 4.50 🔵 4.47 🟢 4.35 🟢 4.45 🟢 4.45 🟢
Gather - - - 3.52 🟡 3.46 🔴 3.37 🔴
Zulip - 3.86 🟡 3.94 🟡 3.66 🟡 3.77 🟡 3.73 🟡
Jabber - - - - - -
Audio streams - 4.27 🟢 4.27 🟢 4.02 🟢 4.21 🟢 4.04 🟢
YouTube streams - 4.24 🟢 4.36 🟢 4.32 🟢 4.36 🟢 4.25 🟢
CONFLICTS
Conflict avoidance 3.93 🟡 3.88 🟡 4.00 🟢 3.90 🟡 3.94 🟡 3.91 🟡
VENUE & ACCOMM
Overall accommodation - - 4.34 🟢 4.07 🟢 4.09 🟢 -
Overall venue - - 3.90 🟡 - -
Location - 4.52 🔵 3.60 🟡 - -
Venue facilities - 4.43 🟢 4.07 🟢 - -
Cost of rooms - 3.54 🟡 2.87 ⚫️ - -
Availability of rooms - 4.32 🟢 4.07 🟢 - -
Options for accommodation 3.68 🟡 4.10 🟢
ONSITE
Overall 4.44 🟢 4.37 🟢 4.50 🔵 4.29 🟢 - -
Badge collection 4.73 🔵 4.71 🔵 4.70 🔵 4.69 🔵 - -
WiFi 4.25 🟢 4.09 🟢 4.17 🟢 3.98 🟡 4.06 🟢 4.10 🟢
QR Codes 4.31 🟢 4.31 🟢 4.18 🟢 4.11 🟢 - -
Break F&B 4.15 🟢 3.75 🟡 4.05 🟢 4.44 🟢 - -
Breakout seating 3.54 🟡 3.93 🟡 3.80 🟡 4.08 🟢 - -
Signage 4.08 🟢 3.96 🟡 4.15 🟢 4.22 🟢 - -
Coffee carts 4.65 🔵 4.08 🟢 4.64 🔵 4.56 🔵 - -
Childcare 4.43 🟢 3.76 🟡 4.12 🟢 4.06 🟢 - -
Welcome reception 4.01 🟢 3.97 🟡 4.08 🟢 - - -
Farewell reception 4.31 🟢 4.15 🟢 4.34 🟢 - - -

Remote participation

We once again asked people why they participated remotely. The set of options has developed over time with some changes in wording and emphasis. The table below therefore combines options that were written differently in different surveys.

A number of the results here appear to vary strongly for reasons outside of the control of the IETF, while others indicate areas that the IETF can continue to work on (childcare, visas, accommodation costs).

We also asked if those who participated remotely would have wanted to participate onsite.

Putting the two together shows a fairly consistent proportion of remote participants would have participated onsite, most of whom were unable to do so because they did not have the funding.

Reasons why people participated remotely and would they want to be onsite
IETF 120 Vancouver IETF 119 Brisbane IETF 118 Prague IETF 117 San Francisco
WHY REMOTE
The travel costs were too high 47.17% - 4.44% 15.25%
The accommodation costs were too high 33.96% - 4.67% 13.56%
It was too expensive to travel to the meeting - 75.00% 60.00% 62.71%
I could not get a visa or getting a visa is too difficult 26.42% 4.69% 4.44% 15.25%
I could not take a week away from home/work 22.64% 39.06% 42.22% 35.59%
It is my preferred way to participate 11.32% 15.63% 28.89% 18.64%
I did not want to participate in the whole meeting 11.32% 10.94% 11.11% 23.73%
I had childcare responsibilities 5.66% 9.38% 15.56% 6.78%
The COVID policy (mask wearing, etc) was too loose 3.77% 3.13% 4.44% 5.08%
I did not want to go to this location 1.89% 4.69% 0% 16.95%
ONSITE PREFERRED
Yes 72.73% 73.44% 68.89% 62.71%
No 3.64% 10.94% 2.22% 10.17%
Unsure 23.64% 15.63% 28.89% 27.12%

Funding

For the second time asked the question "What was the primary source of funding for your onsite participation?", the results of which were:

  • "My employer" 67.86% (Was 75.14%)
  • "A grant or bursary" 4.76% (Was 9.04%)
  • "My own business" 4.37% (Was 5.65%)
  • "My personal funds" 7.94% (Was 5.08%)
  • "One or more clients" 5.56% (Was 5.08%)
  • "Other (please specify)" 2.38% (Was 0%)

New participants

We regularly include a set of questions for those who identify themselves as new participants and these have been used to drive large changes to the new participant program over the last two years. We will provide a more detailed analysis of the results of these questions in a future blog post.

Other good results

We're clearly doing a lot right as overall satisfaction is good and there are clear improvements trends in a number of areas. For those onsite, we're pleased that the quality of coffee is back to excellent.

Other areas for improvement

Breakout seating

We continue to bring in extra seating and place it close to the breakout rooms, but the feedback shows that we're still underestimating just how much close by seating is needed.

Signage

Our signage score has improved but there are several comments in the free text responses recommending further improvements.

Opportunities for social interaction

From the comments, the new games night proved succesful and partially compensated for the lack of a social event. Given corporate budget restrictions, we should expect no more social events for at least a year and so we will continue to identify smaller improvements we can make.

Break food and beverage

While we continue to tweak the food and beverage provisions based on feedback, we are always highly constrained by the choices available at the venue and their food management processes.

Areas we might not be able to improve

Side meetings

The satisfaction score for side meetings remains at the acceptable level, lower than that for most other aspects of the meeting. We get the same feedback on side meetings each time, generally that they should be better supported (i.e. provide Meetecho, avoid scheduling conflicts, integrate into meeting agenda) but after multiple revisits, the IESG is not planning on any further changes.

Conflict avoidance

The satisfaction score for conflict avoidances between sessions has not moved over two years despite significant efforts to improve things. It is likely that we are simply not able to do any better given the number of sessions and the limited days in which we can meet.

Individual comments

The individual comments covered a number of key themes, many of which have already been covered above:

  • Visas. We know of at least 50 participants who applied for visas within the recommended timeframe, but did not receive a response and so were not able to participate onsite. We will be following this up with the Canadian government to see if we can prevent similar issues for future meetings.
  • Cost and availability of accommodation. This meeting was actually booked in 2014 and then renegotiated in 2017 for the IETF 107 meeting in March 2020. However, IETF 107 was switched fully online due to COVID and this meeting was rebooked for 2024. This timing introduced two issues with accommodation. The first is that since we originally booked this meeting, the cost of accommodation in Vancouver during July has risen steeply, and the second is that our guaranteed room rate stayed comparatively cheap and so the hotel were unwilling to release more rooms at that rate to help us meet demand. We do not expect such a confluence of issues for future meetings.
  • COVID. There were a number of comments complaining about the infection risk at this meeting. We recognise that the risk of infection from airborne diseases, of which COVID is just one, is higher at IETF meetings given the concentration of a large number of people who have flown in from different countries, and the reduced level of immunity after a two-year pause in international travel. Consequently, we will amend our communications to make it clear that anyone who may be infectious should either wear a mask or participate remotely. We will not however, consider any return to formal restrictions or data collection, which is where we believe community consensus still sits.
  • Meeting room screens. There were a number of comments about how the screens in meeting rooms are used that will be considered by the meetings team.
  • Other comments. Other comments covered issues such as the finish time on Friday, Meetecho functionality, and more. These will be passed on to the appropriate people to consider.

And finally

Thank you everyone who responded to this survey, your feedback is much appreciated.


Share this page