Core BPF ecosystem standardized in RFC 9669

As outlined in our previous post introducing the working group, BPF is a technology that can run safe, dynamically-loaded programs in privileged contexts such as an operating system kernel. It is used to safely and efficiently extend the capabilities of the kernel without requiring a change of kernel source code or loading kernel modules. The BPF working group has worked over the past 18 months to document the state of the ecosystem and extensions for this technology that has origins in the Linux kernel and is increasingly being used beyond Linux. The result is RFC 9669, “BPF Instruction Set Architecture (ISA)”.

A primary goal for this effort is to make it easier and more straightforward for vendors to build BPF offload into their devices.

Why standardize BPF? 

BPF is already widely deployed on operating systems such as Linux and Microsoft Windows, so some may be wondering why the IETF—which  has traditionally standardized networking protocols—ended up being the standards body for BPF, and why BPF needed to be standardized at all?

The short answer: device offloading.

Some network interface card (NIC) vendors already have devices that are capable of running eXpress Data Path (XDP) programs on the device directly rather than on the CPU. The reasons for this are quite straightforward: if you have to make a networking decision such as to drop a packet, doing so on the NIC rather than on the CPU will save resources such as CPU capacity and reduce power consumption.

Though some vendors have already implemented these BPF offloading capabilities without having a standardized ISA, others are not quite as risk tolerant. For example, certain nonvolatile memory express (NVMe) vendors have expressed an interest in building BPF offloading capabilities for various use cases, such as eXpress Resubmission Path (XRP), which depends on BPF, but they simply can't fund such a project without certain components of BPF being standardized. Hence, the effort to standardize BPF was born.

Of course, in addition to device offload, the standardization effort may well bear fruit in other areas such as software compatibility.

The standardization process

We’re going to skip a lot of details here, and mostly just want to go over the broad strokes to connect where we started to where we are now.

This whole effort really kicked off in earnest at the Linux Storage, Filesystem, MM & BPF Summit  (LSFMM) in 2022, but at that time, the BPF community hadn’t  decided on which organization they would work with to push through the standardization, nor the scope of standardization. Different organizations were considered, including the IETF. Following LSFMM 2022, a side meeting was conducted during the IETF 115 meeting to discuss whether BPF was an appropriate topic for standardization with the IETF. Though there were some skeptics, the takeaway from IETF 115 was that the IETF was eager to be the standardizing body, and that we would figure out how to make it work as we went along.

Fast forward to IETF 116 Yokohama. After more deliberation and community input, the decision was made to form the BPF working group. The working group was categorized as part of the Internet Area under the leadership of Erik Kline as our Area Director. We, David Vernet and Suresh Krishnan, would serve as co-chairs.

At this point there were still some key unanswered questions, such as how the Linux kernel community and the IETF would work together given their vastly different workflow patterns, and how we would reconcile document licensing and copyright. Though it took some patience and persistence, these issues were all ultimately addressed. The group charter was published with a list of documents that were in scope for our roadmap. At the head of this roadmap was the ISA document.

From that point, it was a matter of iterating on and completing the ISA document. A large part of this effort was resolving technical considerations,

such as proposed semantics for new instructions such as indirect calls (CALLX). Other topics were procedural questions, such as what the process should be for adding new instructions, deprecating old instructions, and how to aggregate instructions into conformance groups. 

Though it sometimes required a lot of discussion and time to resolve a topic, we always managed to reach consensus. Likely worth noting as well is that discussions often spanned mailing lists and in-person (or virtual) discussions at IETF meetings.

Eventually, the number of open topics and discussions in the working group was whittled down to zero, and we moved the document to IETF Working Group Last Call in early March of this year (2024). We had to go through more steps in the IETF document lifecycle, such as submitting the document to the Internet Engineering Steering Group (IESG) for evaluation, getting approval from the IESG, and then finally having the RFC Editor review and publish the document. Our efforts were rewarded when the document was officially ratified as RFC 9669 on 31 October 2024.

What's next?

Getting the ISA document ratified as a Proposed Standard is a significant milestone. The primary value proposition of the standardization effort was to incentivize vendors to build BPF offload by de-risking their investment. Now that we have an official Proposed Standard that's been ratified, the plan is to take a step back and see how the industry responds. 

If we observe that vendors have taken notice and either begin investing in BPF offload, or join the IETF discussions to talk about what other standards they would need, we can plan to ramp back up on our time investment in the IETF. Until then, the plan is to try and minimize the overhead of IETF processes, while still optimizing for iterating on documents, and meeting in person as a working group on an ad-hoc basis, when required.

To stress: "minimize" does not mean "cease functioning". Work is continuing in the background on other documents such as the processor-specific application binary interface (psABI) document, and patches and participation in the WG are both accepted, and encouraged.

We want to give a big thank you to everyone who has participated in the IETF BPF working group so far, and helped us to achieve this milestone. There are too many individuals who helped to enumerate everyone, but wewant to thank Dave Thaler in particular for being the primary author of the document, and for putting in so much work to drive the document to completion over the last ~1.5 years.

Congratulations, to everyone who has contributed along the way and thank you for being a part of this.