Progress on AI Preferences

While the first official meeting of the AIPREF working group was at IETF 122 in Bangkok, we recognise that many interested parties weren't able to make it, especially given that the group was formed just beforehand. We also needed more time than an hour or two at an IETF meeting would have afforded. So, in early April we held an interim meeting in Brussels.

At that meeting, we converged on several important ideas. First, we'd start with a very simple vocabulary of terms that describe preferences, and we'd define a way to attach them to content using robots.txt and HTTP headers.

In the subsequent weeks, we adopted two documents as starting points for these work items:

• A vocabulary for expressing AI usage preferences for vocabulary, and
• Indicating Preferences regarding content usage for attachment

Together, these Internet-Drafts (I-Ds) describe how websites could indicate preferences about how content is used for training generative and non-generative AI, search, and inference-time AI usages. With these updates, sites would be able to target specific uses, using the new Content-Usage mechanism in robots.txt and/or HTTP headers, instead of targeting all or specific crawlers.

What We're Working On

Those are just starting points, however. We're still discussing a number of aspects of them, including the need to have vocabulary terms for things like inference-time use of content, how search can be related to AI usage, and other details of the I-Ds (notably, how the ”Text and Data Mining” (TDM) terminology is used). 

Another area where we need more discussion is how preferences should be combined. Since we're defining more than one attachment mechanism and it's likely that others will be defined outside our processes, we need to be able to reconcile them so that their effect is predictable and sensible. 

See the issues list for more details, but note that creating new issues is reserved for the chairs and editors, so if you see something missing, please bring it up on the mailing list.

Setting Expectations

In Brussels, we realised that managing expectations—both by content owners and policymakers—about what these mechanisms can and cannot do is going to be critical. For example, a preference on content is only the preference of the person who put it there; it is not legally enforceable on its own, and that person may or may not have the legal right to control its use. Therefore, a preference cannot be blindly followed, nor can it absolve an AI vendor who forgoes due diligence when creating a model. 

Likewise, preferences are just that: they aren’t technically enforced. That’s something that the legal regime that they operate in will need to take up. It's likely we'll try to document these and other limitations as the work progresses.

We also recognise that the IETF process is new to many stakeholders. In particular, some policymakers are keenly interested in the potential outcomes of this work. So, to help them understand how it is progressing, we've been giving updates to a few interested regulators.

Getting to RFC

Our charter has very ambitious milestones for this work: we want to submit the I-Ds to the Internet Engineering Steering Group (IESG) for publication as a Proposed Standard in August 2025. That accelerated schedule reflects both the need for something that's an improvement over the current hodgepodge of proprietary practices in this area, and the pressure of impending policy requirements for technical means of opt-out from AI crawlers.

The tight schedule also forces us to focus on shipping a minimal standard (albeit with the important caveat that it has to be extensible to meet future requirements), rather than sketching out a vast framework that takes years to define and that may not have any impact on the real world. 

To meet those expectations, we held an online meeting last week, where we discussed the issues outlined above, and we're meeting both online and face-to-face design meeting in London in mid-July for two days, followed by a brief meeting at the IETF meeting in Madrid the following week. 

The progress we've seen so far leaves us optimistic that we'll be able to meet those goals, or only slip slightly in schedule – but of course we won't know until we finish. Part of our optimism here is due to what we observed at the Internet Architecture Board workshop that spawned this work: although the interests and perspectives of content providers and AI vendors may be very different, they have a shared interest in improving communication about AI preferences.

If you'd like to see more of this work or contribute to it, see the group home page.