]> Anthropic

kelleyk-ietf@kelleyk.net

Web and Internet Transport AI Preferences robots.txt AI preferences AI training This document describes how a service operator binds usage preferences (e.g., "no AI training") to paths by writing ordinary robots.txt groups keyed by a per-purpose policy token in the User-Agent line. It introduces no new directive names and no new grammar: the path language, longest-match precedence, and group structure of robots.txt are not changed. About This Document Status information for this document may be found at . Discussion of this document takes place on the AI Preferences Working Group mailing list (), which is archived at . Subscribe at .

Introduction A service operator binds usage preferences (e.g., "no AI training") to paths by writing ordinary robots.txt groups keyed by a per-purpose policy token in the User-Agent line.

A policy-token group

This expresses a preference against AI training for all paths except those under /press-kit/, for which it expresses a preference in favor (). This document introduces no new directive names and no new grammar; the path language, longest-match precedence, and group structure defined by are not changed. As a result, the wide variety of existing parsers deployed across the Internet will continue to interpret traffic management signals as they did previously, and will ignore policy-token groups as they would any group for a user agent they do not recognize.

Goals

1. Path-specific bindings. Preferences can be expressed per path, and not only site-wide ().
2. Minimal distance from RFC 9309. Semantics, and ideally syntax, stay as close to as possible, so that authors and implementers can use the mental model they already have.
3. No large-scale duplication. robots.txt files are size-constrained (), and some deployed files already contain heavily duplicated groups. Expressing preferences does not require restating existing rules ().
4. Unchanged legacy behavior. A parser or consumer that does not implement this mechanism behaves as before. This holds for parsers that are actually deployed, not only for idealized RFC 9309 parsers. Conversely, implementations treat files that do not use this mechanism exactly as before ( and ).
5. Consistent interpretation across consumers. This mechanism does not depend on the divergent parts of deployed parser behavior ().

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

service operator:

An entity that operates an HTTP service, such as a website, and publishes a robots.txt file for it. For preferences expressed through this mechanism, the service operator is the "declaring party" (). calls this party a "service owner".

consumer:

An entity that interprets preference signals that a service operator expresses through the mechanism defined in this document.

parser:

In the sense of ; software that parses robots.txt files and matches product tokens and paths against their rules.

policy token:

A product token () whose RFC 9309 signals a consumer uses to determine how content will be used. A policy token need not appear in the User-Agent header field () of any outbound request. This document introduces a new, cross-vendor policy token ().

proprietary policy token:

A policy token documented by a single consumer. Some service operators already include these in their robots.txt files.

Vocabulary The __usage_ai-training policy token corresponds to the definition that the IETF aipref Working Group has arrived at (). The following definitions are quoted from that document:

AI Training:

Using an asset to modify the learned parameters of a generative AI model. The training of models that are used to perform exclusively non-generative tasks is not included in this category, even if the model is capable of generative tasks.

Generative AI model:

An AI model that is used to generate synthetic content in one or more modalities. Alternatively, an AI model that is made available for use to generate synthetic content in one or more modalities. This definition does not include a model that is used, or made available for use, for classification, ranking, or scoring, even where the model generates rationale for its output.

This document applies those definitions as follows:

asset:

The content retrieved from a path. The preferences that apply to an asset are those expressed in the robots.txt file in effect when the asset was retrieved (; see also ).

This document does not define a parallel vocabulary; it tracks the work of the aipref Working Group. Implementers SHOULD consider updating their definitions, and adding or removing vocabulary terms, as the aipref Working Group reaches consensus in additional areas.

Design

The Policy Token This document defines the policy token __usage_ai-training. The policy token conforms to the product-token syntax in (letters, _, -). Consumers MUST match policy tokens exactly, apart from case, as for any product token. (Note that some deployed parsers match more flexibly; the __ prefix makes accidental matches unlikely.) A single group MAY name several policy tokens (consecutive User-Agent lines; ) to bind its rules to several purposes. Policy tokens and other product tokens MAY be mixed in a group.

Checking for Preferences To determine whether a preference has been expressed for a given path, a consumer looks up the path and policy-token combination as specified in Sections and of . However, wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token. If the lookup finds that the path is disallowed, a preference against the use described by the policy token has been expressed for that path. If it finds that the path is allowed, a preference in favor of that use has been expressed. If no group contains the __usage_ai-training policy token (i.e., there is no User-Agent: __usage_ai-training line in the file), then no preference has been expressed. If such a file contains a proprietary policy token, consumers SHOULD interpret that token as they would in the absence of this document, but MUST NOT treat it as expressing a preference in the context of this document (see ).

Independence from Crawler Rules Consumers MUST interpret the RFC 9309 signals being expressed based on their own product token(s) exactly as they would in the absence of this document. That is, ExampleBot still checks whether there are rules for ExampleBot; if there are, they still express the service operator's preference about crawling. Nothing in this document changes that signaling. Content-usage preferences and traffic management signals can both be expressed in robots.txt, but they are independent and orthogonal.

Effects on Crawling Behavior This policy token is not a directive for crawling behavior. A consumer MAY use preferences to inform crawling. For example, if it crawls only for purposes covered by the policy token, it MAY refrain from retrieving resources for which a preference against those purposes has been expressed. If a service operator wishes to limit crawling, it MAY use groups for crawlers' product tokens in robots.txt to do so.

Precedence Specific expressions SHOULD take precedence over general ones. Consumers MAY document that they interpret rules that match proprietary policy tokens as being more specific than the __usage_ai-training policy token. Private agreements MAY override preferences expressed via this mechanism.

Security Considerations The security considerations of apply to this mechanism.

Privacy Considerations This document does not introduce any new privacy considerations.

IANA Considerations This document has no IANA actions.

References Normative References This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching. Open Future Mozilla This document defines a vocabulary for expressing preferences regarding how digital assets are used by automated processing systems. This vocabulary allows for the declaration of restrictions or permissions for use of digital assets by such systems. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References Google Mozilla Methods are defined for associating usage preferences with content that is obtained using the HTTP protocol. This document defines attachment methods using the Robots Exclusion Protocol and HTTP header fields. This document updates RFC 9309 to allow for the inclusion of usage preferences. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230.

Acknowledgments The author is grateful to the Working Group's chairs, Mark Nottingham (Cloudflare) and Suresh Krishnan (Cisco), to the editors of the vocabulary draft, Martin Thomson (Mozilla) and Paul Keller (Open Future), and to the responsible Area Director, Mike Bishop (Akamai), whose incredible effort to facilitate discussion and endless patience (in addition to their own individual contributions) have made the Working Group's progress possible. The author also thanks the following people for their substantial contributions to shaping the Working Group's discussion and for their partnership. Affiliations are listed for identification only, since IETF participants contribute as individuals.

Name	Affiliation
Farzaneh Badiei	Digital Medusa
Lila Bailey	Internet Archive
Kevin Bankston	Center for Democracy and Technology
Fabrice Canel	Microsoft
Alissa Cooper	Knight-Georgetown Institute
Sonia Cooper	Microsoft
Glenn Deen	Comcast NBCUniversal
Caleb Donaldson	Google
Daniel Etcovitch	Anthropic
Christopher Flammang	Elsevier
Max Gendler	News Corp
Alex Gibson	News Corp
Jordan Gimbel	Microsoft
Mark Gray	OpenAI
Nate Hake	Travel Lemming
Gary Illyes	Google
Meredith Jacob	American University Washington College of Law
Mirja Kühlewind	Ericsson
Laurent Le Meur	EDRLab
Jo Levy	Norton Law
Greg Lindahl	Common Crawl Foundation
Krishna Madhavan	Microsoft
Sarah McKenna	Sequentum
Angela Mills Wade	European Publishers Council
John Mueller	Google
Chris Needham	BBC
Elaine Newton	Amazon Web Services
Victoria Noble	Electronic Frontier Foundation
Sebastian Posth	Liccium
Brendan Quinn	IPTC
Courtney Radsch	Open Markets Institute
Felix Reda	GitHub
Eric Rescorla	Independent
Matt Rogerson	Financial Times
Leah Romm	Cloudflare
Leonard Rosenthol	Adobe
Malikat Rufai	Meta
Achim Schlosser	Bertelsmann
Rony Shalit	Bright Data
Bradley Silver	Advance
Erin Simon	Google
Esther Tetruashvily	OpenAI
Michael Tremante	Cloudflare
Thom Vaughan	Common Crawl Foundation
James Whymark	Meta
Timid Robot Zehta	Creative Commons

Many more people shaped the Working Group's discussion than appear above, and this list has almost certainly left out someone who belongs on it. Any such omission is inadvertent, and the author apologizes in advance.