Machine-Readable Training Preference Signal

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 3 May 2027.¶

Copyright Notice

Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

A service operator binds usage preferences (e.g., "no AI training") to paths by writing ordinary robots.txt groups keyed by a per-purpose policy token in the User-Agent line.¶

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

This expresses a preference against AI training for all paths except those under /press-kit/, for which it expresses a preference in favor (Section 4.2).¶

This document introduces no new directive names and no new grammar; the path language, longest-match precedence, and group structure defined by [ROBOTS] are not changed. As a result, the wide variety of existing parsers deployed across the Internet will continue to interpret traffic management signals as they did previously, and will ignore policy-token groups as they would any group for a user agent they do not recognize.¶

2. Conventions and Definitions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

service operator:

An entity that operates an HTTP service, such as a website, and publishes a robots.txt file for it. For preferences expressed through this mechanism, the service operator is the "declaring party" (Section 2 of [VOCAB]). [ROBOTS] calls this party a "service owner".¶

consumer:

An entity that interprets preference signals that a service operator expresses through the mechanism defined in this document.¶

parser:

In the sense of [ROBOTS]; software that parses robots.txt files and matches product tokens and paths against their rules.¶

policy token:

A product token (Section 2.2.1 of [ROBOTS]) whose RFC 9309 signals a consumer uses to determine how content will be used. A policy token need not appear in the User-Agent header field (Section 10.1.5 of [HTTP]) of any outbound request. This document introduces a new, cross-vendor policy token (Section 4.1).¶

proprietary policy token:

A policy token documented by a single consumer. Some service operators already include these in their robots.txt files.¶

3. Vocabulary

The __usage_ai-training policy token corresponds to the definition that the IETF aipref Working Group has arrived at (Section 4.1 of [VOCAB]). The following definitions are quoted from that document:¶

AI Training:

Using an asset to modify the learned parameters of a generative AI model.¶

The training of models that are used to perform exclusively non-generative tasks is not included in this category, even if the model is capable of generative tasks.¶

Generative AI model:

An AI model that is used to generate synthetic content in one or more modalities. Alternatively, an AI model that is made available for use to generate synthetic content in one or more modalities. This definition does not include a model that is used, or made available for use, for classification, ranking, or scoring, even where the model generates rationale for its output.¶

This document applies those definitions as follows:¶

asset:

The content retrieved from a path. The preferences that apply to an asset are those expressed in the robots.txt file in effect when the asset was retrieved (Section 2.4 of [ROBOTS]; see also [ATTACH]).¶

This document does not define a parallel vocabulary; it tracks the work of the aipref Working Group. Implementers SHOULD consider updating their definitions, and adding or removing vocabulary terms, as the aipref Working Group reaches consensus in additional areas.¶

4. Design

5. Security Considerations

The security considerations of Section 3 of [ROBOTS] apply to this mechanism.¶

6. Privacy Considerations

This document does not introduce any new privacy considerations.¶

7. IANA Considerations

This document has no IANA actions.¶

8. References

Acknowledgments

The author is grateful to the Working Group's chairs, Mark Nottingham (Cloudflare) and Suresh Krishnan (Cisco), to the editors of the vocabulary draft, Martin Thomson (Mozilla) and Paul Keller (Open Future), and to the responsible Area Director, Mike Bishop (Akamai), whose incredible effort to facilitate discussion and endless patience (in addition to their own individual contributions) have made the Working Group's progress possible.¶

The author also thanks the following people for their substantial contributions to shaping the Working Group's discussion and for their partnership. Affiliations are listed for identification only, since IETF participants contribute as individuals.¶

Many more people shaped the Working Group's discussion than appear above, and this list has almost certainly left out someone who belongs on it. Any such omission is inadvertent, and the author apologizes in advance.¶

Author's Address