Fastly

mnot@mnot.net https://www.mnot.net/

General Internet-Draft This specification introduces the HTTP Variants response header field to communicate what representations are available for a given resource. RFC EDITOR: please remove this section before publication The issues list for this draft can be found at https://github.com/mnot/I-D/labels/variant. The most recent (often, unpublished) draft is at https://mnot.github.io/I-D/variant/. Recent changes are listed at https://github.com/mnot/I-D/commits/gh-pages/variant. See also the draft’s current status in the IETF datatracker, at https://datatracker.ietf.org/doc/draft-nottingham-variant/.

HTTP proactive content negotiation (, Section 3.4.1) is starting to be used more widely again. The most widely seen use – determining a response’s content-coding – is being joined by renewed interest in negotiation for language and other, newer attributes (for example, see ). Successfully reusing negotiated responses that have been stored in a HTTP cache requires establishment of a secondary cache key (, Section 4.1) using the Vary header (, Section 7.1.4), which identifies the request headers that form the secondary cache key for a given response. HTTP’s caching model allows a certain amount of latitude in normalising request header fields to match those stored in the cache, so as to increase the chances of a cache hit while still respecting the semantics of that header. However, this is often inadequate; even with understanding of the headers’ semantics to facilitate such normalisation, a cache does not know enough about the possible alternative representations available on the origin server to make an appropriate decision. For example, if a cache has stored the following request/response pair:

Provided that the cache has full knowledge of the semantics of Accept-Language and Content-Language, it will know that a French representation is available and might be able to infer that an English representation is not available. But, it does not know (for example) whether a Japanese representation is available without making another request, thereby incurring possibly unnecessary latency. This specification introduces the HTTP Variants response header field to enumerate the available variant representations on the origin server, to provide clients and caches with enough information to properly satisfy requests – either by selecting a response from cache or by forwarding the request towards the origin. Variants is best used when content negotiation takes place over a constrained set of representations; since each variant needs to be listed in the header field, it is ill-suited for open-ended sets of representations. Likewise, it works best for content negotiation over header fields whose semantics are well-understood, since it requires a selection algorithm to be specified ahead of time.

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This specification uses the Augmented Backus-Naur Form (ABNF) notation of with a list extension, defined in Section 7 of , that allows for compact definition of comma-separated lists using a ‘#’ operator (similar to how the ‘*’ operator indicates repetition). Additionally, it uses the “field-name”, “OWS” and “token” rules from .

The Variant HTTP response header field is used to indicate what other representations are available for a given resource at the time that the response is produced.

Each variant indicates a response header field that carries a value that clients might proactively negotiate for; each parameter on it indicates a value for which there is an available representation on the origin server. So, given this example header field:

a recipient can infer that the only content-coding available for that resource is “gzip” (along with the “identity” non-encoding; see {{content-encoding}). Given:

a recipient can infer that no content-codings are supported. Note that as always with header field names, it is case-insensitive. A more complex example:

Here, recipients can infer that two Content-Encodings are available, as well as two content languages. Note that, as with all HTTP header fields that use the “#” list rule (see , Section 7), they might occur in the same header field or separately, like this:

The ordering of available-values after the field-name is significant, as it might be used by the header’s algorithm for selecting a response. Senders SHOULD consistently send Variant header fields on all cacheable (as per , Section 3) responses for a resource, since its absence will trigger caches to fall back to Vary processing. Likewise, servers MUST send the Content-* response headers nominated by Variants when sending that header.

To be usable with Variants, proactive content negotiation mechanisms need to be specified to take advantage of it. Specifically, they: MUST define a request header field that advertises the clients preferences or capabilities, whose field-name SHOULD begin with “Accept-“. MUST define a response header field that indicates the result of selection, whose field-name SHOULD begin with “Content-“ and whose field-value SHOULD be a token. MUST define an algorithm for selecting a result. It MUST return an ordered list of selected responses, given the incoming request, a list of selected responses, and the list of available values from Variants. If the result is an empty list, it implies that the cache does not contain an appropriate response. fulfils these requirements for some existing proactive content negotiation mechanisms in HTTP. Note that unlike Vary, Variants does not use stored request headers to help select a response; this is why defining a response header to aid identification and selection is required.

Caches that implement the Variants header field and the relevant semantics of the field-name it contains can use that knowledge to either select an appropriate stored representation, or forward the request if no appropriate representation is stored. They do so by running this algorithm (or its functional equivalent) upon receiving a request, incoming-request: Let selected-responses be a list of the stored responses suitable for reuse as defined in Section 4, excepting the requirement to calculate a secondary cache key. Order selected-responses by the “Date” header field, most recent to least recent. If the freshest (as per , Section 4.2) has one or more Variants header field(s): Select one member of selected-responses and let its “Variants” header field-value(s) be Variants. This SHOULD be the most recent response, but MAY be from an older one as long as it is still fresh. For each variant in Variants: If the field-name corresponds to the response header field identified by a content negotiation mechanism that the implementation supports: Let available-values be a list containing all available-value for the variant. Let selected-responses be the result of running the algorithm defined by the content negotiation mechanism with incoming-request, selected-responses and available-values. For the purposes of selecting a response, ignore the content negotiation’s identified request header field-name in the “Vary” header field of each member of selected-responses, if present. Process any member of selected-responses that has a “Vary” response header field whose field-value still contains one or more field-names, removing that members if it does not match (as per , Section 4.1). Return the first member of selected-responses. If selected-responses is empty, return null. This algorithm will either return the appropriate stored response to use, or null if the cache needs to forward the request towards the origin server.

Caches that fully implement this specification MUST ignore request header-fields in the Vary header for the purposes of secondary cache key calculation (, Section 4.1) when their semantics are understood, implemented as per this specification, and their corresponding response header field is listed in Variants. Request header fields listed in Vary that are not implemented in terms of this specification or not present in the Variants field SHOULD still form part of the secondary cache key. The algorithm in implements these requirements.

Given a request/response pair:

Upon receipt of this response, the cache knows that two representations of this resource are available, one with a Content-Language of “en”, and another whose Content-Language is “de”. Subsequent requests (while this response is fresh) will cause the cache to either reuse this response or forward the request, depending on what the selection algorithm Accept-Language and Content-Language determines. So, a request with “en” in Accept-Language is received and its q-value indicates that it is acceptable, the stored response is used. A request that indicates that “de” is acceptable will be forwarded to the origin, thereby populating the cache. A cache receiving a request that indicates both languages are acceptable will use the q-value to make a determination of what response to return. A cache receiving a request that does not list either language as acceptable (or does not contain an Accept-Language at all) will return the “en” representation (possibly fetching it from the origin), since it is listed first in the Variants list. Note that Accept-Language is listed in Vary, to assure backwards-compatibility with caches that do not support Variants. Also, note that is is the response header which is listed in Variants, not the request header (the opposite of Vary).

A more complicated request/response pair:

Here, the cache knows that there are two axes that the response varies upon; Content-Language and Content-Encoding. Thus, there are a total of six possible representations for the resource, and the cache needs to consider the selection algorithms for both axes. Upon a subsequent request, if both selection algorithms return a stored representation, it can be served from cache; otherwise, the request will need to be forwarded to origin.

Now, consider the previous example, but where only one of the varied axes is listed in Variants:

Here, the cache will need to calculate a secondary cache key as per , Section 4.1 – but considering only Accept-Language to be in its field-value – and then continue processing Variants for the set of stored responses that the algorithm described there selects.

This specification registers one value in the Permanent Message Header Field Names registry established by : Header field name: Variants Applicable protocol: http Status: standard Author/Change Controller: IETF Specification document(s): [this document] Related information:

If the number or advertised characteristics of the representations available for a resource are considered sensitive, the Variants header by its nature will leak them. Note that the Variants header is not a commitment to make representations of a certain nature available; the runtime behaviour of the server always overrides hints like Variants.

This protocol is conceptually similar to, but simpler than, Transparent Content Negotiation . Thanks to its authors for their inspiration. It is also a generalisation of a Fastly VCL feature designed by Rogier ‘DocWilco’ Mulhuijzen. Thanks to Hooman Beheshti for his review and input.

In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. The Hypertext Transfer Protocol (HTTP) is a stateless \%application- level protocol for distributed, collaborative, hypertext information systems. This document defines the semantics of HTTP/1.1 messages, as expressed by request methods, request header fields, response status codes, and response header fields, along with the payload of messages (metadata and body content) and mechanisms for content negotiation. The Hypertext Transfer Protocol (HTTP) is a stateless \%application- level protocol for distributed, collaborative, hypertext information systems. This document defines HTTP caches and the associated header fields that control cache behavior or indicate cacheable response messages. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Internet technical specifications often need to define a formal syntax. Over the years, a modified version of Backus-Naur Form (BNF), called Augmented BNF (ABNF), has been popular among many Internet specifications. The current specification documents ABNF. It balances compactness and simplicity with reasonable representational power. The differences between standard BNF and ABNF involve naming rules, repetition, alternatives, order-independence, and value ranges. This specification also supplies additional rule definitions and encoding for a core lexical analyzer of the type common to several Internet specifications. [STANDARDS-TRACK] The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document provides an overview of HTTP architecture and its associated terminology, defines the "http" and "https" Uniform Resource Identifier (URI) schemes, defines the HTTP/1.1 message syntax and parsing requirements, and describes related security concerns for implementations. This document describes a syntax, called a "language-range", for specifying items in a user's list of language preferences. It also describes different mechanisms for comparing and matching these to language tags. Two kinds of matching mechanisms, filtering and lookup, are defined. Filtering produces a (potentially empty) set of language tags, whereas lookup produces a single language tag. Possible applications include language negotiation or content selection. This document, in combination with RFC 4646, replaces RFC 3066, which replaced RFC 1766. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. An increasing diversity of Web-connected devices and software capabilities has created a need to deliver optimized content for each device. This specification defines a set of HTTP request header fields, colloquially known as Client Hints, to address this. They are intended to be used as input to proactive content negotiation; just as the Accept header field allows user agents to indicate what formats they prefer, Client Hints allow user agents to indicate device and agent specific preferences. This specification defines registration procedures for the message header fields used by Internet mail, HTTP, Netnews and other applications. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. HTTP allows web site authors to put multiple versions of the same information under a single URL. Transparent content negotiation is an extensible negotiation mechanism, layered on top of HTTP, for automatically selecting the best version when the URL is accessed. This enables the smooth deployment of new web data formats and markup tags. This memo defines an Experimental Protocol for the Internet community. It does not specify an Internet standard of any kind. Discussion and suggestions for improvement are requested.

This appendix defines the required information to use existing proactive content negotiation mechanisms (as defined in , Section 5.3) with the Variants header field.

When negotiating for the Content-Encoding response header field’s value, the applicable request header field is Accept-Encoding, as per Section 5.3.4. To perform content negotiation for Content-Encoding given an incoming-request, stored-responses and available-values: Let preferred-codings be a list of the codings in the “Accept-Encoding” header field of incoming-request, ordered by their weight, highest to lowest. If “Accept-Encoding” is not present or empty, preferred-codings will be empty. If identity is not a member of preferred-codings, append identity to preferred-codings with a weight of 0.001. Remove any member of preferred-codings whose weight is 0. Append “identity” to available-values. Remove any member of available-values not present in preferred-codings, comparing in a case-insensitive fashion. Let filtered-responses be an empty list. For each available-value of available-values: If there is a member of stored-responses whose “Content-Encoding” field-value has content-codings (, Section 3.1.2.2) that all match members of available-value in a case-insensitive fashion, append that stored response to filtered-responses. If there is a member of stored-responses that does not have a “Content-Encoding” header field, append that stored response to filtered-responses. Return filtered-responses. This algorithm selects the stored response(s) in order of preference by the client; if none are stored in cache, the request will be forwarded towards the origin. It defaults to the “identity” non-encoding. Implementations MAY remove members of filtered-responses based upon their weight or other criteria before returning. For example, they might wish to return an empty list when the client’s most-preferred available response is not stored, so as to populate the cache as well as honour the client’s preferences.

When negotiating for the Content-Language response header field’s value, the applicable request header field is Accept-Language, as per Section 5.3.5. To perform content negotiation for Content-Language given an incoming-request, stored-responses and available-values: Let preferred-langs be a list of the language-ranges in the “Accept-Language” header field (, Section 5.3.5) of incoming-request, ordered by their weight, highest to lowest. If preferred-langs is empty, append “*” with a weight of 0.001. Remove any member of preferred-langs whose weight is 0. Filter available-values using preferred-langs with either the Basic Filtering scheme defined in Section 3.3.1, or the Lookup scheme defined in Section 3.4 of that document. Use the first member of available-values as the default. Let filtered-responses be an empty list. For each available-value of available-values: If there is a member of stored-responses whose “Content-Language” field-value has a language-tag (, Section 3.1.3.2) that matches available-value in a case-insensitive fashion, append that stored response to filtered-responses. Return filtered-responses. This algorithm selects the available response(s) (according to Variants) in order of preference by the client; if none are stored in cache, the request will be forwarded towards the origin. If no preferred language can be selected, the first available-value will be used as the default. Implementations MAY remove members of filtered-responses based upon their weight or other criteria before returning. For example, they might wish to return an empty list when the client’s most-preferred available response is not stored, so as to populate the cache as well as honour the client’s preferences.