]> Mozilla

mt@lowentropy.net

ART HTTP blockchain This document defines a content coding for HTTP that provides strong integrity protection while maintaining the ability to perform some progressive processing. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the HTTP Group mailing list (), which is archived at . Source for this draft and an issue tracker can be found at .

Introduction Hash-based integrity is a powerful tool for ensuring that content matches expectations. Hashes are often the basis of digital signature schemes that can authenticate content, or at least provide verifiable proof of authorization. The main challenge with hashes is that validating integrity requires every byte in the sequence. This document describes a method of hash generation that can be verified incrementally. This enables systems that depend on progressive processing, such as streaming processing of a long sequence of records, or performance-sensitive applications like HTML rendering. The way this works is by splitting a document into chunks, called chiclets. Each chiclet comprises a length, some number of bytes of content, and a hash for the next chiclet. This is shown in .

Overview of Format ]]>

The hash of the entirety of the content can thereby be represented by the hash of the first chiclet, or chi digest. This enables progressive processing of content, maintaining strong integrity for all content bytes, provided that each chiclet is only released after being validated; see . That can reduce latency -- or at least perceived latency -- for applications that are sensitive to delays by allowing processing to start before the entire sequence of bytes is received. The cost of this scheme is that there is a small amount of overhead. Additional bytes for the hash and length fields are interspersed with content bytes. The relative cost of that overhead depends on the size of chiclets; a 4 KiB chiclet size adds overhead of 0.83%, whereas a 65535-byte chiclet adds overhead of 0.055%. Larger chiclets reduce overhead, but add latency; smaller chiclets can improve latency with higher overheads. Though it is possible to process data incrementally, constructing this format requires two passes over the entire sequence of bytes; see .

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. The term "chi digest" refers to the hash of the first chiclet in the format defined in . This document uses the varint encoding and presentation language developed for QUIC Sections and of .

Chained Hash Integrity Format Any continuous stream of bytes can be turned into a stream of variable-length chiclets. This format allows for progressive validation of the integrity of content.

Format Definition The format comprises zero or more non-final chiclets and a final chiclet. Content bytes are therefore interspersed with framing and integrity bytes. A non-final chiclet comprises a non-zero length (for framing), between 1 and 2⁶¹-41 bytes (inclusive) of content (see ), and a 32-byte hash (that is, integrity bytes). Lengths are encoded using the variable-length integer format from . Lengths do not need to be encoded in the minimal form; the chosen encoding will affect the resulting digest. The hash is the output of the SHA-256 algorithm that covers the entirety of the subsequent chiclet. The final chiclet includes a zero-valued length field, up to 2⁶¹-2 bytes of content (less if the length is encoded in more than one byte), and no hash value. The final chiclet ends when content ends, which means that this format can only be used where there is a distinct end to the protected content. That might mean the end of a file, the end of a stream, or the end of an HTTP message; see . The hash value that is computed from the very first chiclet -- the chi digest -- is not written to the output. The chi digest commits to the entire sequence of chiclets and the content that it protects. A chi digest needs to be carried outside the content; see for one way to convey a chi digest. The format is illustrated in using the notation from .

Chained Hash Integrity Format

The format uses SHA-256 for integrity ; see for details.

Consumption and Validation An entity that receives a stream of bytes in this format can use the chi digest to validate the content. That value might be provided via some alternative communication channel; one method for carrying this hash value in HTTP fields is defined in . Without the chi digest, it is not possible to validate the integrity of the content. Content can still be processed by removing the length and hash bytes. The process for progressively validating and releasing content bytes proceeds as follows, starting with an expected hash value set to the chi digest:

1. Initialize a new instance of SHA-256 .
2. Read a variable-length integer as a length. Update the hash instance with each byte as it is read. Abort with an error if the stream ends before reading a complete value.
3. Read the indicated number of bytes from the stream. If the length value is 0, read to the end of the stream instead. Update the hash instance with each byte as it is read. Abort with an error if the stream ends before reading this amount.
4. If the length value is non-zero, read 32 bytes from the stream and assign those bytes to the next hash value. Update the hash instance with each byte as it is read. Abort with an error if the stream ends before reading this amount.
5. Finalize the hash and generate hash output. If the hash output is not identical to the expected hash value, abort with an error.
6. Release the content bytes.
7. If the length value is non-zero, set the expected hash value to the next hash value and return to the start. Otherwise, end the process successfully.

This process ensures that released bytes are all integrity-protected, according to the value that was provided. The Chi-Digest field () provides only a single hash, but other mechanisms could be defined, such as a future extension to . When hashes are obtained via means other than the Chi-Digest field, there is a risk that chiclet layout has changed. A different layout invalidates the hash, so resources should not change the chiclet layout. To account for variations in how chiclets are constructed (see ), content could be matched against one of a set of chi digests. An entity can validate against a set of chi digests by matching the hash of the first chiclet against any value from the provided set. Once the first chiclet is processed, subsequent chiclets are validated against the single hash that was included in that first chiclet.

Construction In order to produce hashes, content needs to be processed in reverse order. Content is split into multiple pieces, which are turned into chiclets. These chiclets are hashed, starting from the last, placing the resulting hash from each chiclet at the end of the chiclet that immediately precedes it; see . When constructing content, if the content is processed in a forward direction, chiclets might be written with placeholder values for each hash. Hashes can then be added in reverse order once the final chiclet is hashed. The construction of chiclets can operate on fixed-sized pieces, or application needs can drive how content is split. For instance, an image that can be displayed at progressively-higher levels of quality can have certain points at which significant quality improvements can be realized. Aligning chiclets to those points might enable better incremental processing. Similarly, a video file might have chiclets aligned to frame boundaries or significant features within frames. The need to produce the encoding in reverse order makes this format ill-suited to situations where content of arbitrary length is streamed. It is also poorly suited to use with dynamically-generated content, especially where content is latency sensitive, because of the need for two processing passes. It is therefore best suited to static resources where the format can be pregenerated and saved.

Length Constraints An implementation that processes input needs to hold an entire chiclet in its memory before validating and releasing it. Implementations that accept untrusted inputs might refuse to support large chiclets to protect against memory exhaustion attacks. Receivers MUST support chiclets with content lengths of up to 65535 bytes (2¹⁶-1). The total size of chiclets will be larger as they also contain a length field of up to 8 bytes and non-final chiclets include a 32-byte hash. Encoders MUST NOT generate larger chiclets without knowing, by means outside of this specification, that the receiver supports the size it uses. Receivers can support chiclets larger than the 65535-byte minimum. A chiclet larger than the supported maximum results in a decode error. A receiver can validate that the length of a non-final chiclet is acceptable before reading further. In contrast, the final chiclet has a zero-valued length prefix, so receivers need to either rely on other length indicators (such as the HTTP Content-Length field) or abort when received bytes exceed their configured maximum. The absolute maximum length of a chiclet is determined by the length encoding in SHA-2, which is expressed in bits as a 64-bit value. Consequently, the maximum length is 2⁶¹-1 bytes, less overheads of 40 bytes for a non-final chiclet and up to 8 bytes for a final chiclet (though this can be a single byte). This is less than the 2⁶²-1 length that could otherwise be expressed in a QUIC variable-length integer. Of course, there is little value in having such large chiclets, which directly act against the goal of incremental processing. An abusive implementation might also seek to generate wasteful work for recipients by constructing many small chiclets. While small chiclets might be necessary for some applications, very small lengths can add significant amounts of overhead. Recipients therefore need to look for abusive patterns, such as receiving a large number of chiclets below a given threshold, rather than have a strict lower bound on chiclet length.

HTTP Usage The use of progressive validation is well-suited to HTTP applications that rely on the streaming and incremental processing of message content. A content coding is defined to enable progressive decoding while maintaining integrity; see . A header field is defined to carry a chi digest; see .

Content Coding The chi content coding indicates that the representation is encoded as a sequence of chiclets, following the construction defined in . Encoding follows the guidance in ; decoding follows the procedure in . A decoder can read a message that uses chi content coding without a target digest, though that provides no integrity guarantee. A client can indicate that it will accept chi content coding in responses using the Accept-Encoding field. A client can attempt to use chi content coding on requests, which could result in an error response that carries Accept-Encoding if it is not understood by the server; see for details. Intermediaries can check, add, or remove chi content coding at their discretion. Adding the coding requires buffering of much of the message before the hash for any non-final chiclet can be sent; see . Intermediaries MUST update any Chi-Digest field when adding or removing chi codings; see . Like other content codings, the chi content coding applies to the entirety of a representation, so it is less useful in combination with range requests (). A client could still benefit from the use of the coding for a series of range requests if they are contiguous with the first byte of the representation. Compression content coding, such as gzip or br, can be applied in combination with chi in any order. Applying integrity after compression ensures that invalid content does not need to be decompressed; whereas applying integrity before compression allows compression to be negotiated without needing fresh integrity values.

Chi-Digest Field The Chi-Digest field is a sf-list header field; see . Each list item is a byte sequence; see . This field carries only chi digests, each value being 32 bytes. Values with other types or lengths MUST be treated as errors, causing the entire field value to be ignored, as recommended by . No semantics are defined for parameters, which MUST be ignored on receipt, as recommended by . Each digest in the Chi-Digest field corresponds to a chi content coding in the order that they are applied (this follows the same rule as content codings; see ). That means that a recipient uses the last Chi-Digest value when removing the outermost chi coding. Use of the Chi-Digest field is not mandatory when using the chi content coding. However, if the Chi-Digest field is present when adding an additional chi content coding, the field MUST either be removed or have a value added. This prevents existing values from being invalidated. An intermediary that removes a chi coding MUST remove the last Chi-Digest value if the field is present. This field offers no means of negotiating different hash algorithms; see . The Chi-Digest field can be carried in a trailer field, but this is strictly worse than a Repr-Digest field with a sha-256 digest, as defined in . If the chi digest is only delivered in a trailer, progressive validation cannot be performed.

Example This example shows a 47-byte sequence split into chiclets of 16 bytes. This chiclet size is far smaller than would be used in practice, but it allows the example to show multiple chiclets. The resulting 3 chiclets are shown in , with the hash of each, starting from the last.

Example Chiclets

The chi digest can be carried in a Chi-Digest field, as shown in .

Example Response Header

The 114 bytes of response content are shown in in hexadecimal.

Example Encoded Content

Empty content is encoded as a single final chiclet, comprising a single byte with a value of 0. The chi digest of a zero-length sequence is therefore the SHA-256 hash of that one byte, as shown in .

Fields for Empty Content

Note that these examples use the minimal encoding for the variable-length integers. Longer encodings for those integers would produce different digest values.

Security Considerations This coding provides integrity protection equivalent to hashing the message content. However, there are subtle differences that need to be noted. In general, integrity is only valuable if the chi digest comes from a source that also has integrity protection. A chi digest might be provided alongside a reference to a resource (such as with ) or it might be signed (for instance, an HTTP Message Signature that covers a Chi-Digest field). The integrity property depends on the hash function chosen (). Recipients are exposed to a risk of message truncation (). There is also a resource exhaustion exposure, particularly for recipients with limited resources (). Finally, the digest value produced is not canonical ().

Hash Choice Integrity protection relies on the second pre-image resistance property of SHA-256 . If a recipient holds a chi digest, an attacker that wishes to substitute content needs to find a second sequence of chiclets that will produce the same chi digest. If the digest is subsequently signed, the collision resistance of SHA-256 is also relied on to prevent message substitution. This document does not define a way to negotiate an alternative hash function. If attacks on SHA-256 are considered an unacceptable risk, new codings and fields can be defined in a new specification. Whether those items reuse the construction in this document with a different hash function can be considered.

Truncation The use of a streaming format means that data might be released to an application for processing, with no guarantee that subsequent chiclets will be delivered intact. Any application that uses this format therefore needs to be robust in the event that its input is truncated. The final chiclet contains a zero-valued length prefix. The prefix of every chiclet is hashed, so a value of zero ensures that the final chiclet is easily distinguished from any non-final chiclets. This ensures that truncation can be reliably detected.

Chiclet Size Denial of Service The requirement to hold an entire chiclet until it is validated creates a denial of service risk; see for discussion and mitigations.

Non-Canonical The same sequence of bytes can produce many different integrity values. As a result, the chi digest cannot act as a canonical identifier for content. Each digest value commits to a specific sequence of chiclets, so cannot be used with a different arrangement of chiclet lengths. Accepting one of a set of values to safeguard against chiclet layout changes increases the chance of a pre-image attack by a negligible amount for each accepted chi digest.

IANA Considerations This document registers a chi content coding and a header field for carrying chi digests.

Content Coding IANA is requested to add the following entry to the "HTTP Content Coding Registry" established by :

Name:

chi

Description:

Chained Hash Integrity Coding

Reference:

of this document

Chi-Digest Header Field IANA is requested to add the following entry to the "Hypertext Transfer Protocol (HTTP) Field Name Registry" established by :

Field Name:

Chi-Digest

Status:

permanent

Structured Type:

List

Reference:

of this document

Comments:

(none)

References Normative References In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This document defines the core of the QUIC transport protocol. QUIC provides applications with flow-controlled streams for structured communication, low-latency connection establishment, and network path migration. QUIC includes security measures that ensure confidentiality, integrity, and availability in a range of deployment circumstances. Accompanying documents describe the integration of TLS for key negotiation, loss detection, and an exemplary congestion control algorithm. Federal Information Processing Standard, FIPS The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230. This document describes a set of data types and associated algorithms that are intended to make it easier and safer to define and handle HTTP header and trailer fields, known as "Structured Fields", "Structured Headers", or "Structured Trailers". It is intended for use by specifications of new HTTP fields. This document obsoletes RFC 8941. Informative References This document defines HTTP fields that support integrity digests. The Content-Digest field can be used for the integrity of HTTP message content. The Repr-Digest field can be used for the integrity of HTTP representations. Want-Content-Digest and Want-Repr-Digest can be used to indicate a sender's interest and preferences for receiving the respective Integrity fields. This document obsoletes RFC 3230 and the Digest and Want-Digest HTTP fields. This document describes a mechanism for creating, encoding, and verifying digital signatures or message authentication codes over components of an HTTP message. This mechanism supports use cases where the full HTTP message may not be known to the signer and where the message may be transformed (e.g., by intermediaries) before reaching the verifier. This document also describes a means for requesting that a signature be applied to a subsequent HTTP message in an ongoing HTTP exchange.

Acknowledgments was a co-author on a previous iteration of this work. suggested the Chi-Digest field, rather than an integration with RFC 9530.