idnits 2.17.1 draft-ietf-httpbis-rand-access-live-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (November 14, 2017) is 2348 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Experimental ---------------------------------------------------------------------------- == Unused Reference: 'RANGE-UNIT-REGISTRY' is defined on line 392, but no explicit reference was found in the text == Unused Reference: 'RFC4234' is defined on line 397, but no explicit reference was found in the text ** Obsolete normative reference: RFC 7230 (Obsoleted by RFC 9110, RFC 9112) ** Obsolete normative reference: RFC 7233 (Obsoleted by RFC 9110) -- Obsolete informational reference (is this intentional?): RFC 4234 (Obsoleted by RFC 5234) Summary: 3 errors (**), 0 flaws (~~), 4 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 HTTP Working Group C. Pratt 3 Internet-Draft CableLabs 4 Intended status: Experimental B. Stark 5 Expires: May 18, 2018 AT&T 6 D. Thakore 7 CableLabs 8 November 14, 2017 10 HTTP Random Access and Live Content 11 draft-ietf-httpbis-rand-access-live-02 13 Abstract 15 To accommodate byte range requests for content that has data appended 16 over time, this document defines semantics that allow a HTTP client 17 and server to perform byte-range GET and HEAD requests that start at 18 an arbitrary byte offset within the representation and ends at an 19 indeterminate offset. 21 Editorial Note (To be removed by RFC Editor before publication) 23 Discussion of this draft takes place on the HTTPBIS working group 24 mailing list (ietf-http-wg@w3.org), which is archived at 25 . 27 Working Group information can be found at ; 28 source code and issues list for this draft can be found at 29 . 31 Status of This Memo 33 This Internet-Draft is submitted in full conformance with the 34 provisions of BCP 78 and BCP 79. 36 Internet-Drafts are working documents of the Internet Engineering 37 Task Force (IETF). Note that other groups may also distribute 38 working documents as Internet-Drafts. The list of current Internet- 39 Drafts is at https://datatracker.ietf.org/drafts/current/. 41 Internet-Drafts are draft documents valid for a maximum of six months 42 and may be updated, replaced, or obsoleted by other documents at any 43 time. It is inappropriate to use Internet-Drafts as reference 44 material or to cite them other than as "work in progress." 46 This Internet-Draft will expire on May 18, 2018. 48 Copyright Notice 50 Copyright (c) 2017 IETF Trust and the persons identified as the 51 document authors. All rights reserved. 53 This document is subject to BCP 78 and the IETF Trust's Legal 54 Provisions Relating to IETF Documents 55 (https://trustee.ietf.org/license-info) in effect on the date of 56 publication of this document. Please review these documents 57 carefully, as they describe your rights and restrictions with respect 58 to this document. Code Components extracted from this document must 59 include Simplified BSD License text as described in Section 4.e of 60 the Trust Legal Provisions and are provided without warranty as 61 described in the Simplified BSD License. 63 Table of Contents 65 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 66 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 67 2. Performing Range requests on Random-Access Aggregating 68 ("live") Content . . . . . . . . . . . . . . . . . . . . . . 3 69 2.1. Establishing the Randomly Accessible Byte Range . . . . . 4 70 2.2. Byte-Range Requests Beyond the Randomly Accessible Byte 71 Range . . . . . . . . . . . . . . . . . . . . . . . . . . 5 72 3. Other Applications of Random-Access Aggregating Content . . . 7 73 3.1. Requests Starting at the Aggregation ("Live") Point . . . 7 74 3.2. Shift Buffer Representations . . . . . . . . . . . . . . 7 75 4. Security Considerations . . . . . . . . . . . . . . . . . . . 8 76 5. References . . . . . . . . . . . . . . . . . . . . . . . . . 9 77 5.1. Normative References . . . . . . . . . . . . . . . . . . 9 78 5.2. Informative References . . . . . . . . . . . . . . . . . 9 79 Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 9 80 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 9 82 1. Introduction 84 Some Hypertext Transfer Protocol (HTTP) clients use byte-range 85 requests (Range requests using the "bytes" Range Unit) to transfer 86 select portions of large representations. And in some cases large 87 representations require content to be continuously or periodically 88 appended - such as representations consisting of live audio or video 89 sources, blockchain databases, and log files. Clients cannot access 90 the appended/live content using a Range request with the bytes range 91 unit using the currently defined byte-range semantics without 92 accepting performance or behavior sacrifices which are not acceptable 93 for many applications. 95 For instance, HTTP clients have the ability to access appended 96 content on an indeterminate-length resource by transferring the 97 entire representation from the beginning and continuing to read the 98 appended content as it's made available. Obviously, this is highly 99 inefficient for cases where the representation is large and only the 100 most recently appended content is needed by the client. 102 Alternatively, clients can also access appended content by sending 103 periodic open-ended bytes Range requests using the last-known end 104 byte position as the range start. Performing low-frequency periodic 105 bytes Range requests in this fashion (polling) introduces latency 106 since the client will necessarily be somewhat behind the aggregated 107 content - mimicking the behavior (and latency) of segmented content 108 representations such as HLS or MPEG-DASH. And while performing these 109 Range requests at higher frequency can reduce this latency, it also 110 incurs more processing overhead and HTTP exchanges as many of the 111 requests will return no content - since content is usually aggregated 112 in groups of bytes (e.g. a video frame, audio sample, block, or log 113 entry). 115 This document describes a usage model for range requests which 116 enables efficient retrieval of representations that are appended to 117 over time by using large values and associated semantics for 118 communicating range end positions. This model allows representations 119 to be progressively delivered by servers as new content is added. It 120 also ensures compatibility with servers and intermediaries that don't 121 support this technique. 123 1.1. Requirements Language 125 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 126 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 127 document are to be interpreted as described in RFC 2119 [RFC2119]. 129 2. Performing Range requests on Random-Access Aggregating ("live") 130 Content 132 This document recommends a two-step process for accessing resources 133 that have indeterminate length representations. Two steps are 134 necessary because of limitations with the Range request header and 135 the Content-Range response header fields. A server cannot know from 136 a range request that a client wishes to receive a response that does 137 not have a definite end. More critically, the header fields do not 138 allow the server to signal that a resource has indeterminate length 139 without also providing a fixed portion of the resource. A client 140 first learns that the resource has a representation of indeterminate 141 length by requesting a range of the resource. The server responds 142 with the range that is available, but indicates that the length of 143 the representation is unknown using the existing Content-Range 144 syntax. See Section 2.1 for details and examples. Once the client 145 knows the resource has indeterminate length, it can request a range 146 with a very large end position from the resource. The client chooses 147 an explicit end value larger than can be transferred in the 148 foreseeable term. A server which supports range requests of 149 indeterminate length signals its understanding of the client's 150 indeterminate range request by indicating that the range it is 151 providing has a range end that exactly matches the client's requested 152 range end rather than a range that is bounded by what is currently 153 available. See Section 2.2 for details. 155 2.1. Establishing the Randomly Accessible Byte Range 157 Establishing if a representation is continuously aggregating ("live") 158 and determining the randomly-accessible byte range can both be 159 determined using the existing definition for an open-ended byte-range 160 request. Specifically, [RFC7233] defines a byte-range request of the 161 form: 163 byte-range-spec = first-byte-pos "-" [ last-byte-pos ] 165 which allows a client to send a HEAD request with a first-byte-pos 166 and leave last-byte-pos absent. A server that receives a satisfiable 167 byte-range request (with first-byte-pos smaller than the current 168 representation length) may respond with a 206 status code (Partial 169 Content) with a Content-Range header indicating the currently 170 satisfiable byte range. For example: 172 HEAD /resource HTTP/1.1 173 Range: bytes=0- 175 returns a response of the form: 177 HTTP/1.1 206 Partial Content 178 Content-Range: bytes 0-1234567/* 180 from the server indicating that (1) the complete representation 181 length is unknown (via the "*" in place of the complete-length field) 182 and (2) that only bytes 0-1234567 were accessable at the time the 183 request was processed by the server. The client can infer from this 184 response that bytes 0-1234567 of the representation can be requested 185 and returned in a timely fashion (the bytes are immediately 186 available). 188 2.2. Byte-Range Requests Beyond the Randomly Accessible Byte Range 190 Once a client has determined that a representation has an 191 indeterminate length and established the byte range that can be 192 accessed, it may want to perform a request with a start position 193 within the randomly-accessible content range and an end position at 194 an indefinite "live" point - a point where the byte-range GET request 195 is fulfilled on-demand as the content is aggregated. 197 For example, for a large video asset, a client may wish to start a 198 content transfer from the video "key" frame immediately before the 199 point of aggregation and continue the content transfer indefinitely 200 as content is aggregated - in order to support low-latency startup of 201 a live video stream. 203 Unlike a byte-range Range request, a byte-range Content-Range 204 response header cannot be "open ended", per [RFC7233]: 206 byte-content-range = bytes-unit SP 207 ( byte-range-resp / unsatisfied-range ) 209 byte-range-resp = byte-range "/" ( complete-length / "*" ) 210 byte-range = first-byte-pos "-" last-byte-pos 211 unsatisfied-range = "*/" complete-length 213 complete-length = 1*DIGIT 215 Specifically, last-byte-pos is required in byte-range. So in order 216 to preserve interoperability with existing HTTP clients, servers, 217 proxies, and caches, this document proposes a mechanism for a client 218 to indicate support for handling an indeterminate-length byte-range 219 response, and a mechanism for a server to indicate if/when it's 220 providing a indeterminate-length response. 222 A client can indicate support for handling indeterminate-length byte- 223 range responses by providing a Very Large Value for the last-byte-pos 224 in the byte-range request. For example, a client can perform a byte- 225 range GET request of the form: 227 GET /resource HTTP/1.1 228 Range: bytes=1230000-999999999999 230 where the last-byte-pos in the Request is much larger than the last- 231 byte-pos returned in response to an open-ended byte-range HEAD 232 request, as described above. 234 In response, a server may indicate that it is supplying a 235 continuously aggregating ("live") response by supplying the client 236 request's last-byte-pos in the Content-Range response header. 238 For example: 240 GET /resource HTTP/1.1 241 Range: bytes=1230000-999999999999 243 returns 245 HTTP/1.1 206 Partial Content 246 Content-Range: bytes 1230000-999999999999/* 248 from the server to indicate that the response will start at byte 249 1230000 and continues indefinitely to include all aggregated content, 250 as it becomes available. 252 A server that doesn't support or supply a continuously aggregating 253 ("live") response will supply the currently satisfiable byte range, 254 as it would with an open-ended byte request. 256 For example: 258 GET /resource HTTP/1.1 259 Range: bytes=1230000-999999999999 261 will return 263 HTTP/1.1 206 Partial Content 264 Content-Range: bytes 1230000-1234567/* 266 from the server to indicate that the response will start at byte 267 1230000 and end at byte 1234567 and will not include any aggregated 268 content. This is the response expected from a typical HTTP server - 269 one that doesn't support byte-range requests on aggregating content. 271 A client that doesn't receive a response indicating it is 272 continuously aggregating must use other means to access aggregated 273 content (e.g. periodic byte-range polling). 275 A server that does return a continuously aggregating ("live") 276 response should return data using chunked transfer coding and not 277 provide a Content-Length header. A 0-length chunk indicates the end 278 of the transfer, per section 4.1 of [RFC7230]. 280 3. Other Applications of Random-Access Aggregating Content 282 3.1. Requests Starting at the Aggregation ("Live") Point 284 A client that wishes to only receive newly-aggregated portions of a 285 resource (i.e., start at the "live" point), can use a HEAD request to 286 learn what range the server has currently available and initiate an 287 indeterminate-length transfer. For example: 289 HEAD /resource HTTP/1.1 290 Range: bytes=0- 292 With the Content-Range response header indicating the (or ranges) 293 available. For example: 295 206 Partial Content 296 Content-Range: bytes 0-1234567/* 298 The client can then issue a request for a range starting at the end 299 value (using a very large value for the end of a range) and receive 300 only new content. 302 GET /resource HTTP/1.1 303 Range: bytes=1234567-999999999999 305 with a server returning a Content-Range response indicating that an 306 indeterminate-length response body will be provided 308 206 Partial Content 309 Content-Range: bytes 1234567-999999999999/* 311 3.2. Shift Buffer Representations 313 Some representations lend themselves to front-end content deletion in 314 addition to aggregation. While still supporting random access, 315 representations of this type have a portion at the beginning (the "0" 316 end) of the randomly-accessible region that become inaccessible over 317 time. Examples of this kind of representation would be an audio- 318 video time-shift buffer or a rolling log file. 320 For example a Range request containing: 322 HEAD /resource HTTP/1.1 323 Range: bytes=0- 325 returns 326 206 Partial Content 327 Content-Range: bytes 1000000-1234567/* 329 indicating that the first 1000000 bytes were not accessible at the 330 time the HEAD request was processed. Subsequent HEAD requests could 331 return: 333 Content-Range: bytes 1000000-1234567/* 335 Content-Range: bytes 1010000-1244567/* 337 Content-Range: bytes 1020000-1254567/* 339 Note though that the difference between the first-byte-pos and last- 340 byte-pos need not be constant. 342 The client could then follow-up with a GET Range request containing 344 GET /resource HTTP/1.1 345 Range: bytes=1020000-999999999999 347 with the server returning 349 206 Partial Content 350 Content-Range: bytes 1020000-999999999999/* 352 with the response body returning bytes 1020000-1254567 immediately 353 and aggregated ("live") data being returned as the content is 354 aggregated. 356 4. Security Considerations 358 One potential issue with this recommendation is related to the use of 359 very-large last-byte-pos values. Some client and server 360 implementations may not be prepared to deal with byte position values 361 of 2^^63 and beyond. So in applications where there's no expectation 362 that the representation will ever exceed 2^^63, a value smaller than 363 this value should be used as the Very Large last-byte-pos in a byte- 364 seek request or content-range response. Also, some implementations 365 (e.g. JavaScript-based clients and servers) are not able to 366 represent all values beyond 2^^53. So similarly, if there's no 367 expectation that a representation will ever exceed 2^^53 bytes, 368 values smaller than this limit should be used for the last-byte-pos 369 in byte-range requests. 371 5. References 373 5.1. Normative References 375 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 376 Requirement Levels", BCP 14, RFC 2119, 377 DOI 10.17487/RFC2119, March 1997, 378 . 380 [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer 381 Protocol (HTTP/1.1): Message Syntax and Routing", 382 RFC 7230, DOI 10.17487/RFC7230, June 2014, 383 . 385 [RFC7233] Fielding, R., Ed., Lafon, Y., Ed., and J. Reschke, Ed., 386 "Hypertext Transfer Protocol (HTTP/1.1): Range Requests", 387 RFC 7233, DOI 10.17487/RFC7233, June 2014, 388 . 390 5.2. Informative References 392 [RANGE-UNIT-REGISTRY] 393 IANA, "Hypertext Transfer Protocol (HTTP) Parameters", 394 2016, . 397 [RFC4234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax 398 Specifications: ABNF", RFC 4234, DOI 10.17487/RFC4234, 399 October 2005, . 401 Appendix A. Acknowledgements 403 Mark Nottingham, Patrick McManus, Julian Reschke, Remy Lebeau, Rodger 404 Combs, Thorsten Lohmar, Martin Thompson, Adrien de Croy, K. Morgan, 405 Roy T. Fielding, Jeremy Poulter. 407 Authors' Addresses 409 Craig Pratt 410 CableLabs 411 858 Coal Creek Circle 412 Louisville, CO 80027 414 Email: pratt@acm.org 415 Barbara Stark 416 AT&T 417 Atlanta, GA 418 US 420 Email: barbara.stark@att.com 422 Darshak Thakore 423 CableLabs 424 858 Coal Creek Circle 425 Louisville, CO 80027 427 Email: d.thakore@cablelabs.com