idnits 2.17.1 draft-ietf-httpbis-rand-access-live-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (March 6, 2019) is 1877 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Experimental ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 7230 (Obsoleted by RFC 9110, RFC 9112) ** Obsolete normative reference: RFC 7233 (Obsoleted by RFC 9110) ** Obsolete normative reference: RFC 7234 (Obsoleted by RFC 9111) Summary: 3 errors (**), 0 flaws (~~), 2 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 HTTP C. Pratt 3 Internet-Draft 4 Intended status: Experimental D. Thakore 5 Expires: September 7, 2019 CableLabs 6 B. Stark 7 AT&T 8 March 6, 2019 10 HTTP Random Access and Live Content 11 draft-ietf-httpbis-rand-access-live-04 13 Abstract 15 To accommodate byte range requests for content that has data appended 16 over time, this document defines semantics that allow a HTTP client 17 and server to perform byte-range GET and HEAD requests that start at 18 an arbitrary byte offset within the representation and ends at an 19 indeterminate offset. 21 Editorial Note (To be removed by RFC Editor before publication) 23 Discussion of this draft takes place on the HTTPBIS working group 24 mailing list (ietf-http-wg@w3.org), which is archived at 25 . 27 Working Group information can be found at ; 28 source code and issues list for this draft can be found at 29 . 31 Status of This Memo 33 This Internet-Draft is submitted in full conformance with the 34 provisions of BCP 78 and BCP 79. 36 Internet-Drafts are working documents of the Internet Engineering 37 Task Force (IETF). Note that other groups may also distribute 38 working documents as Internet-Drafts. The list of current Internet- 39 Drafts is at http://datatracker.ietf.org/drafts/current/. 41 Internet-Drafts are draft documents valid for a maximum of six months 42 and may be updated, replaced, or obsoleted by other documents at any 43 time. It is inappropriate to use Internet-Drafts as reference 44 material or to cite them other than as "work in progress." 46 This Internet-Draft will expire on September 7, 2019. 48 Copyright Notice 50 Copyright (c) 2019 IETF Trust and the persons identified as the 51 document authors. All rights reserved. 53 This document is subject to BCP 78 and the IETF Trust's Legal 54 Provisions Relating to IETF Documents 55 (http://trustee.ietf.org/license-info) in effect on the date of 56 publication of this document. Please review these documents 57 carefully, as they describe your rights and restrictions with respect 58 to this document. Code Components extracted from this document must 59 include Simplified BSD License text as described in Section 4.e of 60 the Trust Legal Provisions and are provided without warranty as 61 described in the Simplified BSD License. 63 Table of Contents 65 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 66 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 67 1.2. Notational Conventions . . . . . . . . . . . . . . . . . 3 68 2. Performing Range requests on Random-Access Aggregating 69 ("live") Content . . . . . . . . . . . . . . . . . . . . . . 4 70 2.1. Establishing the Randomly Accessible Byte Range . . . . . 4 71 2.2. Byte-Range Requests Beyond the Randomly Accessible Byte 72 Range . . . . . . . . . . . . . . . . . . . . . . . . . . 5 73 3. Other Applications of Random-Access Aggregating Content . . . 7 74 3.1. Requests Starting at the Aggregation ("Live") Point . . . 7 75 3.2. Shift Buffer Representations . . . . . . . . . . . . . . 8 76 4. Recommendations for Very Large Values . . . . . . . . . . . . 10 77 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 78 6. Security Considerations . . . . . . . . . . . . . . . . . . . 10 79 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 11 80 7.1. Normative References . . . . . . . . . . . . . . . . . . 11 81 7.2. Informative References . . . . . . . . . . . . . . . . . 11 82 Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 11 83 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 12 85 1. Introduction 87 Some Hypertext Transfer Protocol (HTTP) clients use byte-range 88 requests (Range requests using the "bytes" Range Unit) to transfer 89 select portions of large representations ([RFC7233]). And in some 90 cases large representations require content to be continuously or 91 periodically appended - such as representations consisting of live 92 audio or video sources, blockchain databases, and log files. Clients 93 cannot access the appended/live content using a Range request with 94 the bytes range unit using the currently defined byte-range semantics 95 without accepting performance or behavior sacrifices which are not 96 acceptable for many applications. 98 For instance, HTTP clients have the ability to access appended 99 content on an indeterminate-length resource by transferring the 100 entire representation from the beginning and continuing to read the 101 appended content as it's made available. Obviously, this is highly 102 inefficient for cases where the representation is large and only the 103 most recently appended content is needed by the client. 105 Alternatively, clients can also access appended content by sending 106 periodic open-ended bytes Range requests using the last-known end 107 byte position as the range start. Performing low-frequency periodic 108 bytes Range requests in this fashion (polling) introduces latency 109 since the client will necessarily be somewhat behind the aggregated 110 content - mimicking the behavior (and latency) of segmented content 111 representations such as "HTTP Live Streaming" (HLS, [RFC8216]) or 112 "Dynamic Adaptive Streaming over HTTP" (MPEG-DASH, [DASH]). And 113 while performing these Range requests at higher frequency can reduce 114 this latency, it also incurs more processing overhead and HTTP 115 exchanges as many of the requests will return no content - since 116 content is usually aggregated in groups of bytes (e.g. a video frame, 117 audio sample, block, or log entry). 119 This document describes a usage model for range requests which 120 enables efficient retrieval of representations that are appended to 121 over time by using large values and associated semantics for 122 communicating range end positions. This model allows representations 123 to be progressively delivered by servers as new content is added. It 124 also ensures compatibility with servers and intermediaries that don't 125 support this technique. 127 1.1. Requirements Language 129 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 130 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 131 document are to be interpreted as described in RFC 2119 [RFC2119]. 133 1.2. Notational Conventions 135 This document cites productions in Augmented Backus-Naur Form (ABNF) 136 productions from [RFC7233], using the notation defined in [RFC5234]. 138 2. Performing Range requests on Random-Access Aggregating ("live") 139 Content 141 This document recommends a two-step process for accessing resources 142 that have indeterminate length representations. 144 Two steps are necessary because of limitations with the Range request 145 header fields and the Content-Range response header fields. A server 146 cannot know from a range request that a client wishes to receive a 147 response that does not have a definite end. More critically, the 148 header fields do not allow the server to signal that a resource has 149 indeterminate length without also providing a fixed portion of the 150 resource. 152 A client first learns that the resource has a representation of 153 indeterminate length by requesting a range of the resource. The 154 server responds with the range that is available, but indicates that 155 the length of the representation is unknown using the existing 156 Content-Range syntax. See Section 2.1 for details and examples. 158 Once the client knows the resource has indeterminate length, it can 159 request a range with a very large end position from the resource. 160 The client chooses an explicit end value larger than can be 161 transferred in the foreseeable term. A server which supports range 162 requests of indeterminate length signals its understanding of the 163 client's indeterminate range request by indicating that the range it 164 is providing has a range end that exactly matches the client's 165 requested range end rather than a range that is bounded by what is 166 currently available. See Section 2.2 for details. 168 2.1. Establishing the Randomly Accessible Byte Range 170 Establishing if a representation is continuously aggregating ("live") 171 and determining the randomly-accessible byte range can both be 172 determined using the existing definition for an open-ended byte-range 173 request. Specifically, [RFC7233] defines a byte-range request of the 174 form: 176 byte-range-spec = first-byte-pos "-" [ last-byte-pos ] 178 which allows a client to send a HEAD request with a first-byte-pos 179 and leave last-byte-pos absent. A server that receives a satisfiable 180 byte-range request (with first-byte-pos smaller than the current 181 representation length) may respond with a 206 status code (Partial 182 Content) with a Content-Range header field indicating the currently 183 satisfiable byte range. For example: 185 HEAD /resource HTTP/1.1 186 Host: example.com 187 Range: bytes=0- 189 returns a response of the form: 191 HTTP/1.1 206 Partial Content 192 Content-Range: bytes 0-1234567/* 194 from the server indicating that (1) the complete representation 195 length is unknown (via the "*" in place of the complete-length field) 196 and (2) that only bytes 0-1234567 were accessible at the time the 197 request was processed by the server. The client can infer from this 198 response that bytes 0-1234567 of the representation can be requested 199 and returned in a timely fashion (the bytes are immediately 200 available). 202 2.2. Byte-Range Requests Beyond the Randomly Accessible Byte Range 204 Once a client has determined that a representation has an 205 indeterminate length and established the byte range that can be 206 accessed, it may want to perform a request with a start position 207 within the randomly-accessible content range and an end position at 208 an indefinite "live" point - a point where the byte-range GET request 209 is fulfilled on-demand as the content is aggregated. 211 For example, for a large video asset, a client may wish to start a 212 content transfer from the video "key" frame immediately before the 213 point of aggregation and continue the content transfer indefinitely 214 as content is aggregated - in order to support low-latency startup of 215 a live video stream. 217 Unlike a byte-range Range request, a byte-range Content-Range 218 response header field cannot be "open ended", per [RFC7233]: 220 byte-content-range = bytes-unit SP 221 ( byte-range-resp / unsatisfied-range ) 223 byte-range-resp = byte-range "/" ( complete-length / "*" ) 224 byte-range = first-byte-pos "-" last-byte-pos 225 unsatisfied-range = "*/" complete-length 227 complete-length = 1*DIGIT 229 Specifically, last-byte-pos is required in byte-range. So in order 230 to preserve interoperability with existing HTTP clients, servers, 231 proxies, and caches, this document proposes a mechanism for a client 232 to indicate support for handling an indeterminate-length byte-range 233 response, and a mechanism for a server to indicate if/when it's 234 providing an indeterminate-length response. 236 A client can indicate support for handling indeterminate-length byte- 237 range responses by providing a very large value for the last-byte-pos 238 in the byte-range request. For example, a client can perform a byte- 239 range GET request of the form: 241 GET /resource HTTP/1.1 242 Host: example.com 243 Range: bytes=1230000-999999999999 245 where the last-byte-pos in the Request is much larger than the last- 246 byte-pos returned in response to an open-ended byte-range HEAD 247 request, as described above, and much larger than the expected 248 maximum size of the representation. See Section 6 for range value 249 considerations. 251 In response, a server may indicate that it is supplying a 252 continuously aggregating ("live") response by supplying the client 253 request's last-byte-pos in the Content-Range response header field. 255 For example: 257 GET /resource HTTP/1.1 258 Host: example.com 259 Range: bytes=1230000-999999999999 261 returns 263 HTTP/1.1 206 Partial Content 264 Content-Range: bytes 1230000-999999999999/* 266 from the server to indicate that the response will start at byte 267 1230000 and continues indefinitely to include all aggregated content, 268 as it becomes available. 270 A server that doesn't support or supply a continuously aggregating 271 ("live") response will supply the currently satisfiable byte range, 272 as it would with an open-ended byte request. 274 For example: 276 GET /resource HTTP/1.1 277 Host: example.com 278 Range: bytes=1230000-999999999999 280 will return 282 HTTP/1.1 206 Partial Content 283 Content-Range: bytes 1230000-1234567/* 285 from the server to indicate that the response will start at byte 286 1230000 and end at byte 1234567 and will not include any aggregated 287 content. This is the response expected from a typical HTTP server - 288 one that doesn't support byte-range requests on aggregating content. 290 A client that doesn't receive a response indicating it is 291 continuously aggregating must use other means to access aggregated 292 content (e.g. periodic byte-range polling). 294 A server that does return a continuously aggregating ("live") 295 response should return data using chunked transfer coding and not 296 provide a Content-Length header field. A 0-length chunk indicates 297 the end of the transfer, per [RFC7230]. 299 3. Other Applications of Random-Access Aggregating Content 301 3.1. Requests Starting at the Aggregation ("Live") Point 303 A client that wishes to only receive newly-aggregated portions of a 304 resource (i.e., start at the "live" point), can use a HEAD request to 305 learn what range the server has currently available and initiate an 306 indeterminate-length transfer. For example: 308 HEAD /resource HTTP/1.1 309 Host: example.com 310 Range: bytes=0- 312 With the Content-Range response header field indicating the range (or 313 ranges) available. For example: 315 206 Partial Content 316 Content-Range: bytes 0-1234567/* 318 The client can then issue a request for a range starting at the end 319 value (using a very large value for the end of a range) and receive 320 only new content. 322 GET /resource HTTP/1.1 323 Host: example.com 324 Range: bytes=1234567-999999999999 326 with a server returning a Content-Range response indicating that an 327 indeterminate-length response body will be provided 329 206 Partial Content 330 Content-Range: bytes 1234567-999999999999/* 332 3.2. Shift Buffer Representations 334 Some representations lend themselves to front-end content removal in 335 addition to aggregation. While still supporting random access, 336 representations of this type have a portion at the beginning (the "0" 337 end) of the randomly-accessible region that become inaccessible over 338 time. Examples of this kind of representation would be an audio- 339 video time-shift buffer or a rolling log file. 341 For example a Range request containing: 343 HEAD /resource HTTP/1.1 344 Host: example.com 345 Range: bytes=0- 347 returns 349 206 Partial Content 350 Content-Range: bytes 1000000-1234567/* 352 indicating that the first 1000000 bytes were not accessible at the 353 time the HEAD request was processed. Subsequent HEAD requests could 354 return: 356 Content-Range: bytes 1000000-1234567/* 358 Content-Range: bytes 1010000-1244567/* 360 Content-Range: bytes 1020000-1254567/* 362 Note though that the difference between the first-byte-pos and last- 363 byte-pos need not be constant. 365 The client could then follow-up with a GET Range request containing 366 GET /resource HTTP/1.1 367 Host: example.com 368 Range: bytes=1020000-999999999999 370 with the server returning 372 206 Partial Content 373 Content-Range: bytes 1020000-999999999999/* 375 with the response body returning bytes 1020000-1254567 immediately 376 and aggregated ("live") data being returned as the content is 377 aggregated. 379 A server that doesn't support or supply a continuously aggregating 380 ("live") response will supply the currently satisfiable byte range, 381 as it would with an open-ended byte request. 383 For example: 385 GET /resource HTTP/1.1 386 Host: example.com 387 Range: bytes=0-999999999999 389 will return 391 HTTP/1.1 206 Partial Content 392 Content-Range: bytes 1020000-1254567/* 394 from the server to indicate that the response will start at byte 395 1020000, end at byte 1254567, and will not include any aggregated 396 content. This is the response expected from a typical HTTP server - 397 one that doesn't support byte-range requests on aggregating content. 399 Note that responses to GET requests against shift-buffer 400 representations using Range can be cached by intermediaries, since 401 the Content-Range response header indicates which portion of the 402 representation is being returned in the response body. However GET 403 requests without a Range header cannot be cached since the first byte 404 of the response body can vary from request to request. To ensure 405 Range-less GET requests against shift-buffer representations are not 406 cached, servers hosting a shift-buffer representation should either 407 not return a 200-level response (e.g. sending a 300-level redirect 408 response with a URI that represents the current start of the shift- 409 buffer) or indicate the response is non-cacheable. See HTTP Caching 410 ([RFC7234]) for details on HTTP cache control. 412 4. Recommendations for Very Large Values 414 While it would be ideal to define a single numerical Very Large 415 Value, there's no single value that would work for all applications 416 and platforms. e.g. JavaScript numbers cannot represent all integer 417 values above 2^^53, so a JavaScript application may want to use 418 2^^53-1 for a Very Large Value. This value, however, would not be 419 sufficient for all applications, such as continuously-streaming high- 420 bitrate streams. So the value 2^^53-1 (9007199254740991) is 421 recommended as a Very Large Value unless an application has a good 422 justification to use a smaller or larger value. e.g. If it's always 423 known that the resource won't exceed a value smaller than the 424 recommended Very Large Value for an application, a smaller value can 425 be used. And if it's likely that an application will utilize 426 resources larger than the recommended Very Large Value - such as a 427 continuously aggregating high-bitrate media stream - a larger value 428 should be used. 430 Note that, in accordance with the semantics defined above, servers 431 that support random-access live content will need to return the last- 432 byte-pos provided in the Range request in some cases - even if the 433 last-byte-pos cannot be represented as a numerical value internally 434 by the server. As is the case with any live/continuously aggregating 435 resource, the server should terminate the content transfer when the 436 end of the resource is reached - whether the end is due to 437 termination of the content source or the content length exceeds the 438 server's maximum representation length. 440 5. IANA Considerations 442 This document has no actions for IANA. 444 6. Security Considerations 446 As described above, servers need to be prepared to receive last-byte- 447 pos values in Range requests that are numerically larger than the 448 server implementation supports - and return these values in Content- 449 Range response header fields. Servers should check the last-byte-pos 450 value before converting and storing them into numeric form to ensure 451 the value doesn't cause an overflow or index incorrect data. The 452 simplest way to satisfy the live-range semantics defined in this 453 document without potential overflow issues is to store the last-byte- 454 pos as a string value and return it in the byte-range Content-Range 455 response header's last-byte-pos field. 457 7. References 459 7.1. Normative References 461 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 462 Requirement Levels", BCP 14, RFC 2119, 463 DOI 10.17487/RFC2119, March 1997, . 466 [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer 467 Protocol (HTTP/1.1): Message Syntax and Routing", 468 RFC 7230, DOI 10.17487/RFC7230, June 2014, 469 . 471 [RFC7233] Fielding, R., Ed., Lafon, Y., Ed., and J. Reschke, Ed., 472 "Hypertext Transfer Protocol (HTTP/1.1): Range Requests", 473 RFC 7233, DOI 10.17487/RFC7233, June 2014, 474 . 476 [RFC7234] Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke, 477 Ed., "Hypertext Transfer Protocol (HTTP/1.1): Caching", 478 RFC 7234, DOI 10.17487/RFC7234, June 2014, 479 . 481 7.2. Informative References 483 [DASH] ISO, "Information technology -- Dynamic adaptive streaming 484 over HTTP (DASH) -- Part 1: Media presentation description 485 and segment formats", ISO/IEC 23009-1:2014, May 2014, 486 . 489 [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax 490 Specifications: ABNF", STD 68, RFC 5234, 491 DOI 10.17487/RFC5234, January 2008, . 494 [RFC8216] Pantos, R., Ed. and W. May, "HTTP Live Streaming", 495 RFC 8216, DOI 10.17487/RFC8216, August 2017, 496 . 498 Appendix A. Acknowledgements 500 Mark Nottingham, Patrick McManus, Julian Reschke, Remy Lebeau, Rodger 501 Combs, Thorsten Lohmar, Martin Thompson, Adrien de Croy, K. Morgan, 502 Roy T. Fielding, Jeremy Poulter. 504 Authors' Addresses 506 Craig Pratt 507 Portland, OR 97229 508 US 510 Email: pratt@acm.org 512 Darshak Thakore 513 CableLabs 514 858 Coal Creek Circle 515 Louisville, CO 80027 516 US 518 Email: d.thakore@cablelabs.com 520 Barbara Stark 521 AT&T 522 Atlanta, GA 523 US 525 Email: barbara.stark@att.com