idnits 2.17.1 draft-ietf-httpbis-rand-access-live-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (March 20, 2018) is 2229 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Experimental ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 7230 (Obsoleted by RFC 9110, RFC 9112) ** Obsolete normative reference: RFC 7233 (Obsoleted by RFC 9110) ** Obsolete normative reference: RFC 7234 (Obsoleted by RFC 9111) Summary: 3 errors (**), 0 flaws (~~), 2 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 HTTP Working Group C. Pratt 3 Internet-Draft 4 Intended status: Experimental D. Thakore 5 Expires: September 21, 2018 CableLabs 6 B. Stark 7 AT&T 8 March 20, 2018 10 HTTP Random Access and Live Content 11 draft-ietf-httpbis-rand-access-live-03 13 Abstract 15 To accommodate byte range requests for content that has data appended 16 over time, this document defines semantics that allow a HTTP client 17 and server to perform byte-range GET and HEAD requests that start at 18 an arbitrary byte offset within the representation and ends at an 19 indeterminate offset. 21 Editorial Note (To be removed by RFC Editor before publication) 23 Discussion of this draft takes place on the HTTPBIS working group 24 mailing list (ietf-http-wg@w3.org), which is archived at 25 . 27 Working Group information can be found at ; 28 source code and issues list for this draft can be found at 29 . 31 Status of This Memo 33 This Internet-Draft is submitted in full conformance with the 34 provisions of BCP 78 and BCP 79. 36 Internet-Drafts are working documents of the Internet Engineering 37 Task Force (IETF). Note that other groups may also distribute 38 working documents as Internet-Drafts. The list of current Internet- 39 Drafts is at https://datatracker.ietf.org/drafts/current/. 41 Internet-Drafts are draft documents valid for a maximum of six months 42 and may be updated, replaced, or obsoleted by other documents at any 43 time. It is inappropriate to use Internet-Drafts as reference 44 material or to cite them other than as "work in progress." 46 This Internet-Draft will expire on September 21, 2018. 48 Copyright Notice 50 Copyright (c) 2018 IETF Trust and the persons identified as the 51 document authors. All rights reserved. 53 This document is subject to BCP 78 and the IETF Trust's Legal 54 Provisions Relating to IETF Documents 55 (https://trustee.ietf.org/license-info) in effect on the date of 56 publication of this document. Please review these documents 57 carefully, as they describe your rights and restrictions with respect 58 to this document. Code Components extracted from this document must 59 include Simplified BSD License text as described in Section 4.e of 60 the Trust Legal Provisions and are provided without warranty as 61 described in the Simplified BSD License. 63 Table of Contents 65 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 66 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 67 1.2. Notational Conventions . . . . . . . . . . . . . . . . . 3 68 2. Performing Range requests on Random-Access Aggregating 69 ("live") Content . . . . . . . . . . . . . . . . . . . . . . 3 70 2.1. Establishing the Randomly Accessible Byte Range . . . . . 4 71 2.2. Byte-Range Requests Beyond the Randomly Accessible Byte 72 Range . . . . . . . . . . . . . . . . . . . . . . . . . . 5 73 3. Other Applications of Random-Access Aggregating Content . . . 7 74 3.1. Requests Starting at the Aggregation ("Live") Point . . . 7 75 3.2. Shift Buffer Representations . . . . . . . . . . . . . . 8 76 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9 77 5. Security Considerations . . . . . . . . . . . . . . . . . . . 9 78 6. References . . . . . . . . . . . . . . . . . . . . . . . . . 10 79 6.1. Normative References . . . . . . . . . . . . . . . . . . 10 80 6.2. Informative References . . . . . . . . . . . . . . . . . 10 81 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 11 82 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11 84 1. Introduction 86 Some Hypertext Transfer Protocol (HTTP) clients use byte-range 87 requests (Range requests using the "bytes" Range Unit) to transfer 88 select portions of large representations ([RFC7233]). And in some 89 cases large representations require content to be continuously or 90 periodically appended - such as representations consisting of live 91 audio or video sources, blockchain databases, and log files. Clients 92 cannot access the appended/live content using a Range request with 93 the bytes range unit using the currently defined byte-range semantics 94 without accepting performance or behavior sacrifices which are not 95 acceptable for many applications. 97 For instance, HTTP clients have the ability to access appended 98 content on an indeterminate-length resource by transferring the 99 entire representation from the beginning and continuing to read the 100 appended content as it's made available. Obviously, this is highly 101 inefficient for cases where the representation is large and only the 102 most recently appended content is needed by the client. 104 Alternatively, clients can also access appended content by sending 105 periodic open-ended bytes Range requests using the last-known end 106 byte position as the range start. Performing low-frequency periodic 107 bytes Range requests in this fashion (polling) introduces latency 108 since the client will necessarily be somewhat behind the aggregated 109 content - mimicking the behavior (and latency) of segmented content 110 representations such as "HTTP Live Streaming" (HLS, [RFC8216]) or 111 "Dynamic Adaptive Streaming over HTTP" (MPEG-DASH, [DASH]). And 112 while performing these Range requests at higher frequency can reduce 113 this latency, it also incurs more processing overhead and HTTP 114 exchanges as many of the requests will return no content - since 115 content is usually aggregated in groups of bytes (e.g. a video frame, 116 audio sample, block, or log entry). 118 This document describes a usage model for range requests which 119 enables efficient retrieval of representations that are appended to 120 over time by using large values and associated semantics for 121 communicating range end positions. This model allows representations 122 to be progressively delivered by servers as new content is added. It 123 also ensures compatibility with servers and intermediaries that don't 124 support this technique. 126 1.1. Requirements Language 128 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 129 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 130 document are to be interpreted as described in RFC 2119 [RFC2119]. 132 1.2. Notational Conventions 134 This document cites productions in Augmented Backus-Naur Form (ABNF) 135 productions from [RFC7233], using the notation defined in [RFC5234]. 137 2. Performing Range requests on Random-Access Aggregating ("live") 138 Content 140 This document recommends a two-step process for accessing resources 141 that have indeterminate length representations. 143 Two steps are necessary because of limitations with the Range request 144 header fields and the Content-Range response header fields. A server 145 cannot know from a range request that a client wishes to receive a 146 response that does not have a definite end. More critically, the 147 header fields do not allow the server to signal that a resource has 148 indeterminate length without also providing a fixed portion of the 149 resource. 151 A client first learns that the resource has a representation of 152 indeterminate length by requesting a range of the resource. The 153 server responds with the range that is available, but indicates that 154 the length of the representation is unknown using the existing 155 Content-Range syntax. See Section 2.1 for details and examples. 157 Once the client knows the resource has indeterminate length, it can 158 request a range with a very large end position from the resource. 159 The client chooses an explicit end value larger than can be 160 transferred in the foreseeable term. A server which supports range 161 requests of indeterminate length signals its understanding of the 162 client's indeterminate range request by indicating that the range it 163 is providing has a range end that exactly matches the client's 164 requested range end rather than a range that is bounded by what is 165 currently available. See Section 2.2 for details. 167 2.1. Establishing the Randomly Accessible Byte Range 169 Establishing if a representation is continuously aggregating ("live") 170 and determining the randomly-accessible byte range can both be 171 determined using the existing definition for an open-ended byte-range 172 request. Specifically, Section 2.1 of [RFC7233] defines a byte-range 173 request of the form: 175 byte-range-spec = first-byte-pos "-" [ last-byte-pos ] 177 which allows a client to send a HEAD request with a first-byte-pos 178 and leave last-byte-pos absent. A server that receives a satisfiable 179 byte-range request (with first-byte-pos smaller than the current 180 representation length) may respond with a 206 status code (Partial 181 Content) with a Content-Range header field indicating the currently 182 satisfiable byte range. For example: 184 HEAD /resource HTTP/1.1 185 Host: example.com 186 Range: bytes=0- 188 returns a response of the form: 190 HTTP/1.1 206 Partial Content 191 Content-Range: bytes 0-1234567/* 193 from the server indicating that (1) the complete representation 194 length is unknown (via the "*" in place of the complete-length field) 195 and (2) that only bytes 0-1234567 were accessable at the time the 196 request was processed by the server. The client can infer from this 197 response that bytes 0-1234567 of the representation can be requested 198 and returned in a timely fashion (the bytes are immediately 199 available). 201 2.2. Byte-Range Requests Beyond the Randomly Accessible Byte Range 203 Once a client has determined that a representation has an 204 indeterminate length and established the byte range that can be 205 accessed, it may want to perform a request with a start position 206 within the randomly-accessible content range and an end position at 207 an indefinite "live" point - a point where the byte-range GET request 208 is fulfilled on-demand as the content is aggregated. 210 For example, for a large video asset, a client may wish to start a 211 content transfer from the video "key" frame immediately before the 212 point of aggregation and continue the content transfer indefinitely 213 as content is aggregated - in order to support low-latency startup of 214 a live video stream. 216 Unlike a byte-range Range request, a byte-range Content-Range 217 response header field cannot be "open ended", per Section 4.2 of 218 [RFC7233]: 220 byte-content-range = bytes-unit SP 221 ( byte-range-resp / unsatisfied-range ) 223 byte-range-resp = byte-range "/" ( complete-length / "*" ) 224 byte-range = first-byte-pos "-" last-byte-pos 225 unsatisfied-range = "*/" complete-length 227 complete-length = 1*DIGIT 229 Specifically, last-byte-pos is required in byte-range. So in order 230 to preserve interoperability with existing HTTP clients, servers, 231 proxies, and caches, this document proposes a mechanism for a client 232 to indicate support for handling an indeterminate-length byte-range 233 response, and a mechanism for a server to indicate if/when it's 234 providing a indeterminate-length response. 236 A client can indicate support for handling indeterminate-length byte- 237 range responses by providing a Very Large Value for the last-byte-pos 238 in the byte-range request. For example, a client can perform a byte- 239 range GET request of the form: 241 GET /resource HTTP/1.1 242 Host: example.com 243 Range: bytes=1230000-999999999999 245 where the last-byte-pos in the Request is much larger than the last- 246 byte-pos returned in response to an open-ended byte-range HEAD 247 request, as described above. 249 In response, a server may indicate that it is supplying a 250 continuously aggregating ("live") response by supplying the client 251 request's last-byte-pos in the Content-Range response header field. 253 For example: 255 GET /resource HTTP/1.1 256 Host: example.com 257 Range: bytes=1230000-999999999999 259 returns 261 HTTP/1.1 206 Partial Content 262 Content-Range: bytes 1230000-999999999999/* 264 from the server to indicate that the response will start at byte 265 1230000 and continues indefinitely to include all aggregated content, 266 as it becomes available. 268 A server that doesn't support or supply a continuously aggregating 269 ("live") response will supply the currently satisfiable byte range, 270 as it would with an open-ended byte request. 272 For example: 274 GET /resource HTTP/1.1 275 Host: example.com 276 Range: bytes=1230000-999999999999 278 will return 280 HTTP/1.1 206 Partial Content 281 Content-Range: bytes 1230000-1234567/* 283 from the server to indicate that the response will start at byte 284 1230000 and end at byte 1234567 and will not include any aggregated 285 content. This is the response expected from a typical HTTP server - 286 one that doesn't support byte-range requests on aggregating content. 288 A client that doesn't receive a response indicating it is 289 continuously aggregating must use other means to access aggregated 290 content (e.g. periodic byte-range polling). 292 A server that does return a continuously aggregating ("live") 293 response should return data using chunked transfer coding and not 294 provide a Content-Length header field. A 0-length chunk indicates 295 the end of the transfer, per Section 4.1 of [RFC7230]. 297 3. Other Applications of Random-Access Aggregating Content 299 3.1. Requests Starting at the Aggregation ("Live") Point 301 A client that wishes to only receive newly-aggregated portions of a 302 resource (i.e., start at the "live" point), can use a HEAD request to 303 learn what range the server has currently available and initiate an 304 indeterminate-length transfer. For example: 306 HEAD /resource HTTP/1.1 307 Host: example.com 308 Range: bytes=0- 310 With the Content-Range response header field indicating the range (or 311 ranges) available. For example: 313 206 Partial Content 314 Content-Range: bytes 0-1234567/* 316 The client can then issue a request for a range starting at the end 317 value (using a very large value for the end of a range) and receive 318 only new content. 320 GET /resource HTTP/1.1 321 Host: example.com 322 Range: bytes=1234567-999999999999 324 with a server returning a Content-Range response indicating that an 325 indeterminate-length response body will be provided 327 206 Partial Content 328 Content-Range: bytes 1234567-999999999999/* 330 3.2. Shift Buffer Representations 332 Some representations lend themselves to front-end content removal in 333 addition to aggregation. While still supporting random access, 334 representations of this type have a portion at the beginning (the "0" 335 end) of the randomly-accessible region that become inaccessible over 336 time. Examples of this kind of representation would be an audio- 337 video time-shift buffer or a rolling log file. 339 For example a Range request containing: 341 HEAD /resource HTTP/1.1 342 Host: example.com 343 Range: bytes=0- 345 returns 347 206 Partial Content 348 Content-Range: bytes 1000000-1234567/* 350 indicating that the first 1000000 bytes were not accessible at the 351 time the HEAD request was processed. Subsequent HEAD requests could 352 return: 354 Content-Range: bytes 1000000-1234567/* 356 Content-Range: bytes 1010000-1244567/* 358 Content-Range: bytes 1020000-1254567/* 360 Note though that the difference between the first-byte-pos and last- 361 byte-pos need not be constant. 363 The client could then follow-up with a GET Range request containing 365 GET /resource HTTP/1.1 366 Host: example.com 367 Range: bytes=1020000-999999999999 369 with the server returning 371 206 Partial Content 372 Content-Range: bytes 1020000-999999999999/* 374 with the response body returning bytes 1020000-1254567 immediately 375 and aggregated ("live") data being returned as the content is 376 aggregated. 378 A server that doesn't support or supply a continuously aggregating 379 ("live") response will supply the currently satisfiable byte range, 380 as it would with an open-ended byte request. 382 For example: 384 GET /resource HTTP/1.1 385 Host: example.com 386 Range: bytes=0-999999999999 388 will return 390 HTTP/1.1 206 Partial Content 391 Content-Range: bytes 1020000-1254567/* 393 from the server to indicate that the response will start at byte 394 1020000, end at byte 1254567, and will not include any aggregated 395 content. This is the response expected from a typical HTTP server - 396 one that doesn't support byte-range requests on aggregating content. 398 Note that responses to GET requests against shift-buffer 399 representations using Range can be cached by intermediaries, since 400 the Content-Range response header indicates which portion of the 401 representation is being returned in the response body. However GET 402 requests without a Range header cannot be cached since the first byte 403 of the response body can vary from request to request. To ensure 404 Range-less GET requests against shift-buffer representations are not 405 cached, servers hosting a shift-buffer representation should either 406 not return a 200-level response (e.g. sending a 300-level redirect 407 response with a URI that represents the current start of the shift- 408 buffer) or indicate the response is non-cacheable. See HTTP Caching 409 ([RFC7234]) for details on HTTP cache control. 411 4. IANA Considerations 413 This document has no actions for IANA. 415 5. Security Considerations 417 One potential issue with this recommendation is related to the use of 418 very-large last-byte-pos values. Some client and server 419 implementations may not be prepared to deal with byte position values 420 of 2^^63 and beyond. So in applications where there's no expectation 421 that the representation will ever exceed 2^^63, a value smaller than 422 this value should be used as the Very Large last-byte-pos in a byte- 423 seek request or content-range response. Also, some implementations 424 (e.g. JavaScript-based clients and servers) are not able to 425 represent all values beyond 2^^53. So similarly, if there's no 426 expectation that a representation will ever exceed 2^^53 bytes, 427 values smaller than this limit should be used for the last-byte-pos 428 in byte-range requests. 430 6. References 432 6.1. Normative References 434 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 435 Requirement Levels", BCP 14, RFC 2119, 436 DOI 10.17487/RFC2119, March 1997, 437 . 439 [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer 440 Protocol (HTTP/1.1): Message Syntax and Routing", 441 RFC 7230, DOI 10.17487/RFC7230, June 2014, 442 . 444 [RFC7233] Fielding, R., Ed., Lafon, Y., Ed., and J. Reschke, Ed., 445 "Hypertext Transfer Protocol (HTTP/1.1): Range Requests", 446 RFC 7233, DOI 10.17487/RFC7233, June 2014, 447 . 449 [RFC7234] Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke, 450 Ed., "Hypertext Transfer Protocol (HTTP/1.1): Caching", 451 RFC 7234, DOI 10.17487/RFC7234, June 2014, 452 . 454 6.2. Informative References 456 [DASH] ISO, "Information technology -- Dynamic adaptive streaming 457 over HTTP (DASH) -- Part 1: Media presentation description 458 and segment formats", ISO/IEC 23009-1:2014, May 2014, 459 . 462 [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax 463 Specifications: ABNF", STD 68, RFC 5234, 464 DOI 10.17487/RFC5234, January 2008, 465 . 467 [RFC8216] Pantos, R., Ed. and W. May, "HTTP Live Streaming", 468 RFC 8216, DOI 10.17487/RFC8216, August 2017, 469 . 471 Acknowledgements 473 Mark Nottingham, Patrick McManus, Julian Reschke, Remy Lebeau, Rodger 474 Combs, Thorsten Lohmar, Martin Thompson, Adrien de Croy, K. Morgan, 475 Roy T. Fielding, Jeremy Poulter. 477 Authors' Addresses 479 Craig Pratt 480 Portland, OR 97229 481 US 483 Email: pratt@acm.org 485 Darshak Thakore 486 CableLabs 487 858 Coal Creek Circle 488 Louisville, CO 80027 489 US 491 Email: d.thakore@cablelabs.com 493 Barbara Stark 494 AT&T 495 Atlanta, GA 496 US 498 Email: barbara.stark@att.com