idnits 2.17.1 draft-tarreau-httpbis-network-friendly-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a Security Considerations section. ** There are 10 instances of too long lines in the document, the longest one being 35 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 29, 2012) is 4410 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'RFC2991' is defined on line 889, but no explicit reference was found in the text == Unused Reference: 'RFC4864' is defined on line 892, but no explicit reference was found in the text == Unused Reference: 'RFC6296' is defined on line 896, but no explicit reference was found in the text == Unused Reference: 'RFC6438' is defined on line 899, but no explicit reference was found in the text -- Obsolete informational reference (is this intentional?): RFC 2629 (Obsoleted by RFC 7749) Summary: 2 errors (**), 0 flaws (~~), 6 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group W. Tarreau 3 Internet-Draft Exceliance 4 Expires: September 30, 2012 A. Jeffries 5 Treehouse Networks Ltd. 6 A. de Croy 7 Qbik New Zealand Ltd. 8 P-H. Kamp 9 Varnish Cache Project 10 March 29, 2012 12 Proposal for a Network-Friendly HTTP Upgrade 13 draft-tarreau-httpbis-network-friendly-00 15 Abstract 17 This document proposes an upgrade to HTTP messaging which aims at 18 being faster, more robust and more friendly to mobile networks than 19 the current version, while retaining the same semantics and offering 20 a high enough compatibility level to make it possible to implement 21 highly efficient gateways between existing implementations and this 22 presently described version, thus offering a smooth upgrade path for 23 legacy applications. 25 Status of this Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at http://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on September 30, 2012. 42 Copyright Notice 44 Copyright (c) 2012 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (http://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 60 1.1. Background . . . . . . . . . . . . . . . . . . . . . . . . 3 61 1.2. Improvements . . . . . . . . . . . . . . . . . . . . . . . 4 62 2. Principles of operation . . . . . . . . . . . . . . . . . . . 4 63 2.1. Frame encoding . . . . . . . . . . . . . . . . . . . . . . 5 64 2.1.1. Request Frame (frame type = 2) . . . . . . . . . . . . 7 65 2.1.2. Status Frame (frame type = 3) . . . . . . . . . . . . 8 66 2.1.3. Entity Frame (frame types = 4..7) . . . . . . . . . . 9 67 2.1.4. Abort Frame (frame type = 9) . . . . . . . . . . . . . 12 68 2.1.5. Header fields encoding . . . . . . . . . . . . . . . . 12 69 2.2. Grouping headers . . . . . . . . . . . . . . . . . . . . . 13 70 2.3. Sending Requests . . . . . . . . . . . . . . . . . . . . . 16 71 3. Connection Setup . . . . . . . . . . . . . . . . . . . . . . . 16 72 4. Improving the handshake to save bandwidth . . . . . . . . . . 17 73 5. Improving the handshake to save time . . . . . . . . . . . . . 18 74 6. Directions for future work (TBD) . . . . . . . . . . . . . . . 19 75 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 76 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 20 77 9. Change log [RFC Editor: Please remove] . . . . . . . . . . . . 20 78 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 20 79 10.1. Normative References . . . . . . . . . . . . . . . . . . . 20 80 10.2. Informative References . . . . . . . . . . . . . . . . . . 20 81 Appendix A. Analysis of header field occurrences . . . . . . . . 21 82 Appendix B. Analysis of header field length . . . . . . . . . . . 22 83 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 22 85 1. Introduction 87 HTTP/1.1 relies on a base designed 15 years ago for use in a context 88 which has significantly evolved over the years. Applications have 89 become mostly stateful with sessions spanning over multiple 90 connections. Network intermediaries have been installed everywhere 91 between clients and servers for various purposes ranging from caching 92 and filtering to load-balancing and off-loading. Enterprise networks 93 rely on HTTP for almost all inter-server communications. Mobile 94 networks are becoming prevalent in HTTP traffic, and at the same time 95 they suffer from important constraints imposed by the medium, such as 96 a higher latency and a higher loss rate than wired networks. HTTP 97 itself is a very verbose protocol which magnifies issues specific to 98 these environments. Web usage has changed, with social networks 99 connecting millions of people and resulting in some sites having to 100 deal with hundreds of thousands of concurrent connections, and front 101 end components having to forward incoming requests to the proper 102 server as quickly as possible. 104 Economics have changed too, making it attractive for some groups to 105 attack business-critical sites. DDoS authors rely on the ratio 106 between the cost of processing traffic for their victim versus the 107 cost of building the attack. HTTP has inherited 15 years of 108 improvements and total backwards compatibility with the original 109 design, making it hard to parse and process, with a number of 110 ambiguous situations left to the implementation's choice. Current 111 model's corner cases with its moderately high parsing cost 112 contributes to the success of these attacks by making it quite 113 expensive for server-side components to ignore undesired requests. 115 1.1. Background 117 Many internet users rely on asymmetric links to connect to the net 118 (POTS, ADSL, HSPA, ...). Downstream to upstream ratios of 4:1 are 119 quite common, sometimes reaching high figures like 20:1 or even more 120 in ADSL2+ or HSDPA. 122 HTTP relies on header-based messages in both directions, with bodies 123 more often in the response messages than in request messages, 124 resulting in the upstream traffic being mostly composed of headers. 125 Most header field names and values are repeated unchanged over 126 multiple requests or responses from the same sender. 128 For historical reasons, request headers are much larger than response 129 headers. The User-Agent and Referer header fields usually take a 130 significant size, and cookies can be so large that some sites prefer 131 to register a separate domain for statics to save the browser from 132 sending them when fetching static objects. 134 The one-request-at-a-time model is not suited at all to high BDP 135 links such as the ones used in mobile environments. The only way to 136 fill at least one direction of the link bandwidth on high latency 137 links such as HSDPA is to fetch many objects in parallel. Pipelining 138 enables this but is not supported by all servers, so user agents are 139 often configured to use a large number of concurrent connections 140 instead in order to parallelize objects retrieval, wasting bandwidth 141 with payload-less TCP packets, wasting server resources, and taking 142 more time to converge to the optimal CWND. 144 Many sites involve a large number of small objects to compose a page, 145 typically smaller than 2 kB ([WebMetrics]), which make it hard to 146 fill the downstream link before filling a smaller upstream even when 147 pipelining is used. 149 Still, the shortcomings above are probably transient. With HSPA+ 150 reaching 168 Mbps downstream and 20 Mbps upstream in 3GPP-Rel10 151 ([4gamericas]), and with Google's advice of running TCP stacks with 152 INITCWD=10, it seems reasonable to expect that request header size on 153 the wire will not remain the limiting factor forever, which implies 154 that reducing the number of round trips and header processing costs 155 will become more important than optimizing the network usage 156 reduction alone. 158 1.2. Improvements 160 This proposal focuses on four improvements over HTTP/1.1 : 161 - Binary encoding of headers fields : header field names are 162 encoded and their sizes advertised to speed up lookup 163 - Grouping of common headers fields : a section defines all header 164 fields common to several subsequent messages, avoiding repetition 165 - Request and response multiplexing : requests and responses may 166 be delivered in parallel and out of order 167 - Layering model : more friendly to intermediaries, saves header 168 field lookups and memory copies 170 Backwards compatibility is an absolute requirement so that gateways 171 can be built to present HTTP/1.1 servers to the world with the new 172 protocol version. This should become even more obvious at mobile 173 operators where it is likely that gateways will present the whole 174 HTTP/1.1 internet to mobile users in HTTP/2.0. Therefore, semantics 175 must not be affected. 177 2. Principles of operation 179 This draft proposes a mechanism to exchange messages in parallel over 180 an established bidirectional connection with support for out of order 181 processing and delivery. 183 In order for messages to flow in both directions out of order, some 184 delimiters are needed. Thus, the protocol is a stream of frames 185 which can be of the following types : 186 - Transport Frame : this frame is only allowed once in each 187 direction and advertises a set of header fields that the sender 188 knows are invariant for this connection and that must be 189 considered present for all messages passing over that connection 190 - Common Frame : this frame may appear as often as needed and 191 advertises sets of header fields that the sender thins will be 192 common to several upcoming messages and are worth advertising only 193 once 194 - Message Frame : this frame holds a request or response message 195 with message-specific header fields but without any message body 196 - Entity frame : this frame carries all or part of a message body 197 - Control frame : various control frames such as Ping/Pong/Pause/ 198 Abort/Close are planned but not described here yet (TBD) 200 Frames which are part of the same message will generally include the 201 reference to the request which initiated the frame, which simply 202 corresponds to the request arrival order over the connection. This 203 is particularly important since responses may appear in any order. 205 If we note 'T', 'C', 'Mx' and 'Ex' the Transport Frame, Common Frame, 206 Message Frame number 'x' and Entity Frame number 'x', the stream 207 between a user agent (UA) and an origin server (O) could be 208 represented like this : 210 requests > E4 M4 C M3 M2 M1 C T 211 UA ======================================= O 212 T C M1 E1 M3 C M2 M4 E2 E3 E4 E3 E3 < responses 214 In the diagram above, the client has sent 4 requests and the server 215 has responded to all of them in a slightly different order and with 216 some payload interleaved. In general, over a connection, there will 217 be in each direction zero or one Transport Frame, zero or a few 218 Common Frames, one or more Message Frames, and zero or more Entity 219 Frames. 221 2.1. Frame encoding 223 NOTE: the proposed encoding is a work in progress and subject to 224 change 226 Frames use reasonably low overhead. Some frames will need to 227 indicate a request number, while others won't. All frames start with 228 a frame type octet indicating the frame type and the HTTP version. 230 Frame types between 0 and 31 are standard frames and have their own 231 format. Frames types 32 to 63 are extension frames which all follow 232 the same unambiguous format. Such frames are not described here and 233 are left for future work or may even be dropped if considered 234 unneeded. 236 In order to associate frames to a given request, response frames and 237 Entity frames will include a 16-bit request number. The request 238 number correspond to the arrival order of the request over the 239 connection and automatically wraps past 2^16, meaning that no more 240 than 65536 outstanding requests are supported over a single 241 connection. In practice this should be more than enough considering 242 that : 243 1. current HTTP implementations only support one outstanding 244 request; 245 2. TCP congestion and losses affect all requests at the same 246 time, so it is unlikely that browsers will push more than a few 247 hundreds requests in parallel. 249 The two higher bits of the frame type octet indicate the HTTP 250 version, and the lower 6 bits indicate the frame type : 252 0 1 2 3 4 5 6 7 253 +---+------------+ 254 | V | frame-type | 255 +---+------------+ 257 V stands for the HTTP version. Possible values for these 2 bits are: 258 - 00: HTTP/1.0 259 - 01: HTTP/1.1 260 - 10: HTTP/2.0 261 - 11: other version 262 The frame type is defined below : 264 frame = frame-type frame-body 265 = %x00 tra-frame ; Transport Frame 266 / %x01 com-frame ; Common Frame 267 / %x02 req-frame ; Request Frame 268 / %x03 sts-frame ; Status Frame 269 / %x04 sef-frame ; Small Entity Frame 270 / %x05 mef-frame ; Medium Entity Frame 271 / %x06 lef-frame ; Large Entity Frame 272 / %x07 hef-frame ; Huge Entity Frame 273 / %x08 trl-frame ; Trailers Frame 274 / %x09 abt-frame ; Abort Frame 275 / %x0A-1F ; reserved frame (control etc...) 276 / %x20-3F ext-frame ; extension frame 278 tra-frame = header-list ; Transport Frame 279 com-frame = header-list ; Common Frame 280 trl-frame = header-list ; Trailers Frame 281 ext-frame = frame-len opaque ; extension frame 282 frame-len = 4*OCTETS ; 32-bit frame length encoding 284 2.1.1. Request Frame (frame type = 2) 286 The Request Frame is a Message Frame composed of a bit indicating if 287 an Entity Frame is expected for this request, a method, a URI and an 288 optional header list. 290 0 1 2 3 4 5 6 7 291 +-+-+---+-------+ 292 |E|M|0 0| METH | 293 +-+-+---+-------+ 294 | optional-meth | 295 | (0-16) | 296 +---------------+ 297 | length-prefix | 298 | (1-2) | 299 +---------------+ 300 | URI (1-32767) | 301 +---------------+ 302 | header-list | 303 | (variable) | 304 +---------------+ 306 - E : Entity is present. One or more Entity Frames are expected 307 if this bit is 1, while 0 indicates no entity is attached to this 308 request. 309 - M : 311 0: METH contains the method length minus 1, between 1 and 16 312 bytes, and the method follows in the optional-meth field 313 1: METH contains a method number among the following values and 314 no optional-meth field is provided : 315 0: OPTIONS 316 1: GET 317 2: HEAD 318 3: POST 319 4: PUT 320 5: DELETE 321 6: TRACE 322 7: CONNECT 323 other: TBD 324 - optional-meth: this is the method written in plain text then 325 M=0. 326 - length-prefix: this is the number of octets representing the 327 request URI encoded as a 15-bit quantity between 0 and 32767 on 328 either 1 or 2 octets, using the variable length encoding described 329 in the header field encoding section. 330 - URI: this is the request URI, it is of exactly length-prefix 331 octets 332 - header-list: this is the encoded list of headers specific to 333 this request, see below. 335 In many cases, this frame alone will be enough to send a complete 336 request, which will then be as small as just a frame-type octet 337 followed by 1 byte for the method, one byte for the URI length, the 338 URI itself and the null byte to end the header list. This sums up to 339 the URI length plus 4 bytes. 341 2.1.2. Status Frame (frame type = 3) 343 The Status Frame is composed of a bit indicating if an Entity Frame 344 is expected for this response, a bit indicating if this response is a 345 final response or an interim response, a status and a request number. 347 0 1 348 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 349 +-+-+-------+-------------------+ 350 |E|F|0 0 0 0| Status (10) | 351 +-+-+-------+-------------------+ 352 | R (16) | 353 +-------------------------------+ 354 | header-list (variable) | 355 +-------------------------------+ 357 - E : Entity present. One or more Entity Frames are expected if 358 this bit is 1. 0 indicates no entity is attached to this response. 359 - F : Final response. All responses except those with status 1xx 360 are final and have this bit set. Responses 1xx are not final and 361 have this bit cleared. 362 - Status : This is the HTTP status encoded over 10 bits. 363 - R : this is the associated request number encoded on 16 bits. 364 - header-list: this is the encoded list of header fields specific 365 to this response, see below 367 2.1.3. Entity Frame (frame types = 4..7) 369 The Entity Frame is composed only of payload which in principle is 370 very comparable chunked encoding. The payload length is encoded on a 371 variable size so for this we have 4 types of Entity Frames which are 372 totally similar except for the data length encoding : 373 - Small frames : length is encoded on 6 bits (64 bytes max). 374 These frames are useful for uploading small contents such as 375 credentials, as well as to send an empty final frame. 376 - Medium frames : the length is encoded on 22 bits (4 MB max). 377 These will probably be the most common ones. 378 - Large frames : the length is encoded on 32 bits (4 GB max). 379 These ones might also be very common. 380 - Huge frames : the length is encoded on 64 bits (18 EB max). 381 These ones will probably only be used in CDN environments where 382 use of sendfile() is desirable for very large files, when 383 multiplexing is not involved. 385 An entity length contains a bit indicating if more Entity Frames are 386 expected, a bit indicating if a Trailers Frame is expected, a length, 387 a request number, and data. 389 2.1.3.1. Small Entity Frame (frame type = 4) 391 This is the smallest Entity Frame, which can be used to transfer 392 between 0 and 63 bytes of payload and can be as small as one single 393 byte (0). 395 0 1 2 3 4 5 6 7 396 +-+-+-----------+ 397 |E|T| Length (6)| 398 +-+-+-----------+ 399 | R | 400 | (16) | 401 +---------------+ 402 | DATA | 403 +---------------+ 405 - E : More Entity Frames present. One or more Entity Frames are 406 expected if this bit is 1, while 0 indicates this is the last 407 Entity Frame for this request number. 408 - T : 1 if a Trailers Frame is expected, otherwise zero. 409 - Length : this is the length of the entity data in octets, 410 encoded on 6 bits. 411 - R : this is the associated request number encoded on 16 bits. 412 - DATA (0..Length bytes) : entity payload. 414 2.1.3.2. Medium Entity Frame (frame type = 5) 416 This frame type combines the small length field with 16 more bits to 417 encode up to 22 bits of length. 419 0 1 2 3 4 5 6 7 420 +-+-+-----------+ 421 |E|T| Length ...| 422 +-+-+-----------+ 423 | ... Length | 424 | (22) | 425 +---------------+ 426 | R | 427 | (16) | 428 +---------------+ 429 | DATA | 430 +---------------+ 432 - E : More Entity Frames present. One or more Entity Frames are 433 expected if this bit is 1, while 0 indicates this is the last 434 Entity Frame for this request number. 435 - T : 1 if a Trailers Frame is expected, otherwise zero. 436 - Length : this is the length of the entity data in octets, 437 encoded on 22 bits, with the 6 higher offset bits in the first 438 octet. 439 - R : this is the associated request number encoded on 16 bits. 440 - DATA (0..Length bytes) : entity payload. 442 2.1.3.3. Large Entity Frame (frame type = 6) 444 This frame type only uses a 32-bit length field. 446 0 1 2 3 4 5 6 7 447 +-+-+-----------+ 448 |E|T| 000000 | 449 +-+-+-----------+ 450 | Length | 451 : (32) : 452 +---------------+ 453 | R | 454 | (16) | 455 +---------------+ 456 | DATA | 457 +---------------+ 459 - E : More Entity Frames present. One or more Entity Frames are 460 expected if this bit is 1, while 0 indicates this is the last 461 Entity Frame for this request number. 462 - T : 1 if a Trailers Frame is expected, otherwise zero. 463 - Length : this is the length of the entity data in octets, data 464 encoded on 32 bits. 465 - R : this is the associated request number encoded on 16 bits. 466 - DATA (0..Length bytes) : entity payload. 468 2.1.3.4. Huge Entity Frame (frame type = 7) 470 This is the largest Entity Frame, used to code up to 64-bit lengths. 472 0 1 2 3 4 5 6 7 473 +-+-+-----------+ 474 |E|T| 000000 | 475 +-+-+-----------+ 476 | Length | 477 : (64) : 478 +---------------+ 479 | R | 480 | (16) | 481 +---------------+ 482 | DATA | 483 +---------------+ 485 - E : More Entity Frames present. One or more Entity Frames are 486 expected if this bit is 1, while 0 indicates this is the last 487 Entity Frame for this request number. 488 - T : 1 if a Trailers Frame is expected, otherwise zero. 489 - Length : this is the length of the entity data in octets, 490 encoded on 64 bits. 491 - R : this is the associated request number encoded on 16 bits. 493 - DATA (0..Length bytes) : entity payload. 495 2.1.4. Abort Frame (frame type = 9) 497 The Abort Frame is composed of a status and a request number. It is 498 returned by a server if an error caused the request to be aborted in 499 the middle of a transfer. It may also be emitted by a client which 500 wishes to abort a transfer (either download or upload) without 501 breaking the connection. The receiver of such a frame must 502 immediately stop any communication with this request number and not 503 expect any further data for this request number in the same 504 direction. The connection is not affected and other requests 505 continue their normal work. 507 TBD: it seems to make sense to have an ACK frame (or maybe respond 508 with an ABRT frame) for this frame in case of a client abort so that 509 the client knows the server has really stopped sending anything for 510 this request. 512 0 1 513 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 514 +-----------+-------------------+ 515 |0 0 0 0 0 0| Status (10) | 516 +-----------+-------------------+ 517 | R (16) | 518 +-------------------------------+ 520 - Status : This is the HTTP status encoded over 10 bits in case of 521 a server-initiated abort. TBD: would this be useful to let a 522 client tell the server what it wants to abort ? Maybe 523 intermediaries could tell servers the client is gone. 524 - R : this is the associated request number encoded on 16 bits. 526 2.1.5. Header fields encoding 528 Header fields have two parts, one which is the field-name and one 529 which is the field-value. A header-list is defined as a sequence of 530 header fields terminated by and end-of-headers tag (%x00). 532 Based on the observations from Appendix A, the current proposal 533 suggests to encode header field names either as a registered well- 534 known field-name identifier, or as a 7-bit name length followed by 535 the header's name. This operation will permit to reduce up to 127 536 header names to one single byte each. For optimal efficiency, the 537 assignment of header names to entries has to be done based on wider 538 analysis. It is suggested that no more than half of the possible 539 entries are assigned, in order to leave room for newer headers, or 540 for dynamically assigned header fields. 542 In order to support larger field values, the field-value is encoded 543 as a variable sized length-prefix followed by a value. 545 header-list = *( header-field ) end-of-hdr 546 header-field = field-name field-value 547 field-name = common-hdr / rare-hdr / rsvd-hdr 548 field-value = length-prefix *( octet ) 549 rare-hdr = hdr-len token ; token is [hdr-len] octets 550 hdr-len = %x01-7F ; header names may be up to 127 bytes long 551 common-hdr = %x80-FE ; 127 possible header names 552 rsvd-hdr = %xFF ; for future extensions if needed. 553 end-of-hdr = %x00 ; this was the last header. 555 The length-prefix is used to efficiently encode a length which most 556 of the time is small but sometimes needs to be large. The principle 557 is that small lengths between 0 and 127 are encoded on a single 558 octet, and lengths between 128 and 32727 are encoded on two octets. 559 (TBD: decide if we should encode 128 to 32895 instead). This is 560 appropriate for field-values and for the request-URI : 562 0 1 2 3 4 5 6 7 563 +-+-------------+ 564 |0| LENGTH(7) | 565 +-+-------------+ 567 0 1 568 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 569 +-+-----------------------------+ 570 |1| LENGTH(15) | 571 +-+-----------------------------+ 573 2.2. Grouping headers 575 Observations from Appendix B suggest that it is worth grouping 576 headers for multiple consecutive messages over a single connection. 577 Some of these headers will be connection-specific and should be 578 common to all messages transported over the connection, while other 579 ones will be common to a group of messages. 581 This proposal thus introduces the notion of sections to communicate 582 header fields. These sections have a different lifetime. They are 583 only valid for a hop-by-hop connection, and have no end-to-end 584 meaning. The header fields will be split into three sections : 585 - Transport Header Fields 586 - Common Header Fields 587 - Message Header Fields 589 The Transport Header Fields section holds all headers fields that are 590 specific to the connection and invariant over all the connection. 591 These headers are transmitted in a Transport Frame only once at the 592 beginning of the connection and never after that. The recipient of 593 any message always considers the Transport Header Fields when parsing 594 a message coming over that connection. While a user agent may use 595 this section to present a number of invariant header fields such as 596 the User-Agent, Accept or Host, intermediaries which are able to 597 multiplex requests over a single connection will probably not use it 598 much, maybe only for rare constant header fields such as Via, or even 599 Host if the connection was opened for a specific Host field value. 600 It is important to note that since this header field section only 601 applies to a hop-by-hop connection, only context-specific header 602 fields will be there so all header fields present there must be 603 considered after those of all other sections in order to maintain 604 ordering (eg: chaining multiple Via fields). 606 The Common Header Fields section holds a number of headers which are 607 common for a number of subsequent requests, and may be updated at any 608 time. These header fields are transmitted in a Common Frame. All 609 headers fields contained in the Common Header Fields section are 610 implicitly present in any subsequent message until the next Common 611 Header Fields section is encountered, which voids and replaces any 612 previous Common Header Fields section. Header fields eligible to 613 this section are all those which are expected to appear multiple 614 times over a connection, without necessarily being invariant. A user 615 agent will likely use this section to send Cookie, a Referer or even 616 Authentication credentials. A multiplexing intermediary may use this 617 section when forwarding multiple requests at once from the same user 618 agent, or to store almost invariant headers fields such as Host. All 619 header fields present in this section must be considered after the 620 Message Header Fields section and before the Transport Header Fields 621 section. 623 The Message Header Fields section represents all header fields that 624 are attached to a given message (request, response, trailers...). 625 The recipient of a message will reconstruct the original message 626 headers by concatenating the Message Header Fields section, the last 627 Common Header Fields section and the Transport Header Fields section. 628 Respecting this order is important so that some hop-by-hop header 629 fields are correctly appended last (for instance, Via or X-Forwarded- 630 For). 632 All these sections are proper to a connection only. Each hop is free 633 to rearrange them as it likes for the other side connection if it 634 estimates it is appropriate, provided that the resulting set of 635 header fields remains the same once reassembled. 637 Doing this is not only interesting for the sender which saves 638 upstream bandwidth, but also for the recipient which has to process 639 much less header fields for each message. If an intermediary has to 640 rewrite, insert or delete a header field which is in either the 641 Transport or Common section, it only does so once, and not for every 642 request or response. Common rewriting practices include rewriting 643 the Host header field in requests and removing the Server header 644 field from responses. Another example of CPU savings if gained by 645 not having to perform more layer7 inspection than necessary. For 646 instance, a front load balancer which selects the target server based 647 on the Host header field alone might simply splice the client and the 648 server connection together when it receives a Host header field from 649 the client in the Transport Frame. 651 Example of request path with a client, a load balancer and two 652 servers. All connections are fresh new, both from the client to the 653 LB, and the LB to the servers. Hence, all request numbers start at 0 654 on each connection. TF, CF and RF designate the Transport Frame, the 655 Common Frame and the Request Message Frame respectively. 657 +-----+ 658 | C | 659 +-----+ 660 /_____/ 661 | TF: Host="foo.example.com", UA="foo browser" 662 | CF: Cookie="user=123" 663 | RF: R=0, METH=GET, URI="/" 664 | RF: R=1, METH=GET, URI="/css/style.css" 665 | RF: R=2, METH=GET, URI="/js/menu.js" 666 +------+------+ 667 | LB | 668 +-+---------+-+ 669 | | 670 +------------+ +-----------+ 671 | TF: Host="foo.example.com" | TF: Host="foo.example.com", Via="LB" 672 | Via="LB" | RF: R=0, METH=GET, URI="/css/style.css", 673 | RF: R=0, METH=GET, URI="/", | Cookie="123", UA 674 | Cookie="123", UA | RF: R=1, METH=GET, URI="/js/menu.js", 675 | | Cookie="123", UA 676 +---+---+ +---+---+ 677 |dynamic| |static | 678 +-------+ +-------+ 680 Here, the LB maps the request numbers between the connections : 681 - C: Req #0 <=> dynamic: req #0 682 - C: Req #1 <=> static: req #0 683 - C: Req #2 <=> static: req #1 685 2.3. Sending Requests 687 A client wishing to send requests does not need to verify that the 688 recipient accepts enough requests. It simply writes a new request 689 message to the stream, which implicitly gets a new request number. 690 If the recipient is not reading, the request will just wait somewhere 691 along the path as it does with usual HTTP pipelining. 693 If a client wishes to send a request with a body, it must not send 694 multiple interleaved bodies from different requests unless it has 695 verified that the recipient is willing to process them. Otherwise, 696 it would be possible to enter a deadlock with interleaved partial 697 bodies sent to a server which supports only one outstanding request 698 at a time. The proper way to proceed is to send the first request 699 without prior check, but if other request bodies have to be 700 interleaved before the first request is complete, then the client 701 must first make use of the Expect: 100-Continue header field and wait 702 for the server to send the non-final 100 response corresponding to 703 the same request, thus proving it is able to read multiple requests 704 at once. In practice this is not an issue since clients sending 705 multiple POSTs at once are not common. 707 Note that this restriction does not apply to response bodies from the 708 servers, as the servers will always respond to requests that have 709 been received, so for each response, it is certain that there is a 710 client listening. 712 3. Connection Setup 714 The protocol is designed to operate over various stream-based 715 bidirectional connections, and to be upgradable from HTTP/1.1, 716 offering a smooth upgrade path to existing applications. 718 A client wishing to use this protocol to communicate with an origin 719 server for which the protocol support is unknown will send the first 720 request in HTTP/1.1 format, with an additional Upgrade: HTTP/2.0 721 header : 723 GET / HTTP/1.1 724 Host: www.example.com 725 Connection: Upgrade 726 Upgrade: HTTP/2.0 727 ... 729 If the server does not support the new protocol, it will simply 730 respond to the client using HTTP/1.1 : 732 HTTP/1.1 200 OK 733 Content-length: 243 734 Content-type: text/html 735 ... 737 However, if the server supports the new protocol, it will first emit 738 an interim response then will immediately respond with the final 739 response in HTTP/2.0, just as if it had received the first request in 740 HTTP/2.0 : 742 HTTP/1.1 101 Switching Protocols 743 Connection: Upgrade 744 Upgrade: HTTP/2.0 746 [ tra-frame ] [ com-frame ] [ sts-frame ] ... 748 4. Improving the handshake to save bandwidth 750 In order to save network exchanges, two new hop-by-hop header fields 751 could be registered : 752 - Http2-Th : list of the headers fields to keep in the Transport 753 Header Fields section after the Upgrade 754 - Http2-Ch : list of the headers fields to keep in the Common 755 Header Fields section after the Upgrade 756 This way, a client could make the server keep various information 757 such as the Host and User-Agent in the Transport Header Fields 758 section and the Referer as a Common Header Fields section for next 759 requests, so that only the request-uri has to be sent after the 760 upgrade : 762 GET / HTTP/1.1 763 Host: www.example.com 764 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.16) Gecko/20101210 SeaMonkey/2.0.11 765 Accept: text/css,*/*;q=0.1 766 Accept-Language: en-us,en;q=0.5 767 Accept-Encoding: gzip,deflate 768 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 769 Referer: http://www.ietf.org/meeting/83/index.html 770 Cookie: styleSheet=1 771 Connection: Upgrade, Http2-Th, Http2-Ch 772 Upgrade: HTTP/2.0 773 Http2-Th: Host, User-Agent, Accept, Accept-Language, Accept-Encoding, Accept-Charset 774 Http2-Ch: Referer, Cookie 775 ... 777 5. Improving the handshake to save time 779 Some minimum testing suggests that many consecutive requests will 780 only vary by the request-uri. This is the case for instance, for 781 requests for static objects fetched from a same host. In this 782 situation, the sender would like to benefit from HTTP pipelining/ 783 multiplexing without knowing whether the whole chain supports the 784 protocol upgrade. The solution consists in enumerating the expected 785 upcoming requests in a specific header field, that the recipient will 786 decide to consider as individual requests sharing the same Common 787 Header Fields section and Transport Headers Fields section. 789 These additional requests will take number 1 and onwards. The 790 recipient will just have to indicate in a header field of the 791 handshake response the highest number of the pending requests its is 792 willing to process. If the client does not receive this header field 793 in the response handshake, then it knows that the next hop to the 794 server does not support this optimization and it is free to send 795 these requests individually once the handshake completes. 797 For this we would register two more hop-by-hop headers fields, one 798 for the request and one for the response : 799 - Http2-Reqs : comma-delimited list of request-uri represented as 800 quoted-strings. 801 - Http2-Accepted-Reqs : integer number representing the number of 802 the last accepted request for which a response message will be 803 delivered 805 Example : 807 GET / HTTP/1.1 808 Host: www.example.com 809 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.16) Gecko/20101210 SeaMonkey/2.0.11 810 Accept: text/css,*/*;q=0.1 811 Accept-Language: en-us,en;q=0.5 812 Accept-Encoding: gzip,deflate 813 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 814 Referer: http://www.ietf.org/meeting/83/index.html 815 Cookie: styleSheet=1 816 Connection: Upgrade, Http2-Th, Http2-Ch, Http2-Reqs 817 Upgrade: HTTP/2.0 818 Http2-Th: Host, User-Agent, Accept, Accept-Language, Accept-Encoding, Accept-Charset 819 Http2-Ch: Referer, Cookie 820 Http2-Reqs: "/css/ietf.js", "/css/ietf.css", "/css/ietf4.css", "/css/ietf3.css" 821 ... 823 HTTP/1.1 101 Switching Protocols 824 Connection: Upgrade 825 Upgrade: HTTP/2.0 826 Http2-Accepted-Reqs: 4 828 [ tra-frame ] [ com-frame ] [ sts-frame ] ... 830 6. Directions for future work (TBD) 832 This draft in its state currently lacks a number of things : 833 - the frame encoding could be much better with some specific 834 fields always at the same position (for instance, the request 835 number). 836 - date formats have not been discussed but are expensive to parse 837 at the moment and cause issues with header folding due to the 838 comma. A binary encoding of a single scalar (eg: epoch in 839 milliseconds) would be much more suited. 840 - multiple header occurrences might be better handled by having a 841 repetition of the header value than by keeping the comma inside 842 the header field value. Several options will have to be explored. 843 - watch out other working groups (eg: hybi) to see how extensions 844 may be efficiently added at a low cost (eg: per-frame compression, 845 ...) 846 - determine if some sets of features are more suited to the 847 current most common usage (loading a web page in a graphical 848 browser) than to some other usages such as interactive use of XHR, 849 displaying widgets on a TV, forwarding a request between a load 850 balancer and an origin server, or making one's backups online ; 851 some of the SPDY experience will probably be useful here. 853 - identify what is needed to operate over datagram-based transport 854 protocols such as UDP and if it is worth having a single protocol 855 for all transports. 856 - protocol handshake if another port is to be used. 857 - use delta encoding for header updates ? Would this void the 858 need for Transport Header Fields ? 859 - replace "Host" with "Base" which would include a scheme ? 861 7. IANA Considerations 863 The Upgrade field header value "HTTP/2.0" might require a IANA 864 assignment. 866 8. Acknowledgements 868 This document was produced using the xml2rfc tool [RFC2629]. 870 9. Change log [RFC Editor: Please remove] 872 draft-tarreau-httpbis-network-friendly-00: original version, 873 2012-03-29. 875 10. References 877 10.1. Normative References 879 10.2. Informative References 881 [4gamericas] 882 "4G Mobile Broadband Evolution - 3GPP Release 10 and 883 Beyond", 2011, . 886 [RFC2629] Rose, M., "Writing I-Ds and RFCs using XML", RFC 2629, 887 June 1999. 889 [RFC2991] Thaler, D. and C. Hopps, "Multipath Issues in Unicast and 890 Multicast Next-Hop Selection", RFC 2991, November 2000. 892 [RFC4864] Van de Velde, G., Hain, T., Droms, R., Carpenter, B., and 893 E. Klein, "Local Network Protection for IPv6", RFC 4864, 894 May 2007. 896 [RFC6296] Wasserman, M. and F. Baker, "IPv6-to-IPv6 Network Prefix 897 Translation", RFC 6296, June 2011. 899 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 900 for Equal Cost Multipath Routing and Link Aggregation in 901 Tunnels", RFC 6438, November 2011. 903 [WebMetrics] 904 Ramachandran, S., "Let's make the web faster - Web 905 metrics: Size and number of resources", 2010, 906 . 908 Appendix A. Analysis of header field occurrences 910 An analysis of 30797 requests received by a server located behind a 911 load balancer indicates that a small set of headers is very common : 912 101 different header names were found in requests 913 9.6 headers on average were present in each request 914 headers total 648 bytes per request on average 915 4 header names were present in 100% of the requests (Host, User- 916 Agent, Accept, X-Forwarded-For) 917 4 header names were present in 94% of the requests (Accept- 918 Language, Connection, Accept-Encoding, Referer) 919 1 header name was present in 75% of the requests (Cookie) 920 4 header names were present in more than 10% of the requests 921 (Accept-Charset, UA-CPU, Keep-Alive, Cache-Control) 922 3 header names were present in more than 5% of the requests (Via, 923 If-Modified-Since, If-None-Match) 924 The analysis of the responses was even comparable, with only 22 925 different header names (one single site) : 926 8.6 headers on average were present in each request 927 headers total 257 bytes per request on average 928 3 header names were present in 100% of the requests 929 (Server,Date,Connection) 930 2 header names were present in 97% of the requests (Content- 931 Type,Content-Length) 932 1 header name was present in 67% of the requests (Last-Modified) 933 9 header names were present in more than 10% of the requests 934 (ETag, Accept-Ranges, Expires, Cache-Control, Pragma, P3P, Vary, 935 Content-Encoding, X-Pad) 936 2 header names were present in more than 5% of the requests 937 (Cache-Control, Set-Cookie) 939 It is also worth noting that 40 different header names represent 940 562532 of the 564043 header occurrences (99.73%). These header names 941 alone are responsible for 175 bytes per request on average. 943 Appendix B. Analysis of header field length 945 The analysis above shows that many request headers are almost always 946 identical. Among the 648 bytes per request, we can see that : 947 The User-Agent header is sent with every request yet does not 948 change. This header alone was responsible for 145 bytes on 949 average per request. 950 The Referer header is sent with every request, while it remains 951 unchanged for 9.75 requests on average, sometimes with up to 38 952 requests using the same. This header accounts for 91 bytes per 953 request on average. 954 The Cookie header is sent with 75% of the requests and only 955 changes on average once every 9.6 such requests. It accounts for 956 184 bytes per request. 957 The Accept-Language, Accept-Encoding, Accept-Charset and Accept 958 headers are constant across all requests and account for 121 bytes 959 per request. 960 The transport-specific headers such as Connection, Host, 961 X-Forwarded-For and Keep-alive did not change for a given client. 962 Together they account for 84 bytes per request on average. 963 In the end, only If-Modified-Since and If-None-Match were changed 964 at almost very request. These ones are found in 11% of the 965 requests where they account for 47 bytes on average. 967 The analysis of the responses showed that header values were even 968 more constant, with only the following ones changing with almost 969 every request : 970 Content-Length (found in 94% of the responses) 971 Last-Modified (found in 67% of the responses) 972 ETag (found in 61% of the responses) 974 Authors' Addresses 976 Willy Tarreau 977 Exceliance 978 R&D Produits reseau 979 3 rue du petit Robinson 980 78350 Jouy-en-Josas 981 France 983 Email: w@1wt.eu 984 URI: http://www.exceliance.fr/ 985 Amos Jeffries 986 c/- 130 Fox St 987 Hamilton East 988 Hamilton, 3216 989 New Zealand 991 Phone: +64 21 293 4049 992 Email: amos@treenet.co.nz 993 URI: http://treenet.co.nz/ 995 Adrien de Croy 996 Qbik New Zealand Ltd. 997 28 York St 998 Parnell 999 Auckland 1052 1000 New Zealand 1002 Email: adrien@qbik.com 1003 URI: http://www.wingate.com/ 1005 Poul-Henning Kamp 1006 Herluf Trollesvej 3 1007 Slagelse, DK-4200 1008 Denmark 1010 Phone: +45 21 72 05 25 1011 Email: phk@varnish.org 1012 URI: http://varnish.org/