idnits 2.17.1 draft-ietf-tcpm-fastopen-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The abstract seems to contain references ([RFC2119]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 401 has weird spacing: '... due to incom...' == Line 640 has weird spacing: '... by encrypt...' -- The document date (January 26, 2014) is 3743 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Experimental ---------------------------------------------------------------------------- == Missing Reference: 'RFC2119' is mentioned on line 66, but not defined == Missing Reference: 'RFC3390' is mentioned on line 530, but not defined == Missing Reference: 'RFC 1323' is mentioned on line 651, but not defined ** Obsolete undefined reference: RFC 1323 (Obsoleted by RFC 7323) == Missing Reference: 'BELSHE11' is mentioned on line 750, but not defined == Unused Reference: 'RFC6928' is defined on line 938, but no explicit reference was found in the text == Unused Reference: 'QWGMSS11' is defined on line 977, but no explicit reference was found in the text == Unused Reference: 'BELSHE12' is defined on line 1004, but no explicit reference was found in the text ** Obsolete normative reference: RFC 793 (Obsoleted by RFC 9293) -- Obsolete informational reference (is this intentional?): RFC 1644 (Obsoleted by RFC 6247) -- Obsolete informational reference (is this intentional?): RFC 6013 (Obsoleted by RFC 7805) == Outdated reference: A later version (-10) exists of draft-ietf-tcpm-fastopen-01 Summary: 3 errors (**), 0 flaws (~~), 11 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Draft Y. Cheng 3 draft-ietf-tcpm-fastopen-06.txt J. Chu 4 Intended status: Experimental S. Radhakrishnan 5 Expiration date: February, 2014 A. Jain 6 Google, Inc. 7 January 26, 2014 9 TCP Fast Open 11 Status of this Memo 13 Distribution of this memo is unlimited. 15 This Internet-Draft is submitted in full conformance with the 16 provisions of BCP 78 and BCP 79. 18 Internet-Drafts are working documents of the Internet Engineering 19 Task Force (IETF), its areas, and its working groups. Note that other 20 groups may also distribute working documents as Internet-Drafts. 22 Internet-Drafts are draft documents valid for a maximum of six months 23 and may be updated, replaced, or obsoleted by other documents at any 24 time. It is inappropriate to use Internet-Drafts as reference 25 material or to cite them other than as "work in progress." 27 The list of current Internet-Drafts can be accessed at 28 http://www.ietf.org/1id-abstracts.html 30 The list of Internet-Draft Shadow Directories can be accessed at 31 http://www.ietf.org/shadow.html 33 Copyright Notice 35 Copyright (c) 2014 IETF Trust and the persons identified as the 36 document authors. All rights reserved. 38 This document is subject to BCP 78 and the IETF Trust's Legal 39 Provisions Relating to IETF Documents 40 (http://trustee.ietf.org/license-info) in effect on the date of 41 publication of this document. Please review these documents 42 carefully, as they describe your rights and restrictions with respect 43 to this document. Code Components extracted from this document must 44 include Simplified BSD License text as described in Section 4.e of 45 the Trust Legal Provisions and are provided without warranty as 46 described in the Simplified BSD License. 48 Abstract 49 TCP Fast Open (TFO) allows data to be carried in the SYN and SYN-ACK 50 packets and consumed by the receiving end during the initial 51 connection handshake, thus saving up to one full round trip time 52 (RTT) compared to the standard TCP, which requires a three-way 53 handshake (3WHS) to complete before data can be exchanged. However 54 TFO deviates from the standard TCP semantics the data in the SYN 55 could be replayed to an application in some rare circumstances. 56 Applications should not use TFO unless they can tolerate this issue 57 detailed in the Applicability section. 59 Terminology 61 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 62 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 63 document are to be interpreted as described in RFC 2119 [RFC2119]. 64 TFO refers to TCP Fast Open. Client refers to the TCP's active open 65 side and server refers to the TCP's passive open side. 67 Table of Contents 69 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 70 2. Data In SYN . . . . . . . . . . . . . . . . . . . . . . . . . . 3 71 2.1 Relaxing TCP Semantics on Duplicated SYNs . . . . . . . . . 4 72 2.2. SYNs with Spoofed IP Addresses . . . . . . . . . . . . . . 4 73 3. Protocol Overview . . . . . . . . . . . . . . . . . . . . . . . 5 74 4. Protocol Details . . . . . . . . . . . . . . . . . . . . . . . 7 75 4.1. Fast Open Cookie . . . . . . . . . . . . . . . . . . . . . 7 76 4.1.1. TCP Options . . . . . . . . . . . . . . . . . . . . . . 7 77 4.1.2. Server Cookie Handling . . . . . . . . . . . . . . . . 8 78 4.1.3. Client Cookie Handling . . . . . . . . . . . . . . . . 9 79 4.1.3.1 Client Caching Negative Responses . . . . . . . . . 9 80 4.2. Fast Open Protocol . . . . . . . . . . . . . . . . . . . . 9 81 4.2.1. Fast Open Cookie Request . . . . . . . . . . . . . . . 10 82 4.2.2. TCP Fast Open . . . . . . . . . . . . . . . . . . . . . 11 83 5. Security Considerations . . . . . . . . . . . . . . . . . . . . 13 84 5.1. Resource Exhaustion Attack by SYN Flood with Valid 85 Cookies . . . . . . . . . . . . . . . . . . . . . . . . . . 13 86 5.1.1 Attacks from behind Shared Public IPs (NATs) . . . . . . 14 87 5.2. Amplified Reflection Attack to Random Host . . . . . . . . 15 88 6. TFO's Applicability . . . . . . . . . . . . . . . . . . . . . . 16 89 6.1 Duplicate Data in SYNs . . . . . . . . . . . . . . . . . . . 16 90 6.2 Potential Performance Improvement . . . . . . . . . . . . . 16 91 6.3. Example: Web Clients and Servers . . . . . . . . . . . . . 16 92 6.3.1. HTTP Request Replay . . . . . . . . . . . . . . . . . . 16 93 6.3.2. Speculative Connections by the Applications . . . . . . 17 94 6.3.3. HTTP over TLS (HTTPS) . . . . . . . . . . . . . . . . . 17 95 6.3.4. Comparison with HTTP Persistent Connections . . . . . . 17 96 7. Open Areas for Experimentation . . . . . . . . . . . . . . . . 18 97 7.1. Performance impact due to middle-boxes and NAT . . . . . . 18 98 7.2. Cookie-less Fast Open . . . . . . . . . . . . . . . . . . . 18 99 7.3 Impact on congestion control . . . . . . . . . . . . . . . . 19 100 8. Related Work . . . . . . . . . . . . . . . . . . . . . . . . . 19 101 8.1. T/TCP . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 102 8.2. Common Defenses Against SYN Flood Attacks . . . . . . . . . 19 103 8.3. TCP Cookie Transaction (TCPCT) . . . . . . . . . . . . . . 20 104 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 20 105 10. Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . 20 106 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 20 107 11.1. Normative References . . . . . . . . . . . . . . . . . . . 20 108 11.2. Informative References . . . . . . . . . . . . . . . . . . 21 109 Appendix A. Example Socket API Changes to support TFO . . . . . . 22 110 A.1 Active Open . . . . . . . . . . . . . . . . . . . . . . . . 22 111 A.2 Passive Open . . . . . . . . . . . . . . . . . . . . . . . . 23 112 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 23 114 1. Introduction 116 TCP Fast Open (TFO) enables data to be exchanged safely during TCP's 117 connection handshake. This document describes a design that enables 118 applications to save a round trip while avoiding severe security 119 ramifications. At the core of TFO is a security cookie used by the 120 server side to authenticate a client initiating a TFO connection. 121 This document covers the details of exchanging data during TCP's 122 initial handshake, the protocol for TFO cookies, potential new 123 security vulnerabilities and their mitigation, and the new socket 124 API. 126 TFO is motivated by the performance needs of today's Web 127 applications. Current TCP only permits data exchange after 3WHS 128 [RFC793], which adds one RTT to network latency. For short Web 129 transfers this additional RTT is a significant portion of overall 130 network latency, even when HTTP persistent connection is widely used. 131 For example, the Chrome browser keeps TCP connections idle for up to 132 5 minutes but 35% of Chrome HTTP requests are made on new TCP 133 connections [RCCJR11]. For such Web and Web-like applications placing 134 data in the SYN can yield significant latency improvements. Next we 135 describe how we resolve the challenges that arise upon doing so. 137 2. Data In SYN 139 Standard TCP already allows data to be carried in SYN packets 140 ([RFC793], section 3.4) but forbids the receiver from delivering it 141 to the application until 3WHS is completed. This is because TCP's 142 initial handshake serves to capture old or duplicate SYNs. 144 To enable applications exchange data in TCP handshake, TFO removes 145 the constraint and allows data in SYN packets to be delivered to 146 application. This change of TCP semantic raises two issues discussed 147 in the following subsections, making TFO unsuitable for certain 148 applications. 150 Therefore TCP implementations MUST NOT use TFO by default, but only 151 use TFO if requested explicitly by the application on a per service 152 port basis. Applications need to evaluate TFO applicability described 153 in Section 6 before using TFO. 155 2.1 Relaxing TCP Semantics on Duplicated SYNs 157 TFO allows data to be delivered to the application before 3WHS is 158 completed, thus opening itself to a data integrity issue in either of 159 the two cases below: 161 a) the receiver host receives data in a duplicate SYN after it has 162 forgotten it received the original SYN (e.g. due to a reboot); 164 b) the duplicate is received after the connection created by the 165 original SYN has been closed and the close was initiated by the 166 sender (so the receiver will not be protected by the 2MSL TIMEWAIT 167 state). 169 The now obsoleted T/TCP [RFC1644] attempted to address these issues. 170 It is not successful and not deployed due to various vulnerabilities 171 as described in the Related Work section. Rather than trying to 172 capture all dubious SYN packets to make TFO 100% compatible with TCP 173 semantics, we made a design decision early on to accept old SYN 174 packets with data, i.e., to restrict TFO use to a class of 175 applications (Section 6) that are tolerant of duplicate SYN packets 176 with data. We believe this is the right design trade-off balancing 177 complexity with usefulness. 179 2.2. SYNs with Spoofed IP Addresses 181 Standard TCP suffers from the SYN flood attack [RFC4987] because SYN 182 packets with spoofed source IP addresses can easily fill up a 183 listener's small queue, causing a service port to be blocked 184 completely until timeouts. 186 TFO goes one step further to allow server-side TCP to send up data to 187 the application layer before 3WHS is completed. This opens up serious 188 new vulnerabilities. Applications serving ports that have TFO enabled 189 may waste lots of CPU and memory resources processing the requests 190 and producing the responses. If the response is much larger than the 191 request, the attacker can further mount an amplified reflection 192 attack against victims of choice beyond the TFO server itself. 194 Numerous mitigation techniques against regular SYN flood attacks 195 exist and have been well documented [RFC4987]. Unfortunately none are 196 applicable to TFO. We propose a server-supplied cookie to mitigate 197 these new vulnerabilities in Section 3 and evaluate the effectiveness 198 of the defense in Section 7. 200 3. Protocol Overview 202 The key component of TFO is the Fast Open Cookie (cookie), a message 203 authentication code (MAC) tag generated by the server. The client 204 requests a cookie in one regular TCP connection, then uses it for 205 future TCP connections to exchange data during 3WHS: 207 Requesting a Fast Open Cookie: 209 1. The client sends a SYN with a Fast Open Cookie Request option. 211 2. The server generates a cookie and sends it through the Fast Open 212 Cookie option of a SYN-ACK packet. 214 3. The client caches the cookie for future TCP Fast Open connections 215 (see below). 217 Performing TCP Fast Open: 219 1. The client sends a SYN with Fast Open Cookie option and data. 221 2. The server validates the cookie: 222 a. If the cookie is valid, the server sends a SYN-ACK 223 acknowledging both the SYN and the data. The server then 224 delivers the data to the application. 226 b. Otherwise, the server drops the data and sends a SYN-ACK 227 acknowledging only the SYN sequence number. 229 3. If the server accepts the data in the SYN packet, it may send the 230 response data before the handshake finishes. The max amount is 231 governed by the TCP's congestion control [RFC5681]. 233 4. The client sends an ACK acknowledging the SYN and the server data. 234 If the client's data is not acknowledged, the client retransmits 235 the data in the ACK packet. 237 5. The rest of the connection proceeds like a normal TCP connection. 238 The client can repeat many Fast Open operations once it acquires a 239 cookie (until the cookie is expired by the server). Thus TFO is 240 useful for applications that have temporal locality on client and 241 server connections. 243 Requesting Fast Open Cookie in connection 1: 245 TCP A (Client) TCP B(Server) 246 ______________ _____________ 247 CLOSED LISTEN 249 #1 SYN-SENT ----- ----------> SYN-RCVD 251 #2 ESTABLISHED <---- ---------- SYN-RCVD 252 (caches cookie C) 254 Performing TCP Fast Open in connection 2: 256 TCP A (Client) TCP B(Server) 257 ______________ _____________ 258 CLOSED LISTEN 260 #1 SYN-SENT ----- ----> SYN-RCVD 262 #2 ESTABLISHED <---- ---- SYN-RCVD 264 #3 ESTABLISHED <---- ---- SYN-RCVD 266 #4 ESTABLISHED ----- --------------------> ESTABLISHED 268 #5 ESTABLISHED --- ----------> ESTABLISHED 270 4. Protocol Details 272 4.1. Fast Open Cookie 274 The Fast Open Cookie is designed to mitigate new security 275 vulnerabilities in order to enable data exchange during handshake. 276 The cookie is a message authentication code tag generated by the 277 server and is opaque to the client; the client simply caches the 278 cookie and passes it back on subsequent SYN packets to open new 279 connections. The server can expire the cookie at any time to enhance 280 security. 282 4.1.1. TCP Options 284 Fast Open Cookie Option 286 The server uses this option to grant a cookie to the client in the 287 SYN-ACK packet; the client uses it to pass the cookie back to the 288 server in subsequent SYN packets. 290 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 291 | Kind | Length | 292 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 293 | | 294 ~ Cookie ~ 295 | | 296 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 298 Kind 1 byte: constant TBD (assigned by IANA) 299 Length 1 byte: range 6 to 18 (bytes); limited by 300 remaining space in the options field. 301 The number MUST be even. 302 Cookie 4 to 16 bytes (Length - 2) 303 Options with invalid Length values or without SYN flag set MUST be 304 ignored. The minimum Cookie size is 4 bytes. Although the diagram 305 shows a cookie aligned on 32-bit boundaries, alignment is not 306 required. 308 Fast Open Cookie Request Option 310 The client uses this option in the SYN packet to request a cookie 311 from a TFO-enabled server 312 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 313 | Kind | Length | 314 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 316 Kind 1 byte: same as the Fast Open Cookie option 317 Length 1 byte: constant 2. This distinguishes the option 318 from the Fast Open cookie option. 319 Options with invalid Length values, without SYN flag set, or with ACK 320 flag set MUST be ignored. 322 4.1.2. Server Cookie Handling 324 The server is in charge of cookie generation and authentication. The 325 cookie SHOULD be a message authentication code tag with the following 326 properties: 327 1. The cookie authenticates the client's (source) IP address of the 328 SYN packet. The IP address may be an IPv4 or IPv6 address. 330 2. The cookie can only be generated by the server and can not be 331 fabricated by any other parties including the client. 333 3. The generation and verification are fast relative to the rest of 334 SYN and SYN-ACK processing. 336 4. A server may encode other information in the cookie, and accept 337 more than one valid cookie per client at any given time. But this 338 is all server implementation dependent and transparent to the 339 client. 341 5. The cookie expires after a certain amount of time. The reason for 342 cookie expiration is detailed in the "Security Consideration" 343 section. This can be done by either periodically changing the 344 server key used to generate cookies or including a timestamp when 345 generating the cookie. 347 To gradually invalidate cookies over time, the server can 348 implement key rotation to generate and verify cookies using 349 multiple keys. This approach is useful for large-scale servers to 350 retain Fast Open rolling key updates. We do not specify a 351 particular mechanism because the implementation is server 352 specific. 354 The server supports the cookie generation and verification 355 operations: 357 - GetCookie(IP_Address): returns a (new) cookie 359 - IsCookieValid(IP_Address, Cookie): checks if the cookie is valid, 360 i.e., it has not expired and it authenticates the client IP address. 362 Example Implementation: a simple implementation is to use AES_128 to 363 encrypt the IPv4 (with padding) or IPv6 address and truncate to 64 364 bits. The server can periodically update the key to expire the 365 cookies. AES encryption on recent processors is fast and takes only a 366 few hundred nanoseconds [RCCJR11]. 368 If only one valid cookie is allowed per-IP and the server can 369 regenerate the cookie independently, the best validation process is 370 to simply regenerate a valid cookie and compare it against the 371 incoming cookie. In that case if the incoming cookie fails the check, 372 a valid cookie is readily available to be sent to the client. 374 4.1.3. Client Cookie Handling 376 The client MUST cache cookies from servers for later Fast Open 377 connections. For a multi-homed client, the cookies are both client 378 and server IP dependent. Beside the cookie we RECOMMEND that the 379 client caches the MSS to the server to enhance performance. 381 The MSS advertised by the server is stored in the cache to determine 382 the maximum amount of data that can be supported in the SYN packet. 383 This information is needed because data is sent before the server 384 announces its MSS in the SYN-ACK packet. Without this information, 385 the data size in the SYN packet is limited to the default MSS of 536 386 bytes [RFC1122]. In particular it's known IPv4 receiver advertised 387 MSS less than 536 bytes would result in transmission of an unexpected 388 large segment. If the cache MSS is larger than the typical 1460 389 bytes, the extra large data in SYN segment may have issues that 390 offsets the performance benefit of Fast Open. For examples, the 391 super-size SYN may trigger IP fragmentation and confuses firewall or 392 middle-boxes. Therefore the client MAY limit the cached MSS to 1460 393 bytes. 395 4.1.3.1 Client Caching Negative Responses 397 The client MUST cache negative responses from the server in order to 398 avoid potential connection failures. Negative responses include 399 server not acknowledging the data in SYN, ICMP error messages, and 400 most importantly no response (SYN/ACK) from the server at all, i.e, 401 connection timeout. The last case is likely due to incompatible 402 middle-boxes or firewall blocking the connection completely after it 403 sees data in SYN. If the client does not react to these negative 404 responses and continue to retry Fast Open, the client may never be 405 able to connect to the specific server. 407 For any negative responses, the client SHOULD disables Fast Open on 408 the specific path and port, at least temporarily. Since TFO is 409 enabled on a per-service port basis but cookies are independent of 410 service ports, clients' cache should include remote port numbers too. 412 4.2. Fast Open Protocol 413 One predominant requirement of TFO is to be fully compatible with 414 existing TCP implementations, both on the client and the server 415 sides. 417 The server keeps two variables per listening port: 419 FastOpenEnabled: default is off. It MUST be turned on explicitly by 420 the application. When this flag is off, the server does not perform 421 any TFO related operations and MUST ignore all cookie options. 423 PendingFastOpenRequests: tracks number of TFO connections in SYN-RCVD 424 state. If this variable goes over a preset system limit, the server 425 SHOULD disable TFO for all new connection requests until 426 PendingFastOpenRequests drops below the system limit. This variable 427 is used for defending some vulnerabilities discussed in the "Security 428 Considerations" section. 430 The server keeps a FastOpened flag per TCB to mark if a connection 431 has successfully performed a TFO. 433 4.2.1. Fast Open Cookie Request 435 Any client attempting TFO MUST first request a cookie from the server 436 with the following steps: 438 1. The client sends a SYN packet with a Fast Open Cookie Request 439 option. 441 2. The server SHOULD respond with a SYN-ACK based on the procedures 442 in the "Server Cookie Handling" section. This SYN-ACK SHOULD 443 contain a Fast Open Cookie option if the server currently supports 444 TFO for this listener port. 446 3. If the SYN-ACK contains a Fast Open Cookie option, the client 447 replaces the cookie and other information as described in the 448 "Client Cookie Handling" section. Otherwise, if the SYN-ACK is 449 first seen, i.e.,not a (spurious) retransmission, the client MAY 450 remove the server information from the cookie cache. If the SYN- 451 ACK is a spurious retransmission without valid Fast Open Cookie 452 Option, the client does nothing to the cookie cache for the 453 reasons below. 455 The network or servers may drop the SYN or SYN-ACK packets with the 456 new cookie options, which will cause SYN or SYN-ACK timeouts. We 457 RECOMMEND both the client and the server to retransmit SYN and SYN- 458 ACK without the cookie options on timeouts. This ensures the 459 connections of cookie requests will go through and lowers the latency 460 penalty (of dropped SYN/SYN-ACK packets). The obvious downside for 461 maximum compatibility is that any regular SYN drop will fail the 462 cookie (although one can argue the delay in the data transmission 463 till after 3WHS is justified if the SYN drop is due to network 464 congestion). Next section describes a heuristic to detect such drops 465 when the client receives the SYN-ACK. 467 We also RECOMMEND the client to record servers that failed to respond 468 to cookie requests and only attempt another cookie request after 469 certain period. An alternate proposal is to request cookie in FIN 470 instead since FIN-drop by incompatible middle-box does not affect 471 latency. However such paths are likely to drop SYN packet with data 472 later, and many applications close the connections with RST instead, 473 so the actual benefit of this approach is not clear. 475 4.2.2. TCP Fast Open 477 Once the client obtains the cookie from the target server, it can 478 perform subsequent TFO connections until the cookie is expired by the 479 server. 481 Client: Sending SYN 483 To open a TFO connection, the client MUST have obtained a cookie from 484 the server: 486 1. Send a SYN packet. 488 a. If the SYN packet does not have enough option space for the 489 Fast Open Cookie option, abort TFO and fall back to regular 3WHS. 491 b. Otherwise, include the Fast Open Cookie option with the cookie 492 of the server. Include any data up to the cached server MSS or 493 default 536 bytes. 495 2. Advance to SYN-SENT state and update SND.NXT to include the data 496 accordingly. 498 3. If RTT is available from the cache, seed SYN timer according to 499 [RFC6298]. 501 To deal with network or servers dropping SYN packets with payload or 502 unknown options, when the SYN timer fires, the client SHOULD 503 retransmit a SYN packet without data and Fast Open Cookie options. 505 Server: Receiving SYN and responding with SYN-ACK 507 Upon receiving the SYN packet with Fast Open Cookie option: 509 1. Initialize and reset a local FastOpened flag. If FastOpenEnabled 510 is false, go to step 5. 512 2. If PendingFastOpenRequests is over the system limit, go to step 5. 514 3. If IsCookieValid() in section 4.1.2 returns false, go to step 5. 516 4. Buffer the data and notify the application. Set FastOpened flag 517 and increment PendingFastOpenRequests. 519 5. Send the SYN-ACK packet. The packet MAY include a Fast Open 520 Option. If FastOpened flag is set, the packet acknowledges the SYN 521 and data sequence. Otherwise it acknowledges only the SYN 522 sequence. The server MAY include data in the SYN-ACK packet if the 523 response data is readily available. Some application may favor 524 delaying the SYN-ACK, allowing the application to process the 525 request in order to produce a response, but this is left up to the 526 implementation. 528 6. Advance to the SYN-RCVD state. If the FastOpened flag is set, the 529 server MUST follow the congestion control [RFC5681], in particular 530 the initial congestion window [RFC3390], to send more data 531 packets. 533 If the SYN-ACK timer fires, the server SHOULD retransmit a SYN-ACK 534 segment with neither data nor Fast Open Cookie options for 535 compatibility reasons. 537 A special case is simultaneous open where the SYN receiver is a 538 client in SYN-SENT state. The protocol remains the same because 539 [RFC793] already supports both data in SYN and simultaneous open. But 540 the client's socket may have data available to read before it's 541 connected. This document does not cover the corresponding API change. 543 Client: Receiving SYN-ACK 545 The client SHOULD perform the following steps upon receiving the SYN- 546 ACK: 548 1. Update the cookie cache if the SYN-ACK has a Fast Open Cookie 549 Option or MSS option or both. 551 2. Send an ACK packet. Set acknowledgment number to RCV.NXT and 552 include the data after SND.UNA if data is available. 554 3. Advance to the ESTABLISHED state. 556 Note there is no latency penalty if the server does not acknowledge 557 the data in the original SYN packet. The client SHOULD retransmit any 558 unacknowledged data in the first ACK packet in step 2. The data 559 exchange will start after the handshake like a regular TCP 560 connection. 562 If the client has timed out and retransmitted only regular SYN 563 packets, it can heuristically detect paths that intentionally drop 564 SYN with Fast Open option or data. If the SYN-ACK acknowledges only 565 the initial sequence and does not carry a Fast Open cookie option, 566 presumably it is triggered by a retransmitted (regular) SYN and the 567 original SYN or the corresponding SYN-ACK was lost. 569 Server: Receiving ACK 571 Upon receiving an ACK acknowledging the SYN sequence, the server 572 decrements PendingFastOpenRequests and advances to the ESTABLISHED 573 state. No special handling is required further. 575 5. Security Considerations 577 The Fast Open cookie stops an attacker from trivially flooding 578 spoofed SYN packets with data to burn server resources or to mount an 579 amplified reflection attack on random hosts. The server can defend 580 against spoofed SYN floods with invalid cookies using existing 581 techniques [RFC4987]. We note that although generating bogus cookies 582 is cost-free, the cost of validating the cookies, inherent to any 583 authentication scheme, may not be substantial compared to processing 584 a regular SYN packet. 586 5.1. Resource Exhaustion Attack by SYN Flood with Valid Cookies 588 However, the attacker may still obtain cookies from some compromised 589 hosts, then flood spoofed SYN with data and "valid" cookies (from 590 these hosts or other vantage points). With DHCP, it's possible to 591 obtain cookies of past IP addresses without compromising any host. 592 Below we identify new vulnerabilities of TFO and describe the 593 countermeasures. 595 Like regular TCP handshakes, TFO is vulnerable to such an attack. But 596 the potential damage can be much more severe. Besides causing 597 temporary disruption to service ports under attack, it may exhaust 598 server CPU and memory resources. Such an attack will show up on 599 application server logs as a application level DoS from Bot-nets, 600 triggering other defenses and alerts. 602 For this reason it is crucial for the TFO server to limit the maximum 603 number of total pending TFO connection requests, i.e., 604 PendingFastOpenRequests. When the limit is exceeded, the server 605 temporarily disables TFO entirely as described in "Server Cookie 606 Handling". Then subsequent TFO requests will be downgraded to regular 607 connection requests, i.e., with the data dropped and only SYN 608 acknowledged. This allows regular SYN flood defense techniques 609 [RFC4987] like SYN-cookies to kick in and prevent further service 610 disruption. 612 The main impact of SYN floods against the standard TCP stack is not 613 directly from the floods themselves costing TCP processing overhead 614 or host memory, but rather from the spoofed SYN packets filling up 615 the often small listener's queue. 617 On the other hand, TFO SYN floods can cause damage directly if 618 admitted without limit into the stack. The RST packets from the 619 spoofed host will fuel rather than defeat the SYN floods as compared 620 to the non-TFO case, because the attacker can flood more SYNs with 621 data to cost more data processing resources. For this reason, a TFO 622 server needs to monitor the connections in SYN-RCVD being reset in 623 addition to imposing a reasonable max queue length. Implementations 624 may combine the two, e.g., by continuing to account for those 625 connection requests that have just been reset against the listener's 626 PendingFastOpenRequests until a timeout period has passed. 628 Limiting the maximum number of pending TFO connection requests does 629 make it easy for an attacker to overflow the queue, causing TFO to be 630 disabled. We argue that causing TFO to be disabled is unlikely to be 631 of interest to attackers because the service will remain intact 632 without TFO hence there is hardly any real damage. 634 5.1.1 Attacks from behind Shared Public IPs (NATs) 636 An attacker behind NAT can easily obtain valid cookies to launch the 637 above attack to hurt other clients that share the path. [BRISCOE12] 638 suggested that the server can extend cookie generation to include the 639 TCP timestamp---GetCookie(IP_Address, Timestamp)---and implement it 640 by encrypting the concatenation of the two values to generate the 641 cookie. The client stores both the cookie and its corresponding 642 timestamp, and echoes both in the SYN. The server then implements 643 IsCookieValid(IP_Address, Timestamp, Cookie) by encrypting the IP and 644 timestamp data and comparing it with the cookie value. 646 This enables the server to issue different cookies to clients that 647 share the same IP address, hence can selectively discard those 648 misused cookies from the attacker. However the attacker can simply 649 repeat the attack with new cookies. The server would eventually need 650 to throttle all requests from the IP address just like the current 651 approach. Moreover this approach requires modifying [RFC 1323] to 652 send non-zero Timestamp Echo Reply in SYN, potentially cause firewall 653 issues. Therefore we believe the benefit does not outweigh the 654 drawbacks. 656 5.2. Amplified Reflection Attack to Random Host 658 Limiting PendingFastOpenRequests with a system limit can be done 659 without Fast Open Cookies and would protect the server from resource 660 exhaustion. It would also limit how much damage an attacker can cause 661 through an amplified reflection attack from that server. However, it 662 would still be vulnerable to an amplified reflection attack from a 663 large number of servers. An attacker can easily cause damage by 664 tricking many servers to respond with data packets at once to any 665 spoofed victim IP address of choice. 667 With the use of Fast Open Cookies, the attacker would first have to 668 steal a valid cookie from its target victim. This likely requires the 669 attacker to compromise the victim host or network first. 671 The attacker here has little interest in mounting an attack on the 672 victim host that has already been compromised. But it may be 673 motivated to disrupt the victim's network. Since a stolen cookie is 674 only valid for a single server, it has to steal valid cookies from a 675 large number of servers and use them before they expire to cause 676 sufficient damage without triggering the defense. 678 One can argue that if the attacker has compromised the target network 679 or hosts, it could perform a similar but simpler attack by injecting 680 bits directly. The degree of damage will be identical, but TFO- 681 specific attack allows the attacker to remain anonymous and disguises 682 the attack as from other servers. 684 The best defense is for the server not to respond with data until 685 handshake finishes. In this case the risk of amplification reflection 686 attack is completely eliminated. But the potential latency saving 687 from TFO may diminish if the server application produces responses 688 earlier before the handshake completes. 690 6. TFO's Applicability 692 This section is to help applications considering TFO to evaluate 693 TFO's benefits and drawbacks using the Web client and server 694 applications as an example throughout. Applications here refer 695 specifically to the process that writes data into the socket, i.e., a 696 java script process that sends data to the server. A proposed socket 697 API change is in the Appendix. 699 6.1 Duplicate Data in SYNs 701 It is possible, though uncommon, that using TFO results in the first 702 data written to a socket to be delivered more than once to the 703 application on the remote host(Section 2.1). This replay potential 704 only applies to data in the SYN but not subsequent data exchanges. 705 The client MUST NOT use TFO to send data in the SYN, and the server 706 MUST NOT accept data in the SYN if it cannot handle receiving the 707 same SYN data more than once, due to reasons described before. 709 6.2 Potential Performance Improvement 711 TFO is designed for latency-conscious applications that are sensitive 712 to TCP's initial connection setup delay. To benefit from TFO, the 713 first application data unit (e.g., an HTTP request) needs to be no 714 more than TCP's maximum segment size (minus options used in SYN). 715 Otherwise the remote server can only process the client's application 716 data unit once the rest of it is delivered after the initial 717 handshake, diminishing TFO's benefit. 719 To the extent possible, applications SHOULD reuse the connection to 720 take advantage of TCP's built-in congestion control and reduce 721 connection setup overhead. An application that employs too many 722 short-lived connections will negatively impact network stability, as 723 these connections often exit before TCP's congestion control 724 algorithm takes effect. 726 6.3. Example: Web Clients and Servers 728 6.3.1. HTTP Request Replay 730 While TFO is motivated by Web applications, the browser should not 731 use TFO to send requests in SYNs if those requests cannot tolerate 732 replays. One example is POST requests without application-layer 733 transaction protection (e.g., a unique identifier in the request 734 header). 736 On the other hand, TFO is particularly useful for GET requests. 737 Although not all GET requests are idem-potent, GETs are frequently 738 replayed today across striped TCP connections: after a server 739 receives an HTTP request but before the ACKs of the requests reach 740 the browser, the browser may timeout and retry the same request on 741 another (possibly new) TCP connection. This differs from a TFO replay 742 only in that the replay is initiated by the browser, not by the TCP 743 stack. 745 6.3.2. Speculative Connections by the Applications 747 Some Web browsers maintain a history of the domains for frequently 748 visited web pages. The browsers then speculatively pre-open TCP 749 connections to these domains before the user initiates any requests 750 for them [BELSHE11]. While this technique also saves the handshake 751 latency, it wastes server and network resources by initiating and 752 maintaining idle connections. 754 6.3.3. HTTP over TLS (HTTPS) 756 For TLS over TCP, it is safe and useful to include TLS CLIENT_HELLO 757 in the SYN packet to save 1-RTT in TLS handshake. There is no concern 758 about violating idem-potency. In particular it can be used alone with 759 the speculative connection above. While HTTPS adoption is still low, 760 it will increase over time (especially given the context of recent 761 revelations). Thus the potential importance of TCP Fast Open in 762 decreasing user perceived latency in HTTPS. 764 6.3.4. Comparison with HTTP Persistent Connections 766 Is TFO useful given the wide deployment of HTTP persistent 767 connections? The short answer is yes. Studies [RCCJR11][AERG11] show 768 that the average number of transactions per connection is between 2 769 and 4, based on large-scale measurements from both servers and 770 clients. In these studies, the servers and clients both kept idle 771 connections up to several minutes, well into "human think" time. 773 Keeping connections open and idle even longer risks a greater 774 performance penalty. [HNESSK10][MQXMZ11] show that the majority of 775 home routers and ISPs fail to meet the the 124-minute idle timeout 776 mandated in [RFC5382]. In [MQXMZ11], 35% of mobile ISPs silently 777 timeout idle connections within 30 minutes. End hosts, unaware of 778 silent middle-box timeouts, suffer multi-minute TCP timeouts upon 779 using those long-idle connections. 781 To circumvent this problem, some applications send frequent TCP keep- 782 alive probes. However, this technique drains power on mobile devices 783 [MQXMZ11]. In fact, power has become such a prominent issue in modern 784 LTE devices that mobile browsers close HTTP connections within 785 seconds or even immediately [SOUDERS11]. 787 [RCCJR11] studied Chrome browser performance based on 28 days of 788 global statistics. The Chrome browser keeps idle HTTP persistent 789 connections for 5 to 10 minutes. However the average number of the 790 transactions per connection is only 3.3 and TCP 3WHS accounts for up 791 to 25% of the HTTP transaction network latency. The authors 792 estimated that TFO improves page load time by 10% to 40% on selected 793 popular Web sites. 795 7. Open Areas for Experimentation 797 We now outline some areas that need experimentation in the Internet 798 and under different network scenarios. These experiments should help 799 the community evaluate Fast Open benefits and risks towards further 800 standardization and implementation of Fast Open and its related 801 protocols. 803 7.1. Performance impact due to middle-boxes and NAT 805 [MAF04] found that some middle-boxes and end-hosts may drop packets 806 with unknown TCP options. Studies [LANGLEY06, HNRGHT11] both found 807 that 6% of the probed paths on the Internet drop SYN packets with 808 data or with unknown TCP options. The TFO protocol deals with this 809 problem by falling back to regular TCP handshake and re-transmitting 810 SYN without data or cookie options after the initial SYN timeout. 811 Moreover the implementation is recommended to negatively cache such 812 incidents to avoid recurring timeouts. Further study is required to 813 evaluate the performance impact of these malicious drop behaviors. 815 Another interesting study is the (loss of) TFO performance benefit 816 behind certain carrier-grade NAT. Typically hosts behind a NAT 817 sharing the same IP address will get the same cookie for the same 818 server. This will not prevent TFO from working. But on some carrier- 819 grade NAT configurations where every new TCP connection from the same 820 physical host uses a different public IP address, TFO does not 821 provide latency benefits. However, there is no performance penalty 822 either, as described in Section "Client: Receiving SYN-ACK". 824 7.2. Cookie-less Fast Open 826 The cookie mechanism mitigates resource exhaustion and amplification 827 attacks. However cookies are not necessary if the server has 828 application-level protection or is immune to these attacks. For 829 example a Web server that only replies with a simple HTTP redirect 830 response that fits in the SYN-ACK packet may not care about resource 831 exhaustion. Such an application can safely disable TFO cookie checks. 833 Disabling cookies simplifies both the client and the server, as the 834 client no longer needs to request a cookie and the server no longer 835 needs to check or generate cookies. Disabling cookies also 836 potentially simplifies configuration, as the server no longer needs a 837 key. It may be preferable to enable SYN cookies and disable TFO 838 [RFC4987] when a server is overloaded by a large-scale Bot-net 839 attack. 841 Careful experimentation is necessary to evaluate if cookie-less TFO 842 is practical. The implementation can provide an experimental feature 843 to allow zero length, or null, cookies as opposed to the minimum 4 844 bytes cookies. Thus the server may return a null cookie and the 845 client will send data in SYN with it subsequently. If the server 846 believes it's under a DoS attack through other defense mechanisms, it 847 can switch to regular Fast Open for listener sockets. 849 7.3 Impact on congestion control 851 Although TFO does not directly change the congestion control, there 852 are subtle cases that it may. When SYN-ACK times out, regular TCP 853 reduces the initial congestion window before sending any data 854 [RFC5681]. However in TFO the server may have already sent up to an 855 initial window of data. 857 If the server serves mostly short connections then the losses of SYN- 858 ACKs are not as effective as regular TCP on reducing the congestion 859 window. This could result in an unstable network condition. The 860 connections that experience losses may attempt again and add more 861 load under congestion. A potential solution is to temporarily disable 862 Fast Open if the server observes many SYN-ACK or data losses during 863 the handshake across connections. 865 8. Related Work 867 8.1. T/TCP 869 TCP Extensions for Transactions [RFC1644] attempted to bypass the 870 three-way handshake, among other things, hence shared the same goal 871 but also the same set of issues as TFO. It focused most of its effort 872 battling old or duplicate SYNs, but paid no attention to security 873 vulnerabilities it introduced when bypassing 3WHS [PHRACK98]. 875 As stated earlier, we take a practical approach to focus TFO on the 876 security aspect, while allowing old, duplicate SYN packets with data 877 after recognizing that 100% TCP semantics is likely infeasible. We 878 believe this approach strikes the right tradeoff, and makes TFO much 879 simpler and more appealing to TCP implementers and users. 881 8.2. Common Defenses Against SYN Flood Attacks 883 [RFC4987] studies on mitigating attacks from regular SYN flood, i.e., 884 SYN without data . But from the stateless SYN-cookies to the stateful 885 SYN Cache, none can preserve data sent with SYN safely while still 886 providing an effective defense. 888 The best defense may be to simply disable TFO when a host is 889 suspected to be under a SYN flood attack, e.g., the SYN backlog is 890 filled. Once TFO is disabled, normal SYN flood defenses can be 891 applied. The "Security Consideration" section contains a thorough 892 discussion on this topic. 894 8.3. TCP Cookie Transaction (TCPCT) 896 TCPCT [RFC6013] eliminates server state during initial handshake and 897 defends spoofing DoS attacks. Like TFO, TCPCT allows SYN and SYN-ACK 898 packets to carry data. But the server can only send up to MSS bytes 899 of data during the handshake instead of the initial congestion window 900 unlike TFO. Therefore applications like Web may not receive the 901 latency benefit as TFO. 903 9. IANA Considerations 905 The Fast Open Cookie Option and Fast Open Cookie Request Option 906 define no new namespace. The options require IANA to allocate one 907 value from the TCP option Kind namespace. Early implementation before 908 the IANA allocation SHOULD follow [RFC6994] and use experimental 909 option 254 and magic number 0xF989 (16 bits), then migrate to the new 910 option after the allocation accordingly. 912 10. Acknowledgement 914 We thank Bob Briscoe, Michael Scharf, Gorry Fairhurst, Rick Jones, 915 Roberto Peon, William Chan, Adam Langley, Neal Cardwell, Eric 916 Dumazet, and Matt Mathis for their feedbacks. We especially thank 917 Barath Raghavan for his contribution on the security design of Fast 918 Open and proofreading this draft numerous times. 920 11. References 922 11.1. Normative References 923 [RFC793] Postel, J. "Transmission Control Protocol", RFC 793, 924 September 1981. 926 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - 927 Communication Layers", STD 3, RFC 1122, October 1989. 929 [RFC5382] S. Guha, Ed., Biswas, K., Ford B., Sivakumar S., Srisuresh, 930 P., "NAT Behavioral Requirements for TCP", RFC 5382 932 [RFC5681] Allman, M., Paxson, V. and E. Blanton, "TCP Congestion 933 Control", RFC 5681, September 2009. 935 [RFC6298] Paxson, V., Allman, M., Chu, J. and M. Sargent, "Computing 936 TCP's Retransmission Timer", RFC 6298, June 2011. 938 [RFC6928] Chu, J., Dukkipati, N., Cheng, Y. and M. Mathis, 939 "Increasing TCP's Initial Window", RFC 6928, April 2013. 941 [RFC6994] Touch, Joe, "Shared Use of Experimental TCP Options", 942 RFC6994, August 2013. 944 11.2. Informative References 946 [AERG11] M. Al-Fares, K. Elmeleegy, B. Reed, and I. Gashinsky, 947 "Overclocking the Yahoo! CDN for Faster Web Page Loads". In 948 Proceedings of Internet Measurement Conference, November 949 2011. 951 [HNESSK10] S. Haetoenen, A. Nyrhinen, L. Eggert, S. Strowes, P. 952 Sarolahti, M. Kojo., "An Experimental Study of Home Gateway 953 Characteristics". In Proceedings of Internet Measurement 954 Conference. Octobor 2010 956 [HNRGHT11] M. Honda, Y. Nishida, C. Raiciu, A. Greenhalgh, M. 957 Handley, H. Tokuda, "Is it Still Possible to Extend TCP?". 958 In Proceedings of Internet Measurement Conference. November 959 2011. 961 [LANGLEY06] Langley, A, "Probing the viability of TCP extensions", 962 URL http://www.imperialviolet.org/binary/ecntest.pdf 964 [MAF04] Medina, A., Allman, M., and S. Floyd, "Measuring 965 Interactions Between Transport Protocols and Middleboxes", 966 In Proceedings of Internet Measurement Conference, October 967 2004. 969 [MQXMZ11] Z. Mao, Z. Qian, Q. Xu, Z. Mao, M. Zhang. "An Untold Story 970 of Middleboxes in Cellular Networks", In Proceedings of 971 SIGCOMM. August 2011. 973 [PHRACK98] "T/TCP vulnerabilities", Phrack Magazine, Volume 8, Issue 974 53 artical 6. July 8, 1998. URL 975 http://www.phrack.com/issues.html?issue=53&id=6 977 [QWGMSS11] F. Qian, Z. Wang, A. Gerber, Z. Mao, S. Sen, O. 978 Spatscheck. "Profiling Resource Usage for Mobile 979 Applications: A Cross-layer Approach", In Proceedings of 980 International Conference on Mobile Systems. April 2011. 982 [RCCJR11] Radhakrishnan, S., Cheng, Y., Chu, J., Jain, A. and 983 Raghavan, B., "TCP Fast Open". In Proceedings of 7th ACM 984 CoNEXT Conference, December 2011. 986 [RFC1644] Braden, R., "T/TCP -- TCP Extensions for Transactions 987 Functional Specification", RFC 1644, July 1994. 989 [RFC4987] Eddy, W., "TCP SYN Flooding Attacks and Common 990 Mitigations", RFC 4987, August 2007. 992 [RFC6013] Simpson, W., "TCP Cookie Transactions (TCPCT)", RFC6013, 993 January 2011. 995 [SOUDERS11] S. Souders. "Making A Mobile Connection". 996 http://www.stevesouders.com/blog/2011/09/21/making-a- 997 mobile-connection/ 999 [BRISCOE12] Briscoe, B., "Some ideas building on draft-ietf-tcpm- 1000 fastopen-01", tcpm list, 1001 http://www.ietf.org/mail-archive/web/tcpm/current/ 1002 January 16, 2014msg07192.html 1004 [BELSHE12] Belshe, M., "The era of browser preconnect.", 1005 http://www.belshe.com/2011/02/10/ 1006 the-era-of-browser-preconnect/ 1008 Appendix A. Example Socket API Changes to support TFO 1010 A.1 Active Open 1012 The active open side involves changing or replacing the connect() 1013 call, which does not take a user data buffer argument. We recommend 1014 replacing connect() call to minimize API changes and hence 1015 applications to reduce the deployment hurdle. 1017 One solution implemented in Linux 3.7 is introducing a new flag 1018 MSG_FASTOPEN for sendto() or sendmsg(). MSG_FASTOPEN marks the 1019 attempt to send data in SYN like a combination of connect() and 1020 sendto(), by performing an implicit connect() operation. It blocks 1021 until the handshake has completed and the data is buffered. 1023 For non-blocking socket it returns the number of bytes buffered and 1024 sent in the SYN packet. If the cookie is not available locally, it 1025 returns -1 with errno EINPROGRESS, and sends a SYN with TFO cookie 1026 request automatically. The caller needs to write the data again when 1027 the socket is connected. On errors, it returns the same errno as 1028 connect() if the handshake fails. 1030 An implementation may prefer not to change the sendmsg() because TFO 1031 is a TCP specific feature. A solution is to add a new socket option 1032 TCP_FASTOPEN for TCP sockets. When the option is enabled before a 1033 connect operation, sendmsg() or sendto() will perform Fast Open 1034 operation similar to the MSG_FASTOPEN flag described above. This 1035 approach however requires an extra setsockopt() system call. 1037 A.2 Passive Open 1039 The passive open side change is simpler compared to active open side. 1040 The application only needs to enable the reception of Fast Open 1041 requests via a new TCP_FASTOPEN setsockopt() socket option before 1042 listen(). 1044 The option enables Fast Open on the listener socket. The option value 1045 specifies the PendingFastOpenRequests threshold, i.e., the maximum 1046 length of pending SYNs with data payload. Once enabled, the TCP 1047 implementation will respond with TFO cookies per request. 1049 Traditionally accept() returns only after a socket is connected. But 1050 for a Fast Open connection, accept() returns upon receiving a SYN 1051 with a valid Fast Open cookie and data, and the data is available to 1052 be read through, e.g., recvmsg(), read(). 1054 Authors' Addresses 1055 Yuchung Cheng 1056 Google, Inc. 1057 1600 Amphitheatre Parkway 1058 Mountain View, CA 94043, USA 1059 EMail: ycheng@google.com 1061 Jerry Chu 1062 Google, Inc. 1063 1600 Amphitheatre Parkway 1064 Mountain View, CA 94043, USA 1065 EMail: hkchu@google.com 1067 Sivasankar Radhakrishnan 1068 Department of Computer Science and Engineering 1069 University of California, San Diego 1070 9500 Gilman Dr 1071 La Jolla, CA 92093-0404 1072 EMail: sivasankar@cs.ucsd.edu 1074 Arvind Jain 1075 Google, Inc. 1076 1600 Amphitheatre Parkway 1077 Mountain View, CA 94043, USA 1078 EMail: arvind@google.com