idnits 2.17.1 draft-ietf-tcpm-fastopen-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The abstract seems to contain references ([RFC2119]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 142 has weird spacing: '...certain appli...' == Line 631 has weird spacing: '... by encrypt...' -- The document date (July 15, 2013) is 3938 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Experimental ---------------------------------------------------------------------------- == Missing Reference: 'RFC2119' is mentioned on line 66, but not defined == Missing Reference: 'RFC3390' is mentioned on line 516, but not defined == Missing Reference: 'RFC 1323' is mentioned on line 642, but not defined ** Obsolete undefined reference: RFC 1323 (Obsoleted by RFC 7323) == Missing Reference: 'BELSHE11' is mentioned on line 866, but not defined == Unused Reference: 'RFC6928' is defined on line 906, but no explicit reference was found in the text == Unused Reference: 'QWGMSS11' is defined on line 947, but no explicit reference was found in the text == Unused Reference: 'BELSHE12' is defined on line 978, but no explicit reference was found in the text ** Obsolete normative reference: RFC 793 (Obsoleted by RFC 9293) -- Obsolete informational reference (is this intentional?): RFC 1644 (Obsoleted by RFC 6247) -- Obsolete informational reference (is this intentional?): RFC 6013 (Obsoleted by RFC 7805) == Outdated reference: A later version (-10) exists of draft-ietf-tcpm-fastopen-01 Summary: 3 errors (**), 0 flaws (~~), 11 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Draft Y. Cheng 3 draft-ietf-tcpm-fastopen-04.txt J. Chu 4 Intended status: Experimental S. Radhakrishnan 5 Expiration date: February, 2014 A. Jain 6 Google, Inc. 7 July 15, 2013 9 TCP Fast Open 11 Status of this Memo 13 Distribution of this memo is unlimited. 15 This Internet-Draft is submitted in full conformance with the 16 provisions of BCP 78 and BCP 79. 18 Internet-Drafts are working documents of the Internet Engineering 19 Task Force (IETF), its areas, and its working groups. Note that other 20 groups may also distribute working documents as Internet-Drafts. 22 Internet-Drafts are draft documents valid for a maximum of six months 23 and may be updated, replaced, or obsoleted by other documents at any 24 time. It is inappropriate to use Internet-Drafts as reference 25 material or to cite them other than as "work in progress." 27 The list of current Internet-Drafts can be accessed at 28 http://www.ietf.org/1id-abstracts.html 30 The list of Internet-Draft Shadow Directories can be accessed at 31 http://www.ietf.org/shadow.html 33 Copyright Notice 35 Copyright (c) 2013 IETF Trust and the persons identified as the 36 document authors. All rights reserved. 38 This document is subject to BCP 78 and the IETF Trust's Legal 39 Provisions Relating to IETF Documents 40 (http://trustee.ietf.org/license-info) in effect on the date of 41 publication of this document. Please review these documents 42 carefully, as they describe your rights and restrictions with respect 43 to this document. Code Components extracted from this document must 44 include Simplified BSD License text as described in Section 4.e of 45 the Trust Legal Provisions and are provided without warranty as 46 described in the Simplified BSD License. 48 Abstract 49 TCP Fast Open (TFO) allows data to be carried in the SYN and SYN-ACK 50 packets and consumed by the receiving end during the initial 51 connection handshake, thus saving up to one full round trip time 52 (RTT) compared to the standard TCP, which requires a three-way 53 handshake (3WHS) to complete before data can be exchanged. However 54 TFO deviates from the standard TCP semantics in that the data in the 55 SYN could be replayed to an application in some rare circumstances. 56 Applications should not use TFO unless they can tolerate this issue, 57 which is detailed in the Applicability section. 59 Terminology 61 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 62 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 63 document are to be interpreted as described in RFC 2119 [RFC2119]. 64 TFO refers to TCP Fast Open. Client refers to the TCP's active open 65 side and server refers to the TCP's passive open side. 67 Table of Contents 69 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 70 2. Data In SYN . . . . . . . . . . . . . . . . . . . . . . . . . . 3 71 2.1 Relaxing TCP Semantics on Duplicated SYNs . . . . . . . . . 4 72 2.2. SYNs with Spoofed IP Addresses . . . . . . . . . . . . . . 4 73 3. Protocol Overview . . . . . . . . . . . . . . . . . . . . . . . 5 74 4. Protocol Details . . . . . . . . . . . . . . . . . . . . . . . 7 75 4.1. Fast Open Cookie . . . . . . . . . . . . . . . . . . . . . 7 76 4.1.1. TCP Options . . . . . . . . . . . . . . . . . . . . . . 7 77 4.1.2. Server Cookie Handling . . . . . . . . . . . . . . . . 8 78 4.1.3. Client Cookie Handling . . . . . . . . . . . . . . . . 9 79 4.2. Fast Open Protocol . . . . . . . . . . . . . . . . . . . . 9 80 4.2.1. Fast Open Cookie Request . . . . . . . . . . . . . . . 10 81 4.2.2. TCP Fast Open . . . . . . . . . . . . . . . . . . . . . 11 82 5. Security Considerations . . . . . . . . . . . . . . . . . . . . 13 83 5.1. Resource Exhaustion Attack by SYN Flood with Valid 84 Cookies . . . . . . . . . . . . . . . . . . . . . . . . . . 13 85 5.1.1 Attacks from behind Sharing Public IPs (NATs) . . . . . 14 86 5.2. Amplified Reflection Attack to Random Host . . . . . . . . 15 87 6. TFO's Applicability . . . . . . . . . . . . . . . . . . . . . . 16 88 6.1 Duplicate Data in SYNs . . . . . . . . . . . . . . . . . . . 16 89 6.2 Potential Performance Improvement . . . . . . . . . . . . . 16 90 6.3. Example: Web Clients and Servers . . . . . . . . . . . . . 16 91 6.3.1. HTTP Request Replay . . . . . . . . . . . . . . . . . . 16 92 6.3.2. Comparison with HTTP Persistent Connections . . . . . . 17 93 7. Open Areas for Experimentation . . . . . . . . . . . . . . . . 17 94 7.1. Performance impact due to middle-boxes and NAT . . . . . . 18 95 7.2. Cookie-less Fast Open . . . . . . . . . . . . . . . . . . . 18 96 8. Related Work . . . . . . . . . . . . . . . . . . . . . . . . . 18 97 8.1. T/TCP . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 98 8.2. Common Defenses Against SYN Flood Attacks . . . . . . . . . 19 99 8.3. TCP Cookie Transaction (TCPCT) . . . . . . . . . . . . . . 19 100 8.4. Speculative Connections by the Applications . . . . . . . . 19 101 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 19 102 10. Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . 20 103 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 20 104 11.1. Normative References . . . . . . . . . . . . . . . . . . . 20 105 11.2. Informative References . . . . . . . . . . . . . . . . . . 20 106 Appendix A. Example Socket API Changes to support TFO . . . . . . 22 107 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 22 109 1. Introduction 111 TCP Fast Open (TFO) enables data to be exchanged safely during TCP's 112 connection handshake. This document describes a design that enables 113 applications to save a round trip while avoiding severe security 114 ramifications. At the core of TFO is a security cookie used by the 115 server side to authenticate a client initiating a TFO connection. 116 This document covers the details of exchanging data during TCP's 117 initial handshake, the protocol for TFO cookies, potential new 118 security vulnerabilities and their mitigation, and the new socket 119 API. 121 TFO is motivated by the performance needs of today's Web 122 applications. Current TCP only permits data exchange after 3WHS 123 [RFC793], which adds one RTT to network latency. For short Web 124 transfers this additional RTT is a significant portion of overall 125 network latency [THK98], even when HTTP persistent connection is 126 widely used. For example, the Chrome browser keeps TCP connections 127 idle for up to 5 minutes but 35% of Chrome HTTP requests are made on 128 new TCP connections [RCCJR11]. For such Web and Web-like applications 129 placing data in the SYN can yield significant latency improvements. 130 Next we describe how we resolve the challenges that arise upon doing 131 so. 133 2. Data In SYN 135 Standard TCP already allows data to be carried in SYN packets 136 ([RFC793], section 3.4) but forbids the receiver from delivering it 137 to the application until 3WHS is completed. This is because TCP's 138 initial handshake serves to capture old or duplicate SYNs. 140 Allowing data in SYN packets to be delivered raises two issues 141 discussed in the following subsections. These issues make TFO 142 unsuitable for certain applications. Therefore TCP implementations 143 MUST NOT use TFO by default, but only use TFO if requested explicitly 144 by the application on a per service port basis. Applications need to 145 evaluate TFO applicability described in Section 6 before using TFO. 147 2.1 Relaxing TCP Semantics on Duplicated SYNs 149 TFO allows data to be delivered to the application before 3WHS is 150 completed, thus opening itself to a data integrity issue in either of 151 the two cases below: 153 a) the receiver host receives data in a duplicate SYN after it has 154 forgotten it received the original SYN (e.g. due to a reboot); 156 b) the duplicate is received after the connection created by the 157 original SYN has been closed and the close was initiated by the 158 sender (so the receiver will not be protected by the 2MSL TIMEWAIT 159 state). 161 The now obsoleted T/TCP [RFC1644] attempted to address these issues. 162 It is not successful and not deployed due to various vulnerabilities 163 [PHRACK98]. Rather than trying to capture all dubious SYN packets to 164 make TFO 100% compatible with TCP semantics, we made a design 165 decision early on to accept old SYN packets with data, i.e., to 166 restrict TFO use to a class of applications (Section 6) that are 167 tolerant of duplicate SYN packets with data. We believe this is the 168 right design trade-off balancing complexity with usefulness. 170 2.2. SYNs with Spoofed IP Addresses 172 Standard TCP suffers from the SYN flood attack [RFC4987] because 173 bogus SYN packets, i.e., SYN packets with spoofed source IP addresses 174 can easily fill up a listener's small queue, causing a service port 175 to be blocked completely until timeouts. Secondary damage comes from 176 these SYN requests taking up memory space. Though this is less of an 177 issue today as servers typically have plenty of memory. 179 TFO goes one step further to allow server-side TCP to send up data to 180 the application layer before 3WHS is completed. This opens up serious 181 new vulnerabilities. Applications serving ports that have TFO enabled 182 may waste lots of CPU and memory resources processing the requests 183 and producing the responses. If the response is much larger than the 184 request, the attacker can further mount an amplified reflection 185 attack against victims of choice beyond the TFO server itself. 187 Numerous mitigation techniques against regular SYN flood attacks 188 exist and have been well documented [RFC4987]. Unfortunately none are 189 applicable to TFO. We propose a server-supplied cookie to mitigate 190 these new vulnerabilities in Section 3 and evaluate the effectiveness 191 of the defense in Section 7. 193 3. Protocol Overview 195 The key component of TFO is the Fast Open Cookie (cookie), a message 196 authentication code (MAC) tag generated by the server. The client 197 requests a cookie in one regular TCP connection, then uses it for 198 future TCP connections to exchange data during 3WHS: 200 Requesting a Fast Open Cookie: 202 1. The client sends a SYN with a Fast Open Cookie Request option. 204 2. The server generates a cookie and sends it through the Fast Open 205 Cookie option of a SYN-ACK packet. 207 3. The client caches the cookie for future TCP Fast Open connections 208 (see below). 210 Performing TCP Fast Open: 212 1. The client sends a SYN with Fast Open Cookie option and data. 214 2. The server validates the cookie: 215 a. If the cookie is valid, the server sends a SYN-ACK 216 acknowledging both the SYN and the data. The server then 217 delivers the data to the application. 219 b. Otherwise, the server drops the data and sends a SYN-ACK 220 acknowledging only the SYN sequence number. 222 3. If the server accepts the data in the SYN packet, it may send the 223 response data before the handshake finishes. The max amount is 224 governed by the TCP's congestion control [RFC5681]. 226 4. The client sends an ACK acknowledging the SYN and the server data. 227 If the client's data is not acknowledged, the client retransmits 228 the data in the ACK packet. 230 5. The rest of the connection proceeds like a normal TCP connection. 231 The client can repeat many Fast Open operations once it acquires a 232 cookie (until the cookie is expired by the server). Thus TFO is 233 useful for applications that have temporal locality on client and 234 server connections. 236 Requesting Fast Open Cookie in connection 1: 238 TCP A (Client) TCP B(Server) 239 ______________ _____________ 240 CLOSED LISTEN 242 #1 SYN-SENT ----- ----------> SYN-RCVD 244 #2 ESTABLISHED <---- ---------- SYN-RCVD 245 (caches cookie C) 247 Performing TCP Fast Open in connection 2: 249 TCP A (Client) TCP B(Server) 250 ______________ _____________ 251 CLOSED LISTEN 253 #1 SYN-SENT ----- ----> SYN-RCVD 255 #2 ESTABLISHED <---- ---- SYN-RCVD 257 #3 ESTABLISHED <---- ---- SYN-RCVD 259 #4 ESTABLISHED ----- --------------------> ESTABLISHED 261 #5 ESTABLISHED --- ----------> ESTABLISHED 263 4. Protocol Details 265 4.1. Fast Open Cookie 267 The Fast Open Cookie is designed to mitigate new security 268 vulnerabilities in order to enable data exchange during handshake. 269 The cookie is a message authentication code tag generated by the 270 server and is opaque to the client; the client simply caches the 271 cookie and passes it back on subsequent SYN packets to open new 272 connections. The server can expire the cookie at any time to enhance 273 security. 275 4.1.1. TCP Options 277 Fast Open Cookie Option 279 The server uses this option to grant a cookie to the client in the 280 SYN-ACK packet; the client uses it to pass the cookie back to the 281 server in subsequent SYN packets. 283 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 284 | Kind | Length | 285 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 286 | | 287 ~ Cookie ~ 288 | | 289 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 291 Kind 1 byte: constant TBD (assigned by IANA) 292 Length 1 byte: range 6 to 18 (bytes); limited by 293 remaining space in the options field. 294 The number MUST be even. 295 Cookie 4 to 16 bytes (Length - 2) 296 Options with invalid Length values or without SYN flag set MUST be 297 ignored. The minimum Cookie size is 4 bytes. Although the diagram 298 shows a cookie aligned on 32-bit boundaries, alignment is not 299 required. 301 Fast Open Cookie Request Option 303 The client uses this option in the SYN packet to request a cookie 304 from a TFO-enabled server 305 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 306 | Kind | Length | 307 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 309 Kind 1 byte: same as the Fast Open Cookie option 310 Length 1 byte: constant 2. This distinguishes the option 311 from the Fast Open cookie option. 312 Options with invalid Length values, without SYN flag set, or with ACK 313 flag set MUST be ignored. 315 4.1.2. Server Cookie Handling 317 The server is in charge of cookie generation and authentication. The 318 cookie SHOULD be a message authentication code tag with the following 319 properties: 320 1. The cookie authenticates the client's (source) IP address of the 321 SYN packet. The IP address may be an IPv4 or IPv6 address. 323 2. The cookie can only be generated by the server and can not be 324 fabricated by any other parties including the client. 326 3. The generation and verification are fast relative to the rest of 327 SYN and SYN-ACK processing. 329 4. A server may encode other information in the cookie, and accept 330 more than one valid cookie per client at any given time. But this 331 is all server implementation dependent and transparent to the 332 client. 334 5. The cookie expires after a certain amount of time. The reason for 335 cookie expiration is detailed in the "Security Consideration" 336 section. This can be done by either periodically changing the 337 server key used to generate cookies or including a timestamp when 338 generating the cookie. 340 To gradually invalidate cookies over time, the server can 341 implement key rotation to generate and verify cookies using 342 multiple keys. This approach is useful for large-scale servers to 343 retain Fast Open rolling key updates. We do not specify a 344 particular mechanism because the implementation is server 345 specific. 347 The server supports the cookie generation and verification 348 operations: 350 - GetCookie(IP_Address): returns a (new) cookie 352 - IsCookieValid(IP_Address, Cookie): checks if the cookie is valid, 353 i.e., it has not expired and it authenticates the client IP address. 355 Example Implementation: a simple implementation is to use AES_128 to 356 encrypt the IPv4 (with padding) or IPv6 address and truncate to 64 357 bits. The server can periodically update the key to expire the 358 cookies. AES encryption on recent processors is fast and takes only a 359 few hundred nanoseconds [RCCJR11]. 361 If only one valid cookie is allowed per-IP and the server can 362 regenerate the cookie independently, the best validation process is 363 to simply regenerate a valid cookie and compare it against the 364 incoming cookie. In that case if the incoming cookie fails the check, 365 a valid cookie is readily available to be sent to the client. 367 4.1.3. Client Cookie Handling 369 The client MUST cache cookies from servers for later Fast Open 370 connections. For a multi-homed client, the cookies are both client 371 and server IP dependent. Beside the cookie, we RECOMMEND that the 372 client caches the MSS and RTT to the server to enhance performance. 374 The MSS advertised by the server is stored in the cache to determine 375 the maximum amount of data that can be supported in the SYN packet. 376 This information is needed because data is sent before the server 377 announces its MSS in the SYN-ACK packet. Without this information, 378 the data size in the SYN packet is limited to the default MSS of 536 379 bytes [RFC1122]. The client SHOULD update the cache MSS value 380 whenever it discovers new MSS value, e.g., through path MTU 381 discovery. 383 Caching RTT allows seeding a more accurate SYN timeout than the 384 default value [RFC6298]. This lowers the performance penalty if the 385 network or the server drops the SYN packets with data or the cookie 386 options. 388 The cache replacement algorithm is not specified and is left to the 389 implementations. 391 Note that before TFO sees wide deployment, clients SHOULD cache 392 negative responses from servers in order to reduce the amount of 393 futile TFO attempts. Since TFO is enabled on a per-service port basis 394 but cookies are independent of service ports, clients' cache should 395 include remote port numbers too. 397 4.2. Fast Open Protocol 399 One predominant requirement of TFO is to be fully compatible with 400 existing TCP implementations, both on the client and the server 401 sides. 403 The server keeps two variables per listening port: 405 FastOpenEnabled: default is off. It MUST be turned on explicitly by 406 the application. When this flag is off, the server does not perform 407 any TFO related operations and MUST ignore all cookie options. 409 PendingFastOpenRequests: tracks number of TFO connections in SYN-RCVD 410 state. If this variable goes over a preset system limit, the server 411 SHOULD disable TFO for all new connection requests until 412 PendingFastOpenRequests drops below the system limit. This variable 413 is used for defending some vulnerabilities discussed in the "Security 414 Considerations" section. 416 The server keeps a FastOpened flag per TCB to mark if a connection 417 has successfully performed a TFO. 419 4.2.1. Fast Open Cookie Request 421 Any client attempting TFO MUST first request a cookie from the server 422 with the following steps: 424 1. The client sends a SYN packet with a Fast Open Cookie Request 425 option. 427 2. The server SHOULD respond with a SYN-ACK based on the procedures 428 in the "Server Cookie Handling" section. This SYN-ACK SHOULD 429 contain a Fast Open Cookie option if the server currently supports 430 TFO for this listener port. 432 3. If the SYN-ACK contains a Fast Open Cookie option, the client 433 replaces the cookie and other information as described in the 434 "Client Cookie Handling" section. Otherwise, if the SYN-ACK is 435 first seen, i.e.,not a (spurious) retransmission, the client MAY 436 remove the server information from the cookie cache. If the SYN- 437 ACK is a spurious retransmission without valid Fast Open Cookie 438 Option, the client does nothing to the cookie cache for the 439 reasons below. 441 The network or servers may drop the SYN or SYN-ACK packets with the 442 new cookie options, which will causes SYN or SYN-ACK timeouts. We 443 RECOMMEND both the client and the server to retransmit SYN and SYN- 444 ACK without the cookie options on timeouts. This ensures the 445 connections of cookie requests will go through and lowers the latency 446 penalty (of dropped SYN/SYN-ACK packets). The obvious downside for 447 maximum compatibility is that any regular SYN drop will fail the 448 cookie (although one can argue the delay in the data transmission 449 till after 3WHS is justified if the SYN drop is due to network 450 congestion). Next section describes a heuristic to detect such drops 451 when the client receives the SYN-ACK. 453 We also RECOMMEND the client to record servers that failed to respond 454 to cookie requests and only attempt another cookie request after 455 certain period. An alternate proposal is to request cookie in FIN 456 instead since FIN-drop by incompatible middle-box does not affect 457 latency. However such paths are likely to drop SYN packet with data 458 later, and many applications close the connections with RST instead, 459 so the actual benefit of this approach is not clear. 461 4.2.2. TCP Fast Open 463 Once the client obtains the cookie from the target server, it can 464 perform subsequent TFO connections until the cookie is expired by the 465 server. 467 Client: Sending SYN 469 To open a TFO connection, the client MUST have obtained a cookie from 470 the server: 472 1. Send a SYN packet. 474 a. If the SYN packet does not have enough option space for the 475 Fast Open Cookie option, abort TFO and fall back to regular 3WHS. 477 b. Otherwise, include the Fast Open Cookie option with the cookie 478 of the server. Include any data up to the cached server MSS or 479 default 536 bytes. 481 2. Advance to SYN-SENT state and update SND.NXT to include the data 482 accordingly. 484 3. If RTT is available from the cache, seed SYN timer according to 485 [RFC6298]. 487 To deal with network or servers dropping SYN packets with payload or 488 unknown options, when the SYN timer fires, the client SHOULD 489 retransmit a SYN packet without data and Fast Open Cookie options. 491 Server: Receiving SYN and responding with SYN-ACK 493 Upon receiving the SYN packet with Fast Open Cookie option: 495 1. Initialize and reset a local FastOpened flag. If FastOpenEnabled 496 is false, go to step 5. 498 2. If PendingFastOpenRequests is over the system limit, go to step 5. 500 3. If IsCookieValid() in section 4.1.2 returns false, go to step 5. 502 4. Buffer the data and notify the application. Set FastOpened flag 503 and increment PendingFastOpenRequests. 505 5. Send the SYN-ACK packet. The packet MAY include a Fast Open 506 Option. If FastOpened flag is set, the packet acknowledges the SYN 507 and data sequence. Otherwise it acknowledges only the SYN 508 sequence. The server MAY include data in the SYN-ACK packet if the 509 response data is readily available. Some application may favor 510 delaying the SYN-ACK, allowing the application to process the 511 request in order to produce a response, but this is left up to the 512 implementation. 514 6. Advance to the SYN-RCVD state. If the FastOpened flag is set, the 515 server MUST follow the congestion control [RFC5681], in particular 516 the initial congestion window [RFC3390], to send more data 517 packets. 519 Note that if SYN-ACK is lost, regular TCP reduces the initial 520 congestion window before sending any data. In this case TFO is 521 slightly more aggressive in the first data round trip even though 522 it does not change the congestion control. 524 If the SYN-ACK timer fires, the server SHOULD retransmit a SYN-ACK 525 segment with neither data nor Fast Open Cookie options for 526 compatibility reasons. 528 A special case is simultaneous open where the SYN receiver is a 529 client in SYN-SENT state. The protocol remains the same because 530 [RFC793] already supports both data in SYN and simultaneous open. But 531 the client's socket may have data available to read before it's 532 connected. This document does not cover the corresponding API change. 534 Client: Receiving SYN-ACK 536 The client SHOULD perform the following steps upon receiving the SYN- 537 ACK: 539 1. Update the cookie cache if the SYN-ACK has a Fast Open Cookie 540 Option or MSS option or both. 542 2. Send an ACK packet. Set acknowledgment number to RCV.NXT and 543 include the data after SND.UNA if data is available. 545 3. Advance to the ESTABLISHED state. 547 Note there is no latency penalty if the server does not acknowledge 548 the data in the original SYN packet. The client SHOULD retransmit any 549 unacknowledged data in the first ACK packet in step 2. The data 550 exchange will start after the handshake like a regular TCP 551 connection. 553 If the client has timed out and retransmitted only regular SYN 554 packets, it can heuristically detect paths that intentionally drop 555 SYN with Fast Open option or data. If the SYN-ACK acknowledges only 556 the initial sequence and does not carry a Fast Open cookie option, 557 presumably it is triggered by a retransmitted (regular) SYN and the 558 original SYN or the corresponding SYN-ACK was lost. 560 Server: Receiving ACK 562 Upon receiving an ACK acknowledging the SYN sequence, the server 563 decrements PendingFastOpenRequests and advances to the ESTABLISHED 564 state. No special handling is required further. 566 5. Security Considerations 568 The Fast Open cookie stops an attacker from trivially flooding 569 spoofed SYN packets with data to burn server resources or to mount an 570 amplified reflection attack on random hosts. The server can defend 571 against spoofed SYN floods with invalid cookies using existing 572 techniques [RFC4987]. We note that although generating bogus cookies 573 is cost-free, the cost of validating the cookies, inherent to any 574 authentication scheme, may not be substantial compared to processing 575 a regular SYN packet. 577 5.1. Resource Exhaustion Attack by SYN Flood with Valid Cookies 579 However, the attacker may still obtain cookies from some compromised 580 hosts, then flood spoofed SYN with data and "valid" cookies (from 581 these hosts or other vantage points). With DHCP, it's possible to 582 obtain cookies of past IP addresses without compromising any host. 583 Below we identify new vulnerabilities of TFO and describe the 584 countermeasures. 586 Like regular TCP handshakes, TFO is vulnerable to such an attack. But 587 the potential damage can be much more severe. Besides causing 588 temporary disruption to service ports under attack, it may exhaust 589 server CPU and memory resources. Such an attack will show up on 590 application server logs as a application level DoS from Bot-nets, 591 triggering other defenses and alerts. 593 For this reason it is crucial for the TFO server to limit the maximum 594 number of total pending TFO connection requests, i.e., 595 PendingFastOpenRequests. When the limit is exceeded, the server 596 temporarily disables TFO entirely as described in "Server Cookie 597 Handling". Then subsequent TFO requests will be downgraded to regular 598 connection requests, i.e., with the data dropped and only SYN 599 acknowledged. This allows regular SYN flood defense techniques 600 [RFC4987] like SYN-cookies to kick in and prevent further service 601 disruption. 603 The main impact of SYN floods against the standard TCP stack is not 604 directly from the floods themselves costing TCP processing overhead 605 or host memory, but rather from the spoofed SYN packets filling up 606 the often small listener's queue. 608 On the other hand, TFO SYN floods can cause damage directly if 609 admitted without limit into the stack. The RST packets from the 610 spoofed host will fuel rather than defeat the SYN floods as compared 611 to the non-TFO case, because the attacker can flood more SYNs with 612 data to cost more data processing resources. For this reason, a TFO 613 server needs to monitor the connections in SYN-RCVD being reset in 614 addition to imposing a reasonable max queue length. Implementations 615 may combine the two, e.g., by continuing to account for those 616 connection requests that have just been reset against the listener's 617 PendingFastOpenRequests until a timeout period has passed. 619 Limiting the maximum number of pending TFO connection requests does 620 make it easy for an attacker to overflow the queue, causing TFO to be 621 disabled. We argue that causing TFO to be disabled is unlikely to be 622 of interest to attackers because the service will remain intact 623 without TFO hence there is hardly any real damage. 625 5.1.1 Attacks from behind Sharing Public IPs (NATs) 627 An attacker behind NAT can easily obtain valid cookies to launch the 628 above attack to hurt other clients that share the path. [BRISCOE12] 629 suggested that the server can extend cookie generation to include the 630 TCP timestamp---GetCookie(IP_Address, Timestamp)---and implement it 631 by encrypting the concatenation of the two values to generate the 632 cookie. The client stores both the cookie and its corresponding 633 timestamp, and echoes both in the SYN. The server then implements 634 IsCookieValid(IP_Address, Timestamp, Cookie) by encrypting the IP and 635 timestamp data and comparing it with the cookie value. 637 This enables the server to issue different cookies to clients that 638 share the same IP address, hence can selectively discard those 639 misused cookies from the attacker. However the attacker can simply 640 repeat the attack with new cookies. The server would eventually need 641 to throttle all requests from the IP address just like the current 642 approach. Moreover this approach requires modifying [RFC 1323] to 643 send non-zero Timestamp Echo Reply in SYN, potentially cause firewall 644 issues. Therefore we believe the benefit does not outweigh the 645 drawbacks. 647 5.2. Amplified Reflection Attack to Random Host 649 Limiting PendingFastOpenRequests with a system limit can be done 650 without Fast Open Cookies and would protect the server from resource 651 exhaustion. It would also limit how much damage an attacker can cause 652 through an amplified reflection attack from that server. However, it 653 would still be vulnerable to an amplified reflection attack from a 654 large number of servers. An attacker can easily cause damage by 655 tricking many servers to respond with data packets at once to any 656 spoofed victim IP address of choice. 658 With the use of Fast Open Cookies, the attacker would first have to 659 steal a valid cookie from its target victim. This likely requires the 660 attacker to compromise the victim host or network first. 662 The attacker here has little interest in mounting an attack on the 663 victim host that has already been compromised. But it may be 664 motivated to disrupt the victim's network. Since a stolen cookie is 665 only valid for a single server, it has to steal valid cookies from a 666 large number of servers and use them before they expire to cause 667 sufficient damage without triggering the defense. 669 One can argue that if the attacker has compromised the target network 670 or hosts, it could perform a similar but simpler attack by injecting 671 bits directly. The degree of damage will be identical, but TFO- 672 specific attack allows the attacker to remain anonymous and disguises 673 the attack as from other servers. 675 The best defense is for the server not to respond with data until 676 handshake finishes. In this case the risk of amplification reflection 677 attack is completely eliminated. But the potential latency saving 678 from TFO may diminish if the server application produces responses 679 earlier before the handshake completes. 681 6. TFO's Applicability 683 This section is to help applications considering TFO to evaluate 684 TFO's benefits and drawbacks using the Web client and server 685 applications as an example throughout. A proposed socket API change 686 is in the Appendix. 688 6.1 Duplicate Data in SYNs 690 It is possible, though uncommon, that using TFO results in the first 691 data written to a socket to be delivered more than once to the 692 application on the remote host(Section 2.1). This replay potential 693 only applies to data in the SYN but not subsequent data exchanges. 694 The client MUST NOT use TFO to send data in the SYN, and the server 695 MUST NOT accept data in the SYN if it cannot handle receiving the 696 same SYN data more than once, due to reasons described before. 698 6.2 Potential Performance Improvement 700 TFO is designed for latency-conscious applications that are sensitive 701 to TCP's initial connection setup delay. To benefit from TFO, the 702 first application data unit (e.g., an HTTP request) needs to be no 703 more than TCP's maximum segment size (minus options used in SYN). 704 Otherwise the remote server can only process the client's application 705 data unit once the rest of it is delivered after the initial 706 handshake, diminishing TFO's benefit. 708 To the extent possible, applications SHOULD reuse the connection to 709 take advantage of TCP's built-in congestion control and reduce 710 connection setup overhead. An application employs too many short- 711 lived connections will negatively impact network stability, as these 712 connections often exit before TCP's congestion control algorithm 713 takes effect. 715 6.3. Example: Web Clients and Servers 717 6.3.1. HTTP Request Replay 719 While TFO is motivated by Web applications, the browser should not 720 use TFO to send requests in SYNs if those requests cannot tolerate 721 replays. One example is POST requests without application-layer 722 transaction protection (e.g., a unique identifier in the request 723 header). 725 TFO is particularly useful for GET requests. Even though not all GET 726 requests are idempotent, GETs are frequently replayed today across 727 striped TCP connections. After a server receives an HTTP request but 728 before the ACKs of the requests reach the browser, the browser may 729 timeout and retry the same request on another (possibly new) TCP 730 connection. This differs from a TFO replay only in that the replay is 731 initiated by the browser, not by the TCP stack. 733 Finally, TFO is safe and useful for HTTPS requests because it saves 734 the first SSL handshake RTT and the HTTP request is sent after the 735 connection establishes. 737 6.3.2. Comparison with HTTP Persistent Connections 739 Is TFO useful given the wide deployment of HTTP persistent 740 connections? The short answer is yes. Studies [RCCJR11][AERG11] show 741 that the average number of transactions per connection is between 2 742 and 4, based on large-scale measurements from both servers and 743 clients. In these studies, the servers and clients both kept idle 744 connections up to several minutes, well into "human think" time. 746 Keeping connections open and idle even longer risks a greater 747 performance penalty. [HNESSK10][MQXMZ11] show that the majority of 748 home routers and ISPs fail to meet the the 124-minute idle timeout 749 mandated in [RFC5382]. In [MQXMZ11], 35% of mobile ISPs silently 750 timeout idle connections within 30 minutes. End hosts, unaware of 751 silent middle-box timeouts, suffer multi-minute TCP timeouts upon 752 using those long-idle connections. 754 To circumvent this problem, some applications send frequent TCP keep- 755 alive probes. However, this technique drains power on mobile devices 756 [MQXMZ11]. In fact, power has become such a prominent issue in modern 757 LTE devices that mobile browsers close HTTP connections within 758 seconds or even immediately [SOUDERS11]. 760 [RCCJR11] studied Chrome browser performance based on 28 days of 761 global statistics. The Chrome browser keeps idle HTTP persistent 762 connections for 5 to 10 minutes. However the average number of the 763 transactions per connection is only 3.3 and TCP 3WHS accounts for up 764 to 25% of the HTTP transaction network latency. The authors tested a 765 Linux TFO implementation with TFO enabled Chrome browser on popular 766 web sites in emulated environments such as residential broadband and 767 mobile networks. They showed that TFO improves page load time by 10% 768 to 40%. 770 7. Open Areas for Experimentation 772 We now outline some areas that need experimentation in the Internet 773 and under different network scenarios. These experiments should help 774 the community evaluate Fast Open benefits and risks towards further 775 standardization and implementation of Fast Open and its related 776 protocols. 778 7.1. Performance impact due to middle-boxes and NAT 780 [MAF04] found that some middle-boxes and end-hosts may drop packets 781 with unknown TCP options. Studies [LANGLEY06, HNRGHT11] both found 782 that 6% of the probed paths on the Internet drop SYN packets with 783 data or with unknown TCP options. The TFO protocol deals with this 784 problem by falling back to regular TCP handshake and re-transmitting 785 SYN without data or cookie options after the initial SYN timeout. 786 Moreover the implementation is recommended to negatively cache such 787 incidents to avoid recurring timeouts. Further study is required to 788 evaluate the performance impact of these malicious drop behaviors. 790 Another interesting study is the (loss of) TFO performance benefit 791 behind certain carrier-grade NAT. Typically hosts behind a NAT 792 sharing the same IP address will get the same cookie for the same 793 server. This will not prevent TFO from working. But on some carrier- 794 grade NAT configurations where every new TCP connection from the same 795 physical host uses a different public IP address, TFO does not 796 provide latency benefits. However, there is no performance penalty 797 either, as described in Section "Client: Receiving SYN-ACK". 799 7.2. Cookie-less Fast Open 801 The cookie mechanism mitigates resource exhaustion and amplification 802 attacks. However cookies are not necessary if the server has 803 application-level protection or is immune to these attacks. For 804 example a Web server that only replies with a simple HTTP redirect 805 response that fits in the SYN-ACK packet may not care about resource 806 exhaustion. Such an application can safely disable TFO cookie checks. 808 Disabling cookies simplifies both the client and the server, as the 809 client no longer needs to request a cookie and the server no longer 810 needs to check or generate cookies. Disabling cookies also 811 potentially simplifies configuration, as the server no longer needs a 812 key. It may be preferable to enable SYN cookies and disable TFO 813 [RFC4987] when a server is overloaded by a large-scale Bot-net 814 attack. 816 Careful experimentation is necessary to evaluate if cookie-less TFO 817 is practical. The implementation can provide an experimental feature 818 to allow zero length, or null, cookies as opposed to the minimum 4 819 bytes cookies. Thus the server may return a null cookie and the 820 client will send data in SYN with it subsequently. If the server 821 believes it's under a DoS attack through other defense mechanisms, it 822 can switch to regular Fast Open for listener sockets. 824 8. Related Work 825 8.1. T/TCP 827 TCP Extensions for Transactions [RFC1644] attempted to bypass the 828 three-way handshake, among other things, hence shared the same goal 829 but also the same set of issues as TFO. It focused most of its effort 830 battling old or duplicate SYNs, but paid no attention to security 831 vulnerabilities it introduced when bypassing 3WHS [PHRACK98]. 833 As stated earlier, we take a practical approach to focus TFO on the 834 security aspect, while allowing old, duplicate SYN packets with data 835 after recognizing that 100% TCP semantics is likely infeasible. We 836 believe this approach strikes the right tradeoff, and makes TFO much 837 simpler and more appealing to TCP implementers and users. 839 8.2. Common Defenses Against SYN Flood Attacks 841 [RFC4987] studies on mitigating attacks from regular SYN flood, i.e., 842 SYN without data . But from the stateless SYN-cookies to the stateful 843 SYN Cache, none can preserve data sent with SYN safely while still 844 providing an effective defense. 846 The best defense may be to simply disable TFO when a host is 847 suspected to be under a SYN flood attack, e.g., the SYN backlog is 848 filled. Once TFO is disabled, normal SYN flood defenses can be 849 applied. The "Security Consideration" section contains a thorough 850 discussion on this topic. 852 8.3. TCP Cookie Transaction (TCPCT) 854 TCPCT [RFC6013] eliminates server state during initial handshake and 855 defends spoofing DoS attacks. Like TFO, TCPCT allows SYN and SYN-ACK 856 packets to carry data. But the server can only send up to MSS bytes 857 of data during the handshake instead of the initial congestion window 858 unlike TFO. Therefore applications like Web may not receive the 859 latency benefit as TFO. 861 8.4. Speculative Connections by the Applications 863 Some Web browsers maintain a history of the domains for frequently 864 visited web pages. The browsers then speculatively pre-open TCP 865 connections to these domains before the user initiates any requests 866 for them [BELSHE11]. The downside of this approach is that it wastes 867 server and network resources by initiating and maintaining idle 868 connections; It is also subject to the NAT timeout issues described 869 in Section 6.3.2. TFO offers similar performance improvement without 870 the added overhead. 872 9. IANA Considerations 873 The Fast Open Cookie Option and Fast Open Cookie Request Option 874 define no new namespace. The options require IANA to allocate one 875 value from the TCP option Kind namespace. Early implementation before 876 the IANA allocation SHOULD follow [EXPOPT] and use experimental 877 option 254 and magic number 0xF989 (16 bits), then migrate to the new 878 option after the allocation accordingly. 880 10. Acknowledgement 882 We thank Rick Jones, Bob Briscoe, Adam Langley, Matt Mathis, Neal 883 Cardwell, Roberto Peon, William Chan, Eric Dumazet, and Tom Herbert 884 for their feedbacks. We especially thank Barath Raghavan for his 885 contribution on the security design of Fast Open and proofreading 886 this draft numerous times. 888 11. References 890 11.1. Normative References 891 [RFC793] Postel, J. "Transmission Control Protocol", RFC 793, 892 September 1981. 894 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - 895 Communication Layers", STD 3, RFC 1122, October 1989. 897 [RFC5382] S. Guha, Ed., Biswas, K., Ford B., Sivakumar S., Srisuresh, 898 P., "NAT Behavioral Requirements for TCP", RFC 5382 900 [RFC5681] Allman, M., Paxson, V. and E. Blanton, "TCP Congestion 901 Control", RFC 5681, September 2009. 903 [RFC6298] Paxson, V., Allman, M., Chu, J. and M. Sargent, "Computing 904 TCP's Retransmission Timer", RFC 6298, June 2011. 906 [RFC6928] Chu, J., Dukkipati, N., Cheng, Y. and M. Mathis, 907 "Increasing TCP's Initial Window", RFC 6928, April 2013. 909 11.2. Informative References 911 [AERG11] M. Al-Fares, K. Elmeleegy, B. Reed, and I. Gashinsky, 912 "Overclocking the Yahoo! CDN for Faster Web Page Loads". In 913 Proceedings of Internet Measurement Conference, November 914 2011. 916 [EXPOPT] Touch, Joe, "Shared Use of Experimental TCP Options", 917 Internet-Draft draft-ietf-tcpm-experimental-options (work 918 in progress), June 2013. 920 [HNESSK10] S. Haetoenen, A. Nyrhinen, L. Eggert, S. Strowes, P. 922 Sarolahti, M. Kojo., "An Experimental Study of Home Gateway 923 Characteristics". In Proceedings of Internet Measurement 924 Conference. Octobor 2010 926 [HNRGHT11] M. Honda, Y. Nishida, C. Raiciu, A. Greenhalgh, M. 927 Handley, H. Tokuda, "Is it Still Possible to Extend TCP?". 928 In Proceedings of Internet Measurement Conference. November 929 2011. 931 [LANGLEY06] Langley, A, "Probing the viability of TCP extensions", 932 URL http://www.imperialviolet.org/binary/ecntest.pdf 934 [MAF04] Medina, A., Allman, M., and S. Floyd, "Measuring 935 Interactions Between Transport Protocols and Middleboxes", 936 In Proceedings of Internet Measurement Conference, October 937 2004. 939 [MQXMZ11] Z. Mao, Z. Qian, Q. Xu, Z. Mao, M. Zhang. "An Untold Story 940 of Middleboxes in Cellular Networks", In Proceedings of 941 SIGCOMM. August 2011. 943 [PHRACK98] "T/TCP vulnerabilities", Phrack Magazine, Volume 8, Issue 944 53 artical 6. July 8, 1998. URL 945 http://www.phrack.com/issues.html?issue=53&id=6 947 [QWGMSS11] F. Qian, Z. Wang, A. Gerber, Z. Mao, S. Sen, O. 948 Spatscheck. "Profiling Resource Usage for Mobile 949 Applications: A Cross-layer Approach", In Proceedings of 950 International Conference on Mobile Systems. April 2011. 952 [RCCJR11] Radhakrishnan, S., Cheng, Y., Chu, J., Jain, A. and 953 Raghavan, B., "TCP Fast Open". In Proceedings of 7th ACM 954 CoNEXT Conference, December 2011. 956 [RFC1644] Braden, R., "T/TCP -- TCP Extensions for Transactions 957 Functional Specification", RFC 1644, July 1994. 959 [RFC4987] Eddy, W., "TCP SYN Flooding Attacks and Common 960 Mitigations", RFC 4987, August 2007. 962 [RFC6013] Simpson, W., "TCP Cookie Transactions (TCPCT)", RFC6013, 963 January 2011. 965 [SOUDERS11] S. Souders. "Making A Mobile Connection". 966 http://www.stevesouders.com/blog/2011/09/21/making-a- 967 mobile-connection/ 969 [THK98] Touch, J., Heidemann, J., Obraczka, K., "Analysis of HTTP 970 Performance", USC/ISI Research Report 98-463. December 971 1998. 973 [BRISCOE12] Briscoe, B., "Some ideas building on draft-ietf-tcpm- 974 fastopen-01", tcpm list, 975 http://www.ietf.org/mail-archive/web/tcpm/current/ 976 January 16, 2014msg07192.html 978 [BELSHE12] Belshe, M., "The era of browser preconnect.", 979 http://www.belshe.com/2011/02/10/ 980 the-era-of-browser-preconnect/ 982 Appendix A. Example Socket API Changes to support TFO 984 The design rationale is to minimize changes to the socket API and 985 hence applications, in order to reduce the deployment hurdle. The 986 following changes have been implemented in Linux 3.7 or later 987 kernels. 989 A.1 MSG_FASTOPEN flag for sendto() or sendmsg() 991 MSG_FASTOPEN marks the attempt to send data in SYN like a combination 992 of connect() and sendto(), by performing an implicit connect() 993 operation. It blocks until the handshake has completed and the data 994 is buffered. 996 For non-blocking socket it returns the number of bytes buffered and 997 sent in the SYN packet. If the cookie is not available locally, it 998 returns -1 with errno EINPROGRESS, and sends a SYN with TFO cookie 999 request automatically. The caller needs to write the data again when 1000 the socket is connected. 1002 It returns the same errno as connect() if the handshake fails. 1004 A.2 TCP_FASTOPEN setsockopt() Socket Option 1006 The option enables Fast Open on the listener socket. The option value 1007 specifies the PendingFastOpenRequests threshold, i.e., the maximum 1008 length of pending SYNs with data payload. Once enabled, the TCP 1009 implementation will respond with TFO cookies per request. 1011 Previously accept() returns only after a socket is connected. But for 1012 a Fast Open connection, accept() returns upon receiving a SYN with a 1013 valid Fast Open cookie and data, and the data is available to be read 1014 through, e.g., recvmsg(), read(). 1016 Authors' Addresses 1017 Yuchung Cheng Google, Inc. 1600 Amphitheatre Parkway Mountain View, 1018 CA 94043, USA EMail: ycheng@google.com 1020 Jerry Chu Google, Inc. 1600 Amphitheatre Parkway Mountain View, CA 1021 94043, USA EMail: hkchu@google.com 1023 Sivasankar Radhakrishnan Department of Computer Science and 1024 Engineering University of California, San Diego 9500 Gilman Dr La 1025 Jolla, CA 92093-0404 EMail: sivasankar@cs.ucsd.edu 1027 Arvind Jain Google, Inc. 1600 Amphitheatre Parkway Mountain View, CA 1028 94043, USA EMail: arvind@google.com