idnits 2.17.1 draft-ietf-mptcp-experience-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 1185 has weird spacing: '...o Tokyo on th...' -- The document date (October 27, 2016) is 2709 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 6824 (Obsoleted by RFC 8684) == Outdated reference: A later version (-03) exists of draft-boucadair-mptcp-max-subflow-02 Summary: 1 error (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 MPTCP Working Group O. Bonaventure 3 Internet-Draft UCLouvain 4 Intended status: Informational C. Paasch 5 Expires: April 30, 2017 Apple, Inc. 6 G. Detal 7 Tessares 8 October 27, 2016 10 Use Cases and Operational Experience with Multipath TCP 11 draft-ietf-mptcp-experience-07 13 Abstract 15 This document discusses both use cases and operational experience 16 with Multipath TCP in real networks. It lists several prominent use 17 cases where Multipath TCP has been considered and is being used. It 18 also gives insight to some heuristics and decisions that have helped 19 to realize these use cases and suggests possible improvements. 21 Status of this Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at http://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on April 30, 2017. 38 Copyright Notice 40 Copyright (c) 2016 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents 45 (http://trustee.ietf.org/license-info) in effect on the date of 46 publication of this document. Please review these documents 47 carefully, as they describe your rights and restrictions with respect 48 to this document. Code Components extracted from this document must 49 include Simplified BSD License text as described in Section 4.e of 50 the Trust Legal Provisions and are provided without warranty as 51 described in the Simplified BSD License. 53 Table of Contents 55 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 56 2. Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . 5 57 2.1. Datacenters . . . . . . . . . . . . . . . . . . . . . . . 5 58 2.2. Cellular/WiFi Offload . . . . . . . . . . . . . . . . . . 5 59 2.3. Multipath TCP proxies . . . . . . . . . . . . . . . . . . 9 60 3. Operational Experience . . . . . . . . . . . . . . . . . . . . 11 61 3.1. Middlebox interference . . . . . . . . . . . . . . . . . . 11 62 3.2. Congestion control . . . . . . . . . . . . . . . . . . . . 13 63 3.3. Subflow management . . . . . . . . . . . . . . . . . . . . 13 64 3.4. Implemented subflow managers . . . . . . . . . . . . . . . 14 65 3.5. Subflow destination port . . . . . . . . . . . . . . . . . 16 66 3.6. Closing subflows . . . . . . . . . . . . . . . . . . . . . 17 67 3.7. Packet schedulers . . . . . . . . . . . . . . . . . . . . 18 68 3.8. Segment size selection . . . . . . . . . . . . . . . . . . 19 69 3.9. Interactions with the Domain Name System . . . . . . . . . 19 70 3.10. Captive portals . . . . . . . . . . . . . . . . . . . . . 20 71 3.11. Stateless webservers . . . . . . . . . . . . . . . . . . . 21 72 3.12. Loadbalanced server farms . . . . . . . . . . . . . . . . 22 73 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 74 5. Security Considerations . . . . . . . . . . . . . . . . . . . 24 75 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 26 76 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 27 77 7.1. Normative References . . . . . . . . . . . . . . . . . . . 27 78 7.2. Informative References . . . . . . . . . . . . . . . . . . 27 79 Appendix A. Changelog . . . . . . . . . . . . . . . . . . . . . . 34 80 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 36 82 1. Introduction 84 Multipath TCP was specified in [RFC6824] and five independent 85 implementations have been developed. As of November 2016, Multipath 86 TCP has been or is being implemented on the following platforms: 88 o Linux kernel [MultipathTCP-Linux] 90 o Apple iOS and macOS [Apple-MPTCP] 92 o Citrix load balancers 94 o FreeBSD [FreeBSD-MPTCP] 96 o Oracle Solaris 98 The first three implementations are known to interoperate. Three of 99 these implementations are open-source (Linux kernel, FreeBSD and 100 Apple's iOS and macOS). Apple's implementation is widely deployed. 102 Since the publication of [RFC6824] as an experimental RFC, experience 103 has been gathered by various network researchers and users about the 104 operational issues that arise when Multipath TCP is used in today's 105 Internet. 107 When the MPTCP working group was created, several use cases for 108 Multipath TCP were identified [RFC6182]. Since then, other use cases 109 have been proposed and some have been tested and even deployed. We 110 describe these use cases in Section 2. 112 Section 3 focuses on the operational experience with Multipath TCP. 113 Most of this experience comes from the utilization of the Multipath 114 TCP implementation in the Linux kernel [MultipathTCP-Linux]. This 115 open-source implementation has been downloaded and is used by 116 thousands of users all over the world. Many of these users have 117 provided direct or indirect feedback by writing documents (scientific 118 articles or blog messages) or posting to the mptcp-dev mailing list 119 (see https://listes-2.sipr.ucl.ac.be/sympa/arc/mptcp-dev ). This 120 Multipath TCP implementation is actively maintained and continuously 121 improved. It is used on various types of hosts, ranging from 122 smartphones or embedded routers to high-end servers. 124 The Multipath TCP implementation in the Linux kernel is not, by far, 125 the most widespread deployment of Multipath TCP. Since September 126 2013, Multipath TCP is also supported on smartphones and tablets 127 since iOS7 [IOS7]. There are likely hundreds of millions of 128 Multipath TCP enabled devices. This Multipath TCP implementation is 129 currently only used to support the Siri voice recognition/control 130 application. Some lessons learned from this deployment are described 131 in [IETFJ]. 133 Section 3 is organized as follows. Supporting the middleboxes was 134 one of the difficult issues in designing the Multipath TCP protocol. 135 We explain in Section 3.1 which types of middleboxes the Linux Kernel 136 implementation of Multipath TCP supports and how it reacts upon 137 encountering these. Section 3.2 summarizes the MPTCP specific 138 congestion controls that have been implemented. Section 3.3 to 139 Section 3.7 discuss heuristics and issues with respect to subflow 140 management as well as the scheduling across the subflows. 141 Section 3.8 explains some problems that occurred with subflows having 142 different Maximum Segment Size (MSS) values. Section 3.9 presents 143 issues with respect to content delivery networks and suggests a 144 solution to this issue. Finally, Section 3.10 documents an issue 145 with captive portals where MPTCP will behave sub optimally. 147 2. Use cases 149 Multipath TCP has been tested in several use cases. There is already 150 an abundant scientific literature on Multipath TCP [MPTCPBIB]. 151 Several of the papers published in the scientific literature have 152 identified possible improvements that are worth being discussed here. 154 2.1. Datacenters 156 A first, although initially unexpected, documented use case for 157 Multipath TCP has been in datacenters [HotNets][SIGCOMM11]. Today's 158 datacenters are designed to provide several paths between single- 159 homed servers. The multiplicity of these paths comes from the 160 utilization of Equal Cost Multipath (ECMP) and other load balancing 161 techniques inside the datacenter. Most of the deployed load 162 balancing techniques in datacenters rely on hashes computed over the 163 five tuple. Thus all packets from the same TCP connection follow the 164 same path and so are not reordered. The results in [HotNets] 165 demonstrate by simulations that Multipath TCP can achieve a better 166 utilization of the available network by using multiple subflows for 167 each Multipath TCP session. Although [RFC6182] assumes that at least 168 one of the communicating hosts has several IP addresses, [HotNets] 169 demonstrates that Multipath TCP is beneficial when both hosts are 170 single-homed. This idea is analyzed in more details in [SIGCOMM11] 171 where the Multipath TCP implementation in the Linux kernel is 172 modified to be able to use several subflows from the same IP address. 173 Measurements in a public datacenter show the quantitative benefits of 174 Multipath TCP [SIGCOMM11] in this environment. 176 Although ECMP is widely used inside datacenters, this is not the only 177 environment where there are different paths between a pair of hosts. 178 ECMP and other load balancing techniques such as Link Aggregation 179 Groups (LAG) are widely used in today's networks and having multiple 180 paths between a pair of single-homed hosts is becoming the norm 181 instead of the exception. Although these multiple paths have often 182 the same cost (from an IGP metrics viewpoint), they do not 183 necessarily have the same performance. For example, [IMC13c] reports 184 the results of a long measurement study showing that load balanced 185 Internet paths between that same pair of hosts can have huge delay 186 differences. 188 2.2. Cellular/WiFi Offload 190 A second use case that has been explored by several network 191 researchers is the cellular/WiFi offload use case. Smartphones or 192 other mobile devices equipped with two wireless interfaces are a very 193 common use case for Multipath TCP. In September 2015, this is also 194 the largest deployment of Multipath-TCP enabled devices [IOS7]. It 195 has been briefly discussed during IETF88 [ietf88], but there is no 196 published paper or report that analyses this deployment. For this 197 reason, we only discuss published papers that have mainly used the 198 Multipath TCP implementation in the Linux kernel for their 199 experiments. 201 The performance of Multipath TCP in wireless networks was briefly 202 evaluated in [NSDI12]. One experiment analyzes the performance of 203 Multipath TCP on a client with two wireless interfaces. This 204 evaluation shows that when the receive window is large, Multipath TCP 205 can efficiently use the two available links. However, if the window 206 becomes smaller, then packets sent on a slow path can block the 207 transmission of packets on a faster path. In some cases, the 208 performance of Multipath TCP over two paths can become lower than the 209 performance of regular TCP over the best performing path. Two 210 heuristics, reinjection and penalization, are proposed in [NSDI12] to 211 solve this identified performance problem. These two heuristics have 212 since been used in the Multipath TCP implementation in the Linux 213 kernel. [CONEXT13] explored the problem in more detail and revealed 214 some other scenarios where Multipath TCP can have difficulties in 215 efficiently pooling the available paths. Improvements to the 216 Multipath TCP implementation in the Linux kernel are proposed in 217 [CONEXT13] to cope with some of these problems. 219 The first experimental analysis of Multipath TCP in a public wireless 220 environment was presented in [Cellnet12]. These measurements explore 221 the ability of Multipath TCP to use two wireless networks (real WiFi 222 and 3G networks). Three modes of operation are compared. The first 223 mode of operation is the simultaneous use of the two wireless 224 networks. In this mode, Multipath TCP pools the available resources 225 and uses both wireless interfaces. This mode provides fast handover 226 from WiFi to cellular or the opposite when the user moves. 227 Measurements presented in [CACM14] show that the handover from one 228 wireless network to another is not an abrupt process. When a host 229 moves, there are regions where the quality of one of the wireless 230 networks is weaker than the other, but the host considers this 231 wireless network to still be up. When a mobile host enters such 232 regions, its ability to send packets over another wireless network is 233 important to ensure a smooth handover. This is clearly illustrated 234 from the packet trace discussed in [CACM14]. 236 Many cellular networks use volume-based pricing and users often 237 prefer to use unmetered WiFi networks when available instead of 238 metered cellular networks. [Cellnet12] implements support for the 239 MP_PRIO option to explore two other modes of operation. 241 In the backup mode, Multipath TCP opens a TCP subflow over each 242 interface, but the cellular interface is configured in backup mode. 244 This implies that data flows only over the WiFi interface when both 245 interfaces are considered to be active. If the WiFi interface fails, 246 then the traffic switches quickly to the cellular interface, ensuring 247 a smooth handover from the user's viewpoint [Cellnet12]. The cost of 248 this approach is that the WiFi and cellular interfaces are likely to 249 remain active all the time since all subflows are established over 250 the two interfaces. 252 The single-path mode is slightly different. This mode benefits from 253 the break-before-make capability of Multipath TCP. When an MPTCP 254 session is established, a subflow is created over the WiFi interface. 255 No packet is sent over the cellular interface as long as the WiFi 256 interface remains up [Cellnet12]. This implies that the cellular 257 interface can remain idle and battery capacity is preserved. When 258 the WiFi interface fails, a new subflow is established over the 259 cellular interface in order to preserve the established Multipath TCP 260 sessions. Compared to the backup mode described earlier, 261 measurements reported in [Cellnet12] indicate that this mode of 262 operation is characterized by a throughput drop while the cellular 263 interface is brought up and the subflows are reestablished. 265 From a protocol viewpoint, [Cellnet12] discusses the problem posed by 266 the unreliability of the REMOVE_ADDR option and proposes a small 267 protocol extension to allow hosts to reliably exchange this option. 268 It would be useful to analyze packet traces to understand whether the 269 unreliability of the REMOVE_ADDR option poses an operational problem 270 in real deployments. 272 Another study of the performance of Multipath TCP in wireless 273 networks was reported in [IMC13b]. This study uses laptops connected 274 to various cellular ISPs and WiFi hotspots. It compares various file 275 transfer scenarios. [IMC13b] observes that 4-path MPTCP outperforms 276 2-path MPTCP, especially for larger files. However, for three 277 congestion control algorithms (LIA, OLIA and Reno - see Section 3.2), 278 there is no significant performance difference for file sizes smaller 279 than 4MB. 281 A different study of the performance of Multipath TCP with two 282 wireless networks is presented in [INFOCOM14]. In this study the two 283 networks had different qualities : a good network and a lossy 284 network. When using two paths with different packet loss ratios, the 285 Multipath TCP congestion control scheme moves traffic away from the 286 lossy link that is considered to be congested. However, [INFOCOM14] 287 documents an interesting scenario that is summarized hereafter. 289 client ----------- path1 -------- server 290 | | 291 +--------------- path2 ------------+ 293 Figure 1: Simple network topology 295 Initially, the two paths in Figure 1 have the same quality and 296 Multipath TCP distributes the load over both of them. During the 297 transfer, the path2 becomes lossy, e.g. because the client moves. 298 Multipath TCP detects the packet losses and they are retransmitted 299 over path1. This enables the data transfer to continue over this 300 path. However, the subflow over path2 is still up and transmits one 301 packet from time to time. Although the N packets have been 302 acknowledged over the first subflow (at the MPTCP level), they have 303 not been acknowledged at the TCP level over the second subflow. To 304 preserve the continuity of the sequence numbers over the second 305 subflow, TCP will continue to retransmit these segments until either 306 they are acknowledged or the maximum number of retransmissions is 307 reached. This behavior is clearly inefficient and may lead to 308 blocking since the second subflow will consume window space to be 309 able to retransmit these packets. [INFOCOM14] proposes a new 310 Multipath TCP option to solve this problem. In practice, a new TCP 311 option is probably not required. When the client detects that the 312 data transmitted over the second subflow has been acknowledged over 313 the first subflow, it could decide to terminate the second subflow by 314 sending a RST segment. If the interface associated to this subflow 315 is still up, a new subflow could be immediately reestablished. It 316 would then be immediately usable to send new data and would not be 317 forced to first retransmit the previously transmitted data. As of 318 this writing, this dynamic management of the subflows is not yet 319 implemented in the Multipath TCP implementation in the Linux kernel. 321 Some studies have started to analyze the performance of Multipath TCP 322 on smartphones with real applications. In contrast with the bulk 323 transfers that are used by many publications, many deployed 324 applications do not exchange huge amounts of data and mainly use 325 small connections. [COMMAG2016] proposes a software testing 326 framework that allows to automate Android applications to study their 327 interactions with Multipath TCP. [PAM2016] analyses a one-month 328 packet trace of all the packets exchanged by a dozen of smartphones 329 used by regular users. This analysis reveals that short connections 330 are important on smartphones and that the main benefit of using 331 Multipath TCP on smartphones is the ability to perform seamless 332 handovers between different wireless networks. Long connections 333 benefit from these handovers. 335 2.3. Multipath TCP proxies 337 As Multipath TCP is not yet widely deployed on both clients and 338 servers, several deployments have used various forms of proxies. Two 339 families of solutions are currently being used or tested. 341 A first use case is when a Multipath TCP enabled client wants to use 342 several interfaces to reach a regular TCP server. A typical use case 343 is a smartphone that needs to use both its WiFi and its cellular 344 interface to transfer data. Several types of proxies are possible 345 for this use case. An HTTP proxy deployed on a Multipath TCP capable 346 server would enable the smartphone to use Multipath TCP to access 347 regular web servers. Obviously, this solution only works for 348 applications that rely on HTTP. Another possibility is to use a 349 proxy that can convert any Multipath TCP connection into a regular 350 TCP connection. Multipath TCP-specific proxies have been proposed 351 [HotMiddlebox13b] [HAMPEL]. 353 Another possibility leverages the SOCKS protocol [RFC1928]. SOCKS is 354 often used in enterprise networks to allow clients to reach external 355 servers. For this, the client opens a TCP connection to the SOCKS 356 server that relays it to the final destination. If both the client 357 and the SOCKS server use Multipath TCP, but not the final 358 destination, then Multipath TCP can still be used on the path between 359 the clients and the SOCKS server. At IETF'93, Korea Telecom 360 announced that they have deployed in June 2015 a commercial service 361 that uses Multipath TCP on smartphones. These smartphones access 362 regular TCP servers through a SOCKS proxy. This enables them to 363 achieve throughputs of up to 850 Mbps [KT]. 365 Measurements performed with Android smartphones [Mobicom15] show that 366 popular applications work correctly through a SOCKS proxy and 367 Multipath TCP enabled smartphones. Thanks to Multipath TCP, long- 368 lived connections can be spread over the two available interfaces. 369 However, for short-lived connections, most of the data is sent over 370 the initial subflow that is created over the interface corresponding 371 to the default route and the second subflow is almost not used 372 [PAM2016]. 374 A second use case is when Multipath TCP is used by middleboxes, 375 typically inside access networks. Various network operators are 376 discussing and evaluating solutions for hybrid access networks 377 [TR-348]. Such networks arise when a network operator controls two 378 different access network technologies, e.g. wired and cellular, and 379 wants to combine them to improve the bandwidth offered to the 380 endusers [I-D.lhwxz-hybrid-access-network-architecture]. Several 381 solutions are currently investigated for such networks [TR-348]. 382 Figure 2 shows the organization of such a network. When a client 383 creates a normal TCP connection, it is intercepted by the Hybrid CPE 384 (HPCE) that converts it in a Multipath TCP connection so that it can 385 use the available access networks (DSL and LTE in the example). The 386 Hybrid Access Gateway (HAG) does the opposite to ensure that the 387 regular server sees a normal TCP connection. Some of the solutions 388 currently discussed for hybrid networks use Multipath TCP on the HCPE 389 and the HAG. Other solutions rely on tunnels between the HCPE and 390 the HAG [I-D.lhwxz-gre-notifications-hybrid-access]. 392 client --- HCPE ------ DSL ------- HAG --- internet --- server 393 | | 394 +------- LTE -----------+ 396 Figure 2: Hybrid Access Network 398 3. Operational Experience 400 3.1. Middlebox interference 402 The interference caused by various types of middleboxes has been an 403 important concern during the design of the Multipath TCP protocol. 404 Three studies on the interactions between Multipath TCP and 405 middleboxes are worth discussing. 407 The first analysis appears in [IMC11]. This paper was the main 408 motivation for Multipath TCP incorporating various techniques to cope 409 with middlebox interference. More specifically, Multipath TCP has 410 been designed to cope with middleboxes that : 412 o change source or destination addresses 414 o change source or destination port numbers 416 o change TCP sequence numbers 418 o split or coalesce segments 420 o remove TCP options 422 o modify the payload of TCP segments 424 These middlebox interferences have all been included in the MBtest 425 suite [MBTest]. This test suite is used in [HotMiddlebox13] to 426 verify the reaction of the Multipath TCP implementation in the Linux 427 kernel [MultipathTCP-Linux] when faced with middlebox interference. 428 The test environment used for this evaluation is a dual-homed client 429 connected to a single-homed server. The middlebox behavior can be 430 activated on any of the paths. The main results of this analysis are 431 : 433 o the Multipath TCP implementation in the Linux kernel, is not 434 affected by a middlebox that performs NAT or modifies TCP sequence 435 numbers 437 o when a middlebox removes the MP_CAPABLE option from the initial 438 SYN segment, the Multipath TCP implementation in the Linux kernel 439 falls back correctly to regular TCP 441 o when a middlebox removes the DSS option from all data segments, 442 the Multipath TCP implementation in the Linux kernel falls back 443 correctly to regular TCP 445 o when a middlebox performs segment coalescing, the Multipath TCP 446 implementation in the Linux kernel is still able to accurately 447 extract the data corresponding to the indicated mapping 449 o when a middlebox performs segment splitting, the Multipath TCP 450 implementation in the Linux kernel correctly reassembles the data 451 corresponding to the indicated mapping. [HotMiddlebox13] shows on 452 figure 4 in section 3.3 a corner case with segment splitting that 453 may lead to a desynchronization between the two hosts. 455 The interactions between Multipath TCP and real deployed middleboxes 456 is also analyzed in [HotMiddlebox13] and a particular scenario with 457 the FTP application level gateway running on a NAT is described. 459 Middlebox interference can also be detected by analyzing packet 460 traces on Multipath TCP enabled servers. A closer look at the 461 packets received on the multipath-tcp.org server [TMA2015] shows that 462 among the 184,000 Multipath TCP connections, only 125 of them were 463 falling back to regular TCP. These connections originated from 28 464 different client IP addresses. These include 91 HTTP connections and 465 34 FTP connections. The FTP interference is expected since 466 Application Level Gateways used for FTP modify the TCP payload and 467 the DSS Checksum detects these modifications. The HTTP interference 468 appeared only on the direction from server to client and could have 469 been caused by transparent proxies deployed in cellular or enterprise 470 networks. A longer trace is discussed in [COMCOM2016] and similar 471 conclusions about the middlebox interference are provided. 473 From an operational viewpoint, knowing that Multipath TCP can cope 474 with various types of middlebox interference is important. However, 475 there are situations where the network operators need to gather 476 information about where a particular middlebox interference occurs. 477 The tracebox software [tracebox] described in [IMC13a] is an 478 extension of the popular traceroute software that enables network 479 operators to check at which hop a particular field of the TCP header 480 (including options) is modified. It has been used by several network 481 operators to debug various middlebox interference problems. 482 Experience with tracebox indicates that supporting the ICMP extension 483 defined in [RFC1812] makes it easier to debug middlebox problems in 484 IPv4 networks. 486 Users of the Multipath TCP implementation have reported some 487 experience with middlebox interference. The strangest scenario has 488 been a middlebox that accepts the Multipath TCP options in the SYN 489 segment but later replaces Multipath TCP options with a TCP EOL 490 option [StrangeMbox]. This causes Multipath TCP to perform a 491 fallback to regular TCP without any impact on the application. 493 3.2. Congestion control 495 Congestion control has been an important challenge for Multipath TCP. 496 The congestion control scheme specified for Multipath TCP is defined 497 in [RFC6356]. A detailed description of this algorithm is provided 498 in [NSDI11]. This congestion control scheme has been implemented in 499 the Linux implementation of Multipath TCP. Linux uses a modular 500 architecture to support various congestion control schemes. This 501 architecture is applicable for both regular TCP and Multipath TCP. 502 While the coupled congestion control scheme defined in [RFC6356] is 503 the default congestion control scheme in the Linux implementation, 504 other congestion control schemes have been added. The second 505 congestion control scheme is OLIA [CONEXT12]. This congestion 506 control scheme is also an adaptation of the NewReno single path 507 congestion control scheme to support multiple paths. Simulations and 508 measurements have shown that it provides some performance benefits 509 compared to the default congestion control scheme [CONEXT12]. 510 Measurements over a wide range of parameters reported in [CONEXT13] 511 also indicate some benefits with the OLIA congestion control scheme. 512 Recently, a delay-based congestion control scheme has been ported to 513 the Multipath TCP implementation in the Linux kernel. This 514 congestion control scheme has been evaluated by using simulations in 515 [ICNP12] and measurements in [PaaschPhD]. The fourth congestion 516 control scheme that has been included in the Linux implementation of 517 Multipath TCP is the BALIA scheme that provides a better balance 518 between TCP friendliness, responsiveness, and window oscillation 519 [BALIA]. 521 These different congestion control schemes have been compared in 522 several articles. [CONEXT13] and [PaaschPhD] compare these 523 algorithms in an emulated environment. The evaluation showed that 524 the delay-based congestion control scheme is less able to efficiently 525 use the available links than the three other schemes. 527 3.3. Subflow management 529 The multipath capability of Multipath TCP comes from the utilization 530 of one subflow per path. The Multipath TCP architecture [RFC6182] 531 and the protocol specification [RFC6824] define the basic usage of 532 the subflows and the protocol mechanisms that are required to create 533 and terminate them. However, there are no guidelines on how subflows 534 are used during the lifetime of a Multipath TCP session. Most of the 535 published experiments with Multipath TCP have been performed in 536 controlled environments. Still, based on the experience running them 537 and discussions on the mptcp-dev mailing list, interesting lessons 538 have been learned about the management of these subflows. 540 From a subflow viewpoint, the Multipath TCP protocol is completely 541 symmetrical. Both the clients and the server have the capability to 542 create subflows. However in practice the existing Multipath TCP 543 implementations have opted for a strategy where only the client 544 creates new subflows. The main motivation for this strategy is that 545 often the client resides behind a NAT or a firewall, preventing 546 passive subflow openings on the client. Although there are 547 environments such as datacenters where this problem does not occur, 548 as of this writing, no precise requirement has emerged for allowing 549 the server to create new subflows. 551 3.4. Implemented subflow managers 553 The Multipath TCP implementation in the Linux kernel includes several 554 strategies to manage the subflows that compose a Multipath TCP 555 session. The basic subflow manager is the full-mesh. As the name 556 implies, it creates a full-mesh of subflows between the communicating 557 hosts. 559 The most frequent use case for this subflow manager is a multihomed 560 client connected to a single-homed server. In this case, one subflow 561 is created for each interface on the client. The current 562 implementation of the full-mesh subflow manager is static. The 563 subflows are created immediately after the creation of the initial 564 subflow. If one subflow fails during the lifetime of the Multipath 565 TCP session (e.g. due to excessive retransmissions, or the loss of 566 the corresponding interface), it is not always reestablished. There 567 is ongoing work to enhance the full-mesh path manager to deal with 568 such events. 570 When the server is multihomed, using the full-mesh subflow manager 571 may lead to a large number of subflows being established. For 572 example, consider a dual-homed client connected to a server with 573 three interfaces. In this case, even if the subflows are only 574 created by the client, 6 subflows will be established. This may be 575 excessive in some environments, in particular when the client and/or 576 the server have a large number of interfaces. Implementations should 577 limit the number of subflows that are used. 579 Creating subflows between multihomed clients and servers may 580 sometimes lead to operational issues as observed by discussions on 581 the mptcp-dev mailing list. In some cases the network operators 582 would like to have a better control on how the subflows are created 583 by Multipath TCP [I-D.boucadair-mptcp-max-subflow]. This might 584 require the definition of policy rules to control the operation of 585 the subflow manager. The two scenarios below illustrate some of 586 these requirements. 588 host1 ---------- switch1 ----- host2 589 | | | 590 +-------------- switch2 --------+ 592 Figure 3: Simple switched network topology 594 Consider the simple network topology shown in Figure 3. From an 595 operational viewpoint, a network operator could want to create two 596 subflows between the communicating hosts. From a bandwidth 597 utilization viewpoint, the most natural paths are host1-switch1-host2 598 and host1-switch2-host2. However, a Multipath TCP implementation 599 running on these two hosts may sometimes have difficulties to obtain 600 this result. 602 To understand the difficulty, let us consider different allocation 603 strategies for the IP addresses. A first strategy is to assign two 604 subnets : subnetA (resp. subnetB) contains the IP addresses of 605 host1's interface to switch1 (resp. switch2) and host2's interface to 606 switch1 (resp. switch2). In this case, a Multipath TCP subflow 607 manager should only create one subflow per subnet. To enforce the 608 utilization of these paths, the network operator would have to 609 specify a policy that prefers the subflows in the same subnet over 610 subflows between addresses in different subnets. It should be noted 611 that the policy should probably also specify how the subflow manager 612 should react when an interface or subflow fails. 614 A second strategy is to use a single subnet for all IP addresses. In 615 this case, it becomes more difficult to specify a policy that 616 indicates which subflows should be established. 618 The second subflow manager that is currently supported by the 619 Multipath TCP implementation in the Linux kernel is the ndiffport 620 subflow manager. This manager was initially created to exploit the 621 path diversity that exists between single-homed hosts due to the 622 utilization of flow-based load balancing techniques [SIGCOMM11]. 623 This subflow manager creates N subflows between the same pair of IP 624 addresses. The N subflows are created by the client and differ only 625 in the source port selected by the client. It was not designed to be 626 used on multihomed hosts. 628 A more flexible subflow manager has been proposed, implemented and 629 evaluated in [CONEXT15]. This subflow manager exposes various kernel 630 events to a user space daemon that decides when subflows need to be 631 created and terminated based on various policies. 633 3.5. Subflow destination port 635 The Multipath TCP protocol relies on the token contained in the 636 MP_JOIN option to associate a subflow to an existing Multipath TCP 637 session. This implies that there is no restriction on the source 638 address, destination address and source or destination ports used for 639 the new subflow. The ability to use different source and destination 640 addresses is key to support multihomed servers and clients. The 641 ability to use different destination port numbers is worth discussing 642 because it has operational implications. 644 For illustration, consider a dual-homed client that creates a second 645 subflow to reach a single-homed server as illustrated in Figure 4. 647 client ------- r1 --- internet --- server 648 | | 649 +----------r2-------+ 651 Figure 4: Multihomed-client connected to single-homed server 653 When the Multipath TCP implementation in the Linux kernel creates the 654 second subflow it uses the same destination port as the initial 655 subflow. This choice is motivated by the fact that the server might 656 be protected by a firewall and only accept TCP connections (including 657 subflows) on the official port number. Using the same destination 658 port for all subflows is also useful for operators that rely on the 659 port numbers to track application usage in their network. 661 There have been suggestions from Multipath TCP users to modify the 662 implementation to allow the client to use different destination ports 663 to reach the server. This suggestion seems mainly motivated by 664 traffic shaping middleboxes that are used in some wireless networks. 665 In networks where different shaping rates are associated to different 666 destination port numbers, this could allow Multipath TCP to reach a 667 higher performance. This behavior is valid according to the 668 Multipath TCP specification [RFC6824]. An application could used an 669 enhanced socket API [SOCKET] to behave in this way. 671 However, from an implementation point-of-view supporting different 672 destination ports for the same Multipath TCP connection can cause 673 some issues. A legacy implementation of a TCP stack creates a 674 listening socket to react upon incoming SYN segments. The listening 675 socket is handling the SYN segments that are sent on a specific port 676 number. Demultiplexing incoming segments can thus be done solely by 677 looking at the IP addresses and the port numbers. With Multipath TCP 678 however, incoming SYN segments may have an MP_JOIN option with a 679 different destination port. This means, that all incoming segments 680 that did not match on an existing listening-socket or an already 681 established socket must be parsed for an eventual MP_JOIN option. 682 This imposes an additional cost on servers, previously not existent 683 on legacy TCP implementations. 685 3.6. Closing subflows 687 client server 688 | | 689 MPTCP: established | | MPTCP: established 690 Sub: established | | Sub: established 691 | | 692 | DATA_FIN | 693 MPTCP: close-wait | <------------------------ | close() (step 1) 694 Sub: established | DATA_ACK | 695 | ------------------------> | MPTCP: fin-wait-2 696 | | Sub: established 697 | | 698 | DATA_FIN + subflow-FIN | 699 close()/shutdown() | ------------------------> | MPTCP: time-wait 700 (step 2) | DATA_ACK | Sub: close-wait 701 MPTCP: closed | <------------------------ | 702 Sub: fin-wait-2 | | 703 | | 704 | subflow-FIN | 705 MPTCP: closed | <------------------------ | subflow-close() 706 Sub: time-wait | subflow-ACK | 707 (step 3) | ------------------------> | MPTCP: time-wait 708 | | Sub: closed 709 | | 711 Figure 5: Multipath TCP may not be able to avoid time-wait state on 712 the subflow (indicated as Sub in the drawing), even if enforced by 713 the application on the client-side. 715 Figure 5 shows a very particular issue within Multipath TCP. Many 716 high-performance applications try to avoid Time-Wait state by 717 deferring the closure of the connection until the peer has sent a 718 FIN. That way, the client on the left of Figure 5 does a passive 719 closure of the connection, transitioning from Close-Wait to Last-ACK 720 and finally freeing the resources after reception of the ACK of the 721 FIN. An application running on top of a Multipath TCP enabled Linux 722 kernel might also use this approach. The difference here is that the 723 close() of the connection (Step 1 in Figure 5) only triggers the 724 sending of a DATA_FIN. Nothing guarantees that the kernel is ready 725 to combine the DATA_FIN with a subflow-FIN. The reception of the 726 DATA_FIN will make the application trigger the closure of the 727 connection (step 2), trying to avoid Time-Wait state with this late 728 closure. This time, the kernel might decide to combine the DATA_FIN 729 with a subflow-FIN. This decision will be fatal, as the subflow's 730 state machine will not transition from Close-Wait to Last-Ack, but 731 rather go through Fin-Wait-2 into Time-Wait state. The Time-Wait 732 state will consume resources on the host for at least 2 MSL (Maximum 733 Segment Lifetime). Thus, a smart application that tries to avoid 734 Time-Wait state by doing late closure of the connection actually ends 735 up with one of its subflows in Time-Wait state. A high-performance 736 Multipath TCP kernel implementation should honor the desire of the 737 application to do passive closure of the connection and successfully 738 avoid Time-Wait state - even on the subflows. 740 The solution to this problem lies in an optimistic assumption that a 741 host doing active-closure of a Multipath TCP connection by sending a 742 DATA_FIN will soon also send a FIN on all its subflows. Thus, the 743 passive closer of the connection can simply wait for the peer to send 744 exactly this FIN - enforcing passive closure even on the subflows. 745 Of course, to avoid consuming resources indefinitely, a timer must 746 limit the time our implementation waits for the FIN. 748 3.7. Packet schedulers 750 In a Multipath TCP implementation, the packet scheduler is the 751 algorithm that is executed when transmitting each packet to decide on 752 which subflow it needs to be transmitted. The packet scheduler 753 itself does not have any impact on the interoperability of Multipath 754 TCP implementations. However, it may clearly impact the performance 755 of Multipath TCP sessions. The Multipath TCP implementation in the 756 Linux kernel supports a pluggable architecture for the packet 757 scheduler [PaaschPhD]. As of this writing, two schedulers have been 758 implemented: round-robin and lowest-rtt-first. The second scheduler 759 relies on the round-trip-time (rtt) measured on each TCP subflow and 760 sends first segments over the subflow having the lowest round-trip- 761 time. They are compared in [CSWS14]. The experiments and 762 measurements described in [CSWS14] show that the lowest-rtt-first 763 scheduler appears to be the best compromise from a performance 764 viewpoint. Another study of the packet schedulers is presented in 765 [PAMS2014]. This study relies on simulations with the Multipath TCP 766 implementation in the Linux kernel. They compare the lowest-rtt- 767 first with the round-robin and a random scheduler. They show some 768 situations where the lowest-rtt-first scheduler does not perform as 769 well as the other schedulers, but there are many scenarios where the 770 opposite is true. [PAMS2014] notes that "it is highly likely that 771 the optimal scheduling strategy depends on the characteristics of the 772 paths being used." 774 3.8. Segment size selection 776 When an application performs a write/send system call, the kernel 777 allocates a packet buffer (sk_buff in Linux) to store the data the 778 application wants to send. The kernel will store at most one MSS 779 (Maximum Segment Size) of data per buffer. As the MSS can differ 780 amongst subflows, an MPTCP implementation must select carefully the 781 MSS used to generate application data. The Linux kernel 782 implementation had various ways of selecting the MSS: minimum or 783 maximum amongst the different subflows. However, these heuristics of 784 MSS selection can cause significant performance issues in some 785 environment. Consider the following example. An MPTCP connection 786 has two established subflows that respectively use a MSS of 1420 and 787 1428 bytes. If MPTCP selects the maximum, then the application will 788 generate segments of 1428 bytes of data. An MPTCP implementation 789 will have to split the segment in two ( 1420-byte and 8-byte) 790 segments when pushing on the subflow with the smallest MSS. The 791 latter segment will introduce a large overhead as for a single data 792 segment 2 slots will be used in the congestion window (in packets) 793 therefore reducing by roughly twice the potential throughput (in 794 bytes/s) of this subflow. Taking the smallest MSS does not solve the 795 issue as there might be a case where the subflow with the smallest 796 MSS only sends a few packets therefore reducing the potential 797 throughput of the other subflows. 799 The Linux implementation recently took another approach [DetalMSS]. 800 Instead of selecting the minimum and maximum values, it now 801 dynamically adapts the MSS based on the contribution of all the 802 subflows to the connection's throughput. For this it computes, for 803 each subflow, the potential throughput achieved by selecting each MSS 804 value and by taking into account the lost space in the congestion 805 window. It then selects the MSS that allows to achieve the highest 806 potential throughput. 808 Given the prevalence of middleboxes that clamp the MSS, Multipath TCP 809 implementations must be able to efficiently support subflows with 810 different MSS values. The strategy described above is a possible 811 solution to this problem. 813 3.9. Interactions with the Domain Name System 815 Multihomed clients such as smartphones can send DNS queries over any 816 of their interfaces. When a single-homed client performs a DNS 817 query, it receives from its local resolver the best answer for its 818 request. If the client is multihomed, the answer in response to the 819 DNS query may vary with the interface over which it has been sent. 821 cdn1 822 | 823 client -- cellular -- internet -- cdn3 824 | | 825 +----- wifi --------+ 826 | 827 cdn2 829 Figure 6: Simple network topology 831 If the client sends a DNS query over the WiFi interface, the answer 832 will point to the cdn2 server while the same request sent over the 833 cellular interface will point to the cdn1 server. This might cause 834 problems for CDN providers that locate their servers inside ISP 835 networks and have contracts that specify that the CDN server will 836 only be accessed from within this particular ISP. Assume now that 837 both the client and the CDN servers support Multipath TCP. In this 838 case, a Multipath TCP session from cdn1 or cdn2 would potentially use 839 both the cellular network and the WiFi network. Serving the client 840 from cdn2 over the cellular interface could violate the contract 841 between the CDN provider and the network operators. A similar 842 problem occurs with regular TCP if the client caches DNS replies. 843 For example the client obtains a DNS answer over the cellular 844 interface and then stops this interface and starts to use its WiFi 845 interface. If the client retrieves data from cdn1 over its WiFi 846 interface, this may also violate the contract between the CDN and the 847 network operators. 849 A possible solution to prevent this problem would be to modify the 850 DNS resolution on the client. The client subnet EDNS extension 851 defined in [RFC7871] could be used for this purpose. When the client 852 sends a DNS query from its WiFi interface, it should also send the 853 client subnet corresponding to the cellular interface in this 854 request. This would indicate to the resolver that the answer should 855 be valid for both the WiFi and the cellular interfaces (e.g., the 856 cdn3 server). 858 3.10. Captive portals 860 Multipath TCP enables a host to use different interfaces to reach a 861 server. In theory, this should ensure connectivity when at least one 862 of the interfaces is active. In practice however, there are some 863 particular scenarios with captive portals that may cause operational 864 problems. The reference environment is shown in Figure 7. 866 client ----- network1 867 | 868 +------- internet ------------- server 870 Figure 7: Issue with captive portal 872 The client is attached to two networks : network1 that provides 873 limited connectivity and the entire Internet through the second 874 network interface. In practice, this scenario corresponds to an open 875 WiFi network with a captive portal for network1 and a cellular 876 service for the second interface. On many smartphones, the WiFi 877 interface is preferred over the cellular interface. If the 878 smartphone learns a default route via both interfaces, it will 879 typically prefer to use the WiFi interface to send its DNS request 880 and create the first subflow. This is not optimal with Multipath 881 TCP. A better approach would probably be to try a few attempts on 882 the WiFi interface and then, upon failure of these attempts, try to 883 use the second interface for the initial subflow as well. 885 3.11. Stateless webservers 887 MPTCP has been designed to interoperate with webservers that benefit 888 from SYN-cookies to protect against SYN-flooding attacks [RFC4987]. 889 MPTCP achieves this by echoing the keys negotiated during the 890 MP_CAPABLE handshake in the third ACK of the 3-way handshake. 891 Reception of this third ACK then allows the server to reconstruct the 892 state specific to MPTCP. 894 However, one caveat to this mechanism is the non-reliable nature of 895 the third ACK. Indeed, when the third ACK gets lost, the server will 896 not be able to reconstruct the MPTCP-state. MPTCP will fallback to 897 regular TCP in this case. This is in contrast to regular TCP. When 898 the client starts sending data, the first data segment also includes 899 the SYN-cookie, which allows the server to reconstruct the TCP-state. 900 Further, this data segment will be retransmitted by the client in 901 case it gets lost and thus is resilient against loss. MPTCP does not 902 include the keys in this data segment and thus the server cannot 903 reconstruct the MPTCP state. 905 This issue might be considered as a minor one for MPTCP. Losing the 906 third ACK should only happen when packet loss is high. However, when 907 packet-loss is high MPTCP provides a lot of benefits as it can move 908 traffic away from the lossy link. It is undesirable that MPTCP has a 909 higher chance to fall back to regular TCP in those lossy 910 environments. 912 [I-D.paasch-mptcp-syncookies] discusses this issue and suggests a 913 modified handshake mechanism that ensures reliable delivery of the 914 MP_CAPABLE, following the 3-way handshake. This modification will 915 make MPTCP reliable, even in lossy environments when servers need to 916 use SYN-cookies to protect against SYN-flooding attacks. 918 3.12. Loadbalanced server farms 920 Large-scale server farms typically deploy thousands of servers behind 921 a single virtual IP (VIP). Steering traffic to these servers is done 922 through layer-4 load balancers that ensure that a TCP-flow will 923 always be routed to the same server [Presto08]. 925 As Multipath TCP uses multiple different TCP subflows to steer the 926 traffic across the different paths, load balancers need to ensure 927 that all these subflows are routed to the same server. This implies 928 that the load balancers need to track the MPTCP-related state, 929 allowing them to parse the token in the MP_JOIN and assign those 930 subflows to the appropriate server. However, server farms typically 931 deploy several load balancers for reliability and capacity reasons. 932 As a TCP subflow might get routed to any of these load balancers, 933 they would need to synchronize the MPTCP-related state - a solution 934 that is not feasible at large scale. 936 The token (carried in the MP_JOIN) contains the information 937 indicating which MPTCP-session the subflow belongs to. As the token 938 is a hash of the key, servers are not able to generate the token in 939 such a way that the token can provide the necessary information to 940 the load balancers, which would allow them to route TCP subflows to 941 the appropriate server. [I-D.paasch-mptcp-loadbalancer] discusses 942 this issue in detail and suggests two alternative MP_CAPABLE 943 handshakes to overcome these. 945 4. IANA Considerations 947 There are no IANA considerations in this informational document. 949 5. Security Considerations 951 This informational document discusses use-cases and operational 952 experience with Multipath TCP. An extensive analysis of the 953 remaining security issues in the Multipath TCP specification has been 954 published in [RFC7430], together with suggestions for possible 955 solutions. 957 From a security viewpoint, it is important to note that Multipath 958 TCP, like other multipath solutions such as SCTP, has the ability to 959 send packets belonging to a single connection over different paths. 960 This design feature of Multipath TCP implies that middleboxes that 961 have been deployed on-path assuming that they would observe all the 962 packets exchanged for a given connection in both directions may not 963 function correctly anymore. A typical example are firewalls, IDS or 964 DPIs deployed in enterprise networks. Those devices expect to 965 observe all the packets from all TCP connections. With Multipath 966 TCP, those middleboxes may not observe anymore all packets since some 967 of them may follow a different path. The two examples below 968 illustrate typical deployments of such middleboxes. The first 969 example, Figure 8, shows a Multipath TCP enabled smartphone attached 970 to both an enterprise and a cellular network. If a Multipath TCP 971 connection is established by the smartphone towards a server, some of 972 the packets sent by the smartphone or the server may be transmitted 973 over the cellular network and thus be invisible for the enterprise 974 middlebox. 976 smartphone +----- entreprise net --- MBox----+------ server 977 | | 978 +----- cellular net -------------+ 980 Figure 8: Enterprise Middlebox may not observe all packets from 981 multihomed host 983 The second example, Figure 9, shows a possible issue when multiple 984 middleboxes are deployed inside a network. For simplicity, we assume 985 that network1 is the default IPv4 path while network2 is the default 986 IPv6 path. A similar issue could occur with per-flow load balancing 987 such as ECMP [RFC2992]. With regular TCP, all packets from each 988 connection would either pass through Mbox1 or Mbox2. With Multipath 989 TCP, the client can easily establish a subflow over network1 and 990 another over network2 and each middlebox would only observe a part of 991 the traffic of the end-to-end Multipath TCP connection. 993 client ----R-- network1 --- MBox1 -----R------------- server 994 | | 995 +-- network2 --- MBox2 -----+ 997 Figure 9: Interactions between load balancing and security 998 Middleboxes 1000 In these two cases, it is possible for an attacker to evade some 1001 security measures operating on the TCP byte stream and implemented on 1002 the middleboxes by controlling the bytes that are actually sent over 1003 each subflow and there are tools that ease those kinds of evasion 1004 [PZ15] [PT14]. This is not a security issue for Multipath TCP itself 1005 since Multipath TCP behaves correctly. However, this demonstrates 1006 the difficulty of enforcing security policies by relying only on on- 1007 path middleboxes instead of enforcing them directly on the endpoints. 1009 6. Acknowledgements 1011 This work was partially supported by the FP7-Trilogy2 project. We 1012 would like to thank all the implementers and users of the Multipath 1013 TCP implementation in the Linux kernel. This document has benefited 1014 from the comments of John Ronan, Yoshifumi Nishida, Phil Eardley, 1015 Jaehyun Hwang, Mirja Kuehlewind, Benoit Claise, Jari Arkko, Qin Wu, 1016 Spencer Dawkins and Ben Campbell. 1018 7. References 1020 7.1. Normative References 1022 [RFC6182] Ford, A., Raiciu, C., Handley, M., Barre, S., and J. 1023 Iyengar, "Architectural Guidelines for Multipath TCP 1024 Development", RFC 6182, DOI 10.17487/RFC6182, March 2011, 1025 . 1027 [RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure, 1028 "TCP Extensions for Multipath Operation with Multiple 1029 Addresses", RFC 6824, DOI 10.17487/RFC6824, January 2013, 1030 . 1032 7.2. Informative References 1034 [Apple-MPTCP] 1035 Apple, Inc, ., "iOS - Multipath TCP Support in iOS 7", 1036 n.d., . 1038 [BALIA] Peng, Q., Walid, A., Hwang, J., and S. Low, "Multipath TCP 1039 Analysis, Design, and Implementation", IEEE/ACM Trans. on 1040 Networking, Vol. 24, No. 1 , 2016. 1042 [CACM14] Paasch, C. and O. Bonaventure, "Multipath TCP", 1043 Communications of the ACM, 57(4):51-57 , April 2014, 1044 . 1046 [COMCOM2016] 1047 Tran, V., De Coninck, Q., Hesmans, B., Sadre, R., and O. 1048 Bonaventure, "Observing real Multipath TCP traffic", 1049 Computer Communications , April 2016, . 1053 [COMMAG2016] 1054 De Coninck, Q., Baerts, M., Hesmans, B., and O. 1055 Bonaventure, "Observing Real Smartphone Applications over 1056 Multipath TCP", IEEE Communications Magazine , March 2016, 1057 . 1061 [CONEXT12] 1062 Khalili, R., Gast, N., Popovic, M., Upadhyay, U., and J. 1063 Leboudec, "MPTCP is not pareto-optimal performance issues 1064 and a possible solution", Proceedings of the 8th 1065 international conference on Emerging networking 1066 experiments and technologies (CoNEXT12) , 2012. 1068 [CONEXT13] 1069 Paasch, C., Khalili, R., and O. Bonaventure, "On the 1070 Benefits of Applying Experimental Design to Improve 1071 Multipath TCP", Conference on emerging Networking 1072 EXperiments and Technologies (CoNEXT) , December 2013, . 1077 [CONEXT15] 1078 Hesmans, B., Detal, G., Barre, S., Bauduin, R., and O. 1079 Bonaventure, "SMAPP - Towards Smart Multipath TCP-enabled 1080 APPlications", Proc. Conext 2015, Heidelberg, Germany , 1081 December 2015, . 1084 [CSWS14] Paasch, C., Ferlin, S., Alay, O., and O. Bonaventure, 1085 "Experimental Evaluation of Multipath TCP Schedulers", 1086 SIGCOMM CSWS2014 workshop , August 2014. 1088 [Cellnet12] 1089 Paasch, C., Detal, G., Duchene, F., Raiciu, C., and O. 1090 Bonaventure, "Exploring Mobile/WiFi Handover with 1091 Multipath TCP", ACM SIGCOMM workshop on Cellular Networks 1092 (Cellnet12) , 2012, . 1095 [DetalMSS] 1096 Detal, G., "Adaptive MSS value", Post on the mptcp-dev 1097 mailing list , September 2014, . 1101 [FreeBSD-MPTCP] 1102 Williams, N., "Multipath TCP For FreeBSD Kernel Patch 1103 v0.5", n.d., . 1105 [HAMPEL] Hampel, G., Rana, A., and T. Klein, "Seamless TCP mobility 1106 using lightweight MPTCP proxy", Proceedings of the 11th 1107 ACM international symposium on Mobility management and 1108 wireless access (MobiWac '13). , 2013. 1110 [HotMiddlebox13] 1111 Hesmans, B., Duchene, F., Paasch, C., Detal, G., and O. 1112 Bonaventure, "Are TCP Extensions Middlebox-proof?", CoNEXT 1113 workshop HotMiddlebox , December 2013, . 1117 [HotMiddlebox13b] 1118 Detal, G., Paasch, C., and O. Bonaventure, "Multipath in 1119 the Middle(Box)", HotMiddlebox'13 , December 2013, . 1122 [HotNets] Raiciu, C., Pluntke, C., Barre, S., Greenhalgh, A., 1123 Wischik, D., and M. Handley, "Data center networking with 1124 multipath TCP", Proceedings of the 9th ACM SIGCOMM 1125 Workshop on Hot Topics in Networks (Hotnets-IX) , 2010, 1126 . 1128 [I-D.boucadair-mptcp-max-subflow] 1129 Boucadair, M. and C. Jacquenet, "Negotiating the Maximum 1130 Number of Multipath TCP (MPTCP) Subflows", 1131 draft-boucadair-mptcp-max-subflow-02 (work in progress), 1132 May 2016. 1134 [I-D.lhwxz-gre-notifications-hybrid-access] 1135 Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M. 1136 Zhang, "GRE Notifications for Hybrid Access", 1137 draft-lhwxz-gre-notifications-hybrid-access-01 (work in 1138 progress), January 2015. 1140 [I-D.lhwxz-hybrid-access-network-architecture] 1141 Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M. 1142 Zhang, "Hybrid Access Network Architecture", 1143 draft-lhwxz-hybrid-access-network-architecture-02 (work in 1144 progress), January 2015. 1146 [I-D.paasch-mptcp-loadbalancer] 1147 Paasch, C., Greenway, G., and A. Ford, "Multipath TCP 1148 behind Layer-4 loadbalancers", 1149 draft-paasch-mptcp-loadbalancer-00 (work in progress), 1150 September 2015. 1152 [I-D.paasch-mptcp-syncookies] 1153 Paasch, C., Biswas, A., and D. Haas, "Making Multipath TCP 1154 robust for stateless webservers", 1155 draft-paasch-mptcp-syncookies-02 (work in progress), 1156 October 2015. 1158 [ICNP12] Cao, Y., Xu, M., and X. Fu, "Delay-based congestion 1159 control for multipath TCP", 20th IEEE International 1160 Conference on Network Protocols (ICNP) , 2012. 1162 [IETFJ] Bonaventure, O. and S. Seo, "Multipath TCP Deployments,", 1163 IETF Journal, Vol. 12, Issue 2 , November 2016. 1165 [IMC11] Honda, M., Nishida, Y., Raiciu, C., Greenhalgh, A., 1166 Handley, M., and H. Tokuda, "Is it still possible to 1167 extend TCP?", Proceedings of the 2011 ACM SIGCOMM 1168 conference on Internet measurement conference (IMC '11) , 1169 2011, . 1171 [IMC13a] Detal, G., Hesmans, B., Bonaventure, O., Vanaubel, Y., and 1172 B. Donnet, "Revealing Middlebox Interference with 1173 Tracebox", Proceedings of the 2013 ACM SIGCOMM conference 1174 on Internet measurement conference , 2013, . 1178 [IMC13b] Chen, Y., Lim, Y., Gibbens, R., Nahum, E., Khalili, R., 1179 and D. Towsley, "A measurement-based study of MultiPath 1180 TCP performance over wireless network", Proceedings of the 1181 2013 conference on Internet measurement conference (IMC 1182 '13) , n.d., . 1184 [IMC13c] Pelsser, C., Cittadini, L., Vissicchio, S., and R. Bush, 1185 "From Paris to Tokyo on the suitability of ping to 1186 measure latency", Proceedings of the 2013 conference on 1187 Internet measurement conference (IMC '13) , 2013, 1188 . 1190 [INFOCOM14] 1191 Lim, Y., Chen, Y., Nahum, E., Towsley, D., and K. Lee, 1192 "Cross-Layer Path Management in Multi-path Transport 1193 Protocol for Mobile Devices", IEEE INFOCOM'14 , 2014. 1195 [IOS7] Apple, ., "Multipath TCP Support in iOS 7", January 2014, 1196 . 1198 [KT] Seo, S., "KT's GiGA LTE", July 2015, . 1201 [MBTest] Hesmans, B., "MBTest", 2013, 1202 . 1204 [MPTCPBIB] 1205 Bonaventure, O., "Multipath TCP - An annotated 1206 bibliography", Technical report , April 2015, 1207 . 1209 [Mobicom15] 1210 De Coninck, Q., Baerts, M., Hesmans, B., and O. 1211 Bonaventure, "Poster - Evaluating Android Applications 1212 with Multipath TCP", Mobicom 2015 (Poster) , 1213 September 2015. 1215 [MultipathTCP-Linux] 1216 Paasch, C., Barre, S., and . et al, "Multipath TCP 1217 implementation in the Linux kernel", n.d., 1218 . 1220 [NSDI11] Wischik, D., Raiciu, C., Greenhalgh, A., and M. Handley, 1221 "Design, implementation and evaluation of congestion 1222 control for Multipath TCP", In Proceedings of the 8th 1223 USENIX conference on Networked systems design and 1224 implementation (NSDI11) , 2011. 1226 [NSDI12] Raiciu, C., Paasch, C., Barre, S., Ford, A., Honda, M., 1227 Duchene, F., Bonaventure, O., and M. Handley, "How Hard 1228 Can It Be? Designing and Implementing a Deployable 1229 Multipath TCP", USENIX Symposium of Networked Systems 1230 Design and Implementation (NSDI12) , April 2012, . 1235 [PAM2016] De Coninck, Q., Baerts, M., Hesmans, B., and O. 1236 Bonaventure, "A First Analysis of Multipath TCP on 1237 Smartphones", 17th International Passive and Active 1238 Measurements Conference (PAM2016) , March 2016, . 1242 [PAMS2014] 1243 Arzani, B., Gurney, A., Cheng, S., Guerin, R., and B. Loo, 1244 "Impact of Path Selection and Scheduling Policies on MPTCP 1245 Performance", PAMS2014 , 2014. 1247 [PT14] Pearce, C. and P. Thomas, "Multipath TCP Breaking Today's 1248 Networks with Tomorrow's Protocols", Proc. Blackhat 1249 Briefings , 2014, . 1254 [PZ15] Pearce, C. and S. Zeadally, "Ancillary Impacts of 1255 Multipath TCP on Current and Future Network Security", 1256 IEEE Internet Computing, vol. 19, no. 5, pp. 58-65 , 2015. 1258 [PaaschPhD] 1259 Paasch, C., "Improving Multipath TCP", Ph.D. Thesis , 1260 November 2014, . 1263 [Presto08] 1264 Greenberg, A., Lahiri, P., Maltz, D., Parveen, P., and S. 1265 Sengupta, "Towards a Next Generation Data Center 1266 Architecture - Scalability and Commoditization", ACM 1267 PRESTO 2008 , August 2008, 1268 . 1270 [RFC1812] Baker, F., Ed., "Requirements for IP Version 4 Routers", 1271 RFC 1812, DOI 10.17487/RFC1812, June 1995, 1272 . 1274 [RFC1928] Leech, M., Ganis, M., Lee, Y., Kuris, R., Koblas, D., and 1275 L. Jones, "SOCKS Protocol Version 5", RFC 1928, 1276 DOI 10.17487/RFC1928, March 1996, 1277 . 1279 [RFC2992] Hopps, C., "Analysis of an Equal-Cost Multi-Path 1280 Algorithm", RFC 2992, DOI 10.17487/RFC2992, November 2000, 1281 . 1283 [RFC4987] Eddy, W., "TCP SYN Flooding Attacks and Common 1284 Mitigations", RFC 4987, DOI 10.17487/RFC4987, August 2007, 1285 . 1287 [RFC6356] Raiciu, C., Handley, M., and D. Wischik, "Coupled 1288 Congestion Control for Multipath Transport Protocols", 1289 RFC 6356, DOI 10.17487/RFC6356, October 2011, 1290 . 1292 [RFC7430] Bagnulo, M., Paasch, C., Gont, F., Bonaventure, O., and C. 1293 Raiciu, "Analysis of Residual Threats and Possible Fixes 1294 for Multipath TCP (MPTCP)", RFC 7430, DOI 10.17487/ 1295 RFC7430, July 2015, 1296 . 1298 [RFC7871] Contavalli, C., van der Gaast, W., Lawrence, D., and W. 1299 Kumari, "Client Subnet in DNS Queries", RFC 7871, 1300 DOI 10.17487/RFC7871, May 2016, 1301 . 1303 [SIGCOMM11] 1304 Raiciu, C., Barre, S., Pluntke, C., Greenhalgh, A., 1305 Wischik, D., and M. Handley, "Improving datacenter 1306 performance and robustness with multipath TCP", 1307 Proceedings of the ACM SIGCOMM 2011 conference , n.d., 1308 . 1310 [SOCKET] Hesmans, B. and O. Bonaventure, "An Enhanced Socket API 1311 for Multipath TCP", Proceedings of the 2016 Applied 1312 Networking Research Workshop , July 2016, 1313 . 1315 [StrangeMbox] 1316 Bonaventure, O., "Multipath TCP through a strange 1317 middlebox", Blog post , January 2015, . 1321 [TMA2015] Hesmans, B., Tran Viet, H., Sadre, R., and O. Bonaventure, 1322 "A First Look at Real Multipath TCP Traffic", Traffic 1323 Monitoring and Analysis , 2015, . 1327 [TR-348] Broadband Forum, ., "TR 348 - Hybrid Access Broadband 1328 Network Architecture", July 2016, . 1331 [ietf88] Stewart, L., "IETF'88 Meeting minutes of the MPTCP working 1332 group", n.d., . 1335 [tracebox] 1336 Detal, G. and O. Tilmans, "tracebox", 2013, 1337 . 1339 Appendix A. Changelog 1341 This section should be removed before final publication 1343 o initial version : September 16th, 2014 : Added section Section 3.8 1344 that discusses some performance problems that appeared with the 1345 Linux implementation when using subflows having different MSS 1346 values 1348 o update with a description of the middlebox that replaces an 1349 unknown TCP option with EOL [StrangeMbox] 1351 o version ietf-02 : July 2015, answer to last call comments 1353 * Reorganised text to better separate use cases and operational 1354 experience 1356 * New use case on Multipath TCP proxies in Section 2.3 1358 * Added some text on middleboxes in Section 3.1 1360 * Removed the discussion on SDN 1362 * Restructured text and improved writing in some parts 1364 o version ietf-03 : September 2015, answer to comments from Phil 1365 Eardley 1367 * Improved introduction 1369 * Added details about using SOCKS and Korea Telecom's use-case in 1370 Section 2.3. 1372 * Added issue around clients caching DNS-results in Section 3.9 1374 * Explained issue of MPTCP with stateless webservers Section 3.11 1376 * Added description of MPTCP's use behind layer-4 load balancers 1377 Section 3.12 1379 * Restructured text and improved writing in some parts 1381 o version ietf-04 : April 2016, answer to last comments 1383 * Updated text on measurements with smartphones 1385 * Updated conclusion 1387 o version ietf-06 : August 2016, answer to AD's review 1389 * removed some expired drafts 1391 * removed conclusion 1393 o version ietf-07 : September 2016, answer to IESG comments 1395 * small nits 1397 * added more information in the security considerations as 1398 suggested by Stephen Farrel 1400 Authors' Addresses 1402 Olivier Bonaventure 1403 UCLouvain 1405 Email: Olivier.Bonaventure@uclouvain.be 1407 Christoph Paasch 1408 Apple, Inc. 1410 Email: cpaasch@apple.com 1412 Gregory Detal 1413 Tessares 1415 Email: Gregory.Detal@tessares.net