idnits 2.17.1 draft-ietf-mptcp-experience-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 1171 has weird spacing: '...o Tokyo on th...' -- The document date (July 08, 2016) is 2848 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-03) exists of draft-boucadair-mptcp-max-subflow-02 -- Obsolete informational reference (is this intentional?): RFC 6824 (Obsoleted by RFC 8684) Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 MPTCP Working Group O. Bonaventure 3 Internet-Draft UCLouvain 4 Intended status: Informational C. Paasch 5 Expires: January 9, 2017 Apple, Inc. 6 G. Detal 7 Tessares 8 July 08, 2016 10 Use Cases and Operational Experience with Multipath TCP 11 draft-ietf-mptcp-experience-05 13 Abstract 15 This document discusses both use cases and operational experience 16 with Multipath TCP in real world networks. It lists several 17 prominent use cases for which Multipath TCP has been considered and 18 is being used. It also gives insight to some heuristics and 19 decisions that have helped to realize these use cases. 21 Status of this Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at http://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on January 9, 2017. 38 Copyright Notice 40 Copyright (c) 2016 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents 45 (http://trustee.ietf.org/license-info) in effect on the date of 46 publication of this document. Please review these documents 47 carefully, as they describe your rights and restrictions with respect 48 to this document. Code Components extracted from this document must 49 include Simplified BSD License text as described in Section 4.e of 50 the Trust Legal Provisions and are provided without warranty as 51 described in the Simplified BSD License. 53 Table of Contents 55 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 56 2. Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . 5 57 2.1. Datacenters . . . . . . . . . . . . . . . . . . . . . . . 5 58 2.2. Cellular/WiFi Offload . . . . . . . . . . . . . . . . . . 5 59 2.3. Multipath TCP proxies . . . . . . . . . . . . . . . . . . 9 60 3. Operational Experience . . . . . . . . . . . . . . . . . . . . 11 61 3.1. Middlebox interference . . . . . . . . . . . . . . . . . . 11 62 3.2. Congestion control . . . . . . . . . . . . . . . . . . . . 13 63 3.3. Subflow management . . . . . . . . . . . . . . . . . . . . 13 64 3.4. Implemented subflow managers . . . . . . . . . . . . . . . 14 65 3.5. Subflow destination port . . . . . . . . . . . . . . . . . 16 66 3.6. Closing subflows . . . . . . . . . . . . . . . . . . . . . 17 67 3.7. Packet schedulers . . . . . . . . . . . . . . . . . . . . 18 68 3.8. Segment size selection . . . . . . . . . . . . . . . . . . 19 69 3.9. Interactions with the Domain Name System . . . . . . . . . 19 70 3.10. Captive portals . . . . . . . . . . . . . . . . . . . . . 20 71 3.11. Stateless webservers . . . . . . . . . . . . . . . . . . . 21 72 3.12. Loadbalanced serverfarms . . . . . . . . . . . . . . . . . 22 73 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 74 5. Security Considerations . . . . . . . . . . . . . . . . . . . 24 75 6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 25 76 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 26 77 8. Informative References . . . . . . . . . . . . . . . . . . . . 27 78 Appendix A. Changelog . . . . . . . . . . . . . . . . . . . . . . 34 79 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 35 81 1. Introduction 83 Multipath TCP was standardized in [RFC6824] and five independent 84 implementations have been developed 85 [I-D.eardley-mptcp-implementations-survey]. As of September 2015, 86 Multipath TCP has been or is being implemented on the following 87 platforms: 89 o Linux kernel [MultipathTCP-Linux] 91 o Apple iOS and MacOS [Apple-MPTCP] 93 o Citrix load balancers 95 o FreeBSD [FreeBSD-MPTCP] 97 o Oracle 99 The first three implementations 100 [I-D.eardley-mptcp-implementations-survey] are known to interoperate. 101 The last two are currently being tested and improved against the 102 Linux implementation. Three of these implementations are open- 103 source. Apple's implementation is widely deployed. 105 Since the publication of [RFC6824], experience has been gathered by 106 various network researchers and users about the operational issues 107 that arise when Multipath TCP is used in today's Internet. 109 When the MPTCP working group was created, several use cases for 110 Multipath TCP were identified [RFC6182]. Since then, other use cases 111 have been proposed and some have been tested and even deployed. We 112 describe these use cases in Section 2. 114 Section 3 focuses on the operational experience with Multipath TCP. 115 Most of this experience comes from the utilisation of the Multipath 116 TCP implementation in the Linux kernel [MultipathTCP-Linux]. This 117 open-source implementation has been downloaded and is used by 118 thousands of users all over the world. Many of these users have 119 provided direct or indirect feedback by writing documents (scientific 120 articles or blog messages) or posting to the mptcp-dev mailing list 121 (see https://listes-2.sipr.ucl.ac.be/sympa/arc/mptcp-dev ). This 122 Multipath TCP implementation is actively maintained and continuously 123 improved. It is used on various types of hosts, ranging from 124 smartphones or embedded routers to high-end servers. 126 The Multipath TCP implementation in the Linux kernel is not, by far, 127 the most widespread deployment of Multipath TCP. Since September 128 2013, Multipath TCP is also supported on smartphones and tablets 129 running iOS7 [IOS7]. There are likely hundreds of millions of 130 Multipath TCP enabled devices. However, this particular Multipath 131 TCP implementation is currently only used to support a single 132 application. Unfortunately, there is no public information about the 133 lessons learned from this large scale deployment. 135 Section 3 is organized as follows. Supporting the middleboxes was 136 one of the difficult issues in designing the Multipath TCP protocol. 137 We explain in Section 3.1 which types of middleboxes the Linux Kernel 138 implementation of Multipath TCP supports and how it reacts upon 139 encountering these. Section 3.2 summarises the MPTCP specific 140 congestion controls that have been implemented. Section 3.3 to 141 Section 3.7 discuss heuristics and issues with respect to subflow 142 management as well as the scheduling across the subflows. 143 Section 3.8 explains some problems that occurred with subflows having 144 different Maximum Segment Size (MSS) values. Section 3.9 presents 145 issues with respect to content delivery networks and suggests a 146 solution to this issue. Finally, Section 3.10 documents an issue 147 with captive portals where MPTCP will behave suboptimally. 149 2. Use cases 151 Multipath TCP has been tested in several use cases. There is already 152 an abundant scientific literature on Multipath TCP [MPTCPBIB]. 153 Several of the papers published in the scientific literature have 154 identified possible improvements that are worth being discussed here. 156 2.1. Datacenters 158 A first, although initially unexpected, documented use case for 159 Multipath TCP has been in datacenters [HotNets][SIGCOMM11]. Today's 160 datacenters are designed to provide several paths between single- 161 homed servers. The multiplicity of these paths comes from the 162 utilization of Equal Cost Multipath (ECMP) and other load balancing 163 techniques inside the datacenter. Most of the deployed load 164 balancing techniques in datacenters rely on hashes computed over the 165 five tuple. Thus all packets from the same TCP connection follow the 166 same path and so are not reordered. The results in [HotNets] 167 demonstrate by simulations that Multipath TCP can achieve a better 168 utilization of the available network by using multiple subflows for 169 each Multipath TCP session. Although [RFC6182] assumes that at least 170 one of the communicating hosts has several IP addresses, [HotNets] 171 demonstrates that Multipath TCP is beneficial when both hosts are 172 single-homed. This idea is analysed in more details in [SIGCOMM11] 173 where the Multipath TCP implementation in the Linux kernel is 174 modified to be able to use several subflows from the same IP address. 175 Measurements in a public datacenter show the quantitative benefits of 176 Multipath TCP [SIGCOMM11] in this environment. 178 Although ECMP is widely used inside datacenters, this is not the only 179 environment where there are different paths between a pair of hosts. 180 ECMP and other load balancing techniques such as Link Aggregation 181 Groups (LAG) are widely used in today's networks and having multiple 182 paths between a pair of single-homed hosts is becoming the norm 183 instead of the exception. Although these multiple paths have often 184 the same cost (from an IGP metrics viewpoint), they do not 185 necessarily have the same performance. For example, [IMC13c] reports 186 the results of a long measurement study showing that load balanced 187 Internet paths between that same pair of hosts can have huge delay 188 differences. 190 2.2. Cellular/WiFi Offload 192 A second use case that has been explored by several network 193 researchers is the cellular/WiFi offload use case. Smartphones or 194 other mobile devices equipped with two wireless interfaces are a very 195 common use case for Multipath TCP. In September 2015, this is also 196 the largest deployment of Multipath-TCP enabled devices [IOS7]. It 197 has been briefly discussed during IETF88 [ietf88], but there is no 198 published paper or report that analyses this deployment. For this 199 reason, we only discuss published papers that have mainly used the 200 Multipath TCP implementation in the Linux kernel for their 201 experiments. 203 The performance of Multipath TCP in wireless networks was briefly 204 evaluated in [NSDI12]. One experiment analyzes the performance of 205 Multipath TCP on a client with two wireless interfaces. This 206 evaluation shows that when the receive window is large, Multipath TCP 207 can efficiently use the two available links. However, if the window 208 becomes smaller, then packets sent on a slow path can block the 209 transmission of packets on a faster path. In some cases, the 210 performance of Multipath TCP over two paths can become lower than the 211 performance of regular TCP over the best performing path. Two 212 heuristics, reinjection and penalization, are proposed in [NSDI12] to 213 solve this identified performance problem. These two heuristics have 214 since been used in the Multipath TCP implementation in the Linux 215 kernel. [CONEXT13] explored the problem in more detail and revealed 216 some other scenarios where Multipath TCP can have difficulties in 217 efficiently pooling the available paths. Improvements to the 218 Multipath TCP implementation in the Linux kernel are proposed in 219 [CONEXT13] to cope with some of these problems. 221 The first experimental analysis of Multipath TCP in a public wireless 222 environment was presented in [Cellnet12]. These measurements explore 223 the ability of Multipath TCP to use two wireless networks (real WiFi 224 and 3G networks). Three modes of operation are compared. The first 225 mode of operation is the simultaneous use of the two wireless 226 networks. In this mode, Multipath TCP pools the available resources 227 and uses both wireless interfaces. This mode provides fast handover 228 from WiFi to cellular or the opposite when the user moves. 229 Measurements presented in [CACM14] show that the handover from one 230 wireless network to another is not an abrupt process. When a host 231 moves, there are regions where the quality of one of the wireless 232 networks is weaker than the other, but the host considers this 233 wireless network to still be up. When a mobile host enters such 234 regions, its ability to send packets over another wireless network is 235 important to ensure a smooth handover. This is clearly illustrated 236 from the packet trace discussed in [CACM14]. 238 Many cellular networks use volume-based pricing and users often 239 prefer to use unmetered WiFi networks when available instead of 240 metered cellular networks. [Cellnet12] implements support for the 241 MP_PRIO option to explore two other modes of operation. 243 In the backup mode, Multipath TCP opens a TCP subflow over each 244 interface, but the cellular interface is configured in backup mode. 246 This implies that data only flows over only the WiFi interface when 247 both interfaces are considered to be active. If the WiFi interface 248 fails, then the traffic switches quickly to the cellular interface, 249 ensuring a smooth handover from the user's viewpoint [Cellnet12]. 250 The cost of this approach is that the WiFi and cellular interfaces 251 are likely to remain active all the time since all subflows are 252 established over the two interfaces. 254 The single-path mode is slightly different. This mode benefits from 255 the break-before-make capability of Multipath TCP. When an MPTCP 256 session is established, a subflow is created over the WiFi interface. 257 No packet is sent over the cellular interface as long as the WiFi 258 interface remains up [Cellnet12]. This implies that the cellular 259 interface can remain idle and battery capacity is preserved. When 260 the WiFi interface fails, a new subflow is established over the 261 cellular interface in order to preserve the established Multipath TCP 262 sessions. Compared to the backup mode described earlier, 263 measurements reported in [Cellnet12] indicate that this mode of 264 operation is characterised by a throughput drop while the cellular 265 interface is brought up and the subflows are reestablished. 267 From a protocol viewpoint, [Cellnet12] discusses the problem posed by 268 the unreliability of the REMOVE_ADDR option and proposes a small 269 protocol extension to allow hosts to reliably exchange this option. 270 It would be useful to analyze packet traces to understand whether the 271 unreliability of the REMOVE_ADDR option poses an operational problem 272 in real deployments. 274 Another study of the performance of Multipath TCP in wireless 275 networks was reported in [IMC13b]. This study uses laptops connected 276 to various cellular ISPs and WiFi hotspots. It compares various file 277 transfer scenarios. [IMC13b] observes that 4-path MPTCP outperforms 278 2-path MPTCP, especially for larger files. However, for three 279 congestion control algorithms (LIA, OLIA and Reno - see Section 3.2), 280 there is no significant performance difference for file sizes smaller 281 than 4MB. 283 A different study of the performance of Multipath TCP with two 284 wireless networks is presented in [INFOCOM14]. In this study the two 285 networks had different qualities : a good network and a lossy 286 network. When using two paths with different packet loss ratios, the 287 Multipath TCP congestion control scheme moves traffic away from the 288 lossy link that is considered to be congested. However, [INFOCOM14] 289 documents an interesting scenario that is summarised in Figure 1. 291 client ----------- path1 -------- server 292 | | 293 +--------------- path2 ------------+ 295 Figure 1: Simple network topology 297 Initially, the two paths have the same quality and Multipath TCP 298 distributes the load over both of them. During the transfer, the 299 second path becomes lossy, e.g. because the client moves. Multipath 300 TCP detects the packet losses and they are retransmitted over the 301 first path. This enables the data transfer to continue over the 302 first path. However, the subflow over the second path is still up 303 and transmits one packet from time to time. Although the N packets 304 have been acknowledged over the first subflow (at the MPTCP level), 305 they have not been acknowledged at the TCP level over the second 306 subflow. To preserve the continuity of the sequence numbers over the 307 second subflow, TCP will continue to retransmit these segments until 308 either they are acknowledged or the maximum number of retransmissions 309 is reached. This behavior is clearly inefficient and may lead to 310 blocking since the second subflow will consume window space to be 311 able to retransmit these packets. [INFOCOM14] proposes a new 312 Multipath TCP option to solve this problem. In practice, a new TCP 313 option is probably not required. When the client detects that the 314 data transmitted over the second subflow has been acknowledged over 315 the first subflow, it could decide to terminate the second subflow by 316 sending a RST segment. If the interface associated to this subflow 317 is still up, a new subflow could be immediately reestablished. It 318 would then be immediately usable to send new data and would not be 319 forced to first retransmit the previously transmitted data. As of 320 this writing, this dynamic management of the subflows is not yet 321 implemented in the Multipath TCP implementation in the Linux kernel. 323 Some studies have started to analyse the performance of Multipath TCP 324 on smartphones with real applications. In contrast with the bulk 325 transfers that are used by many publications, real applications do 326 not exchange huge amounts of data and establish a large number of 327 small connections. [COMMAG2016] proposes a software testing 328 framework that allows to automate Android applications to study their 329 interactions with Multipath TCP. [PAM2016] analyses a one-month 330 packet trace of all the packets exchanged by a dozen of smartphones 331 used by regular users. This analysis reveals that short connections 332 are important on smartphones and that the main benefit of using 333 Multipath TCP on smartphones is the ability to perform seamless 334 handovers between different wireless networks. Long connections 335 benefit from these handovers. 337 2.3. Multipath TCP proxies 339 As Multipath TCP is not yet widely deployed on both clients and 340 servers, several deployments have used various forms of proxies. Two 341 families of solutions are currently being used or tested 342 [I-D.deng-mptcp-proxy]. 344 A first use case is when a Multipath TCP enabled client wants to use 345 several interfaces to reach a regular TCP server. A typical use case 346 is a smartphone that needs to use both its WiFi and its cellular 347 interface to transfer data. Several types of proxies are possible 348 for this use case. An HTTP proxy deployed on a Multipath TCP capable 349 server would enable the smartphone to use Multipath TCP to access 350 regular web servers. Obviously, this solution only works for 351 applications that rely on HTTP. Another possibility is to use a 352 proxy that can convert any Multipath TCP connection into a regular 353 TCP connection. Multipath TCP-specific proxies have been proposed 354 [I-D.wei-mptcp-proxy-mechanism] [HotMiddlebox13b] 355 [I-D.hampel-mptcp-proxies-anchors]. 357 Another possibility leverages the SOCKS protocol [RFC1928]. SOCKS is 358 often used in enterprise networks to allow clients to reach external 359 servers. For this, the client opens a TCP connection to the SOCKS 360 server that relays it to the final destination. If both the client 361 and the SOCKS server use Multipath TCP, but not the final 362 destination, then Multipath TCP can still be used on the path between 363 the client and the SOCKS server. At IETF'93, Korea Telecom announced 364 that they have deployed in June 2015 a commercial service that uses 365 Multipath TCP on smartphones. These smartphones access regular TCP 366 servers through a SOCKS proxy. This enables them to achieve 367 throughputs of up to 850 Mbps [KT]. 369 Measurements performed with Android smartphones [Mobicom15] show that 370 popular applications work correctly through a SOCKS proxy and 371 Multipath TCP enabled smartphones. Thanks to Multipath TCP, long- 372 lived connections can be spread over the two available interfaces. 373 However, for short-lived connections, most of the data is sent over 374 the initial subflow that is created over the interface corresponding 375 to the default route and the second subflow is almost not used 376 [PAM2016]. 378 A second use case is when Multipath TCP is used by middleboxes, 379 typically inside access networks. Various network operators are 380 discussing and evaluating solutions for hybrid access networks 381 [BBF-WT348]. Such networks arise when a network operator controls 382 two different access network technologies, e.g. wired and cellular, 383 and wants to combine them to improve the bandwidth offered to the 384 endusers [I-D.lhwxz-hybrid-access-network-architecture]. Several 385 solutions are currently investigated for such networks [BBF-WT348]. 386 Figure 2 shows the organisation of such a network. When a client 387 creates a normal TCP connection, it is intercepted by the Hybrid CPE 388 (HPCE) that converts it in a Multipath TCP connection so that it can 389 use the available access networks (DSL and LTE in the example). The 390 Hybrid Access Gateway (HAG) does the opposite to ensure that the 391 regular server sees a normal TCP connection. Some of the solutions 392 that are currently discussed for hybrid networks use Multipath TCP on 393 the HCPE and the HAG. Other solutions rely on tunnels between the 394 HCPE and the HAG [I-D.lhwxz-gre-notifications-hybrid-access]. 396 client --- HCPE ------ DSL ------- HAG --- internet --- server 397 | | 398 +------- LTE -----------+ 400 Figure 2: Hybrid Access Network 402 3. Operational Experience 404 3.1. Middlebox interference 406 The interference caused by various types of middleboxes has been an 407 important concern during the design of the Multipath TCP protocol. 408 Three studies on the interactions between Multipath TCP and 409 middleboxes are worth discussing. 411 The first analysis appears in [IMC11]. This paper was the main 412 motivation for Multipath TCP incorporating various techniques to cope 413 with middlebox interference. More specifically, Multipath TCP has 414 been designed to cope with middleboxes that : 416 o change source or destination addresses 418 o change source or destination port numbers 420 o change TCP sequence numbers 422 o split or coalesce segments 424 o remove TCP options 426 o modify the payload of TCP segments 428 These middlebox interferences have all been included in the MBtest 429 suite [MBTest]. This test suite is used in [HotMiddlebox13] to 430 verify the reaction of the Multipath TCP implementation in the Linux 431 kernel when faced with middlebox interference. The test environment 432 used for this evaluation is a dual-homed client connected to a 433 single-homed server. The middlebox behavior can be activated on any 434 of the paths. The main results of this analysis are : 436 o the Multipath TCP implementation in the Linux kernel is not 437 affected by a middlebox that performs NAT or modifies TCP sequence 438 numbers 440 o when a middlebox removes the MP_CAPABLE option from the initial 441 SYN segment, the Multipath TCP implementation in the Linux kernel 442 falls back correctly to regular TCP 444 o when a middlebox removes the DSS option from all data segments, 445 the Multipath TCP implementation in the Linux kernel falls back 446 correctly to regular TCP 448 o when a middlebox performs segment coalescing, the Multipath TCP 449 implementation in the Linux kernel is still able to accurately 450 extract the data corresponding to the indicated mapping 452 o when a middlebox performs segment splitting, the Multipath TCP 453 implementation in the Linux kernel correctly reassembles the data 454 corresponding to the indicated mapping. [HotMiddlebox13] shows on 455 figure 4 in section 3.3 a corner case with segment splitting that 456 may lead to a desynchronisation between the two hosts. 458 The interactions between Multipath TCP and real deployed middleboxes 459 is also analyzed in [HotMiddlebox13] and a particular scenario with 460 the FTP application level gateway running on a NAT is described. 462 Middlebox interference can also be detected by analysing packet 463 traces on Multipath TCP enabled servers. A closer look at the 464 packets received on the multipath-tcp.org server [TMA2015] shows that 465 among the 184,000 Multipath TCP connections, only 125 of them were 466 falling back to regular TCP. These connections originated from 28 467 different client IP addresses. These include 91 HTTP connections and 468 34 FTP connections. The FTP interference is expected and due to 469 Application Level Gateways running home routers. The HTTP 470 interference appeared only on the direction from server to client and 471 could have been caused by transparent proxies deployed in cellular or 472 enterprise networks. A longer trace is discussed in [COMCOM2016] and 473 similar conclusions about the middlebox interference are provided. 475 From an operational viewpoint, knowing that Multipath TCP can cope 476 with various types of middlebox interference is important. However, 477 there are situations where the network operators need to gather 478 information about where a particular middlebox interference occurs. 479 The tracebox software [tracebox] described in [IMC13a] is an 480 extension of the popular traceroute software that enables network 481 operators to check at which hop a particular field of the TCP header 482 (including options) is modified. It has been used by several network 483 operators to debug various middlebox interference problems. tracebox 484 includes a scripting language that enables its user to specify 485 precisely which packet (including IP and TCP options) is sent by the 486 source. tracebox sends packets with an increasing TTL/HopLimit and 487 compares the information returned in the ICMP messages with the 488 packet that it sent. This enables tracebox to detect any 489 interference caused by middleboxes on a given path. tracebox works 490 better when routers implement the ICMP extension defined in 491 [RFC1812]. 493 Users of the Multipath TCP implementation have reported some 494 experience with middlebox interference. The strangest scenario has 495 been a middlebox that accepts the Multipath TCP options in the SYN 496 segment but later replaces Multipath TCP options with a TCP EOL 497 option [StrangeMbox]. This causes Multipath TCP to perform a 498 fallback to regular TCP without any impact on the application. 500 3.2. Congestion control 502 Congestion control has been an important problem for Multipath TCP. 503 The standardised congestion control scheme for Multipath TCP is 504 defined in [RFC6356] and [NSDI11]. This congestion control scheme 505 has been implemented in the Linux implementation of Multipath TCP. 506 Linux uses a modular architecture to support various congestion 507 control schemes. This architecture is applicable for both regular 508 TCP and Multipath TCP. While the coupled congestion control scheme 509 defined in [RFC6356] is the default congestion control scheme in the 510 Linux implementation, other congestion control schemes have been 511 added. The second congestion control scheme is OLIA [CONEXT12]. 512 This congestion control scheme is also an adaptation of the NewReno 513 single path congestion control scheme to support multiple paths. 514 Simulations and measurements have shown that it provides some 515 performance benefits compared to the the default congestion control 516 scheme [CONEXT12]. Measurements over a wide range of parameters 517 reported in [CONEXT13] also indicate some benefits with the OLIA 518 congestion control scheme. Recently, a delay-based congestion 519 control scheme has been ported to the Multipath TCP implementation in 520 the Linux kernel. This congestion control scheme has been evaluated 521 by using simulations in [ICNP12]. The fourth congestion control 522 scheme that has been included in the Linux implementation of 523 Multipath TCP is the BALIA scheme 524 [I-D.walid-mptcp-congestion-control]. 526 These different congestion control schemes have been compared in 527 several articles. [CONEXT13] and [PaaschPhD] compare these 528 algorithms in an emulated environment. The evaluation showed that 529 the delay-based congestion control scheme is less able to efficiently 530 use the available links than the three other schemes. Reports from 531 some users indicate that they seem to favor OLIA. 533 3.3. Subflow management 535 The multipath capability of Multipath TCP comes from the utilisation 536 of one subflow per path. The Multipath TCP architecture [RFC6182] 537 and the protocol specification [RFC6824] define the basic usage of 538 the subflows and the protocol mechanisms that are required to create 539 and terminate them. However, there are no guidelines on how subflows 540 are used during the lifetime of a Multipath TCP session. Most of the 541 published experiments with Multipath TCP have been performed in 542 controlled environments. Still, based on the experience running them 543 and discussions on the mptcp-dev mailing list, interesting lessons 544 have been learned about the management of these subflows. 546 From a subflow viewpoint, the Multipath TCP protocol is completely 547 symmetrical. Both the clients and the server have the capability to 548 create subflows. However in practice the existing Multipath TCP 549 implementations [I-D.eardley-mptcp-implementations-survey] have opted 550 for a strategy where only the client creates new subflows. The main 551 motivation for this strategy is that often the client resides behind 552 a NAT or a firewall, preventing passive subflow openings on the 553 client. Although there are environments such as datacenters where 554 this problem does not occur, as of this writing, no precise 555 requirement has emerged for allowing the server to create new 556 subflows. 558 3.4. Implemented subflow managers 560 The Multipath TCP implementation in the Linux kernel includes several 561 strategies to manage the subflows that compose a Multipath TCP 562 session. The basic subflow manager is the full-mesh. As the name 563 implies, it creates a full-mesh of subflows between the communicating 564 hosts. 566 The most frequent use case for this subflow manager is a multihomed 567 client connected to a single-homed server. In this case, one subflow 568 is created for each interface on the client. The current 569 implementation of the full-mesh subflow manager is static. The 570 subflows are created immediately after the creation of the initial 571 subflow. If one subflow fails during the lifetime of the Multipath 572 TCP session (e.g. due to excessive retransmissions, or the loss of 573 the corresponding interface), it is not always reestablished. There 574 is ongoing work to enhance the full-mesh path manager to deal with 575 such events. 577 When the server is multihomed, using the full-mesh subflow manager 578 may lead to a large number of subflows being established. For 579 example, consider a dual-homed client connected to a server with 580 three interfaces. In this case, even if the subflows are only 581 created by the client, 6 subflows will be established. This may be 582 excessive in some environments, in particular when the client and/or 583 the server have a large number of interfaces. A recent draft has 584 proposed a Multipath TCP option to negotiate the maximum number of 585 subflows. However, it should be noted that there have been reports 586 on the mptcp-dev mailing indicating that users rely on Multipath TCP 587 to aggregate more than four different interfaces. Thus, there is a 588 need for supporting many interfaces efficiently. 590 Creating subflows between multihomed clients and servers may 591 sometimes lead to operational issues as observed by discussions on 592 the mptcp-dev mailing list. In some cases the network operators 593 would like to have a better control on how the subflows are created 594 by Multipath TCP [I-D.boucadair-mptcp-max-subflow]. This might 595 require the definition of policy rules to control the operation of 596 the subflow manager. The two scenarios below illustrate some of 597 these requirements. 599 host1 ---------- switch1 ----- host2 600 | | | 601 +-------------- switch2 --------+ 603 Figure 3: Simple switched network topology 605 Consider the simple network topology shown in Figure 3. From an 606 operational viewpoint, a network operator could want to create two 607 subflows between the communicating hosts. From a bandwidth 608 utilization viewpoint, the most natural paths are host1-switch1-host2 609 and host1-switch2-host2. However, a Multipath TCP implementation 610 running on these two hosts may sometimes have difficulties to obtain 611 this result. 613 To understand the difficulty, let us consider different allocation 614 strategies for the IP addresses. A first strategy is to assign two 615 subnets : subnetA (resp. subnetB) contains the IP addresses of 616 host1's interface to switch1 (resp. switch2) and host2's interface to 617 switch1 (resp. switch2). In this case, a Multipath TCP subflow 618 manager should only create one subflow per subnet. To enforce the 619 utilization of these paths, the network operator would have to 620 specify a policy that prefers the subflows in the same subnet over 621 subflows between addresses in different subnets. It should be noted 622 that the policy should probably also specify how the subflow manager 623 should react when an interface or subflow fails. 625 A second strategy is to use a single subnet for all IP addresses. In 626 this case, it becomes more difficult to specify a policy that 627 indicates which subflows should be established. 629 The second subflow manager that is currently supported by the 630 Multipath TCP implementation in the Linux kernel is the ndiffport 631 subflow manager. This manager was initially created to exploit the 632 path diversity that exists between single-homed hosts due to the 633 utilization of flow-based load balancing techniques [SIGCOMM11]. 634 This subflow manager creates N subflows between the same pair of IP 635 addresses. The N subflows are created by the client and differ only 636 in the source port selected by the client. It was not designed to be 637 used on multihomed hosts. 639 A more flexible subflow manager has been proposed, implemented and 640 evaluated in [CONEXT15]. This subflow manager exposes various kernel 641 events to a user space daemon that decides when subflows need to be 642 created and terminated based on various policies. 644 3.5. Subflow destination port 646 The Multipath TCP protocol relies on the token contained in the 647 MP_JOIN option to associate a subflow to an existing Multipath TCP 648 session. This implies that there is no restriction on the source 649 address, destination address and source or destination ports used for 650 the new subflow. The ability to use different source and destination 651 addresses is key to support multihomed servers and clients. The 652 ability to use different destination port numbers is worth discussing 653 because it has operational implications. 655 For illustration, consider a dual-homed client that creates a second 656 subflow to reach a single-homed server as illustrated in Figure 4. 658 client ------- r1 --- internet --- server 659 | | 660 +----------r2-------+ 662 Figure 4: Multihomed-client connected to single-homed server 664 When the Multipath TCP implementation in the Linux kernel creates the 665 second subflow it uses the same destination port as the initial 666 subflow. This choice is motivated by the fact that the server might 667 be protected by a firewall and only accept TCP connections (including 668 subflows) on the official port number. Using the same destination 669 port for all subflows is also useful for operators that rely on the 670 port numbers to track application usage in their network. 672 There have been suggestions from Multipath TCP users to modify the 673 implementation to allow the client to use different destination ports 674 to reach the server. This suggestion seems mainly motivated by 675 traffic shaping middleboxes that are used in some wireless networks. 676 In networks where different shaping rates are associated to different 677 destination port numbers, this could allow Multipath TCP to reach a 678 higher performance. As of this writing, we are not aware of any 679 implementation of this kind of tweaking. 681 However, from an implementation point-of-view supporting different 682 destination ports for the same Multipath TCP connection can cause 683 some issues. A legacy implementation of a TCP stack creates a 684 listening socket to react upon incoming SYN segments. The listening 685 socket is handling the SYN segments that are sent on a specific port 686 number. Demultiplexing incoming segments can thus be done solely by 687 looking at the IP addresses and the port numbers. With Multipath TCP 688 however, incoming SYN segments may have an MP_JOIN option with a 689 different destination port. This means, that all incoming segments 690 that did not match on an existing listening-socket or an already 691 established socket must be parsed for an eventual MP_JOIN option. 692 This imposes an additional cost on servers, previously not existent 693 on legacy TCP implementations. 695 3.6. Closing subflows 697 client server 698 | | 699 MPTCP: established | | MPTCP: established 700 Sub: established | | Sub: established 701 | | 702 | DATA_FIN | 703 MPTCP: close-wait | <------------------------ | close() (step 1) 704 Sub: established | DATA_ACK | 705 | ------------------------> | MPTCP: fin-wait-2 706 | | Sub: established 707 | | 708 | DATA_FIN + subflow-FIN | 709 close()/shutdown() | ------------------------> | MPTCP: time-wait 710 (step 2) | DATA_ACK | Sub: close-wait 711 MPTCP: closed | <------------------------ | 712 Sub: fin-wait-2 | | 713 | | 714 | subflow-FIN | 715 MPTCP: closed | <------------------------ | subflow-close() 716 Sub: time-wait | subflow-ACK | 717 (step 3) | ------------------------> | MPTCP: time-wait 718 | | Sub: closed 719 | | 721 Figure 5: Multipath TCP may not be able to avoid time-wait state on 722 the subflow (indicated as "Sub 724 Figure 5 shows a very particular issue within Multipath TCP. Many 725 high-performance applications try to avoid Time-Wait state by 726 deferring the closure of the connection until the peer has sent a 727 FIN. That way, the client on the left of Figure 5 does a passive 728 closure of the connection, transitioning from Close-Wait to Last-ACK 729 and finally freeing the resources after reception of the ACK of the 730 FIN. An application running on top of a Multipath TCP enabled Linux 731 kernel might also use this approach. The difference here is that the 732 close() of the connection (Step 1 in Figure 5) only triggers the 733 sending of a DATA_FIN. Nothing guarantees that the kernel is ready 734 to combine the DATA_FIN with a subflow-FIN. The reception of the 735 DATA_FIN will make the application trigger the closure of the 736 connection (step 2), trying to avoid Time-Wait state with this late 737 closure. This time, the kernel might decide to combine the DATA_FIN 738 with a subflow-FIN. This decision will be fatal, as the subflow's 739 state machine will not transition from Close-Wait to Last-Ack, but 740 rather go through Fin-Wait-2 into Time-Wait state. The Time-Wait 741 state will consume resources on the host for at least 2 MSL (Maximum 742 Segment Lifetime). Thus, a smart application that tries to avoid 743 Time-Wait state by doing late closure of the connection actually ends 744 up with one of its subflows in Time-Wait state. A high-performance 745 Multipath TCP kernel implementation should honor the desire of the 746 application to do passive closure of the connection and successfully 747 avoid Time-Wait state - even on the subflows. 749 The solution to this problem lies in an optimistic assumption that a 750 host doing active-closure of a Multipath TCP connection by sending a 751 DATA_FIN will soon also send a FIN on all its subflows. Thus, the 752 passive closer of the connection can simply wait for the peer to send 753 exactly this FIN - enforcing passive closure even on the subflows. 754 Of course, to avoid consuming resources indefinitely, a timer must 755 limit the time our implementation waits for the FIN. 757 3.7. Packet schedulers 759 In a Multipath TCP implementation, the packet scheduler is the 760 algorithm that is executed when transmitting each packet to decide on 761 which subflow it needs to be transmitted. The packet scheduler 762 itself does not have any impact on the interoperability of Multipath 763 TCP implementations. However, it may clearly impact the performance 764 of Multipath TCP sessions. The Multipath TCP implementation in the 765 Linux kernel supports a pluggable architecture for the packet 766 scheduler [PaaschPhD]. As of this writing, two schedulers have been 767 implemented: round-robin and lowest-rtt-first. The second scheduler 768 relies on the round-trip-time (rtt) measured on each TCP subflow and 769 sends first segments over the subflow having the lowest round-trip- 770 time. They are compared in [CSWS14]. The experiments and 771 measurements described in [CSWS14] show that the lowest-rtt-first 772 scheduler appears to be the best compromise from a performance 773 viewpoint. Another study of the packet schedulers is presented in 774 [PAMS2014]. This study relies on simulations with the Multipath TCP 775 implementation in the Linux kernel. They compare the lowest-rtt- 776 first with the round-robin and a random scheduler. They show some 777 situations where the lowest-rtt-first scheduler does not perform as 778 well as the other schedulers, but there are many scenarios where the 779 opposite is true. [PAMS2014] notes that "it is highly likely that 780 the optimal scheduling strategy depends on the characteristics of the 781 paths being used." 783 3.8. Segment size selection 785 When an application performs a write/send system call, the kernel 786 allocates a packet buffer (sk_buff in Linux) to store the data the 787 application wants to send. The kernel will store at most one MSS 788 (Maximum Segment Size) of data per buffer. As the MSS can differ 789 amongst subflows, an MPTCP implementation must select carefully the 790 MSS used to generate application data. The Linux kernel 791 implementation had various ways of selecting the MSS: minimum or 792 maximum amongst the different subflows. However, these heuristics of 793 MSS selection can cause significant performance issues in some 794 environment. Consider the following example. An MPTCP connection 795 has two established subflows that respectively use a MSS of 1420 and 796 1428 bytes. If MPTCP selects the maximum, then the application will 797 generate segments of 1428 bytes of data. An MPTCP implementation 798 will have to split the segment in two (a 1420-byte and 8-byte 799 segments) when pushing on the subflow with the smallest MSS. The 800 latter segment will introduce a large overhead as for a single data 801 segment 2 slots will be used in the congestion window (in packets) 802 therefore reducing by roughly twice the potential throughput (in 803 bytes/s) of this subflow. Taking the smallest MSS does not solve the 804 issue as there might be a case where the subflow with the smallest 805 MSS only sends a few packets therefore reducing the potential 806 throughput of the other subflows. 808 The Linux implementation recently took another approach [DetalMSS]. 809 Instead of selecting the minimum and maximum values, it now 810 dynamically adapts the MSS based on the contribution of all the 811 subflows to the connection's throughput. For this it computes, for 812 each subflow, the potential throughput achieved by selecting each MSS 813 value and by taking into account the lost space in the cwnd. It then 814 selects the MSS that allows to achieve the highest potential 815 throughput. 817 3.9. Interactions with the Domain Name System 819 Multihomed clients such as smartphones can send DNS queries over any 820 of their interfaces. When a single-homed client performs a DNS 821 query, it receives from its local resolver the best answer for its 822 request. If the client is multihomed, the answer returned to the DNS 823 query may vary with the interface over which it has been sent. 825 cdn1 826 | 827 client -- cellular -- internet -- cdn3 828 | | 829 +----- wifi --------+ 830 | 831 cdn2 833 Figure 6: Simple network topology 835 If the client sends a DNS query over the WiFi interface, the answer 836 will point to the cdn2 server while the same request sent over the 837 cellular interface will point to the cdn1 server. This might cause 838 problems for CDN providers that locate their servers inside ISP 839 networks and have contracts that specify that the CDN server will 840 only be accessed from within this particular ISP. Assume now that 841 both the client and the CDN servers support Multipath TCP. In this 842 case, a Multipath TCP session from cdn1 or cdn2 would potentially use 843 both the cellular network and the WiFi network. Serving the client 844 from cdn2 over the cellular interface could violate the contract 845 between the CDN provider and the network operators. A similar 846 problem occurs with regular TCP if the client caches DNS replies. 847 For example the client obtains a DNS answer over the cellular 848 interface and then stops this interface and starts to use its WiFi 849 interface. If the client retrieves data from cdn1 over its WiFi 850 interface, this may also violate the contract between the CDN and the 851 network operators. 853 A possible solution to prevent this problem would be to modify the 854 DNS resolution on the client. The client subnet EDNS extension 855 defined in [I-D.ietf-dnsop-edns-client-subnet] could be used for this 856 purpose. When the client sends a DNS query from its WiFi interface, 857 it should also send the client subnet corresponding to the cellular 858 interface in this request. This would indicate to the resolver that 859 the answer should be valid for both the WiFi and the cellular 860 interfaces (e.g., the cdn3 server). 862 3.10. Captive portals 864 Multipath TCP enables a host to use different interfaces to reach a 865 server. In theory, this should ensure connectivity when at least one 866 of the interfaces is active. In practice however, there are some 867 particular scenarios with captive portals that may cause operational 868 problems. The reference environment is shown in Figure 7. 870 client ----- network1 871 | 872 +------- internet ------------- server 874 Figure 7: Issue with captive portal 876 The client is attached to two networks : network1 that provides 877 limited connectivity and the entire Internet through the second 878 network interface. In practice, this scenario corresponds to an open 879 WiFi network with a captive portal for network1 and a cellular 880 service for the second interface. On many smartphones, the WiFi 881 interface is preferred over the cellular interface. If the 882 smartphone learns a default route via both interfaces, it will 883 typically prefer to use the WiFi interface to send its DNS request 884 and create the first subflow. This is not optimal with Multipath 885 TCP. A better approach would probably be to try a few attempts on 886 the WiFi interface and then try to use the second interface for the 887 initial subflow as well. 889 3.11. Stateless webservers 891 MPTCP has been designed to interoperate with webservers that benefit 892 from SYN-cookies to protect against SYN-flooding attacks [RFC4987]. 893 MPTCP achieves this by echoing the keys negotiated during the 894 MP_CAPABLE handshake in the third ACK of the 3-way handshake. 895 Reception of this third ACK then allows the server to reconstruct the 896 state specific to MPTCP. 898 However, one caveat to this mechanism is the non-reliable nature of 899 the third ACK. Indeed, when the third ACK gets lost, the server will 900 not be able to reconstruct the MPTCP-state. MPTCP will fallback to 901 regular TCP in this case. This is in contrast to regular TCP. When 902 the client starts sending data, the first data segment also includes 903 the SYN-cookie, which allows the server to reconstruct the TCP-state. 904 Further, this data segment will be retransmitted by the client in 905 case it gets lost and thus is resilient against loss. MPTCP does not 906 include the keys in this data segment and thus the server cannot 907 reconstruct the MPTCP state. 909 This issue might be considered as a minor one for MPTCP. Losing the 910 third ACK should only happen when packet loss is high. However, when 911 packet-loss is high MPTCP provides a lot of benefits as it can move 912 traffic away from the lossy link. It is undesirable that MPTCP has a 913 higher chance to fall back to regular TCP in those lossy 914 environments. 916 [I-D.paasch-mptcp-syncookies] discusses this issue and suggests a 917 modified handshake mechanism that ensures reliable delivery of the 918 MP_CAPABLE, following the 3-way handshake. This modification will 919 make MPTCP reliable, even in lossy environments when servers need to 920 use SYN-cookies to protect against SYN-flooding attacks. 922 3.12. Loadbalanced serverfarms 924 Large-scale serverfarms typically deploy thousands of servers behind 925 a single virtual IP (VIP). Steering traffic to these servers is done 926 through layer-4 loadbalancers that ensure that a TCP-flow will always 927 be routed to the same server [Presto08]. 929 As Multipath TCP uses multiple different TCP subflows to steer the 930 traffic across the different paths, loadbalancers need to ensure that 931 all these subflows are routed to the same server. This implies that 932 the loadbalancers need to track the MPTCP-related state, allowing 933 them to parse the token in the MP_JOIN and assign those subflows to 934 the appropriate server. However, serverfarms typically deploy 935 multiple of these loadbalancers for reliability and capacity reasons. 936 As a TCP subflow might get routed to any of these loadbalancers, they 937 would need to synchronize the MPTCP-related state - a solution that 938 is not feasible at large scale. 940 The token (carried in the MP_JOIN) contains the information 941 indicating which MPTCP-session the subflow belongs to. As the token 942 is a hash of the key, servers are not able to generate the token in 943 such a way that the token can provide the necessary information to 944 the loadbalancers which would allow them to route TCP subflows to the 945 appropriate server. [I-D.paasch-mptcp-loadbalancer] discusses this 946 issue in detail and suggests two alternative MP_CAPABLE handshakes to 947 overcome these. As of September 2015, it is not yet clear how MPTCP 948 might accomodate such use-case to enable its deployment within 949 loadbalanced serverfarms. 951 4. IANA Considerations 953 There are no IANA considerations in this informational document. 955 5. Security Considerations 957 The security considerations for Multipath TCP have already been 958 documented in [RFC6181], [RFC6182], [RFC6824] and [RFC7430]. 960 6. Conclusion 962 In this document, we have documented a few years of experience with 963 Multipath TCP. The different scientific publications that have been 964 summarised confirm that Multipath TCP works well in different use 965 cases in today's Internet. None of the cited publications has 966 identified major issues with Multipath TCP and its utilisation in the 967 current Internet. Some of these publications list directions for 968 future improvements that mainly affect the subflow managers and 969 packet schedulers. These heuristics affect the performance of 970 Multipath TCP, but not the protocol itself. It is likely that these 971 improvements will be discussed in future IETF documents. 973 Besides the published scientific literature, a number of companies 974 have deployed Multipath TCP at large. One of these deployments uses 975 Multipath TCP on the client and the server side, making it a true 976 end-to-end deployment. This deployment uses Multipath TCP to support 977 fast handover between cellular and WiFi networks. A wider deployment 978 of Multipath TCP on servers seems to be blocked by the necessity to 979 support Multipath TCP on load balancers. Given the influence that 980 middleboxes had on the design of Multipath TCP, it is interesting to 981 note that the other industrial deployments use Multipath TCP inside 982 middleboxes. These middelboxes use Multipath TCP to efficiently 983 combine several access links while still interacting with legacy TCP 984 servers. 986 7. Acknowledgements 988 This work was partially supported by the FP7-Trilogy2 project. We 989 would like to thank all the implementers and users of the Multipath 990 TCP implementation in the Linux kernel. This document has benefited 991 from the comments of John Ronan, Yoshifumi Nishida, Phil Eardley and 992 Jaehyun Hwang. 994 8. Informative References 996 [Apple-MPTCP] 997 Apple, Inc, ., "iOS - Multipath TCP Support in iOS 7", 998 n.d., . 1000 [BBF-WT348] 1001 Fabregas (Ed), G., "WT-348 - Hybrid Access for Broadband 1002 Networks", Broadband Forum, contribution bbf2014.1139.04 , 1003 June 2015. 1005 [CACM14] Paasch, C. and O. Bonaventure, "Multipath TCP", 1006 Communications of the ACM, 57(4):51-57 , April 2014, 1007 . 1009 [COMCOM2016] 1010 "Observing real Multipath TCP traffic", Computer 1011 Communications , April 2016, . 1014 [COMMAG2016] 1015 De Coninck, Q., Baerts, M., Hesmans, B., and O. 1016 Bonaventure, "Observing Real Smartphone Applications over 1017 Multipath TCP", IEEE Communications Magazine , March 2016, 1018 . 1022 [CONEXT12] 1023 Khalili, R., Gast, N., Popovic, M., Upadhyay, U., and J. 1024 Leboudec, "MPTCP is not pareto-optimal performance issues 1025 and a possible solution", Proceedings of the 8th 1026 international conference on Emerging networking 1027 experiments and technologies (CoNEXT12) , 2012. 1029 [CONEXT13] 1030 Paasch, C., Khalili, R., and O. Bonaventure, "On the 1031 Benefits of Applying Experimental Design to Improve 1032 Multipath TCP", Conference on emerging Networking 1033 EXperiments and Technologies (CoNEXT) , December 2013, . 1038 [CONEXT15] 1039 Hesmans, B., Detal, G., Barre, S., Bauduin, R., and O. 1040 Bonaventure, "SMAPP - Towards Smart Multipath TCP-enabled 1041 APPlications", Proc. Conext 2015, Heidelberg, Germany , 1042 December 2015, . 1045 [CSWS14] Paasch, C., Ferlin, S., Alay, O., and O. Bonaventure, 1046 "Experimental Evaluation of Multipath TCP Schedulers", 1047 SIGCOMM CSWS2014 workshop , August 2014. 1049 [Cellnet12] 1050 Paasch, C., Detal, G., Duchene, F., Raiciu, C., and O. 1051 Bonaventure, "Exploring Mobile/WiFi Handover with 1052 Multipath TCP", ACM SIGCOMM workshop on Cellular Networks 1053 (Cellnet12) , 2012, . 1056 [DetalMSS] 1057 Detal, G., "Adaptive MSS value", Post on the mptcp-dev 1058 mailing list , September 2014, . 1062 [FreeBSD-MPTCP] 1063 Williams, N., "Multipath TCP For FreeBSD Kernel Patch 1064 v0.5", n.d., . 1066 [HotMiddlebox13] 1067 Hesmans, B., Duchene, F., Paasch, C., Detal, G., and O. 1068 Bonaventure, "Are TCP Extensions Middlebox-proof?", CoNEXT 1069 workshop HotMiddlebox , December 2013, . 1073 [HotMiddlebox13b] 1074 Detal, G., Paasch, C., and O. Bonaventure, "Multipath in 1075 the Middle(Box)", HotMiddlebox'13 , December 2013, . 1078 [HotNets] Raiciu, C., Pluntke, C., Barre, S., Greenhalgh, A., 1079 Wischik, D., and M. Handley, "Data center networking with 1080 multipath TCP", Proceedings of the 9th ACM SIGCOMM 1081 Workshop on Hot Topics in Networks (Hotnets-IX) , 2010, 1082 . 1084 [I-D.boucadair-mptcp-max-subflow] 1085 Boucadair, M. and C. Jacquenet, "Negotiating the Maximum 1086 Number of Multipath TCP (MPTCP) Subflows", 1087 draft-boucadair-mptcp-max-subflow-02 (work in progress), 1088 May 2016. 1090 [I-D.deng-mptcp-proxy] 1091 Lingli, D., Liu, D., Sun, T., Boucadair, M., and G. 1092 Cauchie, "Use-cases and Requirements for MPTCP Proxy in 1093 ISP Networks", draft-deng-mptcp-proxy-01 (work in 1094 progress), October 2014. 1096 [I-D.eardley-mptcp-implementations-survey] 1097 Eardley, P., "Survey of MPTCP Implementations", 1098 draft-eardley-mptcp-implementations-survey-02 (work in 1099 progress), July 2013. 1101 [I-D.hampel-mptcp-proxies-anchors] 1102 Hampel, G. and T. Klein, "MPTCP Proxies and Anchors", 1103 draft-hampel-mptcp-proxies-anchors-00 (work in progress), 1104 February 2012. 1106 [I-D.ietf-dnsop-edns-client-subnet] 1107 Contavalli, C., Gaast, W., tale, t., and W. Kumari, 1108 "Client Subnet in DNS Queries", 1109 draft-ietf-dnsop-edns-client-subnet-08 (work in progress), 1110 April 2016. 1112 [I-D.lhwxz-gre-notifications-hybrid-access] 1113 Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M. 1114 Zhang, "GRE Notifications for Hybrid Access", 1115 draft-lhwxz-gre-notifications-hybrid-access-01 (work in 1116 progress), January 2015. 1118 [I-D.lhwxz-hybrid-access-network-architecture] 1119 Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M. 1120 Zhang, "Hybrid Access Network Architecture", 1121 draft-lhwxz-hybrid-access-network-architecture-02 (work in 1122 progress), January 2015. 1124 [I-D.paasch-mptcp-loadbalancer] 1125 Paasch, C., Greenway, G., and A. Ford, "Multipath TCP 1126 behind Layer-4 loadbalancers", 1127 draft-paasch-mptcp-loadbalancer-00 (work in progress), 1128 September 2015. 1130 [I-D.paasch-mptcp-syncookies] 1131 Paasch, C., Biswas, A., and D. Haas, "Making Multipath TCP 1132 robust for stateless webservers", 1133 draft-paasch-mptcp-syncookies-02 (work in progress), 1134 October 2015. 1136 [I-D.walid-mptcp-congestion-control] 1137 Walid, A., Peng, Q., Hwang, J., and S. Low, "Balanced 1138 Linked Adaptation Congestion Control Algorithm for MPTCP", 1139 draft-walid-mptcp-congestion-control-04 (work in 1140 progress), January 2016. 1142 [I-D.wei-mptcp-proxy-mechanism] 1143 Wei, X., Xiong, C., and E. Ed, "MPTCP proxy mechanisms", 1144 draft-wei-mptcp-proxy-mechanism-02 (work in progress), 1145 June 2015. 1147 [ICNP12] Cao, Y., Xu, M., and X. Fu, "Delay-based congestion 1148 control for multipath TCP", 20th IEEE International 1149 Conference on Network Protocols (ICNP) , 2012. 1151 [IMC11] Honda, M., Nishida, Y., Raiciu, C., Greenhalgh, A., 1152 Handley, M., and H. Tokuda, "Is it still possible to 1153 extend TCP?", Proceedings of the 2011 ACM SIGCOMM 1154 conference on Internet measurement conference (IMC '11) , 1155 2011, . 1157 [IMC13a] Detal, G., Hesmans, B., Bonaventure, O., Vanaubel, Y., and 1158 B. Donnet, "Revealing Middlebox Interference with 1159 Tracebox", Proceedings of the 2013 ACM SIGCOMM conference 1160 on Internet measurement conference , 2013, . 1164 [IMC13b] Chen, Y., Lim, Y., Gibbens, R., Nahum, E., Khalili, R., 1165 and D. Towsley, "A measurement-based study of MultiPath 1166 TCP performance over wireless network", Proceedings of the 1167 2013 conference on Internet measurement conference (IMC 1168 '13) , n.d., . 1170 [IMC13c] Pelsser, C., Cittadini, L., Vissicchio, S., and R. Bush, 1171 "From Paris to Tokyo on the suitability of ping to 1172 measure latency", Proceedings of the 2013 conference on 1173 Internet measurement conference (IMC '13) , 2013, 1174 . 1176 [INFOCOM14] 1177 Lim, Y., Chen, Y., Nahum, E., Towsley, D., and K. Lee, 1178 "Cross-Layer Path Management in Multi-path Transport 1179 Protocol for Mobile Devices", IEEE INFOCOM'14 , 2014. 1181 [IOS7] Apple, ., "Multipath TCP Support in iOS 7", January 2014, 1182 . 1184 [KT] Seo, S., "KT's GiGA LTE", July 2015, . 1187 [MBTest] Hesmans, B., "MBTest", 2013, 1188 . 1190 [MPTCPBIB] 1191 Bonaventure, O., "Multipath TCP - An annotated 1192 bibliography", Technical report , April 2015, 1193 . 1195 [Mobicom15] 1196 De Coninck, Q., Baerts, M., Hesmans, B., and O. 1197 Bonaventure, "Poster - Evaluating Android Applications 1198 with Multipath TCP", Mobicom 2015 (Poster) , 1199 September 2015. 1201 [MultipathTCP-Linux] 1202 Paasch, C., Barre, S., and . et al, "Multipath TCP 1203 implementation in the Linux kernel", n.d., 1204 . 1206 [NSDI11] Wischik, D., Raiciu, C., Greenhalgh, A., and M. Handley, 1207 "Design, implementation and evaluation of congestion 1208 control for Multipath TCP", In Proceedings of the 8th 1209 USENIX conference on Networked systems design and 1210 implementation (NSDI11) , 2011. 1212 [NSDI12] Raiciu, C., Paasch, C., Barre, S., Ford, A., Honda, M., 1213 Duchene, F., Bonaventure, O., and M. Handley, "How Hard 1214 Can It Be? Designing and Implementing a Deployable 1215 Multipath TCP", USENIX Symposium of Networked Systems 1216 Design and Implementation (NSDI12) , April 2012, . 1221 [PAM2016] De Coninck, Q., Baerts, M., Hesmans, B., and O. 1222 Bonaventure, "A First Analysis of Multipath TCP on 1223 Smartphones", 17th International Passive and Active 1224 Measurements Conference (PAM2016) , March 2016, . 1228 [PAMS2014] 1229 Arzani, B., Gurney, A., Cheng, S., Guerin, R., and B. Loo, 1230 "Impact of Path Selection and Scheduling Policies on MPTCP 1231 Performance", PAMS2014 , 2014. 1233 [PaaschPhD] 1234 Paasch, C., "Improving Multipath TCP", Ph.D. Thesis , 1235 November 2014, . 1238 [Presto08] 1239 Greenberg, A., Lahiri, P., Maltz, D., Parveen, P., and S. 1240 Sengupta, "Towards a Next Generation Data Center 1241 Architecture - Scalability and Commoditization", ACM 1242 PRESTO 2008 , August 2008, 1243 . 1245 [RFC1812] Baker, F., Ed., "Requirements for IP Version 4 Routers", 1246 RFC 1812, DOI 10.17487/RFC1812, June 1995, 1247 . 1249 [RFC1928] Leech, M., Ganis, M., Lee, Y., Kuris, R., Koblas, D., and 1250 L. Jones, "SOCKS Protocol Version 5", RFC 1928, 1251 DOI 10.17487/RFC1928, March 1996, 1252 . 1254 [RFC4987] Eddy, W., "TCP SYN Flooding Attacks and Common 1255 Mitigations", RFC 4987, DOI 10.17487/RFC4987, August 2007, 1256 . 1258 [RFC6181] Bagnulo, M., "Threat Analysis for TCP Extensions for 1259 Multipath Operation with Multiple Addresses", RFC 6181, 1260 DOI 10.17487/RFC6181, March 2011, 1261 . 1263 [RFC6182] Ford, A., Raiciu, C., Handley, M., Barre, S., and J. 1264 Iyengar, "Architectural Guidelines for Multipath TCP 1265 Development", RFC 6182, DOI 10.17487/RFC6182, March 2011, 1266 . 1268 [RFC6356] Raiciu, C., Handley, M., and D. Wischik, "Coupled 1269 Congestion Control for Multipath Transport Protocols", 1270 RFC 6356, DOI 10.17487/RFC6356, October 2011, 1271 . 1273 [RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure, 1274 "TCP Extensions for Multipath Operation with Multiple 1275 Addresses", RFC 6824, DOI 10.17487/RFC6824, January 2013, 1276 . 1278 [RFC7430] Bagnulo, M., Paasch, C., Gont, F., Bonaventure, O., and C. 1279 Raiciu, "Analysis of Residual Threats and Possible Fixes 1280 for Multipath TCP (MPTCP)", RFC 7430, DOI 10.17487/ 1281 RFC7430, July 2015, 1282 . 1284 [SIGCOMM11] 1285 Raiciu, C., Barre, S., Pluntke, C., Greenhalgh, A., 1286 Wischik, D., and M. Handley, "Improving datacenter 1287 performance and robustness with multipath TCP", 1288 Proceedings of the ACM SIGCOMM 2011 conference , n.d., 1289 . 1291 [StrangeMbox] 1292 Bonaventure, O., "Multipath TCP through a strange 1293 middlebox", Blog post , January 2015, . 1297 [TMA2015] Hesmans, B., Tran Viet, H., Sadre, R., and O. Bonaventure, 1298 "A First Look at Real Multipath TCP Traffic", Traffic 1299 Monitoring and Analysis , 2015, . 1303 [ietf88] Stewart, L., "IETF'88 Meeting minutes of the MPTCP working 1304 group", n.d., . 1307 [tracebox] 1308 Detal, G. and O. Tilmans, "tracebox", 2013, 1309 . 1311 Appendix A. Changelog 1313 This section should be removed before final publication 1315 o initial version : September 16th, 2014 : Added section Section 3.8 1316 that discusses some performance problems that appeared with the 1317 Linux implementation when using subflows having different MSS 1318 values 1320 o update with a description of the middlebox that replaces an 1321 unknown TCP option with EOL [StrangeMbox] 1323 o version ietf-02 : July 2015, answer to last call comments 1325 * Reorganised text to better separate use cases and operational 1326 experience 1328 * New use case on Multipath TCP proxies in Section 2.3 1330 * Added some text on middleboxes in Section 3.1 1332 * Removed the discussion on SDN 1334 * Restructured text and improved writing in some parts 1336 o version ietf-03 : September 2015, answer to comments from Phil 1337 Eardley 1339 * Improved introduction 1341 * Added details about using SOCKS and Korea Telecom's use-case in 1342 Section 2.3. 1344 * Added issue around clients caching DNS-results in Section 3.9 1346 * Explained issue of MPTCP with stateless webservers Section 3.11 1348 * Added description of MPTCP's use behind layer-4 loadbalancers 1349 Section 3.12 1351 * Restructured text and improved writing in some parts 1353 o version ietf-04 : April 2016, answer to last comments 1355 * Updated text on measurements with smartphones 1357 * Updated conclusion 1359 Authors' Addresses 1361 Olivier Bonaventure 1362 UCLouvain 1364 Email: Olivier.Bonaventure@uclouvain.be 1366 Christoph Paasch 1367 Apple, Inc. 1369 Email: cpaasch@apple.com 1371 Gregory Detal 1372 Tessares 1374 Email: Gregory.Detal@tessares.net