idnits 2.17.1 draft-ietf-mptcp-experience-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 1114 has weird spacing: '...o Tokyo on th...' -- The document date (August 29, 2016) is 2796 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-03) exists of draft-boucadair-mptcp-max-subflow-02 -- Obsolete informational reference (is this intentional?): RFC 6824 (Obsoleted by RFC 8684) Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 MPTCP Working Group O. Bonaventure 3 Internet-Draft UCLouvain 4 Intended status: Informational C. Paasch 5 Expires: March 2, 2017 Apple, Inc. 6 G. Detal 7 Tessares 8 August 29, 2016 10 Use Cases and Operational Experience with Multipath TCP 11 draft-ietf-mptcp-experience-06 13 Abstract 15 This document discusses both use cases and operational experience 16 with Multipath TCP in real world networks. It lists several 17 prominent use cases for which Multipath TCP has been considered and 18 is being used. It also gives insight to some heuristics and 19 decisions that have helped to realize these use cases. 21 Status of this Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at http://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on March 2, 2017. 38 Copyright Notice 40 Copyright (c) 2016 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents 45 (http://trustee.ietf.org/license-info) in effect on the date of 46 publication of this document. Please review these documents 47 carefully, as they describe your rights and restrictions with respect 48 to this document. Code Components extracted from this document must 49 include Simplified BSD License text as described in Section 4.e of 50 the Trust Legal Provisions and are provided without warranty as 51 described in the Simplified BSD License. 53 Table of Contents 55 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 56 2. Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . 5 57 2.1. Datacenters . . . . . . . . . . . . . . . . . . . . . . . 5 58 2.2. Cellular/WiFi Offload . . . . . . . . . . . . . . . . . . 5 59 2.3. Multipath TCP proxies . . . . . . . . . . . . . . . . . . 9 60 3. Operational Experience . . . . . . . . . . . . . . . . . . . . 11 61 3.1. Middlebox interference . . . . . . . . . . . . . . . . . . 11 62 3.2. Congestion control . . . . . . . . . . . . . . . . . . . . 13 63 3.3. Subflow management . . . . . . . . . . . . . . . . . . . . 13 64 3.4. Implemented subflow managers . . . . . . . . . . . . . . . 14 65 3.5. Subflow destination port . . . . . . . . . . . . . . . . . 16 66 3.6. Closing subflows . . . . . . . . . . . . . . . . . . . . . 17 67 3.7. Packet schedulers . . . . . . . . . . . . . . . . . . . . 18 68 3.8. Segment size selection . . . . . . . . . . . . . . . . . . 19 69 3.9. Interactions with the Domain Name System . . . . . . . . . 19 70 3.10. Captive portals . . . . . . . . . . . . . . . . . . . . . 20 71 3.11. Stateless webservers . . . . . . . . . . . . . . . . . . . 21 72 3.12. Loadbalanced serverfarms . . . . . . . . . . . . . . . . . 22 73 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 74 5. Security Considerations . . . . . . . . . . . . . . . . . . . 24 75 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 25 76 7. Informative References . . . . . . . . . . . . . . . . . . . . 26 77 Appendix A. Changelog . . . . . . . . . . . . . . . . . . . . . . 33 78 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 35 80 1. Introduction 82 Multipath TCP was standardized in [RFC6824] and five independent 83 implementations have been developed. As of September 2015, Multipath 84 TCP has been or is being implemented on the following platforms: 86 o Linux kernel [MultipathTCP-Linux] 88 o Apple iOS and MacOS [Apple-MPTCP] 90 o Citrix load balancers 92 o FreeBSD [FreeBSD-MPTCP] 94 o Oracle 96 The first three implementations are known to interoperate. The last 97 two are currently being tested and improved against the Linux 98 implementation. Three of these implementations are open-source. 99 Apple's implementation is widely deployed. 101 Since the publication of [RFC6824], experience has been gathered by 102 various network researchers and users about the operational issues 103 that arise when Multipath TCP is used in today's Internet. 105 When the MPTCP working group was created, several use cases for 106 Multipath TCP were identified [RFC6182]. Since then, other use cases 107 have been proposed and some have been tested and even deployed. We 108 describe these use cases in Section 2. 110 Section 3 focuses on the operational experience with Multipath TCP. 111 Most of this experience comes from the utilisation of the Multipath 112 TCP implementation in the Linux kernel [MultipathTCP-Linux]. This 113 open-source implementation has been downloaded and is used by 114 thousands of users all over the world. Many of these users have 115 provided direct or indirect feedback by writing documents (scientific 116 articles or blog messages) or posting to the mptcp-dev mailing list 117 (see https://listes-2.sipr.ucl.ac.be/sympa/arc/mptcp-dev ). This 118 Multipath TCP implementation is actively maintained and continuously 119 improved. It is used on various types of hosts, ranging from 120 smartphones or embedded routers to high-end servers. 122 The Multipath TCP implementation in the Linux kernel is not, by far, 123 the most widespread deployment of Multipath TCP. Since September 124 2013, Multipath TCP is also supported on smartphones and tablets 125 running iOS7 [IOS7]. There are likely hundreds of millions of 126 Multipath TCP enabled devices. However, this particular Multipath 127 TCP implementation is currently only used to support a single 128 application. Unfortunately, there is no public information about the 129 lessons learned from this large scale deployment. 131 Section 3 is organized as follows. Supporting the middleboxes was 132 one of the difficult issues in designing the Multipath TCP protocol. 133 We explain in Section 3.1 which types of middleboxes the Linux Kernel 134 implementation of Multipath TCP supports and how it reacts upon 135 encountering these. Section 3.2 summarises the MPTCP specific 136 congestion controls that have been implemented. Section 3.3 to 137 Section 3.7 discuss heuristics and issues with respect to subflow 138 management as well as the scheduling across the subflows. 139 Section 3.8 explains some problems that occurred with subflows having 140 different Maximum Segment Size (MSS) values. Section 3.9 presents 141 issues with respect to content delivery networks and suggests a 142 solution to this issue. Finally, Section 3.10 documents an issue 143 with captive portals where MPTCP will behave suboptimally. 145 2. Use cases 147 Multipath TCP has been tested in several use cases. There is already 148 an abundant scientific literature on Multipath TCP [MPTCPBIB]. 149 Several of the papers published in the scientific literature have 150 identified possible improvements that are worth being discussed here. 152 2.1. Datacenters 154 A first, although initially unexpected, documented use case for 155 Multipath TCP has been in datacenters [HotNets][SIGCOMM11]. Today's 156 datacenters are designed to provide several paths between single- 157 homed servers. The multiplicity of these paths comes from the 158 utilization of Equal Cost Multipath (ECMP) and other load balancing 159 techniques inside the datacenter. Most of the deployed load 160 balancing techniques in datacenters rely on hashes computed over the 161 five tuple. Thus all packets from the same TCP connection follow the 162 same path and so are not reordered. The results in [HotNets] 163 demonstrate by simulations that Multipath TCP can achieve a better 164 utilization of the available network by using multiple subflows for 165 each Multipath TCP session. Although [RFC6182] assumes that at least 166 one of the communicating hosts has several IP addresses, [HotNets] 167 demonstrates that Multipath TCP is beneficial when both hosts are 168 single-homed. This idea is analysed in more details in [SIGCOMM11] 169 where the Multipath TCP implementation in the Linux kernel is 170 modified to be able to use several subflows from the same IP address. 171 Measurements in a public datacenter show the quantitative benefits of 172 Multipath TCP [SIGCOMM11] in this environment. 174 Although ECMP is widely used inside datacenters, this is not the only 175 environment where there are different paths between a pair of hosts. 176 ECMP and other load balancing techniques such as Link Aggregation 177 Groups (LAG) are widely used in today's networks and having multiple 178 paths between a pair of single-homed hosts is becoming the norm 179 instead of the exception. Although these multiple paths have often 180 the same cost (from an IGP metrics viewpoint), they do not 181 necessarily have the same performance. For example, [IMC13c] reports 182 the results of a long measurement study showing that load balanced 183 Internet paths between that same pair of hosts can have huge delay 184 differences. 186 2.2. Cellular/WiFi Offload 188 A second use case that has been explored by several network 189 researchers is the cellular/WiFi offload use case. Smartphones or 190 other mobile devices equipped with two wireless interfaces are a very 191 common use case for Multipath TCP. In September 2015, this is also 192 the largest deployment of Multipath-TCP enabled devices [IOS7]. It 193 has been briefly discussed during IETF88 [ietf88], but there is no 194 published paper or report that analyses this deployment. For this 195 reason, we only discuss published papers that have mainly used the 196 Multipath TCP implementation in the Linux kernel for their 197 experiments. 199 The performance of Multipath TCP in wireless networks was briefly 200 evaluated in [NSDI12]. One experiment analyzes the performance of 201 Multipath TCP on a client with two wireless interfaces. This 202 evaluation shows that when the receive window is large, Multipath TCP 203 can efficiently use the two available links. However, if the window 204 becomes smaller, then packets sent on a slow path can block the 205 transmission of packets on a faster path. In some cases, the 206 performance of Multipath TCP over two paths can become lower than the 207 performance of regular TCP over the best performing path. Two 208 heuristics, reinjection and penalization, are proposed in [NSDI12] to 209 solve this identified performance problem. These two heuristics have 210 since been used in the Multipath TCP implementation in the Linux 211 kernel. [CONEXT13] explored the problem in more detail and revealed 212 some other scenarios where Multipath TCP can have difficulties in 213 efficiently pooling the available paths. Improvements to the 214 Multipath TCP implementation in the Linux kernel are proposed in 215 [CONEXT13] to cope with some of these problems. 217 The first experimental analysis of Multipath TCP in a public wireless 218 environment was presented in [Cellnet12]. These measurements explore 219 the ability of Multipath TCP to use two wireless networks (real WiFi 220 and 3G networks). Three modes of operation are compared. The first 221 mode of operation is the simultaneous use of the two wireless 222 networks. In this mode, Multipath TCP pools the available resources 223 and uses both wireless interfaces. This mode provides fast handover 224 from WiFi to cellular or the opposite when the user moves. 225 Measurements presented in [CACM14] show that the handover from one 226 wireless network to another is not an abrupt process. When a host 227 moves, there are regions where the quality of one of the wireless 228 networks is weaker than the other, but the host considers this 229 wireless network to still be up. When a mobile host enters such 230 regions, its ability to send packets over another wireless network is 231 important to ensure a smooth handover. This is clearly illustrated 232 from the packet trace discussed in [CACM14]. 234 Many cellular networks use volume-based pricing and users often 235 prefer to use unmetered WiFi networks when available instead of 236 metered cellular networks. [Cellnet12] implements support for the 237 MP_PRIO option to explore two other modes of operation. 239 In the backup mode, Multipath TCP opens a TCP subflow over each 240 interface, but the cellular interface is configured in backup mode. 242 This implies that data flows only over the WiFi interface when both 243 interfaces are considered to be active. If the WiFi interface fails, 244 then the traffic switches quickly to the cellular interface, ensuring 245 a smooth handover from the user's viewpoint [Cellnet12]. The cost of 246 this approach is that the WiFi and cellular interfaces are likely to 247 remain active all the time since all subflows are established over 248 the two interfaces. 250 The single-path mode is slightly different. This mode benefits from 251 the break-before-make capability of Multipath TCP. When an MPTCP 252 session is established, a subflow is created over the WiFi interface. 253 No packet is sent over the cellular interface as long as the WiFi 254 interface remains up [Cellnet12]. This implies that the cellular 255 interface can remain idle and battery capacity is preserved. When 256 the WiFi interface fails, a new subflow is established over the 257 cellular interface in order to preserve the established Multipath TCP 258 sessions. Compared to the backup mode described earlier, 259 measurements reported in [Cellnet12] indicate that this mode of 260 operation is characterised by a throughput drop while the cellular 261 interface is brought up and the subflows are reestablished. 263 From a protocol viewpoint, [Cellnet12] discusses the problem posed by 264 the unreliability of the REMOVE_ADDR option and proposes a small 265 protocol extension to allow hosts to reliably exchange this option. 266 It would be useful to analyze packet traces to understand whether the 267 unreliability of the REMOVE_ADDR option poses an operational problem 268 in real deployments. 270 Another study of the performance of Multipath TCP in wireless 271 networks was reported in [IMC13b]. This study uses laptops connected 272 to various cellular ISPs and WiFi hotspots. It compares various file 273 transfer scenarios. [IMC13b] observes that 4-path MPTCP outperforms 274 2-path MPTCP, especially for larger files. However, for three 275 congestion control algorithms (LIA, OLIA and Reno - see Section 3.2), 276 there is no significant performance difference for file sizes smaller 277 than 4MB. 279 A different study of the performance of Multipath TCP with two 280 wireless networks is presented in [INFOCOM14]. In this study the two 281 networks had different qualities : a good network and a lossy 282 network. When using two paths with different packet loss ratios, the 283 Multipath TCP congestion control scheme moves traffic away from the 284 lossy link that is considered to be congested. However, [INFOCOM14] 285 documents an interesting scenario that is summarised in Figure 1. 287 client ----------- path1 -------- server 288 | | 289 +--------------- path2 ------------+ 291 Figure 1: Simple network topology 293 Initially, the two paths have the same quality and Multipath TCP 294 distributes the load over both of them. During the transfer, the 295 second path becomes lossy, e.g. because the client moves. Multipath 296 TCP detects the packet losses and they are retransmitted over the 297 first path. This enables the data transfer to continue over the 298 first path. However, the subflow over the second path is still up 299 and transmits one packet from time to time. Although the N packets 300 have been acknowledged over the first subflow (at the MPTCP level), 301 they have not been acknowledged at the TCP level over the second 302 subflow. To preserve the continuity of the sequence numbers over the 303 second subflow, TCP will continue to retransmit these segments until 304 either they are acknowledged or the maximum number of retransmissions 305 is reached. This behavior is clearly inefficient and may lead to 306 blocking since the second subflow will consume window space to be 307 able to retransmit these packets. [INFOCOM14] proposes a new 308 Multipath TCP option to solve this problem. In practice, a new TCP 309 option is probably not required. When the client detects that the 310 data transmitted over the second subflow has been acknowledged over 311 the first subflow, it could decide to terminate the second subflow by 312 sending a RST segment. If the interface associated to this subflow 313 is still up, a new subflow could be immediately reestablished. It 314 would then be immediately usable to send new data and would not be 315 forced to first retransmit the previously transmitted data. As of 316 this writing, this dynamic management of the subflows is not yet 317 implemented in the Multipath TCP implementation in the Linux kernel. 319 Some studies have started to analyse the performance of Multipath TCP 320 on smartphones with real applications. In contrast with the bulk 321 transfers that are used by many publications, real applications do 322 not exchange huge amounts of data and establish a large number of 323 small connections. [COMMAG2016] proposes a software testing 324 framework that allows to automate Android applications to study their 325 interactions with Multipath TCP. [PAM2016] analyses a one-month 326 packet trace of all the packets exchanged by a dozen of smartphones 327 used by regular users. This analysis reveals that short connections 328 are important on smartphones and that the main benefit of using 329 Multipath TCP on smartphones is the ability to perform seamless 330 handovers between different wireless networks. Long connections 331 benefit from these handovers. 333 2.3. Multipath TCP proxies 335 As Multipath TCP is not yet widely deployed on both clients and 336 servers, several deployments have used various forms of proxies. Two 337 families of solutions are currently being used or tested. 339 A first use case is when a Multipath TCP enabled client wants to use 340 several interfaces to reach a regular TCP server. A typical use case 341 is a smartphone that needs to use both its WiFi and its cellular 342 interface to transfer data. Several types of proxies are possible 343 for this use case. An HTTP proxy deployed on a Multipath TCP capable 344 server would enable the smartphone to use Multipath TCP to access 345 regular web servers. Obviously, this solution only works for 346 applications that rely on HTTP. Another possibility is to use a 347 proxy that can convert any Multipath TCP connection into a regular 348 TCP connection. Multipath TCP-specific proxies have been proposed 349 [HotMiddlebox13b] [HAMPEL]. 351 Another possibility leverages the SOCKS protocol [RFC1928]. SOCKS is 352 often used in enterprise networks to allow clients to reach external 353 servers. For this, the client opens a TCP connection to the SOCKS 354 server that relays it to the final destination. If both the client 355 and the SOCKS server use Multipath TCP, but not the final 356 destination, then Multipath TCP can still be used on the path between 357 the client and the SOCKS server. At IETF'93, Korea Telecom announced 358 that they have deployed in June 2015 a commercial service that uses 359 Multipath TCP on smartphones. These smartphones access regular TCP 360 servers through a SOCKS proxy. This enables them to achieve 361 throughputs of up to 850 Mbps [KT]. 363 Measurements performed with Android smartphones [Mobicom15] show that 364 popular applications work correctly through a SOCKS proxy and 365 Multipath TCP enabled smartphones. Thanks to Multipath TCP, long- 366 lived connections can be spread over the two available interfaces. 367 However, for short-lived connections, most of the data is sent over 368 the initial subflow that is created over the interface corresponding 369 to the default route and the second subflow is almost not used 370 [PAM2016]. 372 A second use case is when Multipath TCP is used by middleboxes, 373 typically inside access networks. Various network operators are 374 discussing and evaluating solutions for hybrid access networks 375 [BBF-WT348]. Such networks arise when a network operator controls 376 two different access network technologies, e.g. wired and cellular, 377 and wants to combine them to improve the bandwidth offered to the 378 endusers [I-D.lhwxz-hybrid-access-network-architecture]. Several 379 solutions are currently investigated for such networks [BBF-WT348]. 380 Figure 2 shows the organisation of such a network. When a client 381 creates a normal TCP connection, it is intercepted by the Hybrid CPE 382 (HPCE) that converts it in a Multipath TCP connection so that it can 383 use the available access networks (DSL and LTE in the example). The 384 Hybrid Access Gateway (HAG) does the opposite to ensure that the 385 regular server sees a normal TCP connection. Some of the solutions 386 that are currently discussed for hybrid networks use Multipath TCP on 387 the HCPE and the HAG. Other solutions rely on tunnels between the 388 HCPE and the HAG [I-D.lhwxz-gre-notifications-hybrid-access]. 390 client --- HCPE ------ DSL ------- HAG --- internet --- server 391 | | 392 +------- LTE -----------+ 394 Figure 2: Hybrid Access Network 396 3. Operational Experience 398 3.1. Middlebox interference 400 The interference caused by various types of middleboxes has been an 401 important concern during the design of the Multipath TCP protocol. 402 Three studies on the interactions between Multipath TCP and 403 middleboxes are worth discussing. 405 The first analysis appears in [IMC11]. This paper was the main 406 motivation for Multipath TCP incorporating various techniques to cope 407 with middlebox interference. More specifically, Multipath TCP has 408 been designed to cope with middleboxes that : 410 o change source or destination addresses 412 o change source or destination port numbers 414 o change TCP sequence numbers 416 o split or coalesce segments 418 o remove TCP options 420 o modify the payload of TCP segments 422 These middlebox interferences have all been included in the MBtest 423 suite [MBTest]. This test suite is used in [HotMiddlebox13] to 424 verify the reaction of the Multipath TCP implementation in the Linux 425 kernel [MultipathTCP-Linux] when faced with middlebox interference. 426 The test environment used for this evaluation is a dual-homed client 427 connected to a single-homed server. The middlebox behavior can be 428 activated on any of the paths. The main results of this analysis are 429 : 431 o the v0.87 Multipath TCP implementation in the Linux kernel is not 432 affected by a middlebox that performs NAT or modifies TCP sequence 433 numbers 435 o when a middlebox removes the MP_CAPABLE option from the initial 436 SYN segment, the v0.87 Multipath TCP implementation in the Linux 437 kernel falls back correctly to regular TCP 439 o when a middlebox removes the DSS option from all data segments, 440 the v0.87 Multipath TCP implementation in the Linux kernel falls 441 back correctly to regular TCP 443 o when a middlebox performs segment coalescing, the v0.87 Multipath 444 TCP implementation in the Linux kernel is still able to accurately 445 extract the data corresponding to the indicated mapping 447 o when a middlebox performs segment splitting, the v0.87 Multipath 448 TCP implementation in the Linux kernel correctly reassembles the 449 data corresponding to the indicated mapping. [HotMiddlebox13] 450 shows on figure 4 in section 3.3 a corner case with segment 451 splitting that may lead to a desynchronisation between the two 452 hosts. 454 The interactions between Multipath TCP and real deployed middleboxes 455 is also analyzed in [HotMiddlebox13] and a particular scenario with 456 the FTP application level gateway running on a NAT is described. 458 Middlebox interference can also be detected by analysing packet 459 traces on Multipath TCP enabled servers. A closer look at the 460 packets received on the multipath-tcp.org server [TMA2015] shows that 461 among the 184,000 Multipath TCP connections, only 125 of them were 462 falling back to regular TCP. These connections originated from 28 463 different client IP addresses. These include 91 HTTP connections and 464 34 FTP connections. The FTP interference is expected and due to 465 Application Level Gateways running home routers. The HTTP 466 interference appeared only on the direction from server to client and 467 could have been caused by transparent proxies deployed in cellular or 468 enterprise networks. A longer trace is discussed in [COMCOM2016] and 469 similar conclusions about the middlebox interference are provided. 471 From an operational viewpoint, knowing that Multipath TCP can cope 472 with various types of middlebox interference is important. However, 473 there are situations where the network operators need to gather 474 information about where a particular middlebox interference occurs. 475 The tracebox software [tracebox] described in [IMC13a] is an 476 extension of the popular traceroute software that enables network 477 operators to check at which hop a particular field of the TCP header 478 (including options) is modified. It has been used by several network 479 operators to debug various middlebox interference problems. 480 Experience with tracebox indicates that supporting the ICMP extension 481 defined in [RFC1812] makes it easier to debug middlebox problems in 482 IPv4 networks. 484 Users of the Multipath TCP implementation have reported some 485 experience with middlebox interference. The strangest scenario has 486 been a middlebox that accepts the Multipath TCP options in the SYN 487 segment but later replaces Multipath TCP options with a TCP EOL 488 option [StrangeMbox]. This causes Multipath TCP to perform a 489 fallback to regular TCP without any impact on the application. 491 3.2. Congestion control 493 Congestion control has been an important challenge for Multipath TCP. 494 The standardised congestion control scheme for Multipath TCP is 495 defined in [RFC6356]. A detailed description of this algorithm is 496 provided in [NSDI11]. This congestion control scheme has been 497 implemented in the Linux implementation of Multipath TCP. Linux uses 498 a modular architecture to support various congestion control schemes. 499 This architecture is applicable for both regular TCP and Multipath 500 TCP. While the coupled congestion control scheme defined in 501 [RFC6356] is the default congestion control scheme in the Linux 502 implementation, other congestion control schemes have been added. 503 The second congestion control scheme is OLIA [CONEXT12]. This 504 congestion control scheme is also an adaptation of the NewReno single 505 path congestion control scheme to support multiple paths. 506 Simulations and measurements have shown that it provides some 507 performance benefits compared to the the default congestion control 508 scheme [CONEXT12]. Measurements over a wide range of parameters 509 reported in [CONEXT13] also indicate some benefits with the OLIA 510 congestion control scheme. Recently, a delay-based congestion 511 control scheme has been ported to the Multipath TCP implementation in 512 the Linux kernel. This congestion control scheme has been evaluated 513 by using simulations in [ICNP12]. The fourth congestion control 514 scheme that has been included in the Linux implementation of 515 Multipath TCP is the BALIA scheme that provides a better balance 516 between TCP friendliness, responsiveness, and window oscillation 517 [BALIA]. 519 These different congestion control schemes have been compared in 520 several articles. [CONEXT13] and [PaaschPhD] compare these 521 algorithms in an emulated environment. The evaluation showed that 522 the delay-based congestion control scheme is less able to efficiently 523 use the available links than the three other schemes. Reports from 524 some users indicate that they seem to favor OLIA. 526 3.3. Subflow management 528 The multipath capability of Multipath TCP comes from the utilisation 529 of one subflow per path. The Multipath TCP architecture [RFC6182] 530 and the protocol specification [RFC6824] define the basic usage of 531 the subflows and the protocol mechanisms that are required to create 532 and terminate them. However, there are no guidelines on how subflows 533 are used during the lifetime of a Multipath TCP session. Most of the 534 published experiments with Multipath TCP have been performed in 535 controlled environments. Still, based on the experience running them 536 and discussions on the mptcp-dev mailing list, interesting lessons 537 have been learned about the management of these subflows. 539 From a subflow viewpoint, the Multipath TCP protocol is completely 540 symmetrical. Both the clients and the server have the capability to 541 create subflows. However in practice the existing Multipath TCP 542 implementations have opted for a strategy where only the client 543 creates new subflows. The main motivation for this strategy is that 544 often the client resides behind a NAT or a firewall, preventing 545 passive subflow openings on the client. Although there are 546 environments such as datacenters where this problem does not occur, 547 as of this writing, no precise requirement has emerged for allowing 548 the server to create new subflows. 550 3.4. Implemented subflow managers 552 The Multipath TCP implementation in the Linux kernel includes several 553 strategies to manage the subflows that compose a Multipath TCP 554 session. The basic subflow manager is the full-mesh. As the name 555 implies, it creates a full-mesh of subflows between the communicating 556 hosts. 558 The most frequent use case for this subflow manager is a multihomed 559 client connected to a single-homed server. In this case, one subflow 560 is created for each interface on the client. The current 561 implementation of the full-mesh subflow manager is static. The 562 subflows are created immediately after the creation of the initial 563 subflow. If one subflow fails during the lifetime of the Multipath 564 TCP session (e.g. due to excessive retransmissions, or the loss of 565 the corresponding interface), it is not always reestablished. There 566 is ongoing work to enhance the full-mesh path manager to deal with 567 such events. 569 When the server is multihomed, using the full-mesh subflow manager 570 may lead to a large number of subflows being established. For 571 example, consider a dual-homed client connected to a server with 572 three interfaces. In this case, even if the subflows are only 573 created by the client, 6 subflows will be established. This may be 574 excessive in some environments, in particular when the client and/or 575 the server have a large number of interfaces. A recent draft has 576 proposed a Multipath TCP option to negotiate the maximum number of 577 subflows. However, it should be noted that there have been reports 578 on the mptcp-dev mailing indicating that users rely on Multipath TCP 579 to aggregate more than four different interfaces. Thus, there is a 580 need for supporting many interfaces efficiently. 582 Creating subflows between multihomed clients and servers may 583 sometimes lead to operational issues as observed by discussions on 584 the mptcp-dev mailing list. In some cases the network operators 585 would like to have a better control on how the subflows are created 586 by Multipath TCP [I-D.boucadair-mptcp-max-subflow]. This might 587 require the definition of policy rules to control the operation of 588 the subflow manager. The two scenarios below illustrate some of 589 these requirements. 591 host1 ---------- switch1 ----- host2 592 | | | 593 +-------------- switch2 --------+ 595 Figure 3: Simple switched network topology 597 Consider the simple network topology shown in Figure 3. From an 598 operational viewpoint, a network operator could want to create two 599 subflows between the communicating hosts. From a bandwidth 600 utilization viewpoint, the most natural paths are host1-switch1-host2 601 and host1-switch2-host2. However, a Multipath TCP implementation 602 running on these two hosts may sometimes have difficulties to obtain 603 this result. 605 To understand the difficulty, let us consider different allocation 606 strategies for the IP addresses. A first strategy is to assign two 607 subnets : subnetA (resp. subnetB) contains the IP addresses of 608 host1's interface to switch1 (resp. switch2) and host2's interface to 609 switch1 (resp. switch2). In this case, a Multipath TCP subflow 610 manager should only create one subflow per subnet. To enforce the 611 utilization of these paths, the network operator would have to 612 specify a policy that prefers the subflows in the same subnet over 613 subflows between addresses in different subnets. It should be noted 614 that the policy should probably also specify how the subflow manager 615 should react when an interface or subflow fails. 617 A second strategy is to use a single subnet for all IP addresses. In 618 this case, it becomes more difficult to specify a policy that 619 indicates which subflows should be established. 621 The second subflow manager that is currently supported by the 622 Multipath TCP implementation in the Linux kernel is the ndiffport 623 subflow manager. This manager was initially created to exploit the 624 path diversity that exists between single-homed hosts due to the 625 utilization of flow-based load balancing techniques [SIGCOMM11]. 626 This subflow manager creates N subflows between the same pair of IP 627 addresses. The N subflows are created by the client and differ only 628 in the source port selected by the client. It was not designed to be 629 used on multihomed hosts. 631 A more flexible subflow manager has been proposed, implemented and 632 evaluated in [CONEXT15]. This subflow manager exposes various kernel 633 events to a user space daemon that decides when subflows need to be 634 created and terminated based on various policies. 636 3.5. Subflow destination port 638 The Multipath TCP protocol relies on the token contained in the 639 MP_JOIN option to associate a subflow to an existing Multipath TCP 640 session. This implies that there is no restriction on the source 641 address, destination address and source or destination ports used for 642 the new subflow. The ability to use different source and destination 643 addresses is key to support multihomed servers and clients. The 644 ability to use different destination port numbers is worth discussing 645 because it has operational implications. 647 For illustration, consider a dual-homed client that creates a second 648 subflow to reach a single-homed server as illustrated in Figure 4. 650 client ------- r1 --- internet --- server 651 | | 652 +----------r2-------+ 654 Figure 4: Multihomed-client connected to single-homed server 656 When the Multipath TCP implementation in the Linux kernel creates the 657 second subflow it uses the same destination port as the initial 658 subflow. This choice is motivated by the fact that the server might 659 be protected by a firewall and only accept TCP connections (including 660 subflows) on the official port number. Using the same destination 661 port for all subflows is also useful for operators that rely on the 662 port numbers to track application usage in their network. 664 There have been suggestions from Multipath TCP users to modify the 665 implementation to allow the client to use different destination ports 666 to reach the server. This suggestion seems mainly motivated by 667 traffic shaping middleboxes that are used in some wireless networks. 668 In networks where different shaping rates are associated to different 669 destination port numbers, this could allow Multipath TCP to reach a 670 higher performance. As of this writing, we are not aware of any 671 implementation of this kind of tweaking. 673 However, from an implementation point-of-view supporting different 674 destination ports for the same Multipath TCP connection can cause 675 some issues. A legacy implementation of a TCP stack creates a 676 listening socket to react upon incoming SYN segments. The listening 677 socket is handling the SYN segments that are sent on a specific port 678 number. Demultiplexing incoming segments can thus be done solely by 679 looking at the IP addresses and the port numbers. With Multipath TCP 680 however, incoming SYN segments may have an MP_JOIN option with a 681 different destination port. This means, that all incoming segments 682 that did not match on an existing listening-socket or an already 683 established socket must be parsed for an eventual MP_JOIN option. 684 This imposes an additional cost on servers, previously not existent 685 on legacy TCP implementations. 687 3.6. Closing subflows 689 client server 690 | | 691 MPTCP: established | | MPTCP: established 692 Sub: established | | Sub: established 693 | | 694 | DATA_FIN | 695 MPTCP: close-wait | <------------------------ | close() (step 1) 696 Sub: established | DATA_ACK | 697 | ------------------------> | MPTCP: fin-wait-2 698 | | Sub: established 699 | | 700 | DATA_FIN + subflow-FIN | 701 close()/shutdown() | ------------------------> | MPTCP: time-wait 702 (step 2) | DATA_ACK | Sub: close-wait 703 MPTCP: closed | <------------------------ | 704 Sub: fin-wait-2 | | 705 | | 706 | subflow-FIN | 707 MPTCP: closed | <------------------------ | subflow-close() 708 Sub: time-wait | subflow-ACK | 709 (step 3) | ------------------------> | MPTCP: time-wait 710 | | Sub: closed 711 | | 713 Figure 5: Multipath TCP may not be able to avoid time-wait state on 714 the subflow (indicated as Sub in the drawing), even if enforced by 715 the application on the client-side. 717 Figure 5 shows a very particular issue within Multipath TCP. Many 718 high-performance applications try to avoid Time-Wait state by 719 deferring the closure of the connection until the peer has sent a 720 FIN. That way, the client on the left of Figure 5 does a passive 721 closure of the connection, transitioning from Close-Wait to Last-ACK 722 and finally freeing the resources after reception of the ACK of the 723 FIN. An application running on top of a Multipath TCP enabled Linux 724 kernel might also use this approach. The difference here is that the 725 close() of the connection (Step 1 in Figure 5) only triggers the 726 sending of a DATA_FIN. Nothing guarantees that the kernel is ready 727 to combine the DATA_FIN with a subflow-FIN. The reception of the 728 DATA_FIN will make the application trigger the closure of the 729 connection (step 2), trying to avoid Time-Wait state with this late 730 closure. This time, the kernel might decide to combine the DATA_FIN 731 with a subflow-FIN. This decision will be fatal, as the subflow's 732 state machine will not transition from Close-Wait to Last-Ack, but 733 rather go through Fin-Wait-2 into Time-Wait state. The Time-Wait 734 state will consume resources on the host for at least 2 MSL (Maximum 735 Segment Lifetime). Thus, a smart application that tries to avoid 736 Time-Wait state by doing late closure of the connection actually ends 737 up with one of its subflows in Time-Wait state. A high-performance 738 Multipath TCP kernel implementation should honor the desire of the 739 application to do passive closure of the connection and successfully 740 avoid Time-Wait state - even on the subflows. 742 The solution to this problem lies in an optimistic assumption that a 743 host doing active-closure of a Multipath TCP connection by sending a 744 DATA_FIN will soon also send a FIN on all its subflows. Thus, the 745 passive closer of the connection can simply wait for the peer to send 746 exactly this FIN - enforcing passive closure even on the subflows. 747 Of course, to avoid consuming resources indefinitely, a timer must 748 limit the time our implementation waits for the FIN. 750 3.7. Packet schedulers 752 In a Multipath TCP implementation, the packet scheduler is the 753 algorithm that is executed when transmitting each packet to decide on 754 which subflow it needs to be transmitted. The packet scheduler 755 itself does not have any impact on the interoperability of Multipath 756 TCP implementations. However, it may clearly impact the performance 757 of Multipath TCP sessions. The Multipath TCP implementation in the 758 Linux kernel supports a pluggable architecture for the packet 759 scheduler [PaaschPhD]. As of this writing, two schedulers have been 760 implemented: round-robin and lowest-rtt-first. The second scheduler 761 relies on the round-trip-time (rtt) measured on each TCP subflow and 762 sends first segments over the subflow having the lowest round-trip- 763 time. They are compared in [CSWS14]. The experiments and 764 measurements described in [CSWS14] show that the lowest-rtt-first 765 scheduler appears to be the best compromise from a performance 766 viewpoint. Another study of the packet schedulers is presented in 767 [PAMS2014]. This study relies on simulations with the Multipath TCP 768 implementation in the Linux kernel. They compare the lowest-rtt- 769 first with the round-robin and a random scheduler. They show some 770 situations where the lowest-rtt-first scheduler does not perform as 771 well as the other schedulers, but there are many scenarios where the 772 opposite is true. [PAMS2014] notes that "it is highly likely that 773 the optimal scheduling strategy depends on the characteristics of the 774 paths being used." 776 3.8. Segment size selection 778 When an application performs a write/send system call, the kernel 779 allocates a packet buffer (sk_buff in Linux) to store the data the 780 application wants to send. The kernel will store at most one MSS 781 (Maximum Segment Size) of data per buffer. As the MSS can differ 782 amongst subflows, an MPTCP implementation must select carefully the 783 MSS used to generate application data. The Linux kernel 784 implementation had various ways of selecting the MSS: minimum or 785 maximum amongst the different subflows. However, these heuristics of 786 MSS selection can cause significant performance issues in some 787 environment. Consider the following example. An MPTCP connection 788 has two established subflows that respectively use a MSS of 1420 and 789 1428 bytes. If MPTCP selects the maximum, then the application will 790 generate segments of 1428 bytes of data. An MPTCP implementation 791 will have to split the segment in two (a 1420-byte and 8-byte 792 segments) when pushing on the subflow with the smallest MSS. The 793 latter segment will introduce a large overhead as for a single data 794 segment 2 slots will be used in the congestion window (in packets) 795 therefore reducing by roughly twice the potential throughput (in 796 bytes/s) of this subflow. Taking the smallest MSS does not solve the 797 issue as there might be a case where the subflow with the smallest 798 MSS only sends a few packets therefore reducing the potential 799 throughput of the other subflows. 801 The Linux implementation recently took another approach [DetalMSS]. 802 Instead of selecting the minimum and maximum values, it now 803 dynamically adapts the MSS based on the contribution of all the 804 subflows to the connection's throughput. For this it computes, for 805 each subflow, the potential throughput achieved by selecting each MSS 806 value and by taking into account the lost space in the cwnd. It then 807 selects the MSS that allows to achieve the highest potential 808 throughput. 810 3.9. Interactions with the Domain Name System 812 Multihomed clients such as smartphones can send DNS queries over any 813 of their interfaces. When a single-homed client performs a DNS 814 query, it receives from its local resolver the best answer for its 815 request. If the client is multihomed, the answer returned to the DNS 816 query may vary with the interface over which it has been sent. 818 cdn1 819 | 820 client -- cellular -- internet -- cdn3 821 | | 822 +----- wifi --------+ 823 | 824 cdn2 826 Figure 6: Simple network topology 828 If the client sends a DNS query over the WiFi interface, the answer 829 will point to the cdn2 server while the same request sent over the 830 cellular interface will point to the cdn1 server. This might cause 831 problems for CDN providers that locate their servers inside ISP 832 networks and have contracts that specify that the CDN server will 833 only be accessed from within this particular ISP. Assume now that 834 both the client and the CDN servers support Multipath TCP. In this 835 case, a Multipath TCP session from cdn1 or cdn2 would potentially use 836 both the cellular network and the WiFi network. Serving the client 837 from cdn2 over the cellular interface could violate the contract 838 between the CDN provider and the network operators. A similar 839 problem occurs with regular TCP if the client caches DNS replies. 840 For example the client obtains a DNS answer over the cellular 841 interface and then stops this interface and starts to use its WiFi 842 interface. If the client retrieves data from cdn1 over its WiFi 843 interface, this may also violate the contract between the CDN and the 844 network operators. 846 A possible solution to prevent this problem would be to modify the 847 DNS resolution on the client. The client subnet EDNS extension 848 defined in [RFC7871] could be used for this purpose. When the client 849 sends a DNS query from its WiFi interface, it should also send the 850 client subnet corresponding to the cellular interface in this 851 request. This would indicate to the resolver that the answer should 852 be valid for both the WiFi and the cellular interfaces (e.g., the 853 cdn3 server). 855 3.10. Captive portals 857 Multipath TCP enables a host to use different interfaces to reach a 858 server. In theory, this should ensure connectivity when at least one 859 of the interfaces is active. In practice however, there are some 860 particular scenarios with captive portals that may cause operational 861 problems. The reference environment is shown in Figure 7. 863 client ----- network1 864 | 865 +------- internet ------------- server 867 Figure 7: Issue with captive portal 869 The client is attached to two networks : network1 that provides 870 limited connectivity and the entire Internet through the second 871 network interface. In practice, this scenario corresponds to an open 872 WiFi network with a captive portal for network1 and a cellular 873 service for the second interface. On many smartphones, the WiFi 874 interface is preferred over the cellular interface. If the 875 smartphone learns a default route via both interfaces, it will 876 typically prefer to use the WiFi interface to send its DNS request 877 and create the first subflow. This is not optimal with Multipath 878 TCP. A better approach would probably be to try a few attempts on 879 the WiFi interface and then try to use the second interface for the 880 initial subflow as well. 882 3.11. Stateless webservers 884 MPTCP has been designed to interoperate with webservers that benefit 885 from SYN-cookies to protect against SYN-flooding attacks [RFC4987]. 886 MPTCP achieves this by echoing the keys negotiated during the 887 MP_CAPABLE handshake in the third ACK of the 3-way handshake. 888 Reception of this third ACK then allows the server to reconstruct the 889 state specific to MPTCP. 891 However, one caveat to this mechanism is the non-reliable nature of 892 the third ACK. Indeed, when the third ACK gets lost, the server will 893 not be able to reconstruct the MPTCP-state. MPTCP will fallback to 894 regular TCP in this case. This is in contrast to regular TCP. When 895 the client starts sending data, the first data segment also includes 896 the SYN-cookie, which allows the server to reconstruct the TCP-state. 897 Further, this data segment will be retransmitted by the client in 898 case it gets lost and thus is resilient against loss. MPTCP does not 899 include the keys in this data segment and thus the server cannot 900 reconstruct the MPTCP state. 902 This issue might be considered as a minor one for MPTCP. Losing the 903 third ACK should only happen when packet loss is high. However, when 904 packet-loss is high MPTCP provides a lot of benefits as it can move 905 traffic away from the lossy link. It is undesirable that MPTCP has a 906 higher chance to fall back to regular TCP in those lossy 907 environments. 909 [I-D.paasch-mptcp-syncookies] discusses this issue and suggests a 910 modified handshake mechanism that ensures reliable delivery of the 911 MP_CAPABLE, following the 3-way handshake. This modification will 912 make MPTCP reliable, even in lossy environments when servers need to 913 use SYN-cookies to protect against SYN-flooding attacks. 915 3.12. Loadbalanced serverfarms 917 Large-scale serverfarms typically deploy thousands of servers behind 918 a single virtual IP (VIP). Steering traffic to these servers is done 919 through layer-4 loadbalancers that ensure that a TCP-flow will always 920 be routed to the same server [Presto08]. 922 As Multipath TCP uses multiple different TCP subflows to steer the 923 traffic across the different paths, loadbalancers need to ensure that 924 all these subflows are routed to the same server. This implies that 925 the loadbalancers need to track the MPTCP-related state, allowing 926 them to parse the token in the MP_JOIN and assign those subflows to 927 the appropriate server. However, serverfarms typically deploy 928 multiple of these loadbalancers for reliability and capacity reasons. 929 As a TCP subflow might get routed to any of these loadbalancers, they 930 would need to synchronize the MPTCP-related state - a solution that 931 is not feasible at large scale. 933 The token (carried in the MP_JOIN) contains the information 934 indicating which MPTCP-session the subflow belongs to. As the token 935 is a hash of the key, servers are not able to generate the token in 936 such a way that the token can provide the necessary information to 937 the loadbalancers which would allow them to route TCP subflows to the 938 appropriate server. [I-D.paasch-mptcp-loadbalancer] discusses this 939 issue in detail and suggests two alternative MP_CAPABLE handshakes to 940 overcome these. As of September 2015, it is not yet clear how MPTCP 941 might accomodate such use-case to enable its deployment within 942 loadbalanced serverfarms. 944 4. IANA Considerations 946 There are no IANA considerations in this informational document. 948 5. Security Considerations 950 The security considerations for Multipath TCP have already been 951 documented in [RFC6181], [RFC6182], [RFC6824] and [RFC7430]. 953 6. Acknowledgements 955 This work was partially supported by the FP7-Trilogy2 project. We 956 would like to thank all the implementers and users of the Multipath 957 TCP implementation in the Linux kernel. This document has benefited 958 from the comments of John Ronan, Yoshifumi Nishida, Phil Eardley, 959 Jaehyun Hwang and Mirja Kuehlewind. 961 7. Informative References 963 [Apple-MPTCP] 964 Apple, Inc, ., "iOS - Multipath TCP Support in iOS 7", 965 n.d., . 967 [BALIA] Peng, Q., Walid, A., Hwang, J., and S. Low, "Multipath TCP 968 Analysis, Design, and Implementation", IEEE/ACM Trans. on 969 Networking, Vol. 24, No. 1 , 2016. 971 [BBF-WT348] 972 Fabregas (Ed), G., "WT-348 - Hybrid Access for Broadband 973 Networks", Broadband Forum, contribution bbf2014.1139.04 , 974 June 2015. 976 [CACM14] Paasch, C. and O. Bonaventure, "Multipath TCP", 977 Communications of the ACM, 57(4):51-57 , April 2014, 978 . 980 [COMCOM2016] 981 "Observing real Multipath TCP traffic", Computer 982 Communications , April 2016, . 985 [COMMAG2016] 986 De Coninck, Q., Baerts, M., Hesmans, B., and O. 987 Bonaventure, "Observing Real Smartphone Applications over 988 Multipath TCP", IEEE Communications Magazine , March 2016, 989 . 993 [CONEXT12] 994 Khalili, R., Gast, N., Popovic, M., Upadhyay, U., and J. 995 Leboudec, "MPTCP is not pareto-optimal performance issues 996 and a possible solution", Proceedings of the 8th 997 international conference on Emerging networking 998 experiments and technologies (CoNEXT12) , 2012. 1000 [CONEXT13] 1001 Paasch, C., Khalili, R., and O. Bonaventure, "On the 1002 Benefits of Applying Experimental Design to Improve 1003 Multipath TCP", Conference on emerging Networking 1004 EXperiments and Technologies (CoNEXT) , December 2013, . 1009 [CONEXT15] 1010 Hesmans, B., Detal, G., Barre, S., Bauduin, R., and O. 1011 Bonaventure, "SMAPP - Towards Smart Multipath TCP-enabled 1012 APPlications", Proc. Conext 2015, Heidelberg, Germany , 1013 December 2015, . 1016 [CSWS14] Paasch, C., Ferlin, S., Alay, O., and O. Bonaventure, 1017 "Experimental Evaluation of Multipath TCP Schedulers", 1018 SIGCOMM CSWS2014 workshop , August 2014. 1020 [Cellnet12] 1021 Paasch, C., Detal, G., Duchene, F., Raiciu, C., and O. 1022 Bonaventure, "Exploring Mobile/WiFi Handover with 1023 Multipath TCP", ACM SIGCOMM workshop on Cellular Networks 1024 (Cellnet12) , 2012, . 1027 [DetalMSS] 1028 Detal, G., "Adaptive MSS value", Post on the mptcp-dev 1029 mailing list , September 2014, . 1033 [FreeBSD-MPTCP] 1034 Williams, N., "Multipath TCP For FreeBSD Kernel Patch 1035 v0.5", n.d., . 1037 [HAMPEL] Hampel, G., Rana, A., and T. Klein, "Seamless TCP mobility 1038 using lightweight MPTCP proxy", Proceedings of the 11th 1039 ACM international symposium on Mobility management and 1040 wireless access (MobiWac '13). , 2013. 1042 [HotMiddlebox13] 1043 Hesmans, B., Duchene, F., Paasch, C., Detal, G., and O. 1044 Bonaventure, "Are TCP Extensions Middlebox-proof?", CoNEXT 1045 workshop HotMiddlebox , December 2013, . 1049 [HotMiddlebox13b] 1050 Detal, G., Paasch, C., and O. Bonaventure, "Multipath in 1051 the Middle(Box)", HotMiddlebox'13 , December 2013, . 1054 [HotNets] Raiciu, C., Pluntke, C., Barre, S., Greenhalgh, A., 1055 Wischik, D., and M. Handley, "Data center networking with 1056 multipath TCP", Proceedings of the 9th ACM SIGCOMM 1057 Workshop on Hot Topics in Networks (Hotnets-IX) , 2010, 1058 . 1060 [I-D.boucadair-mptcp-max-subflow] 1061 Boucadair, M. and C. Jacquenet, "Negotiating the Maximum 1062 Number of Multipath TCP (MPTCP) Subflows", 1063 draft-boucadair-mptcp-max-subflow-02 (work in progress), 1064 May 2016. 1066 [I-D.lhwxz-gre-notifications-hybrid-access] 1067 Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M. 1068 Zhang, "GRE Notifications for Hybrid Access", 1069 draft-lhwxz-gre-notifications-hybrid-access-01 (work in 1070 progress), January 2015. 1072 [I-D.lhwxz-hybrid-access-network-architecture] 1073 Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M. 1074 Zhang, "Hybrid Access Network Architecture", 1075 draft-lhwxz-hybrid-access-network-architecture-02 (work in 1076 progress), January 2015. 1078 [I-D.paasch-mptcp-loadbalancer] 1079 Paasch, C., Greenway, G., and A. Ford, "Multipath TCP 1080 behind Layer-4 loadbalancers", 1081 draft-paasch-mptcp-loadbalancer-00 (work in progress), 1082 September 2015. 1084 [I-D.paasch-mptcp-syncookies] 1085 Paasch, C., Biswas, A., and D. Haas, "Making Multipath TCP 1086 robust for stateless webservers", 1087 draft-paasch-mptcp-syncookies-02 (work in progress), 1088 October 2015. 1090 [ICNP12] Cao, Y., Xu, M., and X. Fu, "Delay-based congestion 1091 control for multipath TCP", 20th IEEE International 1092 Conference on Network Protocols (ICNP) , 2012. 1094 [IMC11] Honda, M., Nishida, Y., Raiciu, C., Greenhalgh, A., 1095 Handley, M., and H. Tokuda, "Is it still possible to 1096 extend TCP?", Proceedings of the 2011 ACM SIGCOMM 1097 conference on Internet measurement conference (IMC '11) , 1098 2011, . 1100 [IMC13a] Detal, G., Hesmans, B., Bonaventure, O., Vanaubel, Y., and 1101 B. Donnet, "Revealing Middlebox Interference with 1102 Tracebox", Proceedings of the 2013 ACM SIGCOMM conference 1103 on Internet measurement conference , 2013, . 1107 [IMC13b] Chen, Y., Lim, Y., Gibbens, R., Nahum, E., Khalili, R., 1108 and D. Towsley, "A measurement-based study of MultiPath 1109 TCP performance over wireless network", Proceedings of the 1110 2013 conference on Internet measurement conference (IMC 1111 '13) , n.d., . 1113 [IMC13c] Pelsser, C., Cittadini, L., Vissicchio, S., and R. Bush, 1114 "From Paris to Tokyo on the suitability of ping to 1115 measure latency", Proceedings of the 2013 conference on 1116 Internet measurement conference (IMC '13) , 2013, 1117 . 1119 [INFOCOM14] 1120 Lim, Y., Chen, Y., Nahum, E., Towsley, D., and K. Lee, 1121 "Cross-Layer Path Management in Multi-path Transport 1122 Protocol for Mobile Devices", IEEE INFOCOM'14 , 2014. 1124 [IOS7] Apple, ., "Multipath TCP Support in iOS 7", January 2014, 1125 . 1127 [KT] Seo, S., "KT's GiGA LTE", July 2015, . 1130 [MBTest] Hesmans, B., "MBTest", 2013, 1131 . 1133 [MPTCPBIB] 1134 Bonaventure, O., "Multipath TCP - An annotated 1135 bibliography", Technical report , April 2015, 1136 . 1138 [Mobicom15] 1139 De Coninck, Q., Baerts, M., Hesmans, B., and O. 1140 Bonaventure, "Poster - Evaluating Android Applications 1141 with Multipath TCP", Mobicom 2015 (Poster) , 1142 September 2015. 1144 [MultipathTCP-Linux] 1145 Paasch, C., Barre, S., and . et al, "Multipath TCP 1146 implementation in the Linux kernel", n.d., 1147 . 1149 [NSDI11] Wischik, D., Raiciu, C., Greenhalgh, A., and M. Handley, 1150 "Design, implementation and evaluation of congestion 1151 control for Multipath TCP", In Proceedings of the 8th 1152 USENIX conference on Networked systems design and 1153 implementation (NSDI11) , 2011. 1155 [NSDI12] Raiciu, C., Paasch, C., Barre, S., Ford, A., Honda, M., 1156 Duchene, F., Bonaventure, O., and M. Handley, "How Hard 1157 Can It Be? Designing and Implementing a Deployable 1158 Multipath TCP", USENIX Symposium of Networked Systems 1159 Design and Implementation (NSDI12) , April 2012, . 1164 [PAM2016] De Coninck, Q., Baerts, M., Hesmans, B., and O. 1165 Bonaventure, "A First Analysis of Multipath TCP on 1166 Smartphones", 17th International Passive and Active 1167 Measurements Conference (PAM2016) , March 2016, . 1171 [PAMS2014] 1172 Arzani, B., Gurney, A., Cheng, S., Guerin, R., and B. Loo, 1173 "Impact of Path Selection and Scheduling Policies on MPTCP 1174 Performance", PAMS2014 , 2014. 1176 [PaaschPhD] 1177 Paasch, C., "Improving Multipath TCP", Ph.D. Thesis , 1178 November 2014, . 1181 [Presto08] 1182 Greenberg, A., Lahiri, P., Maltz, D., Parveen, P., and S. 1183 Sengupta, "Towards a Next Generation Data Center 1184 Architecture - Scalability and Commoditization", ACM 1185 PRESTO 2008 , August 2008, 1186 . 1188 [RFC1812] Baker, F., Ed., "Requirements for IP Version 4 Routers", 1189 RFC 1812, DOI 10.17487/RFC1812, June 1995, 1190 . 1192 [RFC1928] Leech, M., Ganis, M., Lee, Y., Kuris, R., Koblas, D., and 1193 L. Jones, "SOCKS Protocol Version 5", RFC 1928, 1194 DOI 10.17487/RFC1928, March 1996, 1195 . 1197 [RFC4987] Eddy, W., "TCP SYN Flooding Attacks and Common 1198 Mitigations", RFC 4987, DOI 10.17487/RFC4987, August 2007, 1199 . 1201 [RFC6181] Bagnulo, M., "Threat Analysis for TCP Extensions for 1202 Multipath Operation with Multiple Addresses", RFC 6181, 1203 DOI 10.17487/RFC6181, March 2011, 1204 . 1206 [RFC6182] Ford, A., Raiciu, C., Handley, M., Barre, S., and J. 1207 Iyengar, "Architectural Guidelines for Multipath TCP 1208 Development", RFC 6182, DOI 10.17487/RFC6182, March 2011, 1209 . 1211 [RFC6356] Raiciu, C., Handley, M., and D. Wischik, "Coupled 1212 Congestion Control for Multipath Transport Protocols", 1213 RFC 6356, DOI 10.17487/RFC6356, October 2011, 1214 . 1216 [RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure, 1217 "TCP Extensions for Multipath Operation with Multiple 1218 Addresses", RFC 6824, DOI 10.17487/RFC6824, January 2013, 1219 . 1221 [RFC7430] Bagnulo, M., Paasch, C., Gont, F., Bonaventure, O., and C. 1222 Raiciu, "Analysis of Residual Threats and Possible Fixes 1223 for Multipath TCP (MPTCP)", RFC 7430, DOI 10.17487/ 1224 RFC7430, July 2015, 1225 . 1227 [RFC7871] Contavalli, C., van der Gaast, W., Lawrence, D., and W. 1228 Kumari, "Client Subnet in DNS Queries", RFC 7871, 1229 DOI 10.17487/RFC7871, May 2016, 1230 . 1232 [SIGCOMM11] 1233 Raiciu, C., Barre, S., Pluntke, C., Greenhalgh, A., 1234 Wischik, D., and M. Handley, "Improving datacenter 1235 performance and robustness with multipath TCP", 1236 Proceedings of the ACM SIGCOMM 2011 conference , n.d., 1237 . 1239 [StrangeMbox] 1240 Bonaventure, O., "Multipath TCP through a strange 1241 middlebox", Blog post , January 2015, . 1245 [TMA2015] Hesmans, B., Tran Viet, H., Sadre, R., and O. Bonaventure, 1246 "A First Look at Real Multipath TCP Traffic", Traffic 1247 Monitoring and Analysis , 2015, . 1251 [ietf88] Stewart, L., "IETF'88 Meeting minutes of the MPTCP working 1252 group", n.d., . 1255 [tracebox] 1256 Detal, G. and O. Tilmans, "tracebox", 2013, 1257 . 1259 Appendix A. Changelog 1261 This section should be removed before final publication 1263 o initial version : September 16th, 2014 : Added section Section 3.8 1264 that discusses some performance problems that appeared with the 1265 Linux implementation when using subflows having different MSS 1266 values 1268 o update with a description of the middlebox that replaces an 1269 unknown TCP option with EOL [StrangeMbox] 1271 o version ietf-02 : July 2015, answer to last call comments 1273 * Reorganised text to better separate use cases and operational 1274 experience 1276 * New use case on Multipath TCP proxies in Section 2.3 1278 * Added some text on middleboxes in Section 3.1 1280 * Removed the discussion on SDN 1282 * Restructured text and improved writing in some parts 1284 o version ietf-03 : September 2015, answer to comments from Phil 1285 Eardley 1287 * Improved introduction 1289 * Added details about using SOCKS and Korea Telecom's use-case in 1290 Section 2.3. 1292 * Added issue around clients caching DNS-results in Section 3.9 1294 * Explained issue of MPTCP with stateless webservers Section 3.11 1296 * Added description of MPTCP's use behind layer-4 loadbalancers 1297 Section 3.12 1299 * Restructured text and improved writing in some parts 1301 o version ietf-04 : April 2016, answer to last comments 1303 * Updated text on measurements with smartphones 1305 * Updated conclusion 1307 o version ietf-06 : August 2016, answer to AD's review 1309 * removed some expired drafts 1311 * removed conclusion 1313 Authors' Addresses 1315 Olivier Bonaventure 1316 UCLouvain 1318 Email: Olivier.Bonaventure@uclouvain.be 1320 Christoph Paasch 1321 Apple, Inc. 1323 Email: cpaasch@apple.com 1325 Gregory Detal 1326 Tessares 1328 Email: Gregory.Detal@tessares.net