idnits 2.17.1 draft-eardley-mptcp-implementations-survey-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 3 instances of lines with non-RFC2606-compliant FQDNs in the document. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 12, 2013) is 3941 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'RFC6897' is mentioned on line 1703, but not defined == Missing Reference: 'Appendix A' is mentioned on line 1703, but not defined ** Obsolete normative reference: RFC 6824 (Obsoleted by RFC 8684) Summary: 1 error (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group P. Eardley 3 Internet-Draft BT 4 Intended status: Informational July 12, 2013 5 Expires: January 13, 2014 7 Survey of MPTCP Implementations 8 draft-eardley-mptcp-implementations-survey-02 10 Abstract 12 This document presents results from the survey to gather information 13 from people who have implemented MPTCP, in particular to help 14 progress the protocol from Experimental to Standards track. 16 The document currently includes answers from four teams: a Linux 17 implementation from UCLouvain, a FreeBSD implementation from 18 Swinburne, an anonymous implementation in a commercial OS, and a 19 NetScalar Firmware implementation from Citrix Systems, Inc. Thank- 20 you! 22 In summary, we have four independent implementations of all the MPTCP 23 signalling messages, with the exception of address management, and 24 some interoperabiity testing has been done by the other three 25 implementations with the 'reference' Linux implementation. So it 26 appears that the RFC is (at least largely) clear and correct. On 27 address management, we have only one implementation of ADD_ADDR with 28 two teams choosing not to implement it. We have one implementation 29 of the working group's coupled congestion control (RFC6356) and none 30 of the MPTCP-aware API (RFC6897). 32 The main suggested improvements are around 34 o how MPTCP falls back (if the signalling is interrupted by a 35 middlebox): (1) corner cases that are not handled properly, (2) at 36 the IETF, the MPTCP community should work with middlebox vendors, 37 either to reduce or eliminate the need for fallback or to 38 understand the middlebox interactions better. 40 o security: both better MPTCP security (perhaps building on SSL) and 41 a lighter weight mechanism, preferably both in one mechanism. 43 It is hoped that the next version can include information from any 44 other implementations. If you are an implementer and want to 45 contribute your answers, please see the -01 version of this document 46 for a blank survey ready to be filled in. 48 Status of this Memo 49 This Internet-Draft is submitted in full conformance with the 50 provisions of BCP 78 and BCP 79. 52 Internet-Drafts are working documents of the Internet Engineering 53 Task Force (IETF). Note that other groups may also distribute 54 working documents as Internet-Drafts. The list of current Internet- 55 Drafts is at http://datatracker.ietf.org/drafts/current/. 57 Internet-Drafts are draft documents valid for a maximum of six months 58 and may be updated, replaced, or obsoleted by other documents at any 59 time. It is inappropriate to use Internet-Drafts as reference 60 material or to cite them other than as "work in progress." 62 This Internet-Draft will expire on January 13, 2014. 64 Copyright Notice 66 Copyright (c) 2013 IETF Trust and the persons identified as the 67 document authors. All rights reserved. 69 This document is subject to BCP 78 and the IETF Trust's Legal 70 Provisions Relating to IETF Documents 71 (http://trustee.ietf.org/license-info) in effect on the date of 72 publication of this document. Please review these documents 73 carefully, as they describe your rights and restrictions with respect 74 to this document. Code Components extracted from this document must 75 include Simplified BSD License text as described in Section 4.e of 76 the Trust Legal Provisions and are provided without warranty as 77 described in the Simplified BSD License. 79 Table of Contents 81 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 82 2. Survey - summary of replies . . . . . . . . . . . . . . . . . 4 83 3. Interesting aspects of replies . . . . . . . . . . . . . . . . 6 84 3.1. Question 1: Your details . . . . . . . . . . . . . . . . . 6 85 3.2. Question 2: Preliminary information about your 86 implementation . . . . . . . . . . . . . . . . . . . . . . 7 87 3.3. Question 3: Support for MPTCP's Signalling 88 Functionality . . . . . . . . . . . . . . . . . . . . . . 7 89 3.4. Question 4: Fallback from MPTCP . . . . . . . . . . . . . 7 90 3.5. Question 5: Heuristics . . . . . . . . . . . . . . . . . . 8 91 3.6. Question 6: Security . . . . . . . . . . . . . . . . . . . 9 92 3.7. Question 7: IANA . . . . . . . . . . . . . . . . . . . . . 9 93 3.8. Question 8: Congestion control and subflow policy . . . . 9 94 3.9. Question 9: API . . . . . . . . . . . . . . . . . . . . . 10 95 3.10. Question 10: Deployments, use cases and operational 96 experiences . . . . . . . . . . . . . . . . . . . . . . . 10 97 3.11. Question 11: Improvements to RFC6824 . . . . . . . . . . . 11 98 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11 99 5. Security Considerations . . . . . . . . . . . . . . . . . . . 11 100 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 11 101 7. Full survey response for Implementation 1 . . . . . . . . . . 11 102 8. Full survey response for Implementation 2 . . . . . . . . . . 19 103 9. Full survey response for Implementation 3 . . . . . . . . . . 23 104 10. Full survey response for Implementation 4 . . . . . . . . . . 31 105 11. Normative References . . . . . . . . . . . . . . . . . . . . . 38 106 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 38 108 1. Introduction 110 The document reports the results from a survey to gather information 111 from people who have implemented MPTCP. The goal is to help progress 112 the protocol from Experimental to Standards track. 114 Four responses have been received. Thank-you! They are independent 115 implementations: 117 o the Linux implementation from UCLouvain, 119 o the FreeBSD implementation from Swinburne 121 o an anonymous implementation in a commercial OS 123 o a NetScaler Firmware implementation from Citrix Systems, Inc. 125 The Table below presents a highly-compressed summary, with each row 126 corresponding to one question or sub-question of the survey. The 127 following section highlights some interesting aspects of the replies 128 in less compressed form. The full survey responses are in Appendix 129 A, B, C and D. 131 It is hoped that the next version of this document can include 132 information about a further (independent) implementation: 134 o Georg Hampel's user-space implementation (publicly available but 135 not longer maintained) 137 o any other implementations. 139 2. Survey - summary of replies 141 The Table below presents a highly-compressed summary, with each row 142 corresponding to one question or sub-question of the survey. A 143 column is left blank for any future responses. 145 +----------------------------------------------------------------------+ 146 | | 1 | 2 | 3 | 4 | | 147 |Institution | UCLouvain | Swinburne | Anon | Citrix | | 148 | | 149 | Question 2 asks about some preliminary topics, including whether the | 150 | implementation is publicly available and interoperability with the | 151 | Linux implementation (#1). | 152 | | UCLouvain | Swinburne | Anon | Citrix | | 153 |OS |Linux |FreeBSD-10 |Commercial |NetScaler | | 154 |v4 & v6 |Both | IPv4 |Both |Both | | 155 |public |Yes |Yes |No |Yes (pay) | | 156 |independent |Yes |Yes |Yes |Yes | | 157 |interop |Yes(!) |Mostly |Mostly |Yes | | 158 | | 159 | Question 3: Support for MPTCP's signalling functionality | 160 | MPTCP's signalling messages are: MP_CAPABLE, MP_JOIN, Data transfer | 161 | (DSS), ADD_ADDR, REMOVE_ADDR, MP_FASTCLOSE. There are sub-questions | 162 | for MP_JOIN and DSS. | 163 | | UCLouvain | Swinburne | Anon | Citrix | | 164 |MP_CAPABLE |Yes |Yes |Yes |Yes | | 165 |MP_JOIN |Yes |Yes |Yes |Yes | | 166 |initiated by|first end |either end |first end |first end | | 167 |#subflows |32 |8 |no limit |6 | | 168 |DSS |Yes |Yes |Yes |Yes | | 169 |DATA ACK |4 bytes |4 or 8 byte|4 or 8 byte|4 or 8 byte| | 170 |Data seq num|4 bytes |4 or 8 byte|4 or 8 byte|4 or 8 byte| | 171 |DATA_FIN |Yes |Yes |Yes |Yes | | 172 |Checksum |Yes |No |Yes |Yes | | 173 |ADD_ADDR |Yes |No |No (never) |No (never?)| | 174 |REMOVE_ADDR |Yes |No |Partly |Yes | | 175 |FAST_CLOSE |Yes |No |Yes |Yes | | 176 | | 177 | Question 4 asks about fallback from MPTCP: if a middlebox mangles | 178 | MPTCP's signalling by removing MP_CAPABLE, MP_JOIN, DSS or DATA_ACK; | 179 | if data is protected with Checksum in DSS option; if fallback to TCP | 180 | uses an infinte mapping; and if any corner cases have been found. | 181 | | UCLouvain | Swinburne | Anon | Citrix | | 182 |MP_CAPABLE |Yes |Yes |Yes |Yes | | 183 |MP_JOIN |Yes |Yes |Yes |Yes | | 184 |DSS |Yes |No |Yes |Yes | | 185 |DATA_ACK |Yes |No |No | | | 186 |Checksum |Yes |No |Yes |Yes | | 187 |infinte map |Yes |Yes |Yes |Yes | | 188 |corner cases|No | |Yes |Yes | | 189 | | 190 | Question 5 asks about heuristics: aspects that are not required for | 191 | protocol correctness but impact the performance. Questions are about | 192 | sized the receiver and sender buffers, re-transmission policy, if | 193 | additional subflows use the same port number as for the first subflow| 194 | | UCLouvain | Swinburne | Anon | Citrix | | 195 |Recvr buffer|auto-tune |TCP_MAXWIN |no tuning |tuned | | 196 |Sendr buffer|auto-tune |cwnd |no tuning |as TCP | | 197 |Re-transmits|2nd subflow|2nd subflow|2nd subflow|1st subflow| | 198 |Port usage |same ports |same ports |diff local | | | 199 | | 200 | Question 6 asks about what security mechanisms are implemented: the | 201 | one defined in RFC6824 and any others. | 202 | | UCLouvain | Swinburne | Anon | Citrix | | 203 |HMAC-SHA1 |Yes |Yes |Yes |Yes | | 204 |other |Yes |No |No |No | | 205 | | 206 | Question 7 asks whether the implementation follows the IANA-related | 207 | definitions (for TCP Option Kind and sub-registries). | 208 | | UCLouvain | Swinburne | Anon | Citrix | | 209 |RFC6824 |Yes |Yes |Yes |Yes | | 210 | | 211 | Question 8 asks about congestion control and related issues: how | 212 | traffic is shared across multiple subflows; support for 'handover'; | 213 | and support of RFC6356 (or other) coupled congestion control. | 214 | | UCLouvain | Swinburne | Anon | Citrix | | 215 |sharing |shared, RTT|shared |active/back|active/back| | 216 |handover |Yes | |Yes |Yes | | 217 |coupled cc |Yes |No |No |No | | 218 |other ccc |Yes, OLIA |No |No |No | | 219 |MP-PRIO & B |Yes |No |Yes |Yes | | 220 | | 221 | Question 9 is about the API: how legacy applications interact with | 222 | the MPTCP stack, and if implemented the RFC6897 API for MPTCP-aware | 223 | applications. | 224 | | UCLouvain | Swinburne | Anon | Citrix | | 225 |legacy apps |default |sysctl |private API|configured | | 226 |MPTCP API |No |No |No |No | | 227 |advanced API|No |No |No |No | | 228 | | 229 | Question 10 gathers some limited information about operational | 230 | experiences and deployments. | 231 | | UCLouvain | Swinburne | Anon | Citrix | | 232 |Scenario |several |several |mobile |proxy | | 233 |environment |internet |controlled |internet |internet | | 234 |ends / proxy|end hosts |end hosts |end hosts |proxy | | 235 | | 236 +----------------------------------------------------------------------+ 238 3. Interesting aspects of replies 240 This section tries to highlight some interesting comments made in the 241 surveys. The Appendices can be consulted for further detials. 243 3.1. Question 1: Your details 245 Implementation 1 has been implemented by Sebastien Barre, Christoph 246 Paasch and a large team, mainly at UCLouvain. Implementation 2 has 247 been implemented by Lawrence Stewart and Nigel Williams at Swinburne 248 University of Technology. Both these implementations are publicly 249 available. Implementation 3 comes from an anonymous team with a 250 commercial OS. Implementation 4 comes from Citrix Systems, Inc. 252 3.2. Question 2: Preliminary information about your implementation 254 Three of the four implementations are publicly available, two for 255 free (under GPLv2 and BSD licences) and one for a fee (NetScaler 256 Firmware). Implementation 3 (commercial OS) is planned for use in a 257 mobile environment, with MPTCP is used in active/backup mode. 259 All implementations support IPv4 and three of four support IPv6. 261 All implementations are being actively worked on, in order to improve 262 performance and stability and conformance with the RFC. 264 3.3. Question 3: Support for MPTCP's Signalling Functionality 266 Three of the four implementations have implemented all the MPTCP 267 signalling, with the interesting exception of address management, 268 whilst Implementation 2 plans to add support for all those signalling 269 capabilities it does not yet support. 271 On address management, two implementations have decided not to 272 implement ADD_ADDR. (ADD_ADDR allows an MPTCP host to signal a new 273 address explicitly to the other host to allow it to initiate a new 274 subflow - as an alternative to using MP_JOIN to directly start a new 275 subflow). Implementation 3 decided not to support sending ADD_ADDR 276 or processing ADD_ADDR as it is considered a security risk. 277 Implementation 4 decided not to support ADD_ADDR because it didn't 278 think it would be useful as most clients are behind NATing devices. 279 However, both implemented REMOVE_ADDR (in Implementation 3 the client 280 can send a REMOVE_ADDR but ignores incoming REMOVE_ADDR). 282 In Implementations 1, 3 and 4 only the initiator of the original 283 subflow can start a new subflow (a reason mentioned is that NATs make 284 it hard for the server to reach the client). 286 All implementations support 4 bytes "Data ACK" and "Data sequence 287 number" fields, and will interoperate with an implementation sending 288 8 bytes. Implementation 1 uses only 4 bytes fields; if an 289 implementation sends an 8 byte data sequence number it replies with a 290 4 byte data ack. 292 3.4. Question 4: Fallback from MPTCP 294 Question 4 asks about action when there is a problem with MPTCP, for 295 example due to a middlebox mangling MPTCP's signalling. The 296 connection needs to fall back: if the problem is on the first subflow 297 then MPTCP falls back to TCP, whilst if the problem is on an 298 additional subflow then that subflow is closed with a TCP RST, as 299 discussed in [Section 3.6 RFC6824]. 301 Implementations 3 and 4 made several comments about fallback. 303 Implementation 3 suggests that both sender and receiver behaviours 304 could be outlined with more detail, in particular when DSS checksum 305 is not in use and the MPTCP options are stripped. Implementation 3 306 falls back to TCP when there's one sub flow, but not when there are 307 multiple sub flows (MPTCP is used in active/backup mode, and it is 308 assumed that the sub flow transferring data is most likely to be more 309 usable than any other established sub flow, hence the sub flow on 310 which fallback occurred is kept alive and other sub flows are 311 closed). 313 Implementation 4 found a corner case where it is not clear what to 314 do: if a pure ack or data packet without DSS is received in middle of 315 transaction (which can happen if the routing changes and the new path 316 drops MPTCP options). Also, Implementation 4 suggests that 317 clarifying whether the infinite map exchange is unidirectional or 318 bidirectional. 320 Implementation 1 has developed a publicly available test suite that 321 tests MPTCP's traversal of middleboxes. 323 3.5. Question 5: Heuristics 325 Question 5 gathers information about heuristics: aspects that are not 326 required for protocol correctness but impact the performance. We 327 would like to document best practice so that future implementers can 328 learn from the experience of pioneers. 330 There are several differences between the implementations. 332 For receiver buffer, Implementation 1 uses a slightly modified 333 version of Linux's auto-tuning algorithm; Implementation 2 determines 334 the receiver buffer by using "TCP_MAXWIN << tp->rcv_scale" (this is a 335 temporary measure); Implementation 3 uses MPTCP in active/backup 336 mode, so the receive buffer sizes at the MPTCP and subflow level is 337 the same (automatic buffer tuning is turned off); Implementation 4 338 varies the receiver buffer size based on the services and application 339 type. 341 For the sender buffer, Implementation 1 uses Linux auto-tuning, 342 Implementation 2 scales based on occupancy, whilst Implementation 3 343 turns off automatic buffer tuning, and Implementation 4 uses MPTCP- 344 level (sub)flow control that is (almost) the same as regular TCP flow 345 control. 347 Implementations 1, 2 and 3 re-transmit unacknowledged data on a 348 different subflow (and not the same subflow), whilst Implementation 4 349 re-transmits on original subflow for 3 RTOs and then uses another 350 subflow. 352 For port usage, Implementations 1 and 2 uses the same ports for the 353 additional subflows, whilst Implementation 3 uses the same 354 detsination port but a different local port, so that on the wire it 355 looks like two connections to the same remote destination. 357 Implementation 4 suggests that the RFC should more clearly 358 /extensively define failure cases and how to handle unexpected 359 signals. 361 3.6. Question 6: Security 363 Question 6 asks about security related matters. 365 All Implementations have implemented the hash-based, HMAC-SHA1 366 security mechanism defined in [RFC6824]. Implementation 3 suggests 367 that a more secure mechanism could be tied with SSL. Implementation 368 4 suggests that a more secure and lightweight mechanism is needed, as 369 keys are exchanged (in the MP_CAPABLE option) in plain text and the 370 key generation mechanism is highly computational intensive. 371 Implementation 1 has implemented two additional mechanisms in a 372 separate Linux branch - one lightweight and the other SSL-based. 374 3.7. Question 7: IANA 376 All Implementations have followed the IANA-related definitions 377 [Section 8 RFC6824] for: TCP Option Kind number (30); the sub- 378 registry for "MPTCP Option Subtypes"; and the sub-registry for "MPTCP 379 Handshake Algorithms". 381 3.8. Question 8: Congestion control and subflow policy 383 Question 8 asks how is shared across multiple subflows. 385 Implementation 1 has added support for coupled congestion control 386 (both that defined in [RFC6356] and in OLIA, 387 draft-khalili-mptcp-congestion-control. The other implementations do 388 not include coupled congestion control. Whilst Implementation 2 389 plans to add it (currently it uses a simple algorithm spreads traffic 390 across the subflows), Implementations 3 and 4 do not plan to add 391 coupled congestion control - they use one subflow at a time, with 392 others as a backup. Implementation 3 believes it is not currently 393 useful to share load across all network interfaces on a mobile node, 394 as the interfaces have different characteristics for cost, bring-up 395 and power usage. They have both found the B bit (in MP-JOIN) and MP- 396 PRIO option very useful for this active /backup operation. 398 Implementation 2 is also interested in experimenting with congestion 399 control across paths with different path-cost metrics. 401 3.9. Question 9: API 403 Question 9 gathers information about the API. None have implemented 404 the [RFC6897] "basic MPTCP API" for MPTCP-aware applications. For 405 three implementations MPTCP is used for all applications (set by 406 configuration), whilst Implementation 3 uses a private API that 407 allows MPTCP to be used on a per application basis. 409 3.10. Question 10: Deployments, use cases and operational experiences 411 Question 10 takes the opportunity of this survey to gather some 412 limited information about operational experiences and deployments. 414 The Implementations mention different use cases. 416 Implementation 2 is interested in using MPTCP for several use cases: 417 vehicle to infrastructure (V2I) connectivity (to provide a persistent 418 connection using 3G and roadside wifi); multi-homed "home-user" 419 environments; high throughput data transfers. Implementation 3 is 420 interested in the mobile scenario, with MPTCP providing an active 421 /backup mode so achieving session continuity across changing network 422 environments. Implementation 4 is interested in MPTCP giving 423 reliability and fault tolerance via a proxy. Implementation 1 424 already uses MPTCP on www.multipath-tcp.org and for internal ssh 425 servers at UCLouvain. 427 Implementation 4 uses a proxy (MPTCP connections from a client are 428 terminated and the TCP connection established on the other side), 429 whilst the other Implementations are on end hosts. Implementation 2 430 is so far within controlled testbeds, whilst Implementation 3 is on 431 the Internet. 433 Implementation 2 is currently an alpha-quality build, so limited 434 testing so far. 436 Implementation 3 suggests working at the IETF with firewall vendors, 437 to get them to change their defaults to allow MPTCP signals. This 438 would also reduce the "over-engineering" needed to handle fallback 439 cases. Implementation 1 suggests retrieving logs from middleboxes, 440 as the best approach to understanding the interactions of MPTCP 441 signalling with middleboxes. 443 Implementation 3 discusses a scenario that should be handled better. 444 A backup subflow may never sent data. If the initial subflow fails, 445 data is retransmitted on the backup subflow, but that path has a 446 middlebox stripping options. Then it may not be possible to recover 447 the MPTCP session. 449 3.11. Question 11: Improvements to RFC6824 451 Question 11 asks if there are any areas where RFC6824 could be 452 improved. The main topics have been mentioned earlier: 454 o fallback: the need for more clarity in the fallback cases is 455 mentioned by Implementations 3 and 4. 457 o security: the need for both a more secure and a more lightweight 458 mechanism is mentioned. 460 Implementation 3 also suggests several potential improvements, which 461 are outside the scope of RFC6824: support for sub flow level 462 automatic buffer scaling, varying QoS support, and varying window 463 scaling support on each sub flow; also, additional work on option 464 signlling will be brought up in future discussions. 466 4. IANA Considerations 468 This document makes no request of IANA. 470 5. Security Considerations 472 This survey does not impact the security of MPTCP, except to the 473 extent that it uncovers security issues that can be tackled in a 474 future version of the protocol. 476 6. Acknowledgements 478 Many thanks to the people who replied to the survey: Christoph 479 Paasch, Nigel Williams, anon, and Krishna Khanal. Very many thanks 480 to all of the teams who actually did the implementation and testing 481 and are continuing to improve them. 483 7. Full survey response for Implementation 1 485 Question 1: Your details Question 1 gathers some information about 486 the team that has implemented MPTCP. 488 1. Your institution: UCLouvain, IP Networking Lab 489 (http://inl.info.ucl.ac.be) 491 2. Name(s) of people in your implementation and test teams: Initial 492 design from Sebastien Barre. Since then, numerous code-contributors 493 (ordered by number of commits): Christoph Paasch (UCLouvain) Gregory 494 Detal (UCLouvain) Jakko Korkeaniemi (Aalto University) Mihai P. 495 Andrei (Intel) Fabien Duchene (UCLouvain) Andreas Seelinger (RWTH 496 Aachen) Stefan Sicleru (Intel) Lavkesh Lahngir Catalin Nicutar (PUB 497 Bucharest) Andrei Maruseac (Intel) Andreas Ripke (NEC) Vlad Dogaru 498 (Intel) Octavian Purdila (Intel) Niels Laukens (VRT Belgium) John 499 Ronan (TSSG) Brandon Heller (Stanford University) Conformance 500 Testing: Yvan Coene (UCLouvain) 502 3. Do you want your answers to Question 1.1 and 1.2 above to be 503 anonymised? No. 505 3.2. Question 2: Preliminary information about your implementation 506 Question 2 gathers some preliminary information. 508 1. What OS is your implementation for? (or is it application layer?) 509 Linux Kernel. 511 2. Do you support IPv4 or IPv6 addresses or both? We support both. 513 3. Is it publicly available (or will it be?) (for free or a fee?) 514 Publicly available (GPLv2) at www.multipath-tcp.org 516 4. Overall, what are you implementation and testing plans? (details 517 can be given against individual items later) We plan to continue to 518 align our implementation with the IETF specifications and improve its 519 performance and stability. 521 5. Is it an independent implementation? Or does it build on another 522 MPTCP implementation -which one? Independent implementation. 524 6. Have you already done some interop tests, for example with 525 UCLouvain's "reference" Linux implementation? / 527 7. Would you be prepared to take part in an interop event, for 528 example adjacent to IETF-87 in Berlin? Yes. We are also ready to 529 help in organising such an event if needed. 531 3.3. Question 3: Support for MPTCP's Signalling Functionality 532 Question 3 asks about support for the various signalling messages 533 that the MPTCP protocol defines. *** For each message, please give a 534 little information about the status of your implementation: for 535 example, you may have implemented it and fully tested it; the 536 implementation may be in progress; you have not yet implemented it 537 but plan to soon (timescale?); you may you have no intention to 538 implement it (why?); etc. 540 1. Connection initiation (MP_CAPABLE) [Section 3.1 RFC6824] 542 a. What is the status of your implementation? Fully support the 543 MP_CAPABLE exchange. 545 b. Any other comments or information? We generate the random key as 546 a hash of the 5-tuple, sequence number and a local secret. This 547 significantly improves the performance, instead of using a pseudo- 548 random number generator. The performance benefit has been shown 549 during IETF85 550 http://tools.ietf.org/agenda/85/slides/slides-85-mptcp-2.pdf 552 2. Starting a new subflow (MP_JOIN) [Section 3.2 RFC6824] 554 a. What is the status of your implementation? Fully support the 555 MP_JOIN exchange. 557 b. Can either end of the connection start a new subflow (or only the 558 initiator of the original subflow)? Currently, only the initiator of 559 the original subflow starts a new subflow. Given the widespread 560 deployment of NATs, it is often difficult for the server to reach the 561 client. This is the main reason why the server currently does not 562 start new subflows in our implementation. But, the initiator would 563 accept a SYN+MP_JOIN if sent by another implementation. 565 c. What is the maximum number of subflows your implementation can 566 support? Currently 32. 568 d. Any other comments or information? 570 3. Data transfer (DSS) [Section 3.3 RFC6824] 572 a. What is the status of your implementation? Fully working 573 implementation of data transfer. 575 b. The "Data ACK" field can be 4 or 8 octets. Which one(s) have you 576 implemented? We use 4 bytes for the DATA-ACK field. 578 c. The "Data sequence number" field can be 4 or 8 octets. Which 579 one(s) have you implemented? We use 4 bytes for the data sequence 580 number. 582 d. Does your implementation support the "DATA_FIN" operation to 583 close an MPTCP connection? Yes. 585 e. Does your implementation support the "Checksum" field (which is 586 negotiated in the MP_CAPABLE handshake)? Yes. This is configurable 587 via a sysctl. 589 f. Any other comments or information? We support interoperability 590 with implementations that do send 64-bit data sequence numbers and 591 data acks. However, even if the peer sends 64-bit data sequence 592 numbers, we will only reply with a 32-bit data-ack. We do not have 593 heuristics to trigger the sending of DATA_ACKs. We simply send the 594 DATA_ACK in each packet. 596 4. Address management (ADD_ADDR and REMOVE_ADDR) [Section 3.4 597 RFC6824] 599 a. What is the status of your implementation? We support ADD_ADDR/ 600 REMOVE_ADDR messages. 602 b. Can your implementation do ADD_ADDRESS for addresses that appear 603 *after* the connection has been established? Yes, as shown in: 604 "Exploring Mobile/WiFi Handover with Multipath TCP", C. Paasch et. 605 al, ACM SIGCOMM workshop on Cellular Networks (Cellnet'12), 2012. 607 c. Any other comments or information? We do not send out TCP 608 keepalive-messages upon the reception of a REMOVE_ADDR-message. 610 5. Fast close (MP_FASTCLOSE) [Section 3.5 RFC6824] 612 a. What is the status of your implementation? We support the 613 MP_FASTCLOSE implementation. 615 b. Any other comments or information? 617 3.4. Question 4: Fallback from MPTCP Question 4 asks about action 618 when there is a problem with MPTCP, for example due to a middlebox 619 mangling MPTCP's signalling. The connection needs to fall back: if 620 the problem is on the first subflow then MPTCP falls back to TCP, 621 whilst if the problem is on an additional subflow then that subflow 622 is closed with a TCP RST, as discussed in [Section 3.6 RFC6824]. 624 1. If the MP_CAPABLE option is removed by a middlebox, does your 625 implementation fall back to TCP? Yes. 627 2. If the MP_JOIN option does not get through on the SYNs, does your 628 implementation close the additional subflow? Yes. 630 3. If the DSS option does not get through on the first data 631 segment(s), does your implementation fall back? (either falling back 632 to MPTCP (if the issue is on the first subflow) or closing the 633 additional subflow (if the issue is on an additional subflow)) Yes. 634 On the initial subflow we do a seamless fallback, additional subflows 635 will be closed by a RST. 637 4. Similarly, if the "DATA ACK" field does not correctly acknowledge 638 the first data segment(s), does your implementation fall back? Yes. 639 Same as above. 641 5. Does your implementation protect data with the "Checksum" field 642 in the DSS option [Section 3.3 RFC6824]? If the checksum fails 643 (because the subflow has been affected by a middlebox), does your 644 implementation immediately close the affected subflow (with a TCP 645 RST) with the MP_FAIL Option? If the checksum fails and there is a 646 single subflow, does your implementation handle this as a special 647 case, as described in [Section 3.6 RFC6824]? Yes, we support the 648 DSS-checksum. If the checksum is wrong and there exist other 649 subflows, we close the current subflow with an RST. If there is no 650 other subflow, we send an ACK + MP_FAIL and do a fallback to infinite 651 mapping. This fallback has successfully been tested with different 652 type of NAT middleboxes, while using FTP. 654 6. Does your implementation fall back to TCP by using an "infinite 655 mapping" [Section 3.3.1 RFC6824] (so that the subflow-level data is 656 mapped to the connection-level data for the remainder of the 657 connection)? Yes. 659 7. Did you find any corner cases where MPTCP's fallback didn't 660 happen properly? No. We have developped a test-suite to test the 661 middlebox-traversal of MPTCP, available at 662 http://multipath-tcp.org/pmwiki.php/Users/AboutMeasures 664 8. Any other comments or information about fallback? 666 3.5. Question 5: Heuristics Question 5 gathers information about 667 heuristics: aspects that are not required for protocol correctness 668 but impact the performance. We would like to document best practice 669 so that future implementers can learn from the experience of 670 pioneers. The references contain some initial comments about each 671 topic. 673 1. Receiver considerations [S3.3.4, RFC6824]: What receiver buffer 674 have you used? Does this depend on the retransmission strategy? 675 What advice should we give about the receiver? Linux includes an 676 autuning algorithm for the TCP receiver buffer. This algorithm has 677 been slightly modified for Multipath TCP. The receive-buffer does 678 not depend on the retransmission strategy. 680 2. Sender considerations [S3.3.5, RFC6824]: How do you determine how 681 much data a sender is allowed to send and how big the sender buffer 682 is? What advice should we give about the sender? The send-buffer is 683 autotuned similarly as the receive-buffer (see above). We send as 684 much data as possible, filling the congestion windows of each 685 subflow. The sender deploys the "Opportunistic Retransmission" and 686 "Penalization" algorithms from the paper: "How Hard Can It Be? 687 Designing and Implementing a Deployable Multipath TCP", C. Raiciu et. 688 al, NSDI 2012. 690 3. Reliability and retransmissions [S3.3.6, RFC6824]: What is your 691 retransmission policy? (when do you retransmit on the original 692 subflow vs on another subflow or subflows?) When do you decide that 693 a subflow is underperforming and should be reset, and what do you 694 then do? What advice should we give about this issue? Upon an RTO 695 on subflow A, we reinject all the unacknowledged data of subflow A on 696 another subflows. We do not currently have a mechanism to detect 697 that a subflow is underperforming. 699 4. Port usage [S3.3.8.1, RFC6824]: Does your implementation use the 700 same port number for additional subflows as for the first subflow? 701 Have you used the ability to define a specific port in the Add 702 Address option? What advice should we give about this issue? We 703 always use the same port number as for the first subflow. Except, if 704 the ADD_ADDRESS option that has been received contained a specific 705 port. We do not have a means to configure the specific port in the 706 ADD_ADDRESS option, but we support reception of the port. 708 5. Delayed subflow start [S3.3.8.2, RFC6824]: What factors does your 709 implementation consider when deciding about opening additional 710 subflows? What advice should we give about this issue? As soon as 711 we are sure that the initial subflow is fully MPTCP-capable 712 (reception of a DATA_ACK), we create a full mesh among all IP- 713 addresses between the two hosts. We do not explicitly delay the 714 creation of new subflows. 716 6. Failure handling [S3.3.8.3, RFC6824]: Whilst the protocol defines 717 how to handle some unexpected signals, the behaviour after other 718 unexpected signals is not defined. What advice should we give about 719 this issue? We did not implement the caching mentioned in Section 720 3.8.3. 722 7. Use of TCP options: As discussed in [Appendix A, RFC6824], the 723 TCP option space is limited, but a brief study found there was enough 724 room to fit all the MPTCP options. However there are constraints on 725 which MPTCP option(s) can be included in packets with other TCP 726 options - do the suggestions in Appendix A need amending or 727 expanding? We do not implement specific heuristics to reduce the TCP 728 option-space usage. If timestamp is enabled we will only be able to 729 send two SACK-blocks, because the DATA_ACK consumes the remaining 730 bytes. 732 8. What other heuristics should we give advice about? Any other 733 comments or information? 735 3.6. Question 6: Security Question 6 asks about Security related 736 matters [Section 5 RFC6824]. 738 1. Does your implementation use the hash-based, HMAC-SHA1 security 739 mechanism defined in [RFC6824]? Yes. 741 2. Does your implementation support any other handshake algorithms? 742 We have in a separate branch, an implementation of 743 draft-paasch-mptcp-lowoverhead and draft-paasch-mptcp-ssl. 745 3. It has been suggested that a Standards-track MPTCP needs a more 746 secure mechanism. Do you have any views about how to achieve this? 747 We believe that the solution described in draft-paasch-mptcp-ssl 748 would be a good starting point since it leverages the security of the 749 upper layer. 751 4. Any other comments or information? 753 3.7. Question 7: IANA Question 7 asks about IANA related matters. 755 1. Does your implementation follow the IANA-related definitions? 756 [Section 8 RFC6824] defines: TCP Option Kind number (30); the sub- 757 registry for "MPTCP Option Subtypes"; and the sub-registry for "MPTCP 758 Handshake Algorithms" Yes. 760 2. Any other comments or information? 762 3.8. Question 8: Congestion control and subflow policy Question 8 763 asks about how you share traffic across multiple subflows. 765 1. How does your implementation share traffic over the available 766 paths? For example: as a spare path on standby ('all-or- nothing'), 767 as an 'overflow', etc? Does it have the ability to send /receive 768 traffic across multiple subflows simultaneously? The implementation 769 is able to send and receive traffic on all subflows simultaneously. 770 Our scheduler first tries to send traffic on the subflow with the 771 lowest RTT. As this subflow's congestion window is full, we pick the 772 subflow with the next lower RTT. 774 2. Does your implementation support "handover" from one subflow to 775 another when losing an interface? Yes, as described in: "Exploring 776 Mobile/WiFi Handover with Multipath TCP", C. Paasch et. al, ACM 777 SIGCOMM workshop on Cellular Networks (Cellnet'12), 2012. 779 3. Does your implementation support the coupled congestion control 780 defined in [RFC6356]? Yes. 782 4. Does your implementation support some other coupled congestion 783 control (ie that balances traffic on multiple paths according to 784 feedback)? We also support the OLIA congestion control 785 (draft-khalili-mptcp-congestion-control-00). 787 5. The MP_JOIN (Starting a new subflow) Option includes the "B" bit, 788 which allows the sender to indicate whether it wishes the new subflow 789 to be used immediately or as a backup if other path(s) fail. The 790 MP_PRIO Option is a request to change the "B" bit - either on the 791 subflow on which it is sent, or (by setting the optional Address ID 792 field) on other subflows. Does your implementation support the "B" 793 bit and MP_PRIO mechanisms? Do you think they're useful, or have 794 another suggestion? Yes, we support the "B"-bit of the MP_JOIN and 795 the MP_PRIO option. It is configurable on a per-interface basis. 796 Experiences with the "B"-bit can be found in our paper: "Exploring 797 Mobile/WiFi Handover with Multipath TCP", C. Paasch et. al, ACM 798 SIGCOMM workshop on Cellular Networks (Cellnet'12), 2012. 800 6. Any other comments or information or suggestions about the advice 801 we should give about congestion control [S3.3.7 RFC6824] and subflow 802 policy [S3.3.8 RFC6824]? 804 3.9. Question 9: API Question 9 gathers information about your API. 805 [RFC6897] considers the MPTCP Application Interface. 807 1. With your implementation, can legacy applications use (the 808 existing sockets API to use) MPTCP? How does the implementation 809 decide whether to use MPTCP? Should the advice in [Section 4, 810 RFC6897] be modified or expanded? Yes, a standard TCP socket API can 811 be used. By default MPTCP is enabled on all connections. 813 2. The "basic MPTCP API" enables MPTCP-aware applications to 814 interact with the MPTCP stack via five new socket options. For each 815 one, have you implemented it? has it been useful? None of them are 816 part of the current stable release MPTCP v0.86. 817 http://multipath-tcp.org/pmwiki.php?n=Main.Release86 a. 818 TCP_MULTIPATH_ENABLE? b. TCP_MULTIPATH_ADD? c. 819 TCP_MULTIPATH_REMOVE? d. TCP_MULTIPATH_SUBFLOWS? e. 820 TCP_MULTIPATH_CONNID? 822 3. Have you implemented any aspects of an "advanced MPTCP API"? 823 ([Appendix A, RFC6897] hints at what it might include.) No. 825 4. Any other comments or information? 827 3.10. Question 10: Deployments, use cases and operational 828 experiences Question 10 takes the opportunity of this survey to 829 gather some limited information about operational experiences and 830 deployments. Any very brief information would be appreciated, for 831 example: 1. What deployment scenarios are you most interested in? 2. 832 Is your deployment on "the Internet" or in a controlled environment? 833 3. Is your deployment on end hosts or with a MPTCP-enabled proxy (at 834 one or both ends?)? 4. What do you see as the most important 835 benefits of MPTCP in your scenario(s)? 5. How extensively have you 836 deployed and experimented with MPTCP so far? 838 Our implementation is open-source and has been discussed for various 839 types of tests/deployments based on the messages received on the 840 mptcp-dev mailing list. We currently use Multipath TCP on 841 www.multipath-tcp.org and also on internal ssh servers at UCLouvain. 842 6. MPTCP's design seeks to maximise the chances that the signalling 843 works through middleboxes. Did you find cases where middleboxes 844 blocked MPTCP signalling? We have implemented a test suite based on 845 a slightly modified version of the Multipath TCP implementation that 846 allows to check the interoperability between Multipath TCP and 847 middleboxes. We have used it over Internet paths and identified some 848 potential problems. However, the best approach to test these 849 interactions would be to control the middlebox and analyse its logs 850 during the Multipath TCP test. The test suite can be retrieved from 851 http://multipath-tcp.org/pmwiki.php/Users/AboutMeasures 853 7. MPTCP's design seeks to ensure that, if there is a problem with 854 MPTCP signalling, then the connection either falls back to TCP or 855 removes the problematic subflow. Did you find any corner cases where 856 this didn't happen properly? See above. 858 8. Have you encountered any issues or drawbacks with MPTCP? 860 9. Any other comments or information? 862 3.11. Question 11: Improvements to RFC6824 864 1. Are there any areas where [RFC6824] could be improved, either in 865 technical content or clarity? 2. Any other issues you want to raise? 867 8. Full survey response for Implementation 2 869 Question 1: Your details 870 ------------------------------------------------------------- 871 1.1 Swinburne University of Technology, Hawthorn, Victoria, Australia 873 1.2 Lawrence Stewart, Nigel Williams 875 1.3 No 877 Question 2: Preliminary information about your implementation 878 ------------------------------------------------------------- 880 2.1 FreeBSD-10 882 2.2 Currently IPv4 only (IPv6 support will eventually be added) 884 2.3 Publicly available (http://caia.swin.edu.au/urp/newtcp/mptcp/). 885 The code is released under the BSD license. 2.3 887 2.5 Independent 889 2.6 Yes, some limited testing to establish interoperability. 891 2.7 Yes, with some additional work this should be possible (if not 892 then IETF-88). Q 894 uestion 3: Support for MPTCP's Signaling Functionality 895 ------------------------------------------------------------- 897 3.1 a) MP_CAPABLE Implemented 899 b) Do not currently honour checksum flag (to be implemented) 901 3.2 a) MP_JOIN Implemented 903 b) Either end can initiate a MP_JOIN 905 c) 8 (controlled via sysctl) 907 d) Currently do not include HMAC verification during handshake, but 908 this will be enabled in the next patch (several weeks from time of 909 submission) 911 3.3 a) DSS Implemented 913 b) 4 (default) and 8 915 c) 4 (default) and 8 917 d) Yes, however the connection tear-down exchange is not fully 918 implemented - the connection shuts down but the DFIN may not be 919 correctly acknowledged. 921 e) No. This will be supported eventually (time-frame unknown) 923 3.4 a) ADD_ADDR implemented, REMOVE_ADDR not implemented (to be done, 924 timeframe unknown) 926 b) No. Functionality to be added 928 3.5 MP_FASTCLOSE not implemented. Plan to implement eventually 930 Question 4: Fallback from MPTCP 931 ------------------------------------------------------------- 933 4.1 Yes 935 4.2 The subflow PCBs remain allocated, however the subflow is not 936 used to send data. 938 4.3 No, tbd 940 4.4 No, tbd 942 4.5 No, checksumming not implemented 944 4.6 Yes 946 4.8 Fallback hasn't really been put through any structured tests yet 948 Question 5: Heuristics 949 ------------------------------------------------------------- 951 5.1 We use "TCP_MAXWIN << tp->rcv_scale". This is temporary and we 952 will use a call into the "multipath" control layer to determine this 953 value in future releases (we need to investigate a suitable way of 954 calculating this). 956 5.2 cwnd determines the amount of data to send (given that rcv window 957 is always very large). Sendbuffer is scaled based on occupancy. 959 5.3 We currently don't have Data-level retransmits enabled. However 960 our policy is to retransmit on the next subflow that requests data to 961 send that is suitable. There is no intelligence in the packet 962 schedular currently, 964 5.4 The same port numbers are re-used for additional subflows. 966 Question 6: Security 967 ------------------------------------------------------------- 969 6.1 Yes 971 6.2 No 973 Question 7: IANA 974 ------------------------------------------------------------- 976 7.1 Yes 978 Question 8: Congestion Control and subflow policy 979 ------------------------------------------------------------- 981 8.1 A simple algorithm is used to divide the send buffer between 982 subflows, so that traffic is spread across the subflows. 984 8.3 No. (to be added) 986 8.4 No 988 8.5 No 990 Question 9: API 991 ------------------------------------------------------------- 993 9.1 Legacy applications are able to use MPTCP. MPTCP is set globally 994 via a sysctl variable. 996 9.2 No 998 9.3 No 1000 Question 10: API 1001 ------------------------------------------------------------- 1003 10.1 Some current project work is based on MPTCPs use in vehicle to 1004 infrastructure (V2I) connectivity (to provide a persistent connection 1005 using 3G and roadside wifi). Other interests are in multi-homed 1006 "home-user" environments, high throughput data transfers.... We are 1007 also interested in experimenting with congestion control across paths 1008 with different path-cost metrics. 1010 10.2 So far only within controlled testbeds 1012 10.3 End hosts 1014 10.4 Depending on the scenario, connection persistence, throughput... 1016 10.5 Still an alpha-quality build, so limited testing so far. 1018 9. Full survey response for Implementation 3 1020 Survey 3.1. Question 1: Your details Question 1 gathers some 1021 information about the team that has implemented MPTCP. 1023 1. Your institution: anonymized. 1025 2. Name(s) of people in your implementation and test teams: There 1026 were several folks involved in the implementation and testing. 1028 3. Do you want your answers to Question 1.1 and 1.2 above to be 1029 anonymised? Yes. 1031 3.2. Question 2: Preliminary information about your implementation 1032 Question 2 gathers some preliminary information. 1034 1. What OS is your implementation for? (or is it application layer?) 1035 anonymized (commercial OS) 1037 2. Do you support IPv4 or IPv6 addresses or both? Both. 1039 3. Is it publicly available (or will it be?) (for free or a fee?) 1040 No. 1042 4. Overall, what are you implementation and testing plans? (details 1043 can be given against individual items later) We plan to use it in a 1044 mobile environment. 1046 5. Is it an independent implementation? Or does it build on another 1047 MPTCP implementation -which one? It is an independent 1048 implementation. 1050 6. Have you already done some interop tests, for example with 1051 UCLouvain's "reference" Linux implementation? Most MPTCP option 1052 formats were tested with the reference Linux implementation. 1054 7. Would you be prepared to take part in an interop event, for 1055 example adjacent to IETF-87 in Berlin? Unsure at this point. 1057 3.3. Question 3: Support for MPTCP's Signalling Functionality 1058 Question 3 asks about support for the various signalling messages 1059 that the MPTCP protocol defines. *** For each message, please give a 1060 little information about the status of your implementation: for 1061 example, you may have implemented it and fully tested it; the 1062 implementation may be in progress; you have not yet implemented it 1063 but plan to soon (timescale?); you may you have no intention to 1064 implement it (why?); etc. 1066 1. Connection initiation (MP_CAPABLE) [Section 3.1 RFC6824] a. What 1067 is the status of your implementation? Fully implemented and tested 1068 against the reference Linux implementation. 1070 b. Any other comments or information? 1072 2. Starting a new subflow (MP_JOIN) [Section 3.2 RFC6824] a. What 1073 is the status of your implementation? Fully implemented and tested 1074 against the reference Linux implementation. 1076 b. Can either end of the connection start a new subflow (or only the 1077 initiator of the original subflow)? Only the initiator of the 1078 original sub flow can start other sub flows. 1080 c. What is the maximum number of subflows your implementation can 1081 support? There is no hard limit. 1083 d. Any other comments or information? 1085 3. Data transfer (DSS) [Section 3.3 RFC6824] a. What is the status 1086 of your implementation? Fully implemented and tested. 1088 b. The "Data ACK" field can be 4 or 8 octets. Which one(s) have you 1089 implemented? Both have been implemented but the use of the 4-byte 1090 field is the default. When an 8 byte DSS is received, an 8 byte Data 1091 ACK is sent in response. 1093 c. The "Data sequence number" field can be 4 or 8 octets. Which 1094 one(s) have you implemented? Both have been implemented but the use 1095 of the 4-byte field is the default. When a wraparound of the lower 1096 32-bit part of the DSS is detected, the full 8 byte DSS is sent. 1098 d. Does your implementation support the "DATA_FIN" operation to 1099 close an MPTCP connection? Yes. There are cases however where the 1100 sub flows are closed (TCP FIN'd) but the DATA_FIN is not sent - in 1101 this case the MPTCP connection must be closed through a garbage 1102 collector after some idle time. 1104 e. Does your implementation support the "Checksum" field (which is 1105 negotiated in the MP_CAPABLE handshake)? Yes. 1107 f. Any other comments or information? 1109 4. Address management (ADD_ADDR and REMOVE_ADDR) a. What is the 1110 status of your implementation? It does not support sending ADD_ADDR 1111 or processing ADD_ADDR as it is considered a security risk. Also, we 1112 only have a client side implementation at the moment which always 1113 initiates the sub flows. The remote end does not send ADD_ADDR in 1114 our configuration. The client can send REMOVE_ADDR however when one 1115 of the established sub flow's source address goes away. The client 1116 ignores incoming REMOVE_ADDR options also. 1118 b. Can your implementation do ADD_ADDRESS for addresses that appear 1119 *after* the connection has been established? No. c. Any other 1120 comments or information? 1122 5. Fast close (MP_FASTCLOSE) [Section 3.5 RFC6824] a. What is the 1123 status of your implementation? It is supported. Though 1124 Retransmission of Fast close is not supported yet. 1126 b. Any other comments or information? 1128 3.4. Question 4: Fallback from MPTCP Question 4 asks about action 1129 when there is a problem with MPTCP, for example due to a middlebox 1130 mangling MPTCP's signalling. The connection needs to fall back: if 1131 the problem is on the first subflow then MPTCP falls back to TCP, 1132 whilst if the problem is on an additional subflow then that subflow 1133 is closed with a TCP RST, as discussed in [Section 3.6 RFC6824]. 1135 1. If the MP_CAPABLE option is removed by a middlebox, does your 1136 implementation fall back to TCP? Yes. 1138 2. If the MP_JOIN option does not get through on the SYNs, does your 1139 implementation close the additional subflow? Yes. 1141 3. If the DSS option does not get through on the first data 1142 segment(s), does your implementation fall back? (either falling back 1143 to MPTCP (if the issue is on the first subflow) or closing the 1144 additional subflow (if the issue is on an additional subflow)) Yes it 1145 falls back to TCP when there's one sub flow. When there are multiple 1146 sub flows, since MPTCP is used in active/backup mode, it is assumed 1147 that the sub flow transferring data is most likely to be more usable 1148 than any other established sub flow. So the sub flow on which 1149 fallback occurred is kept alive and other sub flows are closed. 1150 Fallback though is not guaranteed to occur safely when there are more 1151 than one sub flows because the infinite mapping option may be 1152 stripped like other DSS options and the MP_FAIL option if used in 1153 scenarios other than for reporting checksum failure can also be 1154 stripped. 1156 4. Similarly, if the "DATA ACK" field does not correctly acknowledge 1157 the first data segment(s), does your implementation fall back? No. 1158 Current implementation just ignores the unexpected data ack. 1160 5. Does your implementation protect data with the "Checksum" field 1161 in the DSS option [Section 3.3 RFC6824]? If the checksum fails 1162 (because the subflow has been affected by a middlebox), does your 1163 implementation immediately close the affected subflow (with a TCP 1164 RST) with the MP_FAIL Option? If the checksum fails and there is a 1165 single subflow, does your implementation handle this as a special 1166 case, as described in [Section 3.6 RFC6824]? Yes. 1168 6. Does your implementation fall back to TCP by using an "infinite 1169 mapping" [Section 3.3.1 RFC6824] (so that the subflow-level data is 1170 mapped to the connection-level data for the remainder of the 1171 connection)? Yes. 1173 7. Did you find any corner cases where MPTCP's fallback didn't 1174 happen properly? If the very first sub flow does not send any data 1175 and is disconnected right away, then the current implementation 1176 allows a join to occur with the addition of another sub flow which 1177 then becomes a fully mp capable sub flow. Thus we allow break before 1178 make by letting additional sub flows to be joined if the very first 1179 one disconnected even without sending any data. This is a very 1180 corner case but an instance where we do not follow the rules of 1181 fallback (allow second sub flow even when first sub flow did not 1182 send/receive data/data acks). 1184 8. Any other comments or information about fallback? Fallback after 1185 connection establishment and after a few data packets were 1186 transferred with MPTCP options is complicated. The spec does not 1187 clearly cover the cases of options being stripped by middle boxes. 1188 It goes into good detail about what to do when the DSS checksum 1189 fails, but not when DSS checksum is not in use and the MPTCP options 1190 are stripped. Both sender/receiver behaviors could be outlined with 1191 more detail. 1193 3.5. Question 5: Heuristics Question 5 gathers information about 1194 heuristics: aspects that are not required for protocol correctness 1195 but impact the performance. We would like to document best practice 1196 so that future implementers can learn from the experience of 1197 pioneers. The references contain some initial comments about each 1198 topic. 1200 1. Receiver considerations [S3.3.4, RFC6824]: What receiver buffer 1201 have you used? Does this depend on the retransmission strategy? 1202 What advice should we give about the receiver? We are just using 1203 MPTCP in active/backup mode. This mode is simpler wrt receive buffer 1204 utilization. The receive buffer sizes at the MPTCP and sub flow 1205 level is the same. Automatic buffer tuning is turned off when MPTCP 1206 is in use. 1208 2. Sender considerations [S3.3.5, RFC6824]: How do you determine how 1209 much data a sender is allowed to send and how big the sender buffer 1210 is? What advice should we give about the sender? Automatic buffer 1211 tuning is turned off when MPTCP is in use. 1213 3. Reliability and retransmissions [S3.3.6, RFC6824]: What is your 1214 retransmission policy? (when do you retransmit on the original 1215 subflow vs on another subflow or subflows?) When do you decide that 1216 a subflow is underperforming and should be reset, and what do you 1217 then do? What advice should we give about this issue? 1218 Retransmissions at MPTCP level do not occur on the same sub flow 1219 except when MP_FAIL option is received. A sub flow is said to be 1220 underperforming when its network connectivity goes away. 1222 4. Port usage [S3.3.8.1, RFC6824]: Does your implementation use the 1223 same port number for additional subflows as for the first subflow? 1224 Have you used the ability to define a specific port in the Add 1225 Address option? What advice should we give about this issue? The 1226 destination port is the same. The local port changes for additional 1227 sub flows so on the wire it is like two tcp connections to the same 1228 remote destination. We have not used Add Address option at all. 1230 5. Delayed subflow start [S3.3.8.2, RFC6824]: What factors does your 1231 implementation consider when deciding about opening additional 1232 subflows? What advice should we give about this issue? The client 1233 implementation is aware of network interfaces coming up or going down 1234 and establishes new sub flows or removes existing sub flows 1235 accordingly. 1237 6. Failure handling [S3.3.8.3, RFC6824]: Whilst the protocol defines 1238 how to handle some unexpected signals, the behaviour after other 1239 unexpected signals is not defined. What advice should we give about 1240 this issue? Fallback, post establishment is probably a case that 1241 needs to be more clearly defined. 1243 7. Use of TCP options: As discussed in [Appendix A, RFC6824], the 1244 TCP option space is limited, but a brief study found there was enough 1245 room to fit all the MPTCP options. However there are constraints on 1246 which MPTCP option(s) can be included in packets with other TCP 1247 options - do the suggestions in Appendix A need amending or 1248 expanding? Looks good already. 1250 8. What other heuristics should we give advice about? Any other 1251 comments or information? 1253 3.6. Question 6: Security Question 6 asks about Security related 1254 matters [Section 5 RFC6824]. 1256 1. Does your implementation use the hash-based, HMACSHA1 security 1257 mechanism defined in [RFC6824]? Yes. 1259 2. Does your implementation support any other handshake algorithms? 1260 No. 1262 3. It has been suggested that a Standards-track MPTCP needs a more 1263 secure mechanism. Do you have any views about how to achieve this? 1264 No. But the mechanism could be tied with SSL because SSL is used 1265 wherever security is deemed important. 1267 4. Any other comments or information? 1269 3.7. Question 7: IANA Question 7 asks about IANA related matters. 1271 1. Does your implementation follow the IANA-related definitions? 1272 [Section 8 RFC6824] defines: TCP Option Kind number (30); the sub- 1273 registry for "MPTCP Option Subtypes"; and the Page 12 of 17 Survey 1274 6/22/13, 5:55 PM sub-registry for "MPTCP Handshake Algorithms" Yes. 1276 2. Any other comments or information? No. 1278 3.8. Question 8: Congestion control and subflow policy Question 8 1279 asks about how you share traffic across multiple subflows. 1281 1. How does your implementation share traffic over the available 1282 paths? For example: as a spare path on standby ('all-ornothing'), as 1283 an 'overflow', etc? Does it have the ability to send /receive 1284 traffic across multiple subflows simultaneously? It uses active/ 1285 backup where one sub flow is preferred or has higher priority over 1286 other sub flows. When the preferred sub flow fails or begins to 1287 experience retransmission timeouts, the other sub flows are used. 1289 2. Does your implementation support "handover" from one subflow to 1290 another when losing an interface? Yes. 1292 3. Does your implementation support the coupled congestion control 1293 defined in [RFC6356]? No. 1295 4. Does your implementation support some other coupled congestion 1296 control (ie that balances traffic on multiple paths according to 1297 feedback)? No. 1299 5. The MP_JOIN (Starting a new subflow) Option includes the "B" bit, 1300 which allows the sender to indicate whether it wishes the new subflow 1301 to be used immediately or as a backup if other path(s) fail. The 1302 MP_PRIO Option is a request to change the "B" bit - either on the 1303 subflow on which it is sent, or (by setting the optional Address ID 1304 field) on other subflows. Does your implementation support the "B" 1305 bit and MP_PRIO mechanisms? Do you think they're useful, or have 1306 another suggestion? Yes the implementation uses the B bit and the 1307 MP_PRIO option. They are very useful for the active/backup mode of 1308 operation. 1310 6. Any other comments or information or suggestions about the advice 1311 we should give about congestion control [S3.3.7 RFC6824] and subflow 1312 policy [S3.3.8 RFC6824]? 1314 3.9. Question 9: API Question 9 gathers information about your API. 1315 [RFC6897] considers the MPTCP Application Interface. 1317 1. With your implementation, can legacy applications use (the 1318 existing sockets API to use) MPTCP? How does the implementation 1319 decide whether to use MPTCP? Should the advice in [Section 4, 1320 RFC6897] be modified or expanded? The implementation does not 1321 support MPTCP with existing sockets API. MPTCP is exposed through a 1322 private SPI today. If MPTCP becomes prolific over the next few 1323 years, MPTCP use shall be expanded. 1325 2. The "basic MPTCP API" enables MPTCP-aware applications to 1326 interact with the MPTCP stack via five new socket options. For each 1327 one, have you implemented it? has it been useful? a. 1328 TCP_MULTIPATH_ENABLE? b. TCP_MULTIPATH_ADD? c. 1329 TCP_MULTIPATH_REMOVE? d. TCP_MULTIPATH_SUBFLOWS? e. 1330 TCP_MULTIPATH_CONNID? This mode of API is not used. Proprietary 1331 methods are used for achieving these basic operations. 1333 3. Have you implemented any aspects of an "advanced MPTCP API"? 1334 ([Appendix A, RFC6897] hints at what it might include.) No. 1336 4. Any other comments or information? 1338 3.10. Question 10: Deployments, use cases and operational 1339 experiences Question 10 takes the opportunity of this survey to 1340 gather some limited information about operational experiences and 1341 deployments. Any very brief information would be appreciated, for 1342 example: 1344 1. What deployment scenarios are you most interested in? MPTCP in 1345 mobile environments is very powerful when used in the active/backup 1346 mode. Since the network interfaces available on mobile devices have 1347 different cost characteristics as well as different bring up and 1348 power usage characteristics, it is not useful to share load across 1349 all available network interfaces - at least not currently. Providing 1350 session continuity across changing network environments is the key 1351 deployment scenario. 1353 2. Is your deployment on "the Internet" or in a controlled 1354 environment? The deployment is on the Internet. 1356 3. Is your deployment on end hosts or with a MPTCPenabled proxy (at 1357 one or both ends?)? The deployment supports MPTCP on both ends. 1359 4. What do you see as the most important benefits of MPTCP in your 1360 scenario(s)? Described in point 1 of this section. 1362 5. How extensively have you deployed and experimented with MPTCP so 1363 far? Deployment is still in early stages. We have been 1364 experimenting with MPTCP for about a year. 1366 6. MPTCP's design seeks to maximise the chances that the signalling 1367 works through middleboxes. Did you find cases where middleboxes 1368 blocked MPTCP signalling? Corporate firewalls block MPTCP signaling 1369 by default. IETF is one venue where Cisco, and other firewall 1370 vendors can be asked to change their defaults to allow MPTCP signals. 1372 7. MPTCP's design seeks to ensure that, if there is a problem with 1373 MPTCP signalling, then the connection either falls back to TCP or 1374 removes the problematic subflow. Did you find any corner cases where 1375 this didn't happen properly? This has been covered a bit in the 1376 Fallback section. When using two sub flows in active/backup mode, 1377 there is a possibility that a backup sub flow that never sent data 1378 starts being used for retransmitting data that is not going through 1379 on the active path. While it is preferable to keep the initial sub 1380 flow that successfully sent MPTCP options and drop the backup path, 1381 the initial sub flow may be the failing one, and we may want to move 1382 to the backup path. But the backup path can be retransmitting data 1383 that did not get sent successfully on the active path and if there is 1384 a middle box in the backup sub flow's path stripping options, then we 1385 have a case where the MPTCP session may not be recoverable as it may 1386 not be evident from what point in the MPTCP sequence space, data was 1387 being sent. The spec does talk of retaining the initial sub flow and 1388 closing the failed flow. So perhaps doing the reverse is not 1389 recommended, however, it would certainly be advantageous to support 1390 MPTCP better in such a failing environment. Also, in parallel 1391 working with firewall vendors to allow MPTCP options always to not 1392 have to over-engineer these cases. 1394 8. Have you encountered any issues or drawbacks with MPTCP? 1396 9. Any other comments or information? 1398 3.11. Question 11: Improvements to RFC6824 1. Are there any areas 1399 where [RFC6824] could be improved, either in technical content or 1400 clarity? Discussed in the fallback section. Other areas around 1401 MPTCP performance such as support for sub flow level automatic buffer 1402 scaling, varying QoS support, varying window scaling support on each 1403 sub flow may be worth discussing further, although they are outside 1404 the scope of the current spec. 1406 2. Any other issues you want to raise? Some additional work on 1407 option signaling that we will bring up in future discussions. 1409 10. Full survey response for Implementation 4 1411 1. Your institution: Citrix Systems, Inc. 1413 2. Name(s) of people in your implementation and test teams: NA 1415 3. Do you want your answers to Question 1.1 and 1.2 above to be 1416 anonymised? No 1418 3.2. Question 2: Preliminary information about your implementation 1419 Question 2 gathers some preliminary information. 1421 1. What OS is your implementation for? (or is it application layer?) 1422 NetScaler Firmware 1424 2. Do you support IPv4 or IPv6 addresses or both? Both 1426 3. Is it publicly available (or will it be?) (for free or a fee?) 1427 It is available for purchase 1429 4. Overall, what are you implementation and testing plans? (details 1430 can be given against individual items later) 1432 5. Is it an independent implementation? Or does it build on another 1433 MPTCP implementation -which one? It is an independent implementation 1435 6. Have you already done some interop tests, for example with 1436 UCLouvain's "reference" Linux implementation? Yes, our 1437 implementation is extensively tested with Linux reference 1438 implementation 1440 7. Would you be prepared to take part in an interop event, for 1441 example adjacent to IETF-87 in Berlin? 1443 3.3. Question 3: Support for MPTCP's Signalling Functionality 1444 Question 3 asks about support for the various signalling messages 1445 that the MPTCP protocol defines. *** For each message, please give a 1446 little information about the status of your implementation: for 1447 example, you may have implemented it and fully tested it; the 1448 implementation may be in progress; you have not yet implemented it 1449 but plan to soon (timescale?); you may you have no intention to 1450 implement it (why?); etc. 1452 1. Connection initiation (MP_CAPABLE) [Section 3.1 RFC6824] a. What 1453 is the status of your implementation? Fully implemented and tested 1455 b. Any other comments or information? One security concern here is 1456 that the keys are exchanged in plain text which is prone to attacks 1457 and also the key generation mechanism is highly computational 1458 intensive 1460 2. Starting a new subflow (MP_JOIN) [Section 3.2 RFC6824] a. What 1461 is the status of your implementation? Fully implemented and tested 1463 b. Can either end of the connection start a new subflow (or only the 1464 initiator of the original subflow)? Only the initiator of the 1465 original subflow can initiate additional subflows. 1467 c. What is the maximum number of subflows your implementation can 1468 support? we support maximum 6 subflows. 1470 d. Any other comments or information? 1472 3. Data transfer (DSS) [Section 3.3 RFC6824] a. What is the status 1473 of your implementation? Fully implemented and tested 1475 b. The "Data ACK" field can be 4 or 8 octets. Which one(s) have you 1476 implemented? Our implementation supports both 4 or 8 Octets Data Ack 1477 in both the directions 1479 c. The "Data sequence number" field can be 4 or 8 octets. Which 1480 one(s) have you implemented? Our implementation supports both 4 or 8 1481 Octets DSN in both the directions 1483 d. Does your implementation support the "DATA_FIN" operation to 1484 close an MPTCP connection? YES 1486 e. Does your implementation support the "Checksum" field (which is 1487 negotiated in the MP_CAPABLE handshake)? YES 1489 f. Any other comments or information? 1491 4. Address management (ADD_ADDR and REMOVE_ADDR) [Section 3.4 1492 RFC6824] 1494 a. What is the status of your implementation? REMOVE_ADDR is 1495 implemented and tested 1496 b. Can your implementation do ADD_ADDRESS for addresses that appear 1497 *after* the connection has been established? NO 1499 c. Any other comments or information? ADD_ADDRESS may not be much 1500 useful in the real environment situation given that most of the 1501 clients are behind the NATing devices. 1503 5. Fast close (MP_FASTCLOSE) [Section 3.5 RFC6824] a. What is the 1504 status of your implementation? Implemented and tested b. Any other 1505 comments or information? 1507 3.4. Question 4: Fallback from MPTCP Question 4 asks about action 1508 when there is a problem with MPTCP, for example due to a middlebox 1509 mangling MPTCP's signalling. The connection needs to fall back: if 1510 the problem is on the first subflow then MPTCP falls back to TCP, 1511 whilst if the problem is on an additional subflow then that subflow 1512 is closed with a TCP RST, as discussed in [Section 3.6 RFC6824]. 1514 1. If the MP_CAPABLE option is removed by a middlebox, does your 1515 implementation fall back to TCP? YES 1517 2. If the MP_JOIN option does not get through on the SYNs, does your 1518 implementation close the additional subflow? YES 1520 3. If the DSS option does not get through on the first data 1521 segment(s), does your implementation fall back? (either falling back 1522 to MPTCP (if the issue is on the first subflow) or closing the 1523 additional subflow (if the issue is on an additional subflow)) YES 1525 4. Similarly, if the "DATA ACK" field does not correctly acknowledge 1526 the first data segment(s), does your implementation fall back? If 1527 the sender receives pure ack for its first DSS packet then it 1528 fallsback to regular TCP. 1530 5. Does your implementation protect data with the "Checksum" field 1531 in the DSS option [Section 3.3 RFC6824]? If the checksum fails 1532 (because the subflow has been affected by a middlebox), does your 1533 implementation immediately close the affected subflow (with a TCP 1534 RST) with the MP_FAIL Option? If the checksum fails and there is a 1535 single subflow, does your implementation handle this as a special 1536 case, as described in [Section 3.6 RFC6824]? Yes, our implementation 1537 supports DSS checksum and will close the subflow with RST if the 1538 checksum validation fails and there are more than one subflows and 1539 sends MP_FAIL if there is a single subflow expecting infinite map 1540 from the peer. 1542 6. Does your implementation fall back to TCP by using an "infinite 1543 mapping" [Section 3.3.1 RFC6824] (so that the subflow-level data is 1544 mapped to the connection-level data for the remainder of the 1545 connection)? YES. 1547 7. Did you find any corner cases where MPTCP's fallback didn't 1548 happen properly? We have found few cases where the draft is not 1549 clear about the recommended action and fallback strategy, like: 1. 1550 what is the expected behavior when pure ack or data packet without 1551 dss is received in middle of transaction? How the hosts should 1552 fallback in this case? This can happen if the routing changes and 1553 the new path drops mptcp options. In this case MP_FAIL/infinite map 1554 exchange may not be possible and so could not decide whether both 1555 parties are in sync to fallback to tcp. 2. whether infinite map is 1556 unidirectional or bidirectional? If one host is sending infinite map 1557 to peer, does the peer also needs to send infinite map to the host? 1558 Exchanging infinite map and falling back to TCP from both ends is 1559 easy from implementation point of view. 8. Any other comments or 1560 information about fallback? 1562 3.5. Question 5: Heuristics Question 5 gathers information about 1563 heuristics: aspects that are not required for protocol correctness 1564 but impact the performance. We would like to document best practice 1565 so that future implementers can learn from the experience of 1566 pioneers. The references contain some initial comments about each 1567 topic. 1569 1. Receiver considerations [S3.3.4, RFC6824]: What receiver buffer 1570 have you used? Does this depend on the retransmission strategy? 1571 What advice should we give about the receiver? Our implementation 1572 uses varying buffer size based on the services and application type. 1574 2. Sender considerations [S3.3.5, RFC6824]: How do you determine how 1575 much data a sender is allowed to send and how big the sender buffer 1576 is? What advice should we give about the sender? The send side flow 1577 control is handled at mptcp level and is independent to subflows. 1578 The mptcp level flow control is (almost) same as the regular TCP flow 1579 control. 1581 3. Reliability and retransmissions [S3.3.6, RFC6824]: What is your 1582 retransmission policy? (when do you retransmit on the original 1583 subflow vs on another subflow or subflows?) When do you decide that 1584 a subflow is underperforming and should be reset, and what do you 1585 then do? What advice should we give about this issue? The 1586 retransmission is done by the subflows as long as the subflow is 1587 alive and is not removed by the REM_ADDR/RST/.. . If 3 RTO happens 1588 on the subflow doing retransmission and multiple subflows are 1589 available then the mptcp starts retransmission from additional 1590 subflow. The original subflow continues retransmission for 7RTO and 1591 will be closed after that with RST. 1593 4. Port usage [S3.3.8.1, RFC6824]: Does your implementation use the 1594 same port number for additional subflows as for the first subflow? 1595 Have you used the ability to define a specific port in the Add 1596 Address option? What advice should we give about this issue? Our 1597 current implementation doesnot support ADD_ADDR and subflow 1598 initiation. 1600 5. Delayed subflow start [S3.3.8.2, RFC6824]: What factors does your 1601 implementation consider when deciding about opening additional 1602 subflows? What advice should we give about this issue? NA 1604 6. Failure handling [S3.3.8.3, RFC6824]: Whilst the protocol defines 1605 how to handle some unexpected signals, the behaviour after other 1606 unexpected signals is not defined. What advice should we give about 1607 this issue? RFC should clearly define failure case handling 1608 otherwise it creates interoperability problems among various 1609 implementations. Our strategy in most of the unexpected failuire 1610 case is to send MP_FAIL RST with expected DSN if there are multiple 1611 subflows and MP_FAIL if there is a single subflow expecting infinite 1612 map from the peer. 1614 7. Use of TCP options: As discussed in [Appendix A, RFC6824], the 1615 TCP option space is limited, but a brief study found there was enough 1616 room to fit all the MPTCP options. However there are constraints on 1617 which MPTCP option(s) can be included in packets with other TCP 1618 options - do the suggestions in Appendix A need amending or 1619 expanding? Looks fine now. Atleast timestamp can be included with 1620 every dss packet (28bytes for dss and 12bytes for Timestamp), but if 1621 there are any other options which needs to be included in data 1622 packets then the implementation has to choose which one to include 1623 among them. 1625 8. What other heuristics should we give advice about? Any other 1626 comments or information? 1628 3.6. Question 6: Security Question 6 asks about Security related 1629 matters [Section 5 RFC6824]. 1631 1. Does your implementation use the hash-based, HMAC-SHA1 security 1632 mechanism defined in [RFC6824]? YES. 1634 2. Does your implementation support any other handshake algorithms? 1635 NO. 1637 3. It has been suggested that a Standards-track MPTCP needs a more 1638 secure mechanism. Do you have any views about how to achieve this? 1639 Yes we also feel more secure and light weight mechanism is required. 1641 4. Any other comments or information? 1643 3.7. Question 7: IANA Question 7 asks about IANA related matters. 1645 1. Does your implementation follow the IANA-related definitions? 1646 [Section 8 RFC6824] defines: TCP Option Kind number (30); the sub- 1647 registry for "MPTCP Option Subtypes"; and the sub-registry for "MPTCP 1648 Handshake Algorithms" YES. 2. Any other comments or information? 1650 3.8. Question 8: Congestion control and subflow policy Question 8 1651 asks about how you share traffic across multiple subflows. 1653 1. How does your implementation share traffic over the available 1654 paths? For example: as a spare path on standby ('all-or- nothing'), 1655 as an 'overflow', etc? Does it have the ability to send /receive 1656 traffic across multiple subflows simultaneously? We give preference 1657 to the path that client is currently using to send data/ack and also 1658 has policy based on primary/backup setup. We accept data from 1659 multiple subflows simultaneously but don't send it simultaneously 1660 out. 1662 2. Does your implementation support "handover" from one subflow to 1663 another when losing an interface? YES. 1665 3. Does your implementation support the coupled congestion control 1666 defined in [RFC6356]? NO. 1668 4. Does your implementation support some other coupled congestion 1669 control (ie that balances traffic on multiple paths according to 1670 feedback)? NO. 1672 5. The MP_JOIN (Starting a new subflow) Option includes the "B" bit, 1673 which allows the sender to indicate whether it wishes the new subflow 1674 to be used immediately or as a backup if other path(s) fail. The 1675 MP_PRIO Option is a request to change the "B" bit - either on the 1676 subflow on which it is sent, or (by setting the optional Address ID 1677 field) on other subflows. Does your implementation support the "B" 1678 bit and MP_PRIO mechanisms? Do you think they're useful, or have 1679 another suggestion? YES, our implementation supports both 'B' flag 1680 and MP_PRIO options, they are much useful to change the priority of 1681 the subflows and to decide which subflow to use for data transfer. 1683 6. Any other comments or information or suggestions about the advice 1684 we should give about congestion control [S3.3.7 RFC6824] and subflow 1685 policy [S3.3.8 RFC6824]? 1687 3.9. Question 9: API Question 9 gathers information about your API. 1688 [RFC6897] considers the MPTCP Application Interface. 1690 1. With your implementation, can legacy applications use (the 1691 existing sockets API to use) MPTCP? How does the implementation 1692 decide whether to use MPTCP? Should the advice in [Section 4, 1693 RFC6897] be modified or expanded? NA. 1695 2. The "basic MPTCP API" enables MPTCP-aware applications to 1696 interact with the MPTCP stack via five new socket options. For each 1697 one, have you implemented it? has it been useful? a. 1698 TCP_MULTIPATH_ENABLE? b. TCP_MULTIPATH_ADD? c. 1699 TCP_MULTIPATH_REMOVE? d. TCP_MULTIPATH_SUBFLOWS? e. 1700 TCP_MULTIPATH_CONNID? NA. 1702 3. Have you implemented any aspects of an "advanced MPTCP API"? 1703 ([Appendix A, RFC6897] hints at what it might include.) NA. 4. Any 1704 other comments or information? 1706 3.10. Question 10: Deployments, use cases and operational 1707 experiences Question 10 takes the opportunity of this survey to 1708 gather some limited information about operational experiences and 1709 deployments. Any very brief information would be appreciated, for 1710 example: 1712 1. What deployment scenarios are you most interested in? MPTCP 1713 Proxy deployment where the mptcp connections from the clients are 1714 terminated and the tcp connection is established on the other side. 1716 2. Is your deployment on "the Internet" or in a controlled 1717 environment? Targeted for the Internet deployment. 1719 3. Is your deployment on end hosts or with a MPTCP-enabled proxy (at 1720 one or both ends?)? Proxy. 1722 4. What do you see as the most important benefits of MPTCP in your 1723 scenario(s)? Reliability and fault tolerance. 1725 5. How extensively have you deployed and experimented with MPTCP so 1726 far? 1728 6. MPTCP's design seeks to maximise the chances that the signalling 1729 works through middleboxes. Did you find cases where middleboxes 1730 blocked MPTCP signalling? Yes some firewalls seem dropping MPTCP 1731 options. 1733 7. MPTCP's design seeks to ensure that, if there is a problem with 1734 MPTCP signalling, then the connection either falls back to TCP or 1735 removes the problematic subflow. Did you find any corner cases where 1736 this didn't happen properly? Few cases listed above. 1738 8. Have you encountered any issues or drawbacks with MPTCP? 9. Any 1739 other comments or information? 1741 3.11. Question 11: Improvements to RFC6824 1743 1. Are there any areas where [RFC6824] could be improved, either in 1744 technical content or clarity? More clarity required in fallback 1745 cases. 1747 2. Any other issues you want to raise? 1749 11. Normative References 1751 [RFC6356] Raiciu, C., Handley, M., and D. Wischik, "Coupled 1752 Congestion Control for Multipath Transport Protocols", 1753 RFC 6356, October 2011. 1755 [RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure, 1756 "TCP Extensions for Multipath Operation with Multiple 1757 Addresses", RFC 6824, January 2013. 1759 Author's Address 1761 Philip Eardley 1762 BT