wdiff draft-eardley-mptcp-implementations-survey-01.txt draft-eardley-mptcp-implementations-survey-02.txt

Network Working Group P. Eardley
Internet-Draft BT
Intended status: Informational May 02, July 12, 2013
Expires: November 03, 2013 January 13, 2014

Survey of MPTCP Implementations (blank version for implementers to fill
in)
draft-eardley-mptcp-implementations-survey-01
draft-eardley-mptcp-implementations-survey-02

Abstract

This document presents results from the survey gathers to gather information
from people who have implemented MPTCP, in particular to help
progress the protocol from Experimental to Standards track.

The document currently includes answers from four teams: a Linux
implementation from UCLouvain, a FreeBSD implementation from
Swinburne, an anonymous implementation in a commercial OS, and a
NetScalar Firmware implementation from Citrix Systems, Inc. Thank-
you!

In summary, we have four independent implementations of all the MPTCP
signalling messages, with the exception of address management, and
some interoperabiity testing has been done by the other three
implementations with the 'reference' Linux implementation. So it
appears that the RFC is (at least largely) clear and correct. On
address management, we have only one implementation of ADD_ADDR with
two teams choosing not to implement it. We have one implementation
of the working group's coupled congestion control (RFC6356) and none
of the MPTCP-aware API (RFC6897).

The main suggested improvements are around

o how MPTCP falls back (if the signalling is interrupted by a
middlebox): (1) corner cases that are not handled properly, (2) at
the IETF, the MPTCP community should work with middlebox vendors,
either to reduce or eliminate the need for fallback or to
understand the middlebox interactions better.

o security: both better MPTCP security (perhaps building on SSL) and
a lighter weight mechanism, preferably both in one mechanism.

It is hoped that the next version can include information from any
other implementations. If you are an implementer and want to
contribute your answers, please see the -01 version of this document
for a blank survey ready to be filled in, if you are an implementer. Thank-you! in.

Status of This this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."

This Internet-Draft will expire on November 03, 2013. January 13, 2014.

This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.

Table of Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Guidelines for filling in the survey . . . . . . 4
2. Survey - summary of replies . . . . . . 3
3. Survey . . . . . . . . . . . 4
3. Interesting aspects of replies . . . . . . . . . . . . . . . . 3 6
3.1. Question 1: Your details . . . . . . . . . . . . . . . . 3 . 6
3.2. Question 2: Preliminary information about your
implementation . . . . . . . . . . . . . . . . . . . . . 4 . 7
3.3. Question 3: Support for MPTCP's Signalling
Functionality 4 . . . . . . . . . . . . . . . . . . . . . . 7
3.4. Question 4: Fallback from MPTCP . . . . . . . . . . . . . 5 7
3.5. Question 5: Heuristics . . . . . . . . . . . . . . . . . 6 . 8
3.6. Question 6: Security . . . . . . . . . . . . . . . . . . 7 . 9
3.7. Question 7: IANA . . . . . . . . . . . . . . . . . . . . 8 . 9
3.8. Question 8: Congestion control and subflow policy . . . . 8 9
3.9. Question 9: API . . . . . . . . . . . . . . . . . . . . . 8 10
3.10. Question 10: Deployments, use cases and operational
experiences . . . . . . . . . . . . . . . . . . . . . . . 9 10
3.11. Question 11: Improvements to RFC6824 . . . . . . . . . . 10 . 11
4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 11
5. Security Considerations . . . . . . . . . . . . . . . . . . . 10 11
6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 10 . 11
7. Full survey response for Implementation 1 . . . . . . . . . . 11
8. Full survey response for Implementation 2 . . . . . . . . . . 19
9. Full survey response for Implementation 3 . . . . . . . . . . 23
10. Full survey response for Implementation 4 . . . . . . . . . . 31
11. Normative References . . . . . . . . . . . . . . . . . . . . 10 . 38
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 10 . 38

1. Introduction

The goal of this document reports the results from a survey is to gather information
from people who have implemented MPTCP, in particular MPTCP. The goal is to help progress
the protocol from Experimental to Standards track.

Four responses have been received. Thank-you! They are independent
implementations:

o the Linux implementation from UCLouvain,

o the FreeBSD implementation from Swinburne

o an anonymous implementation in a commercial OS

o a NetScaler Firmware implementation from Citrix Systems, Inc.

The MPTCP WG's current charter (2013-03-14) states: "The primary goal Table below presents a highly-compressed summary, with each row
corresponding to one question or sub-question of the working group survey. The
following section highlights some interesting aspects of the replies
in less compressed form. The full survey responses are in Appendix
A, B, C and D.

It is to create a bis hoped that the next version of the protocol this document on can include
information about a further (independent) implementation:

o Georg Hampel's user-space implementation (publicly available but
not longer maintained)

o any other implementations.

2. Survey - summary of replies

The Table below presents a highly-compressed summary, with each row
corresponding to one question or sub-question of the Standards track. This develops survey. A
column is left blank for any future responses.

+----------------------------------------------------------------------+
| | 1 | 2 | 3 | 4 | |
|Institution | UCLouvain | Swinburne | Anon | Citrix | |
| |
| Question 2 asks about some preliminary topics, including whether the current
Experimental document ... incorporating experience from (for example)
implementations, |
| implementation is publicly available and interoperability events, experiments, usage
scenarios, protocol corner cases, with the |
| Linux implementation (#1). |
| | UCLouvain | Swinburne | Anon | Citrix | |
|OS |Linux |FreeBSD-10 |Commercial |NetScaler | |
|v4 & v6 |Both | IPv4 |Both |Both | |
|public |Yes |Yes |No |Yes (pay) | |
|independent |Yes |Yes |Yes |Yes | |
|interop |Yes(!) |Mostly |Mostly |Yes | |
| |
| Question 3: Support for MPTCP's signalling functionality |
| MPTCP's signalling messages are: MP_CAPABLE, MP_JOIN, Data transfer |
| (DSS), ADD_ADDR, REMOVE_ADDR, MP_FASTCLOSE. There are sub-questions |
| for MP_JOIN and feedback DSS. |
| | UCLouvain | Swinburne | Anon | Citrix | |
|MP_CAPABLE |Yes |Yes |Yes |Yes | |
|MP_JOIN |Yes |Yes |Yes |Yes | |
|initiated by|first end |either end |first end |first end | |
|#subflows |32 |8 |no limit |6 | |
|DSS |Yes |Yes |Yes |Yes | |
|DATA ACK |4 bytes |4 or 8 byte|4 or 8 byte|4 or 8 byte| |
|Data seq num|4 bytes |4 or 8 byte|4 or 8 byte|4 or 8 byte| |
|DATA_FIN |Yes |Yes |Yes |Yes | |
|Checksum |Yes |No |Yes |Yes | |
|ADD_ADDR |Yes |No |No (never) |No (never?)| |
|REMOVE_ADDR |Yes |No |Partly |Yes | |
|FAST_CLOSE |Yes |No |Yes |Yes | |
| |
| Question 4 asks about fallback from TCPM ... [Also,
the] working group will document MPTCP: if a middlebox mangles |
| MPTCP's signalling by removing MP_CAPABLE, MP_JOIN, DSS or DATA_ACK; |
| if data is protected with Checksum in DSS option; if fallback to TCP |
| uses an infinte mapping; and if any corner cases have been found. |
| | UCLouvain | Swinburne | Anon | Citrix | |
|MP_CAPABLE |Yes |Yes |Yes |Yes | |
|MP_JOIN |Yes |Yes |Yes |Yes | |
|DSS |Yes |No |Yes |Yes | |
|DATA_ACK |Yes |No |No | | |
|Checksum |Yes |No |Yes |Yes | |
|infinte map |Yes |Yes |Yes |Yes | |
|corner cases|No | |Yes |Yes | |
| |
| Question 5 asks about heuristics: aspects that are not required for |
| protocol correctness but impact the performance. Questions are about |
| sized the receiver and sender buffers, re-transmission policy, if |
| additional subflows use the same port number as for the first subflow|
| | UCLouvain | Swinburne | Anon | Citrix | |
|Recvr buffer|auto-tune |TCP_MAXWIN |no tuning |tuned | |
|Sendr buffer|auto-tune |cwnd |no tuning |as TCP | |
|Re-transmits|2nd subflow|2nd subflow|2nd subflow|1st subflow| |
|Port usage |same ports |same ports |diff local | | |
| |
| Question 6 asks about what security mechanisms are implemented: the |
| one defined in RFC6824 and any others. |
| | UCLouvain | Swinburne | Anon | Citrix | |
|HMAC-SHA1 |Yes |Yes |Yes |Yes | |
|other |Yes |No |No |No | |
| |
| Question 7 asks whether the implementation advice. follows the IANA-related |
| definitions (for TCP Option Kind and sub-registries). |
| | UCLouvain | Swinburne | Anon | Citrix | |
|RFC6824 |Yes |Yes |Yes |Yes | |
| |
| Question 8 asks about congestion control and related issues: how |
| traffic is shared across multiple subflows; support for 'handover'; |
| and support of RFC6356 (or other) coupled congestion control. |
| | UCLouvain | Swinburne | Anon | Citrix | |
|sharing |shared, RTT|shared |active/back|active/back| |
|handover |Yes | |Yes |Yes | |
|coupled cc |Yes |No |No |No | |
|other ccc |Yes, OLIA |No |No |No | |
|MP-PRIO & B |Yes |No |Yes |Yes | |
| |
| Question 9 is about the API: how legacy applications interact with |
| the MPTCP stack, and if implemented the RFC6897 API for MPTCP-aware |
| applications. |
| | UCLouvain | Swinburne | Anon | Citrix | |
|legacy apps |default |sysctl |private API|configured | |
|MPTCP API |No |No |No |No | |
|advanced API|No |No |No |No | |
| |
| Question 10 gathers some limited information about operational |
| experiences and deployments. |
| | UCLouvain | Swinburne | Anon | Citrix | |
|Scenario |several |several |mobile |proxy | |
|environment |internet |controlled |internet |internet | |
|ends / proxy|end hosts |end hosts |end hosts |proxy | |
| |
+----------------------------------------------------------------------+

3. Interesting aspects of replies

This section tries to highlight some interesting comments made in the
surveys. The current
documents have several points where Appendices can be consulted for further detials.

3.1. Question 1: Your details

Implementation 1 has been implemented by Sebastien Barre, Christoph
Paasch and a large team, mainly at UCLouvain. Implementation 2 has
been implemented by Lawrence Stewart and Nigel Williams at Swinburne
University of Technology. Both these implementations are publicly
available. Implementation 3 comes from an implementer may benefit anonymous team with a
commercial OS. Implementation 4 comes from
guidance, for example Citrix Systems, Inc.

3.2. Question 2: Preliminary information about heuristics such your implementation

Three of the four implementations are publicly available, two for
free (under GPLv2 and BSD licences) and one for a fee (NetScaler
Firmware). Implementation 3 (commercial OS) is planned for use in a
mobile environment, with MPTCP is used in active/backup mode.

All implementations support IPv4 and three of four support IPv6.

All implementations are being actively worked on, in order to improve
performance and stability and conformance with the RFC.

3.3. Question 3: Support for MPTCP's Signalling Functionality

Three of the four implementations have implemented all the MPTCP
signalling, with the interesting exception of address management,
whilst Implementation 2 plans to add support for all those signalling
capabilities it does not yet support.

On address management, two implementations have decided not to
implement ADD_ADDR. (ADD_ADDR allows an MPTCP host to signal a new
address explicitly to the other host to allow it to initiate a new
subflow - as buffer sizing, an alternative to using MP_JOIN to directly start a new
subflow). Implementation 3 decided not to support sending ADD_ADDR
or processing ADD_ADDR as it is considered a security risk.
Implementation 4 decided not to support ADD_ADDR because it didn't
think it would be useful as most clients are behind NATing devices.
However, both implemented REMOVE_ADDR (in Implementation 3 the client
can send a REMOVE_ADDR but ignores incoming REMOVE_ADDR).

In Implementations 1, 3 and 4 only the initiator of the original
subflow can start a new subflow (a reason mentioned is that NATs make
it hard for the server to reach the client).

All implementations support 4 bytes "Data ACK" and "Data sequence
number" fields, and will interoperate with an implementation sending
8 bytes. Implementation 1 uses only 4 bytes fields; if an
implementation sends an 8 byte data sequence number it replies with a
4 byte data ack.

3.4. Question 4: Fallback from
advice MPTCP

Question 4 asks about alternative implementations such as bump-in-the-stack." action when there is a problem with MPTCP, for
example due to a middlebox mangling MPTCP's signalling. The survey
connection needs to fall back: if the problem is on the first subflow
then MPTCP falls back to TCP, whilst if the problem is on an
additional subflow then that subflow is closed with a TCP RST, as
discussed in [Section 3.6 RFC6824].

Implementations 3 and 4 made several comments about fallback.

Implementation 3 suggests that both sender and receiver behaviours
could be outlined with more detail, in particular when DSS checksum
is not in use and the MPTCP options are stripped. Implementation 3
falls back to TCP when there's one sub flow, but not when there are
multiple sub flows (MPTCP is used in active/backup mode, and it is
assumed that the sub flow transferring data is most likely to be more
usable than any other established sub flow, hence the sub flow on
which fallback occurred is kept alive and other sub flows are
closed).

Implementation 4 found a corner case where it is not clear what to
do: if a pure ack or data packet without DSS is received in middle of
transaction (which can happen if the routing changes and the new path
drops MPTCP options). Also, Implementation 4 suggests that
clarifying whether the infinite map exchange is unidirectional or
bidirectional.

Implementation 1 has developed a publicly available test suite that
tests MPTCP's traversal of middleboxes.

3.5. Question 5: Heuristics

Question 5 gathers relevant information to help this, about heuristics: aspects that are not
required for example
about: protocol correctness but impact the performance. We
would like to document best practice so that future implementers can
learn from the existence experience of running code (especially pioneers.

There are several independent,
interoperable implementations); differences between the parts implementations.

For receiver buffer, Implementation 1 uses a slightly modified
version of Linux's auto-tuning algorithm; Implementation 2 determines
the experimental spec receiver buffer by using "TCP_MAXWIN << tp->rcv_scale" (this is a
temporary measure); Implementation 3 uses MPTCP in active/backup
mode, so the receive buffer sizes at the MPTCP and subflow level is
the same (automatic buffer tuning is turned off); Implementation 4
varies the receiver buffer size based on the services and application
type.

For the sender buffer, Implementation 1 uses Linux auto-tuning,
Implementation 2 scales based on occupancy, whilst Implementation 3
turns off automatic buffer tuning, and Implementation 4 uses MPTCP-
level (sub)flow control that have is (almost) the same as regular TCP flow
control.

Implementations 1, 2 and 3 re-transmit unacknowledged data on a
different subflow (and not been the same subflow), whilst Implementation 4
re-transmits on original subflow for 3 RTOs and then uses another
subflow.

For port usage, Implementations 1 and 2 uses the same ports for the
additional subflows, whilst Implementation 3 uses the same
detsination port but a different local port, so that on the wire it
looks like two connections to the same remote destination.

Implementation 4 suggests that the RFC should more clearly
/extensively define failure cases and how to handle unexpected
signals.

3.6. Question 6: Security

Question 6 asks about security related matters.

All Implementations have implemented (or not been used); the parts hash-based, HMAC-SHA1
security mechanism defined in [RFC6824]. Implementation 3 suggests
that
need improvement, a more secure mechanism could be tied with SSL. Implementation
4 suggests that a more secure and lightweight mechanism is needed, as
keys are exchanged (in the MP_CAPABLE option) in terms of functionality or clarity. plain text and the
key generation mechanism is highly computational intensive.
Implementation 1 has implemented two additional mechanisms in a
separate Linux branch - one lightweight and the other SSL-based.

3.7. Question 7: IANA

All Implementations have followed the IANA-related definitions
[Section 8 RFC6824] for: TCP Option Kind number (30); the sub-
registry for "MPTCP Option Subtypes"; and the sub-registry for "MPTCP
Handshake Algorithms".

3.8. Question 8: Congestion control and subflow policy

Question 8 asks how is shared across multiple subflows.

Implementation 1 has added support for coupled congestion control
(both that defined in [RFC6356] and in OLIA,
draft-khalili-mptcp-congestion-control. The survey other implementations do
not include coupled congestion control. Whilst Implementation 2
plans to add it (currently it uses a simple algorithm spreads traffic
across the subflows), Implementations 3 and 4 do not plan to add
coupled congestion control - they use one subflow at a time, with
others as a backup. Implementation 3 believes it is not currently
useful to share load across all network interfaces on a mobile node,
as the interfaces have different characteristics for cost, bring-up
and power usage. They have both found the B bit (in MP-JOIN) and MP-
PRIO option very useful for this active /backup operation.

Implementation 2 is also interested in experimenting with congestion
control across paths with different path-cost metrics.

3.9. Question 9: API

Question 9 gathers information about the API. None have implemented
the [RFC6897] "basic MPTCP API" for MPTCP-aware applications. For
three implementations MPTCP is used for all applications (set by
configuration), whilst Implementation 3 uses a private API that
allows MPTCP to be used on a per application basis.

3.10. Question 10: Deployments, use cases and operational experiences

Question 10 takes the opportunity of this survey to gather some
limited information about operational experiences and deployments.

The WG Implementations mention different use cases.

Implementation 2 is very grateful interested in using MPTCP for several use cases:
vehicle to those implementers taking infrastructure (V2I) connectivity (to provide a persistent
connection using 3G and roadside wifi); multi-homed "home-user"
environments; high throughput data transfers. Implementation 3 is
interested in the trouble to
fill mobile scenario, with MPTCP providing an active
/backup mode so achieving session continuity across changing network
environments. Implementation 4 is interested in this survey - MPTCP giving
reliability and fault tolerance via a proxy. Implementation 1
already uses MPTCP on www.multipath-tcp.org and for internal ssh
servers at UCLouvain.

Implementation 4 uses a proxy (MPTCP connections from a client are
terminated and the TCP connection established on the other side),
whilst the other Implementations are on end hosts. Implementation 2
is so far within controlled testbeds, whilst Implementation 3 is on
the Internet.

Implementation 2 is currently an alpha-quality build, so limited
testing so far.

Implementation 3 suggests working at the IETF with firewall vendors,
to get them to change their defaults to allow MPTCP signals. This
would also reduce the "over-engineering" needed to handle fallback
cases. Implementation 1 suggests retrieving logs from middleboxes,
as well, the best approach to understanding the interactions of course, as their effort in creating MPTCP
signalling with middleboxes.

Implementation 3 discusses a scenario that should be handled better.
A backup subflow may never sent data. If the implementation! initial subflow fails,
data is retransmitted on the backup subflow, but that path has a
middlebox stripping options. Then it may not be possible to recover
the MPTCP session.

3.11. Question 11: Improvements to RFC6824

Question 11 asks if there are any areas where RFC6824 could be
improved. The intention main topics have been mentioned earlier:

o fallback: the need for more clarity in the fallback cases is
mentioned by Implementations 3 and 4.

o security: the need for both a more secure and a more lightweight
mechanism is mentioned.

Implementation 3 also suggests several potential improvements, which
are outside the scope of RFC6824: support for sub flow level
automatic buffer scaling, varying QoS support, and varying window
scaling support on each sub flow; also, additional work on option
signlling will be brought up in future discussions.

4. IANA Considerations

This document makes no request of IANA.

5. Security Considerations

This survey does not impact the security of MPTCP, except to produce the
extent that it uncovers security issues that can be tackled in a revised
future version of this draft the protocol.

6. Acknowledgements

Many thanks to the people who replied to the survey: Christoph
Paasch, Nigel Williams, anon, and Krishna Khanal. Very many thanks
to all of the teams who actually did the implementation and testing
and are continuing to improve them.

7. Full survey response for Implementation 1

Question 1: Your details Question 1 gathers some information about
the team that
incorporates a summary has implemented MPTCP.

1. Your institution: UCLouvain, IP Networking Lab
(http://inl.info.ucl.ac.be)

2. Name(s) of people in your implementation and test teams: Initial
design from Sebastien Barre. Since then, numerous code-contributors
(ordered by number of commits): Christoph Paasch (UCLouvain) Gregory
Detal (UCLouvain) Jakko Korkeaniemi (Aalto University) Mihai P.
Andrei (Intel) Fabien Duchene (UCLouvain) Andreas Seelinger (RWTH
Aachen) Stefan Sicleru (Intel) Lavkesh Lahngir Catalin Nicutar (PUB
Bucharest) Andrei Maruseac (Intel) Andreas Ripke (NEC) Vlad Dogaru
(Intel) Octavian Purdila (Intel) Niels Laukens (VRT Belgium) John
Ronan (TSSG) Brandon Heller (Stanford University) Conformance
Testing: Yvan Coene (UCLouvain)

3. Do you want your answers to Question 1.1 and 1.2 above to be
anonymised? No.

3.2. Question 2: Preliminary information about your implementation
Question 2 gathers some preliminary information.

1. What OS is your implementation for? (or is it application layer?)
Linux Kernel.

2. Do you support IPv4 or IPv6 addresses or both? We support both.

3. Is it publicly available (or will it be?) (for free or a fee?)
Publicly available (GPLv2) at www.multipath-tcp.org

4. Overall, what are you implementation and testing plans? (details
can be given against individual items later) We plan to continue to
align our implementation with the replies IETF specifications and improve its
performance and stability.

5. Is it an appendix independent implementation? Or does it build on another
MPTCP implementation -which one? Independent implementation.

6. Have you already done some interop tests, for example with all
UCLouvain's "reference" Linux implementation? /

7. Would you be prepared to take part in an interop event, for
example adjacent to IETF-87 in Berlin? Yes. We are also ready to
help in organising such an event if needed.

3.3. Question 3: Support for MPTCP's Signalling Functionality
Question 3 asks about support for the
full responses. various signalling messages
that the MPTCP protocol defines. *** For each message, please give a
little information about the status of your implementation: for
example, you may have implemented it and fully tested it; the
implementation may be in progress; you have not yet implemented it
but plan to soon (timescale?); you may you have no intention to
implement it (why?); etc.

1. Connection initiation (MP_CAPABLE) [Section 3.1 RFC6824]

a. What is the status of your implementation? Fully support the
MP_CAPABLE exchange.

b. Any other comments or information? We generate the random key as
a hash of the 5-tuple, sequence number and a local secret. This
significantly improves the performance, instead of using a pseudo-
random number generator. The performance benefit has been shown
during IETF85
http://tools.ietf.org/agenda/85/slides/slides-85-mptcp-2.pdf

2. Guidelines Starting a new subflow (MP_JOIN) [Section 3.2 RFC6824]

a. What is the status of your implementation? Fully support the
MP_JOIN exchange.

b. Can either end of the connection start a new subflow (or only the
initiator of the original subflow)? Currently, only the initiator of
the original subflow starts a new subflow. Given the widespread
deployment of NATs, it is often difficult for filling the server to reach the
client. This is the main reason why the server currently does not
start new subflows in our implementation. But, the survey

Please could initiator would
accept a SYN+MP_JOIN if sent by another implementation.

c. What is the maximum number of subflows your implementation can
support? Currently 32.

d. Any other comments or information?

3. Data transfer (DSS) [Section 3.3 RFC6824]

a. What is the status of your implementation? Fully working
implementation of data transfer.

b. The "Data ACK" field can be 4 or 8 octets. Which one(s) have you
implemented? We use 4 bytes for the DATA-ACK field.

c. The "Data sequence number" field can be 4 or 8 octets. Which
one(s) have you implemented? We use 4 bytes for the data sequence
number.

d. Does your implementation support the "DATA_FIN" operation to
close an MPTCP connection? Yes.

e. Does your implementation support the "Checksum" field (which is
negotiated in the MP_CAPABLE handshake)? Yes. This is configurable
via a sysctl.

f. Any other comments or information? We support interoperability
with implementations that do send 64-bit data sequence numbers and
data acks. However, even if the peer sends 64-bit data sequence
numbers, we will only reply with a 32-bit data-ack. We do not have
heuristics to trigger the sending of DATA_ACKs. We simply send the
DATA_ACK in each packet.

4. Address management (ADD_ADDR and REMOVE_ADDR) [Section 3.4
RFC6824]

a. What is the status of your implementation? We support ADD_ADDR/
REMOVE_ADDR messages.

b. Can your implementation team fill do ADD_ADDRESS for addresses that appear
*after* the connection has been established? Yes, as shown in:
"Exploring Mobile/WiFi Handover with Multipath TCP", C. Paasch et.
al, ACM SIGCOMM workshop on Cellular Networks (Cellnet'12), 2012.

c. Any other comments or information? We do not send out TCP
keepalive-messages upon the reception of a REMOVE_ADDR-message.

5. Fast close (MP_FASTCLOSE) [Section 3.5 RFC6824]

a. What is the status of your implementation? We support the
MP_FASTCLOSE implementation.

b. Any other comments or information?

3.4. Question 4: Fallback from MPTCP Question 4 asks about action
when there is a problem with MPTCP, for example due to a middlebox
mangling MPTCP's signalling. The connection needs to fall back: if
the problem is on the first subflow then MPTCP falls back to TCP,
whilst if the problem is on an additional subflow then that subflow
is closed with a TCP RST, as discussed in [Section 3.6 RFC6824].

1. If the survey. MP_CAPABLE option is removed by a middlebox, does your
implementation fall back to TCP? Yes.

2. If you
cannot answer the MP_JOIN option does not get through on the SYNs, does your
implementation close the additional subflow? Yes.

3. If the DSS option does not get through on the first data
segment(s), does your implementation fall back? (either falling back
to MPTCP (if the issue is on the first subflow) or closing the
additional subflow (if the issue is on an additional subflow)) Yes.
On the initial subflow we do a question just skip it, seamless fallback, additional subflows
will be closed by a RST.

4. Similarly, if the "DATA ACK" field does not correctly acknowledge
the first data segment(s), does your implementation fall back? Yes.
Same as partial responses are also
very valuable.

Please return above.

5. Does your completed survey implementation protect data with the "Checksum" field
in the DSS option [Section 3.3 RFC6824]? If the checksum fails
(because the subflow has been affected by email attachment a middlebox), does your
implementation immediately close the affected subflow (with a TCP
RST) with the MP_FAIL Option? If the checksum fails and there is a
single subflow, does your implementation handle this as a special
case, as described in [Section 3.6 RFC6824]? Yes, we support the
DSS-checksum. If the checksum is wrong and there exist other
subflows, we close the current subflow with an RST. If there is no
other subflow, we send an ACK + MP_FAIL and do a fallback to infinite
mapping. This fallback has successfully been tested with different
type of NAT middleboxes, while using FTP.

6. Does your implementation fall back to TCP by using an "infinite
mapping" [Section 3.3.1 RFC6824] (so that the MPTCP
WG Chairs: Philip Eardley <philip.eardley@bt.com> subflow-level data is
mapped to the connection-level data for the remainder of the
connection)? Yes.

7. Did you find any corner cases where MPTCP's fallback didn't
happen properly? No. We have developped a test-suite to test the
middlebox-traversal of MPTCP, available at
http://multipath-tcp.org/pmwiki.php/Users/AboutMeasures

8. Any other comments or information about fallback?

3.5. Question 5: Heuristics Question 5 gathers information about
heuristics: aspects that are not required for protocol correctness
but impact the performance. We would like to document best practice
so that future implementers can learn from the experience of
pioneers. The references contain some initial comments about each
topic.

1. Receiver considerations [S3.3.4, RFC6824]: What receiver buffer
have you used? Does this depend on the retransmission strategy?
What advice should we give about the receiver? Linux includes an
autuning algorithm for the TCP receiver buffer. This algorithm has
been slightly modified for Multipath TCP. The receive-buffer does
not depend on the retransmission strategy.

2. Sender considerations [S3.3.5, RFC6824]: How do you determine how
much data a sender is allowed to send and Yoshifumi
Nishida <nishida@sfc.wide.ad.jp>. how big the sender buffer
is? What advice should we give about the sender? The send-buffer is
autotuned similarly as the receive-buffer (see above). We send as
much data as possible, filling the congestion windows of each
subflow. The sender deploys the "Opportunistic Retransmission" and
"Penalization" algorithms from the paper: "How Hard Can It Be?
Designing and Implementing a Deployable Multipath TCP", C. Raiciu et.
al, NSDI 2012.

3. Reliability and retransmissions [S3.3.6, RFC6824]: What is your
retransmission policy? (when do you retransmit on the original
subflow vs on another subflow or subflows?) When do you decide that
a subflow is underperforming and should be reset, and what do you
then do? What advice should we give about this issue? Upon an RTO
on subflow A, we reinject all the unacknowledged data of subflow A on
another subflows. We do not currently have a mechanism to detect
that a subflow is underperforming.

4. Port usage [S3.3.8.1, RFC6824]: Does your implementation use the
same port number for additional subflows as for the first subflow?
Have you used the ability to define a specific port in the Add
Address option? What advice should we give about this issue? We
always use the same port number as for the first subflow. Except, if
the ADD_ADDRESS option that has been received contained a specific
port. We do not have a means to configure the specific port in the
ADD_ADDRESS option, but we support reception of the port.

5. Delayed subflow start [S3.3.8.2, RFC6824]: What factors does your
implementation consider when deciding about opening additional
subflows? What advice should we give about this issue? As soon as
we are sure that the initial subflow is fully MPTCP-capable
(reception of a DATA_ACK), we create a full mesh among all IP-
addresses between the two hosts. We do not explicitly delay the
creation of new subflows.

7. Use of TCP options: As discussed in [Appendix A, RFC6824], the
TCP option space is limited, but a brief study found there was enough
room to fit all the MPTCP options. However there are constraints on
which MPTCP option(s) can be included in packets with other TCP
options - do the suggestions in Appendix A need amending or
expanding? We do not implement specific heuristics to reduce the TCP
option-space usage. If timestamp is enabled we will only be able to
send two SACK-blocks, because the DATA_ACK consumes the remaining
bytes.

8. What other heuristics should we give advice about? Any other
comments or information?

3.6. Question 6: Security Question 6 asks about Security related
matters [Section 5 RFC6824].

1. Does your implementation use the hash-based, HMAC-SHA1 security
mechanism defined in [RFC6824]? Yes.

2. Does your implementation support any other handshake algorithms?
We have in a separate branch, an implementation of
draft-paasch-mptcp-lowoverhead and draft-paasch-mptcp-ssl.

3. It has been suggested that a Standards-track MPTCP needs a more
secure mechanism. Do you wish have any views about how to achieve this?
We believe that the solution described in draft-paasch-mptcp-ssl
would be a good starting point since it leverages the security of the
upper layer.

4. Any other comments or information?

3.7. Question 7: IANA Question 7 asks about IANA related matters.

1. Does your implementation follow the IANA-related definitions?
[Section 8 RFC6824] defines: TCP Option Kind number (30); the sub-
registry for "MPTCP Option Subtypes"; and the sub-registry for "MPTCP
Handshake Algorithms" Yes.

2. Any other comments or information?

3.8. Question 8: Congestion control and subflow policy Question 8
asks about how you share traffic across multiple subflows.

1. How does your implementation share traffic over the available
paths? For example: as a spare path on standby ('all-or- nothing'),
as an 'overflow', etc? Does it have the ability to send /receive
traffic across multiple subflows simultaneously? The implementation
is able to send and receive traffic on all subflows simultaneously.
Our scheduler first tries to send traffic on the subflow with the
lowest RTT. As this subflow's congestion window is full, we pick the
subflow with the next lower RTT.

2. Does your implementation support "handover" from one subflow to
another when losing an interface? Yes, as described in: "Exploring
Mobile/WiFi Handover with Multipath TCP", C. Paasch et. al, ACM
SIGCOMM workshop on Cellular Networks (Cellnet'12), 2012.

3. Does your implementation support the coupled congestion control
defined in [RFC6356]? Yes.

4. Does your implementation support some other coupled congestion
control (ie that balances traffic on multiple paths according to
feedback)? We also support the OLIA congestion control
(draft-khalili-mptcp-congestion-control-00).

5. The MP_JOIN (Starting a new subflow) Option includes the "B" bit,
which allows the sender to remain anonymous, please indicate -- we'll replace whether it wishes the new subflow
to be used immediately or as a backup if other path(s) fail. The
MP_PRIO Option is a request to change the "B" bit - either on the
subflow on which it is sent, or (by setting the optional Address ID
field) on other subflows. Does your answers implementation support the "B"
bit and MP_PRIO mechanisms? Do you think they're useful, or have
another suggestion? Yes, we support the "B"-bit of the MP_JOIN and
the MP_PRIO option. It is configurable on a per-interface basis.
Experiences with the "B"-bit can be found in our paper: "Exploring
Mobile/WiFi Handover with Multipath TCP", C. Paasch et. al, ACM
SIGCOMM workshop on Cellular Networks (Cellnet'12), 2012.

6. Any other comments or information or suggestions about the advice
we should give about congestion control [S3.3.7 RFC6824] and subflow
policy [S3.3.8 RFC6824]?

3.9. Question 1 by "Anonymous Implementation #n", but
publish 9: API Question 9 gathers information about your API.
[RFC6897] considers the MPTCP Application Interface.

1. With your implementation, can legacy applications use (the
existing sockets API to use) MPTCP? How does the implementation
decide whether to use MPTCP? Should the advice in [Section 4,
RFC6897] be modified or expanded? Yes, a standard TCP socket API can
be used. By default MPTCP is enabled on all connections.

2. The "basic MPTCP API" enables MPTCP-aware applications to
interact with the MPTCP stack via five new socket options. For each
one, have you implemented it? has it been useful? None of them are
part of the current stable release MPTCP v0.86.
http://multipath-tcp.org/pmwiki.php?n=Main.Release86 a.
TCP_MULTIPATH_ENABLE? b. TCP_MULTIPATH_ADD? c.
TCP_MULTIPATH_REMOVE? d. TCP_MULTIPATH_SUBFLOWS? e.
TCP_MULTIPATH_CONNID?

3. Have you implemented any aspects of an "advanced MPTCP API"?
([Appendix A, RFC6897] hints at what it might include.) No.

4. Any other comments or information?

3.10. Question 10: Deployments, use cases and operational
experiences Question 10 takes the opportunity of this survey to
gather some limited information about operational experiences and
deployments. Any very brief information would be appreciated, for
example: 1. What deployment scenarios are you most interested in? 2.
Is your deployment on "the Internet" or in a controlled environment?
3. Is your deployment on end hosts or with a MPTCP-enabled proxy (at
one or both ends?)? 4. What do you see as the rest most important
benefits of MPTCP in your answers unaltered, unless scenario(s)? 5. How extensively have you ask
deployed and experimented with MPTCP so far?

Our implementation is open-source and has been discussed for
further obfuscation. Please note various
types of tests/deployments based on the messages received on the
mptcp-dev mailing list. We currently use Multipath TCP on
www.multipath-tcp.org and also on internal ssh servers at UCLouvain.
6. MPTCP's design seeks to maximise the chances that Philip Eardley the signalling
works for BT through middleboxes. Did you find cases where middleboxes
blocked MPTCP signalling? We have implemented a test suite based on
a slightly modified version of the Multipath TCP implementation that
allows to check the interoperability between Multipath TCP and Yoshifumi Nishida works for GE;
middleboxes. We have used it over Internet paths and identified some
potential problems. However, the best approach to test these
interactions would be to control the middlebox and analyse its logs
during the Multipath TCP test. The test suite can be retrieved from
http://multipath-tcp.org/pmwiki.php/Users/AboutMeasures

7. MPTCP's design seeks to ensure that, if there is a problem with
MPTCP signalling, then the connection either falls back to TCP or
removes the problematic subflow. Did you wish find any corner cases where
this didn't happen properly? See above.

8. Have you encountered any issues or drawbacks with MPTCP?

9. Any other comments or information?

3.11. Question 11: Improvements to remain anonymous RFC6824

1. Are there any areas where [RFC6824] could be improved, either in
technical content or clarity? 2. Any other issues you want to raise?

8. Full survey response for Implementation 2

Question 1: Your details
-------------------------------------------------------------
1.1 Swinburne University of Technology, Hawthorn, Victoria, Australia

1.2 Lawrence Stewart, Nigel Williams

1.3 No

Question 2: Preliminary information about your implementation
-------------------------------------------------------------

2.1 FreeBSD-10

2.2 Currently IPv4 only (IPv6 support will eventually be added)

2.3 Publicly available (http://caia.swin.edu.au/urp/newtcp/mptcp/).
The code is released under the BSD license. 2.3

2.5 Independent

2.6 Yes, some limited testing to establish interoperability.

2.7 Yes, with some additional work this should be possible (if not
then IETF-88). Q

uestion 3: Support for MPTCP's Signaling Functionality
-------------------------------------------------------------

3.1 a) MP_CAPABLE Implemented

b) Do not currently honour checksum flag (to be implemented)

3.2 a) MP_JOIN Implemented

b) Either end can initiate a MP_JOIN

c) 8 (controlled via sysctl)

d) Currently do not include HMAC verification during handshake, but
this will be enabled in the next patch (several weeks from one time of us, please return it
submission)

3.3 a) DSS Implemented

b) 4 (default) and 8

c) 4 (default) and 8

d) Yes, however the connection tear-down exchange is not fully
implemented - the connection shuts down but the DFIN may not be
correctly acknowledged.

e) No. This will be supported eventually (time-frame unknown)

3.4 a) ADD_ADDR implemented, REMOVE_ADDR not implemented (to be done,
timeframe unknown)

b) No. Functionality to be added

3.5 MP_FASTCLOSE not implemented. Plan to implement eventually

Question 4: Fallback from MPTCP
-------------------------------------------------------------

4.1 Yes

4.2 The subflow PCBs remain allocated, however the other.

3. subflow is not
used to send data.

4.3 No, tbd

4.4 No, tbd

4.5 No, checksumming not implemented

4.6 Yes

4.8 Fallback hasn't really been put through any structured tests yet

Question 5: Heuristics
-------------------------------------------------------------

5.1 We use "TCP_MAXWIN << tp->rcv_scale". This is temporary and we
will use a call into the "multipath" control layer to determine this
value in future releases (we need to investigate a suitable way of
calculating this).

5.2 cwnd determines the amount of data to send (given that rcv window
is always very large). Sendbuffer is scaled based on occupancy.

5.3 We currently don't have Data-level retransmits enabled. However
our policy is to retransmit on the next subflow that requests data to
send that is suitable. There is no intelligence in the packet
schedular currently,

5.4 The same port numbers are re-used for additional subflows.

Question 6: Security
-------------------------------------------------------------

6.1 Yes

6.2 No

Question 7: IANA
-------------------------------------------------------------

7.1 Yes

Question 8: Congestion Control and subflow policy
-------------------------------------------------------------

8.1 A simple algorithm is used to divide the send buffer between
subflows, so that traffic is spread across the subflows.

8.3 No. (to be added)

8.4 No

8.5 No

Question 9: API
-------------------------------------------------------------

9.1 Legacy applications are able to use MPTCP. MPTCP is set globally
via a sysctl variable.

9.2 No

9.3 No

Question 10: API
-------------------------------------------------------------

10.1 Some current project work is based on MPTCPs use in vehicle to
infrastructure (V2I) connectivity (to provide a persistent connection
using 3G and roadside wifi). Other interests are in multi-homed
"home-user" environments, high throughput data transfers.... We are
also interested in experimenting with congestion control across paths
with different path-cost metrics.

10.2 So far only within controlled testbeds

10.3 End hosts

10.4 Depending on the scenario, connection persistence, throughput...

10.5 Still an alpha-quality build, so limited testing so far.

9. Full survey response for Implementation 3

Survey 3.1. Question 1: Your details Question 1 gathers some
information about the team that has implemented MPTCP.

1. Your institution: anonymized.

2. Name(s) of people in your implementation and test teams: There
were several folks involved in the implementation and testing.

3. Do you want your answers to Question 1.1 and 1.2 above to be
anonymised? Yes.

3.2. Question 2: Preliminary information about your implementation
Question 2 gathers some preliminary information.

1. What OS is your implementation for? (or is it application layer?)
anonymized (commercial OS)

2. Do you support IPv4 or IPv6 addresses or both? Both.

3. Is it publicly available (or will it be?) (for free or a fee?)
No.

4. Overall, what are you implementation and testing plans? (details
can be given against individual items later) We plan to use it in a
mobile environment.

5. Is it an independent implementation? Or does it build on another
MPTCP implementation -which one? It is an independent
implementation.

6. Have you already done some interop tests, for example with
UCLouvain's "reference" Linux implementation? Most MPTCP option
formats were tested with the reference Linux implementation.

7. Would you be prepared to take part in an interop event, for
example adjacent to IETF-87 in Berlin? Unsure at this point.

3.3. Question 3: Support for MPTCP's Signalling Functionality
Question 3 asks about support for the various signalling messages
that the MPTCP protocol defines. *** For each message, please give a
little information about the status of your implementation: for
example, you may have implemented it and fully tested it; the
implementation may be in progress; you have not yet implemented it
but plan to soon (timescale?); you may you have no intention to
implement it (why?); etc.

1. Connection initiation (MP_CAPABLE) [Section 3.1 RFC6824] a. What
is the status of your implementation? Fully implemented and tested
against the reference Linux implementation.

b. Any other comments or information?

2. Starting a new subflow (MP_JOIN) [Section 3.2 RFC6824] a. What
is the status of your implementation? Fully implemented and tested
against the reference Linux implementation.

b. Can either end of the connection start a new subflow (or only the
initiator of the original subflow)? Only the initiator of the
original sub flow can start other sub flows.

c. What is the maximum number of subflows your implementation can
support? There is no hard limit.

d. Any other comments or information?

3. Data transfer (DSS) [Section 3.3 RFC6824] a. What is the status
of your implementation? Fully implemented and tested.

b. The "Data ACK" field can be 4 or 8 octets. Which one(s) have you
implemented? Both have been implemented but the use of the 4-byte
field is the default. When an 8 byte DSS is received, an 8 byte Data
ACK is sent in response.

c. The "Data sequence number" field can be 4 or 8 octets. Which
one(s) have you implemented? Both have been implemented but the use
of the 4-byte field is the default. When a wraparound of the lower
32-bit part of the DSS is detected, the full 8 byte DSS is sent.

d. Does your implementation support the "DATA_FIN" operation to
close an MPTCP connection? Yes. There are cases however where the
sub flows are closed (TCP FIN'd) but the DATA_FIN is not sent - in
this case the MPTCP connection must be closed through a garbage
collector after some idle time.

e. Does your implementation support the "Checksum" field (which is
negotiated in the MP_CAPABLE handshake)? Yes.

f. Any other comments or information?

4. Address management (ADD_ADDR and REMOVE_ADDR) [Section 3.4
RFC6824] a. What is the
status of your implementation? It does not support sending ADD_ADDR
or processing ADD_ADDR as it is considered a security risk. Also, we
only have a client side implementation at the moment which always
initiates the sub flows. The remote end does not send ADD_ADDR in
our configuration. The client can send REMOVE_ADDR however when one
of the established sub flow's source address goes away. The client
ignores incoming REMOVE_ADDR options also.

b. Can your implementation do ADD_ADDRESS for addresses that appear
*after* the connection has been established? No. c. Any other
comments or information?

5. Fast close (MP_FASTCLOSE) [Section 3.5 RFC6824] a. What is the
status of your implementation? It is supported. Though
Retransmission of Fast close is not supported yet.

b. Any other comments or information?

1. If the MP_CAPABLE option is removed by a middlebox, does your
implementation fall back to TCP? Yes.

2. If the MP_JOIN option does not get through on the SYNs, does your
implementation close the additional subflow? Yes.

3. If the DSS option does not get through on the first data
segment(s), does your implementation fall back? (either falling back
to MPTCP (if the issue is on the first subflow) or closing the
additional subflow (if the issue is on an additional subflow)) Yes it
falls back to TCP when there's one sub flow. When there are multiple
sub flows, since MPTCP is used in active/backup mode, it is assumed
that the sub flow transferring data is most likely to be more usable
than any other established sub flow. So the sub flow on which
fallback occurred is kept alive and other sub flows are closed.
Fallback though is not guaranteed to occur safely when there are more
than one sub flows because the infinite mapping option may be
stripped like other DSS options and the MP_FAIL option if used in
scenarios other than for reporting checksum failure can also be
stripped.

4. Similarly, if the "DATA ACK" field does not correctly acknowledge
the first data segment(s), does your implementation fall back? No.
Current implementation just ignores the unexpected data ack.

6. Does your implementation fall back to TCP by using an "infinite
mapping" [Section 3.3.1 RFC6824] (so that the subflow-level data is
mapped to the connection-level data for the remainder of the
connection)? Yes.

7. Did you find any corner cases where MPTCP's fallback didn't
happen properly? If the very first sub flow does not send any data
and is disconnected right away, then the current implementation
allows a join to occur with the addition of another sub flow which
then becomes a fully mp capable sub flow. Thus we allow break before
make by letting additional sub flows to be joined if the very first
one disconnected even without sending any data. This is a very
corner case but an instance where we do not follow the rules of
fallback (allow second sub flow even when first sub flow did not
send/receive data/data acks).

8. Any other comments or information about fallback? Fallback after
connection establishment and after a few data packets were
transferred with MPTCP options is complicated. The spec does not
clearly cover the cases of options being stripped by middle boxes.
It goes into good detail about what to do when the DSS checksum
fails, but not when DSS checksum is not in use and the MPTCP options
are stripped. Both sender/receiver behaviors could be outlined with
more detail.

1. Receiver considerations [S3.3.4, RFC6824]: What receiver buffer
have you used? Does this depend on the retransmission strategy?
What advice should we give about the receiver? We are just using
MPTCP in active/backup mode. This mode is simpler wrt receive buffer
utilization. The receive buffer sizes at the MPTCP and sub flow
level is the same. Automatic buffer tuning is turned off when MPTCP
is in use.

2. Sender considerations [S3.3.5, RFC6824]: How do you determine how
much data a sender is allowed to send and how big the sender buffer
is? What advice should we give about the sender? Automatic buffer
tuning is turned off when MPTCP is in use.

3. Reliability and retransmissions [S3.3.6, RFC6824]: What is your
retransmission policy? (when do you retransmit on the original
subflow vs on another subflow or subflows?) When do you decide that
a subflow is underperforming and should be reset, and what do you
then do? What advice should we give about this issue?
Retransmissions at MPTCP level do not occur on the same sub flow
except when MP_FAIL option is received. A sub flow is said to be
underperforming when its network connectivity goes away.

4. Port usage [S3.3.8.1, RFC6824]: Does your implementation use the
same port number for additional subflows as for the first subflow?
Have you used the ability to define a specific port in the Add
Address option? What advice should we give about this issue? The
destination port is the same. The local port changes for additional
sub flows so on the wire it is like two tcp connections to the same
remote destination. We have not used Add Address option at all.

5. Delayed subflow start [S3.3.8.2, RFC6824]: What factors does your
implementation consider when deciding about opening additional
subflows? What advice should we give about this issue? The client
implementation is aware of network interfaces coming up or going down
and establishes new sub flows or removes existing sub flows
accordingly.

8. What other heuristics should we give advice about? Any other
comments or information?

3.6. Question 6: Security Question 6 asks about Security related
matters [Section 5 RFC6824].

1. Does your implementation use the hash-based, HMAC-SHA1 HMACSHA1 security
mechanism defined in [RFC6824]? Yes.

2. Does your implementation support any other handshake algorithms?
No.

3. It has been suggested that a Standards-track MPTCP needs a more
secure mechanism. Do you have any views about how to achieve this?
No. But the mechanism could be tied with SSL because SSL is used
wherever security is deemed important.

4. Any other comments or information?

3.7. Question 7: IANA Question 7 asks about IANA related matters.

1. Does your implementation follow the IANA-related definitions?
[Section 8 RFC6824] defines: TCP Option Kind number (30); the
sub-registry sub-
registry for "MPTCP Option Subtypes"; and the Page 12 of 17 Survey
6/22/13, 5:55 PM sub-registry for "MPTCP Handshake Algorithms" Yes.

2. Any other comments or information? No.

3.8. Question 8: Congestion control and subflow policy Question 8
asks about how you share traffic across multiple subflows.

1. How does your implementation share traffic over the available
paths? For example: as a spare path on standby ('all-or-
nothing'), ('all-ornothing'), as
an 'overflow', etc? Does it have the ability to send /receive
traffic across multiple subflows simultaneously? It uses active/
backup where one sub flow is preferred or has higher priority over
other sub flows. When the preferred sub flow fails or begins to
experience retransmission timeouts, the other sub flows are used.

2. Does your implementation support "handover" from one subflow to
another when losing an interface? Yes.

3. Does your implementation support the coupled congestion control
defined in [RFC6356]? No.

4. Does your implementation support some other coupled congestion
control (ie that balances traffic on multiple paths according to
feedback)? No.

5. The MP_JOIN (Starting a new subflow) Option includes the "B" bit,
which allows the sender to indicate whether it wishes the new subflow
to be used immediately or as a backup if other path(s) fail. The
MP_PRIO Option is a request to change the "B" bit - either on the
subflow on which it is sent, or (by setting the optional Address ID
field) on other subflows. Does your implementation support the "B"
bit and MP_PRIO mechanisms? Do you think they're useful, or have
another suggestion? Yes the implementation uses the B bit and the
MP_PRIO option. They are very useful for the active/backup mode of
operation.

6. Any other comments or information or suggestions about the advice
we should give about congestion control [S3.3.7 RFC6824] and subflow
policy [S3.3.8 RFC6824]?

3.9. Question 9: API Question 9 gathers information about your API.
[RFC6897] considers the MPTCP Application Interface.

1. With your implementation, can legacy applications use (the
existing sockets API to use) MPTCP? How does the implementation
decide whether to use MPTCP? Should the advice in [Section 4,
RFC6897] be modified or expanded? The implementation does not
support MPTCP with existing sockets API. MPTCP is exposed through a
private SPI today. If MPTCP becomes prolific over the next few
years, MPTCP use shall be expanded.

2. The "basic MPTCP API" enables MPTCP-aware applications to
interact with the MPTCP stack via five new socket options. For each
one, have you implemented it? has it been useful? a.
TCP_MULTIPATH_ENABLE? b. TCP_MULTIPATH_ADD? c.
TCP_MULTIPATH_REMOVE? d. TCP_MULTIPATH_SUBFLOWS? e.
TCP_MULTIPATH_CONNID? This mode of API is not used. Proprietary
methods are used for achieving these basic operations.

3. Have you implemented any aspects of an "advanced MPTCP API"?
([Appendix A, RFC6897] hints at what it might include.) No.

4. Any other comments or information?

1. What deployment scenarios are you most interested in? MPTCP in
mobile environments is very powerful when used in the active/backup
mode. Since the network interfaces available on mobile devices have
different cost characteristics as well as different bring up and
power usage characteristics, it is not useful to share load across
all available network interfaces - at least not currently. Providing
session continuity across changing network environments is the key
deployment scenario.

2. Is your deployment on "the Internet" or in a controlled
environment? The deployment is on the Internet.

3. Is your deployment on end hosts or with a MPTCP-enabled MPTCPenabled proxy (at
one or both ends?)? The deployment supports MPTCP on both ends.

4. What do you see as the most important benefits of MPTCP in your
scenario(s)? Described in point 1 of this section.

5. How extensively have you deployed and experimented with MPTCP so
far? Deployment is still in early stages. We have been
experimenting with MPTCP for about a year.

6. MPTCP's design seeks to maximise the chances that the signalling
works through middleboxes. Did you find cases where middleboxes
blocked MPTCP signalling? Corporate firewalls block MPTCP signaling
by default. IETF is one venue where Cisco, and other firewall
vendors can be asked to change their defaults to allow MPTCP signals.

7. MPTCP's design seeks to ensure that, if there is a problem with
MPTCP signalling, then the connection either falls back to TCP or
removes the problematic subflow. Did you find any corner cases where
this didn't happen properly? This has been covered a bit in the
Fallback section. When using two sub flows in active/backup mode,
there is a possibility that a backup sub flow that never sent data
starts being used for retransmitting data that is not going through
on the active path. While it is preferable to keep the initial sub
flow that successfully sent MPTCP options and drop the backup path,
the initial sub flow may be the failing one, and we may want to move
to the backup path. But the backup path can be retransmitting data
that did not get sent successfully on the active path and if there is
a middle box in the backup sub flow's path stripping options, then we
have a case where the MPTCP session may not be recoverable as it may
not be evident from what point in the MPTCP sequence space, data was
being sent. The spec does talk of retaining the initial sub flow and
closing the failed flow. So perhaps doing the reverse is not
recommended, however, it would certainly be advantageous to support
MPTCP better in such a failing environment. Also, in parallel
working with firewall vendors to allow MPTCP options always to not
have to over-engineer these cases.

8. Have you encountered any issues or drawbacks with MPTCP?

9. Any other comments or information?

3.11. Question 11: Improvements to RFC6824 1. Are there any areas
where [RFC6824] could be improved, either in technical content or
clarity? Discussed in the fallback section. Other areas around
MPTCP performance such as support for sub flow level automatic buffer
scaling, varying QoS support, varying window scaling support on each
sub flow may be worth discussing further, although they are outside
the scope of the current spec.

2. Any other issues you want to raise? Some additional work on
option signaling that we will bring up in future discussions.

10. Full survey response for Implementation 4

1. Your institution: Citrix Systems, Inc.

2. Name(s) of people in your implementation and test teams: NA

3. Do you want your answers to Question 1.1 and 1.2 above to be
anonymised? No

3.2. Question 2: Preliminary information about your implementation
Question 2 gathers some preliminary information.

1. What OS is your implementation for? (or is it application layer?)
NetScaler Firmware

2. Do you support IPv4 or IPv6 addresses or both? Both

3. Is it publicly available (or will it be?) (for free or a fee?)
It is available for purchase

4. IANA Considerations

This document makes Overall, what are you implementation and testing plans? (details
can be given against individual items later)

5. Is it an independent implementation? Or does it build on another
MPTCP implementation -which one? It is an independent implementation

6. Have you already done some interop tests, for example with
UCLouvain's "reference" Linux implementation? Yes, our
implementation is extensively tested with Linux reference
implementation

7. Would you be prepared to take part in an interop event, for
example adjacent to IETF-87 in Berlin?

1. Connection initiation (MP_CAPABLE) [Section 3.1 RFC6824] a. What
is the status of IANA. your implementation? Fully implemented and tested

b. Any other comments or information? One security concern here is
that the keys are exchanged in plain text which is prone to attacks
and also the key generation mechanism is highly computational
intensive

2. Starting a new subflow (MP_JOIN) [Section 3.2 RFC6824] a. What
is the status of your implementation? Fully implemented and tested

b. Can either end of the connection start a new subflow (or only the
initiator of the original subflow)? Only the initiator of the
original subflow can initiate additional subflows.

c. What is the maximum number of subflows your implementation can
support? we support maximum 6 subflows.

d. Any other comments or information?

3. Data transfer (DSS) [Section 3.3 RFC6824] a. What is the status
of your implementation? Fully implemented and tested

b. The "Data ACK" field can be 4 or 8 octets. Which one(s) have you
implemented? Our implementation supports both 4 or 8 Octets Data Ack
in both the directions

c. The "Data sequence number" field can be 4 or 8 octets. Which
one(s) have you implemented? Our implementation supports both 4 or 8
Octets DSN in both the directions

d. Does your implementation support the "DATA_FIN" operation to
close an MPTCP connection? YES

e. Does your implementation support the "Checksum" field (which is
negotiated in the MP_CAPABLE handshake)? YES

f. Any other comments or information?

4. Address management (ADD_ADDR and REMOVE_ADDR) [Section 3.4
RFC6824]

a. What is the status of your implementation? REMOVE_ADDR is
implemented and tested
b. Can your implementation do ADD_ADDRESS for addresses that appear
*after* the connection has been established? NO

c. Any other comments or information? ADD_ADDRESS may not be much
useful in the real environment situation given that most of the
clients are behind the NATing devices.

5. Security Considerations Fast close (MP_FASTCLOSE) [Section 3.5 RFC6824] a. What is the
status of your implementation? Implemented and tested b. Any other
comments or information?

1. If the MP_CAPABLE option is removed by a middlebox, does your
implementation fall back to TCP? YES

2. If the MP_JOIN option does not get through on the SYNs, does your
implementation close the additional subflow? YES

4. Similarly, if the "DATA ACK" field does not correctly acknowledge
the first data segment(s), does your implementation fall back? If
the sender receives pure ack for its first DSS packet then it
fallsback to regular TCP.

5. Does your implementation protect data with the "Checksum" field
in the DSS option [Section 3.3 RFC6824]? If the checksum fails
(because the subflow has been affected by a middlebox), does your
implementation immediately close the affected subflow (with a TCP
RST) with the MP_FAIL Option? If the checksum fails and there is a
single subflow, does your implementation handle this as a special
case, as described in [Section 3.6 RFC6824]? Yes, our implementation
supports DSS checksum and will close the subflow with RST if the
checksum validation fails and there are more than one subflows and
sends MP_FAIL if there is a single subflow expecting infinite map
from the peer.

7. Did you find any corner cases where MPTCP's fallback didn't
happen properly? We have found few cases where the draft is not
clear about the recommended action and fallback strategy, like: 1.
what is the expected behavior when pure ack or data packet without
dss is received in middle of transaction? How the hosts should
fallback in this case? This survey can happen if the routing changes and
the new path drops mptcp options. In this case MP_FAIL/infinite map
exchange may not be possible and so could not decide whether both
parties are in sync to fallback to tcp. 2. whether infinite map is
unidirectional or bidirectional? If one host is sending infinite map
to peer, does the peer also needs to send infinite map to the host?
Exchanging infinite map and falling back to TCP from both ends is
easy from implementation point of view. 8. Any other comments or
information about fallback?

3.5. Question 5: Heuristics Question 5 gathers information about
heuristics: aspects that are not required for protocol correctness
but impact the security performance. We would like to document best practice
so that future implementers can learn from the experience of MPTCP, except
pioneers. The references contain some initial comments about each
topic.

2. Sender considerations [S3.3.5, RFC6824]: How do you determine how
much data a sender is allowed to send and how big the
extent sender buffer
is? What advice should we give about the sender? The send side flow
control is handled at mptcp level and is independent to subflows.
The mptcp level flow control is (almost) same as the regular TCP flow
control.

3. Reliability and retransmissions [S3.3.6, RFC6824]: What is your
retransmission policy? (when do you retransmit on the original
subflow vs on another subflow or subflows?) When do you decide that
a subflow is underperforming and should be reset, and what do you
then do? What advice should we give about this issue? The
retransmission is done by the subflows as long as the subflow is
alive and is not removed by the REM_ADDR/RST/.. . If 3 RTO happens
on the subflow doing retransmission and multiple subflows are
available then the mptcp starts retransmission from additional
subflow. The original subflow continues retransmission for 7RTO and
will be closed after that with RST.

5. Delayed subflow start [S3.3.8.2, RFC6824]: What factors does your
implementation consider when deciding about opening additional
subflows? What advice should we give about this issue? NA

6. Failure handling [S3.3.8.3, RFC6824]: Whilst the protocol defines
how to handle some unexpected signals, the behaviour after other
unexpected signals is not defined. What advice should we give about
this issue? RFC should clearly define failure case handling
otherwise it uncovers creates interoperability problems among various
implementations. Our strategy in most of the unexpected failuire
case is to send MP_FAIL RST with expected DSN if there are multiple
subflows and MP_FAIL if there is a single subflow expecting infinite
map from the peer.

7. Use of TCP options: As discussed in [Appendix A, RFC6824], the
TCP option space is limited, but a brief study found there was enough
room to fit all the MPTCP options. However there are constraints on
which MPTCP option(s) can be included in packets with other TCP
options - do the suggestions in Appendix A need amending or
expanding? Looks fine now. Atleast timestamp can be included with
every dss packet (28bytes for dss and 12bytes for Timestamp), but if
there are any other options which needs to be included in data
packets then the implementation has to choose which one to include
among them.

8. What other heuristics should we give advice about? Any other
comments or information?

3.6. Question 6: Security Question 6 asks about Security related
matters [Section 5 RFC6824].

1. Does your implementation use the hash-based, HMAC-SHA1 security issues
mechanism defined in [RFC6824]? YES.

2. Does your implementation support any other handshake algorithms?
NO.

3. It has been suggested that a Standards-track MPTCP needs a more
secure mechanism. Do you have any views about how to achieve this?
Yes we also feel more secure and light weight mechanism is required.

4. Any other comments or information?

3.7. Question 7: IANA Question 7 asks about IANA related matters.

3.8. Question 8: Congestion control and subflow policy Question 8
asks about how you share traffic across multiple subflows.

1. How does your implementation share traffic over the available
paths? For example: as a spare path on standby ('all-or- nothing'),
as an 'overflow', etc? Does it have the ability to send /receive
traffic across multiple subflows simultaneously? We give preference
to the path that client is currently using to send data/ack and also
has policy based on primary/backup setup. We accept data from
multiple subflows simultaneously but don't send it simultaneously
out.

2. Does your implementation support "handover" from one subflow to
another when losing an interface? YES.

3. Does your implementation support the coupled congestion control
defined in [RFC6356]? NO.

4. Does your implementation support some other coupled congestion
control (ie that balances traffic on multiple paths according to
feedback)? NO.

5. The MP_JOIN (Starting a new subflow) Option includes the "B" bit,
which allows the sender to indicate whether it wishes the new subflow
to be used immediately or as a backup if other path(s) fail. The
MP_PRIO Option is a request to change the "B" bit - either on the
subflow on which it is sent, or (by setting the optional Address ID
field) on other subflows. Does your implementation support the "B"
bit and MP_PRIO mechanisms? Do you think they're useful, or have
another suggestion? YES, our implementation supports both 'B' flag
and MP_PRIO options, they are much useful to change the priority of
the subflows and to decide which subflow to use for data transfer.

6. Any other comments or information or suggestions about the advice
we should give about congestion control [S3.3.7 RFC6824] and subflow
policy [S3.3.8 RFC6824]?

3.9. Question 9: API Question 9 gathers information about your API.
[RFC6897] considers the MPTCP Application Interface.

3. Have you implemented any aspects of an "advanced MPTCP API"?
([Appendix A, RFC6897] hints at what it might include.) NA. 4. Any
other comments or information?

1. What deployment scenarios are you most interested in? MPTCP
Proxy deployment where the mptcp connections from the clients are
terminated and the tcp connection is established on the other side.

2. Is your deployment on "the Internet" or in a
future version of controlled
environment? Targeted for the protocol. Internet deployment.

3. Is your deployment on end hosts or with a MPTCP-enabled proxy (at
one or both ends?)? Proxy.

4. What do you see as the most important benefits of MPTCP in your
scenario(s)? Reliability and fault tolerance.

5. How extensively have you deployed and experimented with MPTCP so
far?

6. Acknowledgements MPTCP's design seeks to maximise the chances that the signalling
works through middleboxes. Did you find cases where middleboxes
blocked MPTCP signalling? Yes some firewalls seem dropping MPTCP
options.

8. Have you encountered any issues or drawbacks with MPTCP? 9. Any
other comments or information?

3.11. Question 11: Improvements to RFC6824

1. Are there any areas where [RFC6824] could be improved, either in
technical content or clarity? More clarity required in fallback
cases.

2. Any other issues you want to raise?

11. Normative References

[RFC6356] Raiciu, C., Handley, M., and D. Wischik, "Coupled
Congestion Control for Multipath Transport Protocols",
RFC 6356, October 2011.

[RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure,
"TCP Extensions for Multipath Operation with Multiple
Addresses", RFC 6824, January 2013.

Author's Address

Philip Eardley
BT