< draft-eardley-mptcp-implementations-survey-01.txt		draft-eardley-mptcp-implementations-survey-02.txt >
	Network Working Group P. Eardley		Network Working Group P. Eardley
	Internet-Draft BT		Internet-Draft BT
	Intended status: Informational May 02, 2013		Intended status: Informational July 12, 2013
	Expires: November 03, 2013		Expires: January 13, 2014
	Survey of MPTCP Implementations (blank version for implementers to fill		Survey of MPTCP Implementations
	in)		draft-eardley-mptcp-implementations-survey-02
	draft-eardley-mptcp-implementations-survey-01
	Abstract		Abstract
	This survey gathers information from people who have implemented		This document presents results from the survey to gather information
	MPTCP, in particular to help progress the protocol from Experimental		from people who have implemented MPTCP, in particular to help
	to Standards track.		progress the protocol from Experimental to Standards track.
	It is ready to be filled in, if you are an implementer. Thank-you!		The document currently includes answers from four teams: a Linux
			implementation from UCLouvain, a FreeBSD implementation from
			Swinburne, an anonymous implementation in a commercial OS, and a
			NetScalar Firmware implementation from Citrix Systems, Inc. Thank-
			you!
	Status of This Memo		In summary, we have four independent implementations of all the MPTCP
			signalling messages, with the exception of address management, and
			some interoperabiity testing has been done by the other three
			implementations with the 'reference' Linux implementation. So it
			appears that the RFC is (at least largely) clear and correct. On
			address management, we have only one implementation of ADD_ADDR with
			two teams choosing not to implement it. We have one implementation
			of the working group's coupled congestion control (RFC6356) and none
			of the MPTCP-aware API (RFC6897).
			The main suggested improvements are around
			o how MPTCP falls back (if the signalling is interrupted by a
			middlebox): (1) corner cases that are not handled properly, (2) at
			the IETF, the MPTCP community should work with middlebox vendors,
			either to reduce or eliminate the need for fallback or to
			understand the middlebox interactions better.
			o security: both better MPTCP security (perhaps building on SSL) and
			a lighter weight mechanism, preferably both in one mechanism.
			It is hoped that the next version can include information from any
			other implementations. If you are an implementer and want to
			contribute your answers, please see the -01 version of this document
			for a blank survey ready to be filled in.
			Status of this Memo
	This Internet-Draft is submitted in full conformance with the		This Internet-Draft is submitted in full conformance with the
	provisions of BCP 78 and BCP 79.		provisions of BCP 78 and BCP 79.
	Internet-Drafts are working documents of the Internet Engineering		Internet-Drafts are working documents of the Internet Engineering
	Task Force (IETF). Note that other groups may also distribute		Task Force (IETF). Note that other groups may also distribute
	working documents as Internet-Drafts. The list of current Internet-		working documents as Internet-Drafts. The list of current Internet-
	Drafts is at http://datatracker.ietf.org/drafts/current/.		Drafts is at http://datatracker.ietf.org/drafts/current/.
	Internet-Drafts are draft documents valid for a maximum of six months		Internet-Drafts are draft documents valid for a maximum of six months
	and may be updated, replaced, or obsoleted by other documents at any		and may be updated, replaced, or obsoleted by other documents at any
	time. It is inappropriate to use Internet-Drafts as reference		time. It is inappropriate to use Internet-Drafts as reference
	material or to cite them other than as "work in progress."		material or to cite them other than as "work in progress."
	This Internet-Draft will expire on November 03, 2013.		This Internet-Draft will expire on January 13, 2014.
	Copyright Notice		Copyright Notice
	Copyright (c) 2013 IETF Trust and the persons identified as the		Copyright (c) 2013 IETF Trust and the persons identified as the
	document authors. All rights reserved.		document authors. All rights reserved.
	This document is subject to BCP 78 and the IETF Trust's Legal		This document is subject to BCP 78 and the IETF Trust's Legal
	Provisions Relating to IETF Documents		Provisions Relating to IETF Documents
	(http://trustee.ietf.org/license-info) in effect on the date of		(http://trustee.ietf.org/license-info) in effect on the date of
	publication of this document. Please review these documents		publication of this document. Please review these documents
	carefully, as they describe your rights and restrictions with respect		carefully, as they describe your rights and restrictions with respect
	to this document. Code Components extracted from this document must		to this document. Code Components extracted from this document must
	include Simplified BSD License text as described in Section 4.e of		include Simplified BSD License text as described in Section 4.e of
	the Trust Legal Provisions and are provided without warranty as		the Trust Legal Provisions and are provided without warranty as
	described in the Simplified BSD License.		described in the Simplified BSD License.
	Table of Contents		Table of Contents
	1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2		1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
	2. Guidelines for filling in the survey . . . . . . . . . . . . 3		2. Survey - summary of replies . . . . . . . . . . . . . . . . . 4
	3. Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . 3		3. Interesting aspects of replies . . . . . . . . . . . . . . . . 6
	3.1. Question 1: Your details . . . . . . . . . . . . . . . . 3		3.1. Question 1: Your details . . . . . . . . . . . . . . . . . 6
	3.2. Question 2: Preliminary information about your		3.2. Question 2: Preliminary information about your
	implementation . . . . . . . . . . . . . . . . . . . . . 4		implementation . . . . . . . . . . . . . . . . . . . . . . 7
	3.3. Question 3: Support for MPTCP's Signalling Functionality 4		3.3. Question 3: Support for MPTCP's Signalling
	3.4. Question 4: Fallback from MPTCP . . . . . . . . . . . . . 5		Functionality . . . . . . . . . . . . . . . . . . . . . . 7
	3.5. Question 5: Heuristics . . . . . . . . . . . . . . . . . 6		3.4. Question 4: Fallback from MPTCP . . . . . . . . . . . . . 7
	3.6. Question 6: Security . . . . . . . . . . . . . . . . . . 7		3.5. Question 5: Heuristics . . . . . . . . . . . . . . . . . . 8
	3.7. Question 7: IANA . . . . . . . . . . . . . . . . . . . . 8		3.6. Question 6: Security . . . . . . . . . . . . . . . . . . . 9
	3.8. Question 8: Congestion control and subflow policy . . . . 8		3.7. Question 7: IANA . . . . . . . . . . . . . . . . . . . . . 9
	3.9. Question 9: API . . . . . . . . . . . . . . . . . . . . . 8		3.8. Question 8: Congestion control and subflow policy . . . . 9
			3.9. Question 9: API . . . . . . . . . . . . . . . . . . . . . 10
	3.10. Question 10: Deployments, use cases and operational		3.10. Question 10: Deployments, use cases and operational
	experiences . . . . . . . . . . . . . . . . . . . . . . . 9		experiences . . . . . . . . . . . . . . . . . . . . . . . 10
	3.11. Question 11: Improvements to RFC6824 . . . . . . . . . . 10		3.11. Question 11: Improvements to RFC6824 . . . . . . . . . . . 11
	4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10		4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11
	5. Security Considerations . . . . . . . . . . . . . . . . . . . 10		5. Security Considerations . . . . . . . . . . . . . . . . . . . 11
	6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 10		6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 11
	7. Normative References . . . . . . . . . . . . . . . . . . . . 10		7. Full survey response for Implementation 1 . . . . . . . . . . 11
	Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 10		8. Full survey response for Implementation 2 . . . . . . . . . . 19
			9. Full survey response for Implementation 3 . . . . . . . . . . 23
			10. Full survey response for Implementation 4 . . . . . . . . . . 31
			11. Normative References . . . . . . . . . . . . . . . . . . . . . 38
			Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 38
	1. Introduction		1. Introduction
	The goal of this survey is to gather information from people who have		The document reports the results from a survey to gather information
	implemented MPTCP, in particular to help progress the protocol from		from people who have implemented MPTCP. The goal is to help progress
	Experimental to Standards track.		the protocol from Experimental to Standards track.
	The MPTCP WG's current charter (2013-03-14) states: "The primary goal		Four responses have been received. Thank-you! They are independent
	of the working group is to create a bis version of the protocol		implementations:
	document on the Standards track. This develops the current
	Experimental document ... incorporating experience from (for example)
	implementations, interoperability events, experiments, usage
	scenarios, protocol corner cases, and feedback from TCPM ... [Also,
	the] working group will document implementation advice. The current
	documents have several points where an implementer may benefit from
	guidance, for example about heuristics such as buffer sizing, or from
	advice about alternative implementations such as bump-in-the-stack."
	The survey gathers relevant information to help this, for example		o the Linux implementation from UCLouvain,
	about: the existence of running code (especially several independent,
	interoperable implementations); the parts of the experimental spec
	that have not been implemented (or not been used); the parts that
	need improvement, in terms of functionality or clarity.
	The survey also takes the opportunity to gather some limited		o the FreeBSD implementation from Swinburne
	information about operational experiences and deployments.
	The WG is very grateful to those implementers taking the trouble to		o an anonymous implementation in a commercial OS
	fill in this survey - as well, of course, as their effort in creating
	the implementation!
	The intention is to produce a revised version of this draft that		o a NetScaler Firmware implementation from Citrix Systems, Inc.
	incorporates a summary of the replies and an appendix with all the
	full responses.
	2. Guidelines for filling in the survey		The Table below presents a highly-compressed summary, with each row
			corresponding to one question or sub-question of the survey. The
			following section highlights some interesting aspects of the replies
			in less compressed form. The full survey responses are in Appendix
			A, B, C and D.
	Please could each implementation team fill in the survey. If you		It is hoped that the next version of this document can include
	cannot answer a question just skip it, as partial responses are also		information about a further (independent) implementation:
	very valuable.
	Please return your completed survey by email attachment to the MPTCP		o Georg Hampel's user-space implementation (publicly available but
	WG Chairs: Philip Eardley <philip.eardley@bt.com> and Yoshifumi		not longer maintained)
	Nishida <nishida@sfc.wide.ad.jp>.
	If you wish to remain anonymous, please indicate -- we'll replace		o any other implementations.
	your answers in Question 1 by "Anonymous Implementation #n", but
	publish the rest of your answers unaltered, unless you ask for
	further obfuscation. Please note that Philip Eardley works for BT
	and Yoshifumi Nishida works for GE; if you wish to remain anonymous
	from one of us, please return it to the other.
	3. Survey		2. Survey - summary of replies
	3.1. Question 1: Your details		The Table below presents a highly-compressed summary, with each row
			corresponding to one question or sub-question of the survey. A
			column is left blank for any future responses.
	Question 1 gathers some information about the team that has		+----------------------------------------------------------------------+
	implemented MPTCP.		| | 1 | 2 | 3 | 4 | |
			|Institution | UCLouvain | Swinburne | Anon | Citrix | |
			| |
			| Question 2 asks about some preliminary topics, including whether the |
			| implementation is publicly available and interoperability with the |
			| Linux implementation (#1). |
			| | UCLouvain | Swinburne | Anon | Citrix | |
			|OS |Linux |FreeBSD-10 |Commercial |NetScaler | |
			|v4 & v6 |Both | IPv4 |Both |Both | |
			|public |Yes |Yes |No |Yes (pay) | |
			|independent |Yes |Yes |Yes |Yes | |
			|interop |Yes(!) |Mostly |Mostly |Yes | |
			| |
			| Question 3: Support for MPTCP's signalling functionality |
			| MPTCP's signalling messages are: MP_CAPABLE, MP_JOIN, Data transfer |
			| (DSS), ADD_ADDR, REMOVE_ADDR, MP_FASTCLOSE. There are sub-questions |
			| for MP_JOIN and DSS. |
			| | UCLouvain | Swinburne | Anon | Citrix | |
			|MP_CAPABLE |Yes |Yes |Yes |Yes | |
			|MP_JOIN |Yes |Yes |Yes |Yes | |
			|initiated by|first end |either end |first end |first end | |
			|#subflows |32 |8 |no limit |6 | |
			|DSS |Yes |Yes |Yes |Yes | |
			|DATA ACK |4 bytes |4 or 8 byte|4 or 8 byte|4 or 8 byte| |
			|Data seq num|4 bytes |4 or 8 byte|4 or 8 byte|4 or 8 byte| |
			|DATA_FIN |Yes |Yes |Yes |Yes | |
			|Checksum |Yes |No |Yes |Yes | |
			|ADD_ADDR |Yes |No |No (never) |No (never?)| |
			|REMOVE_ADDR |Yes |No |Partly |Yes | |
			|FAST_CLOSE |Yes |No |Yes |Yes | |
			| |
			| Question 4 asks about fallback from MPTCP: if a middlebox mangles |
			| MPTCP's signalling by removing MP_CAPABLE, MP_JOIN, DSS or DATA_ACK; |
			| if data is protected with Checksum in DSS option; if fallback to TCP |
			| uses an infinte mapping; and if any corner cases have been found. |
			| | UCLouvain | Swinburne | Anon | Citrix | |
			|MP_CAPABLE |Yes |Yes |Yes |Yes | |
			|MP_JOIN |Yes |Yes |Yes |Yes | |
			|DSS |Yes |No |Yes |Yes | |
			|DATA_ACK |Yes |No |No | | |
			|Checksum |Yes |No |Yes |Yes | |
			|infinte map |Yes |Yes |Yes |Yes | |
			|corner cases|No | |Yes |Yes | |
			| |
			| Question 5 asks about heuristics: aspects that are not required for |
			| protocol correctness but impact the performance. Questions are about |
			| sized the receiver and sender buffers, re-transmission policy, if |
			| additional subflows use the same port number as for the first subflow|
			| | UCLouvain | Swinburne | Anon | Citrix | |
			|Recvr buffer|auto-tune |TCP_MAXWIN |no tuning |tuned | |
			|Sendr buffer|auto-tune |cwnd |no tuning |as TCP | |
			|Re-transmits|2nd subflow|2nd subflow|2nd subflow|1st subflow| |
			|Port usage |same ports |same ports |diff local | | |
			| |
			| Question 6 asks about what security mechanisms are implemented: the |
			| one defined in RFC6824 and any others. |
			| | UCLouvain | Swinburne | Anon | Citrix | |
			|HMAC-SHA1 |Yes |Yes |Yes |Yes | |
			|other |Yes |No |No |No | |
			| |
			| Question 7 asks whether the implementation follows the IANA-related |
			| definitions (for TCP Option Kind and sub-registries). |
			| | UCLouvain | Swinburne | Anon | Citrix | |
			|RFC6824 |Yes |Yes |Yes |Yes | |
			| |
			| Question 8 asks about congestion control and related issues: how |
			| traffic is shared across multiple subflows; support for 'handover'; |
			| and support of RFC6356 (or other) coupled congestion control. |
			| | UCLouvain | Swinburne | Anon | Citrix | |
			|sharing |shared, RTT|shared |active/back|active/back| |
			|handover |Yes | |Yes |Yes | |
			|coupled cc |Yes |No |No |No | |
			|other ccc |Yes, OLIA |No |No |No | |
			|MP-PRIO & B |Yes |No |Yes |Yes | |
			| |
			| Question 9 is about the API: how legacy applications interact with |
			| the MPTCP stack, and if implemented the RFC6897 API for MPTCP-aware |
			| applications. |
			| | UCLouvain | Swinburne | Anon | Citrix | |
			|legacy apps |default |sysctl |private API|configured | |
			|MPTCP API |No |No |No |No | |
			|advanced API|No |No |No |No | |
			| |
			| Question 10 gathers some limited information about operational |
			| experiences and deployments. |
			| | UCLouvain | Swinburne | Anon | Citrix | |
			|Scenario |several |several |mobile |proxy | |
			|environment |internet |controlled |internet |internet | |
			|ends / proxy|end hosts |end hosts |end hosts |proxy | |
			| |
			+----------------------------------------------------------------------+
	1. Your institution:		3. Interesting aspects of replies
	2. Name(s) of people in your implementation and test teams:		This section tries to highlight some interesting comments made in the
			surveys. The Appendices can be consulted for further detials.
	3. Do you want your answers to Question 1.1 and 1.2 above to be		3.1. Question 1: Your details
	anonymised?
			Implementation 1 has been implemented by Sebastien Barre, Christoph
			Paasch and a large team, mainly at UCLouvain. Implementation 2 has
			been implemented by Lawrence Stewart and Nigel Williams at Swinburne
			University of Technology. Both these implementations are publicly
			available. Implementation 3 comes from an anonymous team with a
			commercial OS. Implementation 4 comes from Citrix Systems, Inc.
	3.2. Question 2: Preliminary information about your implementation		3.2. Question 2: Preliminary information about your implementation
			Three of the four implementations are publicly available, two for
			free (under GPLv2 and BSD licences) and one for a fee (NetScaler
			Firmware). Implementation 3 (commercial OS) is planned for use in a
			mobile environment, with MPTCP is used in active/backup mode.
			All implementations support IPv4 and three of four support IPv6.
			All implementations are being actively worked on, in order to improve
			performance and stability and conformance with the RFC.
			3.3. Question 3: Support for MPTCP's Signalling Functionality
			Three of the four implementations have implemented all the MPTCP
			signalling, with the interesting exception of address management,
			whilst Implementation 2 plans to add support for all those signalling
			capabilities it does not yet support.
			On address management, two implementations have decided not to
			implement ADD_ADDR. (ADD_ADDR allows an MPTCP host to signal a new
			address explicitly to the other host to allow it to initiate a new
			subflow - as an alternative to using MP_JOIN to directly start a new
			subflow). Implementation 3 decided not to support sending ADD_ADDR
			or processing ADD_ADDR as it is considered a security risk.
			Implementation 4 decided not to support ADD_ADDR because it didn't
			think it would be useful as most clients are behind NATing devices.
			However, both implemented REMOVE_ADDR (in Implementation 3 the client
			can send a REMOVE_ADDR but ignores incoming REMOVE_ADDR).
			In Implementations 1, 3 and 4 only the initiator of the original
			subflow can start a new subflow (a reason mentioned is that NATs make
			it hard for the server to reach the client).
			All implementations support 4 bytes "Data ACK" and "Data sequence
			number" fields, and will interoperate with an implementation sending
			8 bytes. Implementation 1 uses only 4 bytes fields; if an
			implementation sends an 8 byte data sequence number it replies with a
			4 byte data ack.
			3.4. Question 4: Fallback from MPTCP
			Question 4 asks about action when there is a problem with MPTCP, for
			example due to a middlebox mangling MPTCP's signalling. The
			connection needs to fall back: if the problem is on the first subflow
			then MPTCP falls back to TCP, whilst if the problem is on an
			additional subflow then that subflow is closed with a TCP RST, as
			discussed in [Section 3.6 RFC6824].
			Implementations 3 and 4 made several comments about fallback.
			Implementation 3 suggests that both sender and receiver behaviours
			could be outlined with more detail, in particular when DSS checksum
			is not in use and the MPTCP options are stripped. Implementation 3
			falls back to TCP when there's one sub flow, but not when there are
			multiple sub flows (MPTCP is used in active/backup mode, and it is
			assumed that the sub flow transferring data is most likely to be more
			usable than any other established sub flow, hence the sub flow on
			which fallback occurred is kept alive and other sub flows are
			closed).
			Implementation 4 found a corner case where it is not clear what to
			do: if a pure ack or data packet without DSS is received in middle of
			transaction (which can happen if the routing changes and the new path
			drops MPTCP options). Also, Implementation 4 suggests that
			clarifying whether the infinite map exchange is unidirectional or
			bidirectional.
			Implementation 1 has developed a publicly available test suite that
			tests MPTCP's traversal of middleboxes.
			3.5. Question 5: Heuristics
			Question 5 gathers information about heuristics: aspects that are not
			required for protocol correctness but impact the performance. We
			would like to document best practice so that future implementers can
			learn from the experience of pioneers.
			There are several differences between the implementations.
			For receiver buffer, Implementation 1 uses a slightly modified
			version of Linux's auto-tuning algorithm; Implementation 2 determines
			the receiver buffer by using "TCP_MAXWIN << tp->rcv_scale" (this is a
			temporary measure); Implementation 3 uses MPTCP in active/backup
			mode, so the receive buffer sizes at the MPTCP and subflow level is
			the same (automatic buffer tuning is turned off); Implementation 4
			varies the receiver buffer size based on the services and application
			type.
			For the sender buffer, Implementation 1 uses Linux auto-tuning,
			Implementation 2 scales based on occupancy, whilst Implementation 3
			turns off automatic buffer tuning, and Implementation 4 uses MPTCP-
			level (sub)flow control that is (almost) the same as regular TCP flow
			control.
			Implementations 1, 2 and 3 re-transmit unacknowledged data on a
			different subflow (and not the same subflow), whilst Implementation 4
			re-transmits on original subflow for 3 RTOs and then uses another
			subflow.
			For port usage, Implementations 1 and 2 uses the same ports for the
			additional subflows, whilst Implementation 3 uses the same
			detsination port but a different local port, so that on the wire it
			looks like two connections to the same remote destination.
			Implementation 4 suggests that the RFC should more clearly
			/extensively define failure cases and how to handle unexpected
			signals.
			3.6. Question 6: Security
			Question 6 asks about security related matters.
			All Implementations have implemented the hash-based, HMAC-SHA1
			security mechanism defined in [RFC6824]. Implementation 3 suggests
			that a more secure mechanism could be tied with SSL. Implementation
			4 suggests that a more secure and lightweight mechanism is needed, as
			keys are exchanged (in the MP_CAPABLE option) in plain text and the
			key generation mechanism is highly computational intensive.
			Implementation 1 has implemented two additional mechanisms in a
			separate Linux branch - one lightweight and the other SSL-based.
			3.7. Question 7: IANA
			All Implementations have followed the IANA-related definitions
			[Section 8 RFC6824] for: TCP Option Kind number (30); the sub-
			registry for "MPTCP Option Subtypes"; and the sub-registry for "MPTCP
			Handshake Algorithms".
			3.8. Question 8: Congestion control and subflow policy
			Question 8 asks how is shared across multiple subflows.
			Implementation 1 has added support for coupled congestion control
			(both that defined in [RFC6356] and in OLIA,
			draft-khalili-mptcp-congestion-control. The other implementations do
			not include coupled congestion control. Whilst Implementation 2
			plans to add it (currently it uses a simple algorithm spreads traffic
			across the subflows), Implementations 3 and 4 do not plan to add
			coupled congestion control - they use one subflow at a time, with
			others as a backup. Implementation 3 believes it is not currently
			useful to share load across all network interfaces on a mobile node,
			as the interfaces have different characteristics for cost, bring-up
			and power usage. They have both found the B bit (in MP-JOIN) and MP-
			PRIO option very useful for this active /backup operation.
			Implementation 2 is also interested in experimenting with congestion
			control across paths with different path-cost metrics.
			3.9. Question 9: API
			Question 9 gathers information about the API. None have implemented
			the [RFC6897] "basic MPTCP API" for MPTCP-aware applications. For
			three implementations MPTCP is used for all applications (set by
			configuration), whilst Implementation 3 uses a private API that
			allows MPTCP to be used on a per application basis.
			3.10. Question 10: Deployments, use cases and operational experiences
			Question 10 takes the opportunity of this survey to gather some
			limited information about operational experiences and deployments.
			The Implementations mention different use cases.
			Implementation 2 is interested in using MPTCP for several use cases:
			vehicle to infrastructure (V2I) connectivity (to provide a persistent
			connection using 3G and roadside wifi); multi-homed "home-user"
			environments; high throughput data transfers. Implementation 3 is
			interested in the mobile scenario, with MPTCP providing an active
			/backup mode so achieving session continuity across changing network
			environments. Implementation 4 is interested in MPTCP giving
			reliability and fault tolerance via a proxy. Implementation 1
			already uses MPTCP on www.multipath-tcp.org and for internal ssh
			servers at UCLouvain.
			Implementation 4 uses a proxy (MPTCP connections from a client are
			terminated and the TCP connection established on the other side),
			whilst the other Implementations are on end hosts. Implementation 2
			is so far within controlled testbeds, whilst Implementation 3 is on
			the Internet.
			Implementation 2 is currently an alpha-quality build, so limited
			testing so far.
			Implementation 3 suggests working at the IETF with firewall vendors,
			to get them to change their defaults to allow MPTCP signals. This
			would also reduce the "over-engineering" needed to handle fallback
			cases. Implementation 1 suggests retrieving logs from middleboxes,
			as the best approach to understanding the interactions of MPTCP
			signalling with middleboxes.
			Implementation 3 discusses a scenario that should be handled better.
			A backup subflow may never sent data. If the initial subflow fails,
			data is retransmitted on the backup subflow, but that path has a
			middlebox stripping options. Then it may not be possible to recover
			the MPTCP session.
			3.11. Question 11: Improvements to RFC6824
			Question 11 asks if there are any areas where RFC6824 could be
			improved. The main topics have been mentioned earlier:
			o fallback: the need for more clarity in the fallback cases is
			mentioned by Implementations 3 and 4.
			o security: the need for both a more secure and a more lightweight
			mechanism is mentioned.
			Implementation 3 also suggests several potential improvements, which
			are outside the scope of RFC6824: support for sub flow level
			automatic buffer scaling, varying QoS support, and varying window
			scaling support on each sub flow; also, additional work on option
			signlling will be brought up in future discussions.
			4. IANA Considerations
			This document makes no request of IANA.
			5. Security Considerations
			This survey does not impact the security of MPTCP, except to the
			extent that it uncovers security issues that can be tackled in a
			future version of the protocol.
			6. Acknowledgements
			Many thanks to the people who replied to the survey: Christoph
			Paasch, Nigel Williams, anon, and Krishna Khanal. Very many thanks
			to all of the teams who actually did the implementation and testing
			and are continuing to improve them.
			7. Full survey response for Implementation 1
			Question 1: Your details Question 1 gathers some information about
			the team that has implemented MPTCP.
			1. Your institution: UCLouvain, IP Networking Lab
			(http://inl.info.ucl.ac.be)
			2. Name(s) of people in your implementation and test teams: Initial
			design from Sebastien Barre. Since then, numerous code-contributors
			(ordered by number of commits): Christoph Paasch (UCLouvain) Gregory
			Detal (UCLouvain) Jakko Korkeaniemi (Aalto University) Mihai P.
			Andrei (Intel) Fabien Duchene (UCLouvain) Andreas Seelinger (RWTH
			Aachen) Stefan Sicleru (Intel) Lavkesh Lahngir Catalin Nicutar (PUB
			Bucharest) Andrei Maruseac (Intel) Andreas Ripke (NEC) Vlad Dogaru
			(Intel) Octavian Purdila (Intel) Niels Laukens (VRT Belgium) John
			Ronan (TSSG) Brandon Heller (Stanford University) Conformance
			Testing: Yvan Coene (UCLouvain)
			3. Do you want your answers to Question 1.1 and 1.2 above to be
			anonymised? No.
			3.2. Question 2: Preliminary information about your implementation
	Question 2 gathers some preliminary information.		Question 2 gathers some preliminary information.
	1. What OS is your implementation for? (or is it application		1. What OS is your implementation for? (or is it application layer?)
	layer?)		Linux Kernel.
	2. Do you support IPv4 or IPv6 addresses or both?		2. Do you support IPv4 or IPv6 addresses or both? We support both.
	3. Is it publicly available (or will it be?) (for free or a fee?)		3. Is it publicly available (or will it be?) (for free or a fee?)
			Publicly available (GPLv2) at www.multipath-tcp.org
	4. Overall, what are you implementation and testing plans? (details		4. Overall, what are you implementation and testing plans? (details
	can be given against individual items later)		can be given against individual items later) We plan to continue to
			align our implementation with the IETF specifications and improve its
			performance and stability.
	5. Is it an independent implementation? Or does it build on another		5. Is it an independent implementation? Or does it build on another
	MPTCP implementation -which one?		MPTCP implementation -which one? Independent implementation.
	6. Have you already done some interop tests, for example with		6. Have you already done some interop tests, for example with
	UCLouvain's "reference" Linux implementation?		UCLouvain's "reference" Linux implementation? /
	7. Would you be prepared to take part in an interop event, for		7. Would you be prepared to take part in an interop event, for
	example adjacent to IETF-87 in Berlin?		example adjacent to IETF-87 in Berlin? Yes. We are also ready to
			help in organising such an event if needed.
	3.3. Question 3: Support for MPTCP's Signalling Functionality
			3.3. Question 3: Support for MPTCP's Signalling Functionality
	Question 3 asks about support for the various signalling messages		Question 3 asks about support for the various signalling messages
	that the MPTCP protocol defines.		that the MPTCP protocol defines. *** For each message, please give a
			little information about the status of your implementation: for
	*** For each message, please give a little information about the		example, you may have implemented it and fully tested it; the
	status of your implementation: for example, you may have implemented		implementation may be in progress; you have not yet implemented it
	it and fully tested it; the implementation may be in progress; you		but plan to soon (timescale?); you may you have no intention to
	have not yet implemented it but plan to soon (timescale?); you may		implement it (why?); etc.
	you have no intention to implement it (why?); etc.
	1. Connection initiation (MP_CAPABLE) [Section 3.1 RFC6824]		1. Connection initiation (MP_CAPABLE) [Section 3.1 RFC6824]
	a. What is the status of your implementation?		a. What is the status of your implementation? Fully support the
			MP_CAPABLE exchange.
	b. Any other comments or information?		b. Any other comments or information? We generate the random key as
			a hash of the 5-tuple, sequence number and a local secret. This
			significantly improves the performance, instead of using a pseudo-
			random number generator. The performance benefit has been shown
			during IETF85
			http://tools.ietf.org/agenda/85/slides/slides-85-mptcp-2.pdf
	2. Starting a new subflow (MP_JOIN) [Section 3.2 RFC6824]		2. Starting a new subflow (MP_JOIN) [Section 3.2 RFC6824]
	a. What is the status of your implementation?		a. What is the status of your implementation? Fully support the
	b. Can either end of the connection start a new subflow (or only		MP_JOIN exchange.
	the initiator of the original subflow)?
	c. What is the maximum number of subflows your implementation		b. Can either end of the connection start a new subflow (or only the
	can support?		initiator of the original subflow)? Currently, only the initiator of
			the original subflow starts a new subflow. Given the widespread
			deployment of NATs, it is often difficult for the server to reach the
			client. This is the main reason why the server currently does not
			start new subflows in our implementation. But, the initiator would
			accept a SYN+MP_JOIN if sent by another implementation.
	d. Any other comments or information?		c. What is the maximum number of subflows your implementation can
			support? Currently 32.
			d. Any other comments or information?
	3. Data transfer (DSS) [Section 3.3 RFC6824]		3. Data transfer (DSS) [Section 3.3 RFC6824]
	a. What is the status of your implementation?		a. What is the status of your implementation? Fully working
			implementation of data transfer.
	b. The "Data ACK" field can be 4 or 8 octets. Which one(s) have		b. The "Data ACK" field can be 4 or 8 octets. Which one(s) have you
	you implemented?		implemented? We use 4 bytes for the DATA-ACK field.
	c. The "Data sequence number" field can be 4 or 8 octets. Which		c. The "Data sequence number" field can be 4 or 8 octets. Which
	one(s) have you implemented?		one(s) have you implemented? We use 4 bytes for the data sequence
			number.
	d. Does your implementation support the "DATA_FIN" operation to		d. Does your implementation support the "DATA_FIN" operation to
	close an MPTCP connection?		close an MPTCP connection? Yes.
	e. Does your implementation support the "Checksum" field (which		e. Does your implementation support the "Checksum" field (which is
	is negotiated in the MP_CAPABLE handshake)?		negotiated in the MP_CAPABLE handshake)? Yes. This is configurable
			via a sysctl.
	f. Any other comments or information?		f. Any other comments or information? We support interoperability
			with implementations that do send 64-bit data sequence numbers and
			data acks. However, even if the peer sends 64-bit data sequence
			numbers, we will only reply with a 32-bit data-ack. We do not have
			heuristics to trigger the sending of DATA_ACKs. We simply send the
			DATA_ACK in each packet.
	4. Address management (ADD_ADDR and REMOVE_ADDR) [Section 3.4		4. Address management (ADD_ADDR and REMOVE_ADDR) [Section 3.4
	RFC6824]		RFC6824]
	a. What is the status of your implementation?		a. What is the status of your implementation? We support ADD_ADDR/
			REMOVE_ADDR messages.
	b. Can your implementation do ADD_ADDRESS for addresses that		b. Can your implementation do ADD_ADDRESS for addresses that appear
	appear *after* the connection has been established?		*after* the connection has been established? Yes, as shown in:
			"Exploring Mobile/WiFi Handover with Multipath TCP", C. Paasch et.
			al, ACM SIGCOMM workshop on Cellular Networks (Cellnet'12), 2012.
	c. Any other comments or information?		c. Any other comments or information? We do not send out TCP
			keepalive-messages upon the reception of a REMOVE_ADDR-message.
	5. Fast close (MP_FASTCLOSE) [Section 3.5 RFC6824]		5. Fast close (MP_FASTCLOSE) [Section 3.5 RFC6824]
	a. What is the status of your implementation?		a. What is the status of your implementation? We support the
			MP_FASTCLOSE implementation.
	b. Any other comments or information?
	3.4. Question 4: Fallback from MPTCP		b. Any other comments or information?
	Question 4 asks about action when there is a problem with MPTCP, for		3.4. Question 4: Fallback from MPTCP Question 4 asks about action
	example due to a middlebox mangling MPTCP's signalling. The		when there is a problem with MPTCP, for example due to a middlebox
	connection needs to fall back: if the problem is on the first subflow		mangling MPTCP's signalling. The connection needs to fall back: if
	then MPTCP falls back to TCP, whilst if the problem is on an		the problem is on the first subflow then MPTCP falls back to TCP,
	additional subflow then that subflow is closed with a TCP RST, as		whilst if the problem is on an additional subflow then that subflow
	discussed in [Section 3.6 RFC6824].		is closed with a TCP RST, as discussed in [Section 3.6 RFC6824].
	1. If the MP_CAPABLE option is removed by a middlebox, does your		1. If the MP_CAPABLE option is removed by a middlebox, does your
	implementation fall back to TCP?		implementation fall back to TCP? Yes.
	2. If the MP_JOIN option does not get through on the SYNs, does your		2. If the MP_JOIN option does not get through on the SYNs, does your
	implementation close the additional subflow?		implementation close the additional subflow? Yes.
	3. If the DSS option does not get through on the first data		3. If the DSS option does not get through on the first data
	segment(s), does your implementation fall back? (either falling		segment(s), does your implementation fall back? (either falling back
	back to MPTCP (if the issue is on the first subflow) or closing		to MPTCP (if the issue is on the first subflow) or closing the
	the additional subflow (if the issue is on an additional		additional subflow (if the issue is on an additional subflow)) Yes.
	subflow))		On the initial subflow we do a seamless fallback, additional subflows
			will be closed by a RST.
	4. Similarly, if the "DATA ACK" field does not correctly acknowledge		4. Similarly, if the "DATA ACK" field does not correctly acknowledge
	the first data segment(s), does your implementation fall back?		the first data segment(s), does your implementation fall back? Yes.
			Same as above.
	5. Does your implementation protect data with the "Checksum" field		5. Does your implementation protect data with the "Checksum" field
	in the DSS option [Section 3.3 RFC6824]? If the checksum fails		in the DSS option [Section 3.3 RFC6824]? If the checksum fails
	(because the subflow has been affected by a middlebox), does your		(because the subflow has been affected by a middlebox), does your
	implementation immediately close the affected subflow (with a TCP		implementation immediately close the affected subflow (with a TCP
	RST) with the MP_FAIL Option? If the checksum fails and there is		RST) with the MP_FAIL Option? If the checksum fails and there is a
	a single subflow, does your implementation handle this as a		single subflow, does your implementation handle this as a special
	special case, as described in [Section 3.6 RFC6824]?		case, as described in [Section 3.6 RFC6824]? Yes, we support the
			DSS-checksum. If the checksum is wrong and there exist other
			subflows, we close the current subflow with an RST. If there is no
			other subflow, we send an ACK + MP_FAIL and do a fallback to infinite
			mapping. This fallback has successfully been tested with different
			type of NAT middleboxes, while using FTP.
	6. Does your implementation fall back to TCP by using an "infinite		6. Does your implementation fall back to TCP by using an "infinite
	mapping" [Section 3.3.1 RFC6824] (so that the subflow-level data		mapping" [Section 3.3.1 RFC6824] (so that the subflow-level data is
	is mapped to the connection-level data for the remainder of the		mapped to the connection-level data for the remainder of the
	connection)?		connection)? Yes.
	7. Did you find any corner cases where MPTCP's fallback didn't		7. Did you find any corner cases where MPTCP's fallback didn't
	happen properly?		happen properly? No. We have developped a test-suite to test the
			middlebox-traversal of MPTCP, available at
			http://multipath-tcp.org/pmwiki.php/Users/AboutMeasures
	8. Any other comments or information about fallback?		8. Any other comments or information about fallback?
	3.5. Question 5: Heuristics		3.5. Question 5: Heuristics Question 5 gathers information about
			heuristics: aspects that are not required for protocol correctness
	Question 5 gathers information about heuristics: aspects that are not		but impact the performance. We would like to document best practice
	required for protocol correctness but impact the performance. We		so that future implementers can learn from the experience of
	would like to document best practice so that future implementers can		pioneers. The references contain some initial comments about each
	learn from the experience of pioneers. The references contain some		topic.
	initial comments about each topic.
	1. Receiver considerations [S3.3.4, RFC6824]: What receiver buffer		1. Receiver considerations [S3.3.4, RFC6824]: What receiver buffer
	have you used? Does this depend on the retransmission strategy?		have you used? Does this depend on the retransmission strategy?
	What advice should we give about the receiver?		What advice should we give about the receiver? Linux includes an
			autuning algorithm for the TCP receiver buffer. This algorithm has
			been slightly modified for Multipath TCP. The receive-buffer does
			not depend on the retransmission strategy.
	2. Sender considerations [S3.3.5, RFC6824]: How do you determine how		2. Sender considerations [S3.3.5, RFC6824]: How do you determine how
	much data a sender is allowed to send and how big the sender		much data a sender is allowed to send and how big the sender buffer
	buffer is? What advice should we give about the sender?		is? What advice should we give about the sender? The send-buffer is
			autotuned similarly as the receive-buffer (see above). We send as
			much data as possible, filling the congestion windows of each
			subflow. The sender deploys the "Opportunistic Retransmission" and
			"Penalization" algorithms from the paper: "How Hard Can It Be?
			Designing and Implementing a Deployable Multipath TCP", C. Raiciu et.
			al, NSDI 2012.
	3. Reliability and retransmissions [S3.3.6, RFC6824]: What is your		3. Reliability and retransmissions [S3.3.6, RFC6824]: What is your
	retransmission policy? (when do you retransmit on the original		retransmission policy? (when do you retransmit on the original
	subflow vs on another subflow or subflows?) When do you decide		subflow vs on another subflow or subflows?) When do you decide that
	that a subflow is underperforming and should be reset, and what		a subflow is underperforming and should be reset, and what do you
	do you then do? What advice should we give about this issue?		then do? What advice should we give about this issue? Upon an RTO
			on subflow A, we reinject all the unacknowledged data of subflow A on
			another subflows. We do not currently have a mechanism to detect
			that a subflow is underperforming.
	4. Port usage [S3.3.8.1, RFC6824]: Does your implementation use the		4. Port usage [S3.3.8.1, RFC6824]: Does your implementation use the
	same port number for additional subflows as for the first		same port number for additional subflows as for the first subflow?
	subflow? Have you used the ability to define a specific port in		Have you used the ability to define a specific port in the Add
	the Add Address option? What advice should we give about this		Address option? What advice should we give about this issue? We
	issue?		always use the same port number as for the first subflow. Except, if
			the ADD_ADDRESS option that has been received contained a specific
			port. We do not have a means to configure the specific port in the
			ADD_ADDRESS option, but we support reception of the port.
	5. Delayed subflow start [S3.3.8.2, RFC6824]: What factors does your		5. Delayed subflow start [S3.3.8.2, RFC6824]: What factors does your
	implementation consider when deciding about opening additional		implementation consider when deciding about opening additional
	subflows? What advice should we give about this issue?		subflows? What advice should we give about this issue? As soon as
			we are sure that the initial subflow is fully MPTCP-capable
			(reception of a DATA_ACK), we create a full mesh among all IP-
			addresses between the two hosts. We do not explicitly delay the
			creation of new subflows.
	6. Failure handling [S3.3.8.3, RFC6824]: Whilst the protocol defines		6. Failure handling [S3.3.8.3, RFC6824]: Whilst the protocol defines
	how to handle some unexpected signals, the behaviour after other		how to handle some unexpected signals, the behaviour after other
	unexpected signals is not defined. What advice should we give		unexpected signals is not defined. What advice should we give about
	about this issue?		this issue? We did not implement the caching mentioned in Section
			3.8.3.
	7. Use of TCP options: As discussed in [Appendix A, RFC6824], the		7. Use of TCP options: As discussed in [Appendix A, RFC6824], the
	TCP option space is limited, but a brief study found there was		TCP option space is limited, but a brief study found there was enough
	enough room to fit all the MPTCP options. However there are		room to fit all the MPTCP options. However there are constraints on
	constraints on which MPTCP option(s) can be included in packets		which MPTCP option(s) can be included in packets with other TCP
	with other TCP options - do the suggestions in Appendix A need		options - do the suggestions in Appendix A need amending or
	amending or expanding?		expanding? We do not implement specific heuristics to reduce the TCP
			option-space usage. If timestamp is enabled we will only be able to
			send two SACK-blocks, because the DATA_ACK consumes the remaining
			bytes.
	8. What other heuristics should we give advice about? Any other		8. What other heuristics should we give advice about? Any other
	comments or information?		comments or information?
	3.6. Question 6: Security
	Question 6 asks about Security related matters [Section 5 RFC6824].		3.6. Question 6: Security Question 6 asks about Security related
			matters [Section 5 RFC6824].
	1. Does your implementation use the hash-based, HMAC-SHA1 security		1. Does your implementation use the hash-based, HMAC-SHA1 security
	mechanism defined in [RFC6824]?		mechanism defined in [RFC6824]? Yes.
	2. Does your implementation support any other handshake algorithms?		2. Does your implementation support any other handshake algorithms?
			We have in a separate branch, an implementation of
			draft-paasch-mptcp-lowoverhead and draft-paasch-mptcp-ssl.
	3. It has been suggested that a Standards-track MPTCP needs a more		3. It has been suggested that a Standards-track MPTCP needs a more
	secure mechanism. Do you have any views about how to achieve		secure mechanism. Do you have any views about how to achieve this?
	this?		We believe that the solution described in draft-paasch-mptcp-ssl
			would be a good starting point since it leverages the security of the
			upper layer.
	4. Any other comments or information?		4. Any other comments or information?
	3.7. Question 7: IANA		3.7. Question 7: IANA Question 7 asks about IANA related matters.
	Question 7 asks about IANA related matters.
	1. Does your implementation follow the IANA-related definitions?		1. Does your implementation follow the IANA-related definitions?
	[Section 8 RFC6824] defines: TCP Option Kind number (30); the		[Section 8 RFC6824] defines: TCP Option Kind number (30); the sub-
	sub-registry for "MPTCP Option Subtypes"; and the sub-registry		registry for "MPTCP Option Subtypes"; and the sub-registry for "MPTCP
	for "MPTCP Handshake Algorithms"		Handshake Algorithms" Yes.
	2. Any other comments or information?		2. Any other comments or information?
	3.8. Question 8: Congestion control and subflow policy		3.8. Question 8: Congestion control and subflow policy Question 8
			asks about how you share traffic across multiple subflows.
	Question 8 asks about how you share traffic across multiple subflows.
	1. How does your implementation share traffic over the available		1. How does your implementation share traffic over the available
	paths? For example: as a spare path on standby ('all-or-		paths? For example: as a spare path on standby ('all-or- nothing'),
	nothing'), as an 'overflow', etc? Does it have the ability to		as an 'overflow', etc? Does it have the ability to send /receive
	send /receive traffic across multiple subflows simultaneously?		traffic across multiple subflows simultaneously? The implementation
			is able to send and receive traffic on all subflows simultaneously.
			Our scheduler first tries to send traffic on the subflow with the
			lowest RTT. As this subflow's congestion window is full, we pick the
			subflow with the next lower RTT.
	2. Does your implementation support "handover" from one subflow to		2. Does your implementation support "handover" from one subflow to
	another when losing an interface?		another when losing an interface? Yes, as described in: "Exploring
			Mobile/WiFi Handover with Multipath TCP", C. Paasch et. al, ACM
			SIGCOMM workshop on Cellular Networks (Cellnet'12), 2012.
	3. Does your implementation support the coupled congestion control		3. Does your implementation support the coupled congestion control
	defined in [RFC6356]?		defined in [RFC6356]? Yes.
	4. Does your implementation support some other coupled congestion		4. Does your implementation support some other coupled congestion
	control (ie that balances traffic on multiple paths according to		control (ie that balances traffic on multiple paths according to
	feedback)?		feedback)? We also support the OLIA congestion control
			(draft-khalili-mptcp-congestion-control-00).
	5. The MP_JOIN (Starting a new subflow) Option includes the "B" bit,		5. The MP_JOIN (Starting a new subflow) Option includes the "B" bit,
	which allows the sender to indicate whether it wishes the new		which allows the sender to indicate whether it wishes the new subflow
	subflow to be used immediately or as a backup if other path(s)		to be used immediately or as a backup if other path(s) fail. The
	fail. The MP_PRIO Option is a request to change the "B" bit -		MP_PRIO Option is a request to change the "B" bit - either on the
	either on the subflow on which it is sent, or (by setting the		subflow on which it is sent, or (by setting the optional Address ID
	optional Address ID field) on other subflows. Does your		field) on other subflows. Does your implementation support the "B"
	implementation support the "B" bit and MP_PRIO mechanisms? Do		bit and MP_PRIO mechanisms? Do you think they're useful, or have
	you think they're useful, or have another suggestion?		another suggestion? Yes, we support the "B"-bit of the MP_JOIN and
			the MP_PRIO option. It is configurable on a per-interface basis.
			Experiences with the "B"-bit can be found in our paper: "Exploring
			Mobile/WiFi Handover with Multipath TCP", C. Paasch et. al, ACM
			SIGCOMM workshop on Cellular Networks (Cellnet'12), 2012.
	6. Any other comments or information or suggestions about the advice		6. Any other comments or information or suggestions about the advice
	we should give about congestion control [S3.3.7 RFC6824] and		we should give about congestion control [S3.3.7 RFC6824] and subflow
	subflow policy [S3.3.8 RFC6824]?		policy [S3.3.8 RFC6824]?
	3.9. Question 9: API		3.9. Question 9: API Question 9 gathers information about your API.
	Question 9 gathers information about your API. [RFC6897] considers		[RFC6897] considers the MPTCP Application Interface.
	the MPTCP Application Interface.
	1. With your implementation, can legacy applications use (the		1. With your implementation, can legacy applications use (the
	existing sockets API to use) MPTCP? How does the implementation		existing sockets API to use) MPTCP? How does the implementation
	decide whether to use MPTCP? Should the advice in [Section 4,		decide whether to use MPTCP? Should the advice in [Section 4,
	RFC6897] be modified or expanded?		RFC6897] be modified or expanded? Yes, a standard TCP socket API can
			be used. By default MPTCP is enabled on all connections.
	2. The "basic MPTCP API" enables MPTCP-aware applications to		2. The "basic MPTCP API" enables MPTCP-aware applications to
	interact with the MPTCP stack via five new socket options. For		interact with the MPTCP stack via five new socket options. For each
	each one, have you implemented it? has it been useful?		one, have you implemented it? has it been useful? None of them are
			part of the current stable release MPTCP v0.86.
			http://multipath-tcp.org/pmwiki.php?n=Main.Release86 a.
			TCP_MULTIPATH_ENABLE? b. TCP_MULTIPATH_ADD? c.
			TCP_MULTIPATH_REMOVE? d. TCP_MULTIPATH_SUBFLOWS? e.
			TCP_MULTIPATH_CONNID?
	a. TCP_MULTIPATH_ENABLE?		3. Have you implemented any aspects of an "advanced MPTCP API"?
			([Appendix A, RFC6897] hints at what it might include.) No.
	b. TCP_MULTIPATH_ADD?		4. Any other comments or information?
	c. TCP_MULTIPATH_REMOVE?		3.10. Question 10: Deployments, use cases and operational
			experiences Question 10 takes the opportunity of this survey to
			gather some limited information about operational experiences and
			deployments. Any very brief information would be appreciated, for
			example: 1. What deployment scenarios are you most interested in? 2.
			Is your deployment on "the Internet" or in a controlled environment?
			3. Is your deployment on end hosts or with a MPTCP-enabled proxy (at
			one or both ends?)? 4. What do you see as the most important
			benefits of MPTCP in your scenario(s)? 5. How extensively have you
			deployed and experimented with MPTCP so far?
	d. TCP_MULTIPATH_SUBFLOWS?		Our implementation is open-source and has been discussed for various
			types of tests/deployments based on the messages received on the
			mptcp-dev mailing list. We currently use Multipath TCP on
			www.multipath-tcp.org and also on internal ssh servers at UCLouvain.
			6. MPTCP's design seeks to maximise the chances that the signalling
			works through middleboxes. Did you find cases where middleboxes
			blocked MPTCP signalling? We have implemented a test suite based on
			a slightly modified version of the Multipath TCP implementation that
			allows to check the interoperability between Multipath TCP and
			middleboxes. We have used it over Internet paths and identified some
			potential problems. However, the best approach to test these
			interactions would be to control the middlebox and analyse its logs
			during the Multipath TCP test. The test suite can be retrieved from
			http://multipath-tcp.org/pmwiki.php/Users/AboutMeasures
	e. TCP_MULTIPATH_CONNID?		7. MPTCP's design seeks to ensure that, if there is a problem with
			MPTCP signalling, then the connection either falls back to TCP or
			removes the problematic subflow. Did you find any corner cases where
			this didn't happen properly? See above.
	3. Have you implemented any aspects of an "advanced MPTCP API"?		8. Have you encountered any issues or drawbacks with MPTCP?
	([Appendix A, RFC6897] hints at what it might include.)
			9. Any other comments or information?
			3.11. Question 11: Improvements to RFC6824
			1. Are there any areas where [RFC6824] could be improved, either in
			technical content or clarity? 2. Any other issues you want to raise?
			8. Full survey response for Implementation 2
			Question 1: Your details
			-------------------------------------------------------------
			1.1 Swinburne University of Technology, Hawthorn, Victoria, Australia
			1.2 Lawrence Stewart, Nigel Williams
			1.3 No
			Question 2: Preliminary information about your implementation
			-------------------------------------------------------------
			2.1 FreeBSD-10
			2.2 Currently IPv4 only (IPv6 support will eventually be added)
			2.3 Publicly available (http://caia.swin.edu.au/urp/newtcp/mptcp/).
			The code is released under the BSD license. 2.3
			2.5 Independent
			2.6 Yes, some limited testing to establish interoperability.
			2.7 Yes, with some additional work this should be possible (if not
			then IETF-88). Q
			uestion 3: Support for MPTCP's Signaling Functionality
			-------------------------------------------------------------
			3.1 a) MP_CAPABLE Implemented
			b) Do not currently honour checksum flag (to be implemented)
			3.2 a) MP_JOIN Implemented
			b) Either end can initiate a MP_JOIN
			c) 8 (controlled via sysctl)
			d) Currently do not include HMAC verification during handshake, but
			this will be enabled in the next patch (several weeks from time of
			submission)
			3.3 a) DSS Implemented
			b) 4 (default) and 8
			c) 4 (default) and 8
			d) Yes, however the connection tear-down exchange is not fully
			implemented - the connection shuts down but the DFIN may not be
			correctly acknowledged.
			e) No. This will be supported eventually (time-frame unknown)
			3.4 a) ADD_ADDR implemented, REMOVE_ADDR not implemented (to be done,
			timeframe unknown)
			b) No. Functionality to be added
			3.5 MP_FASTCLOSE not implemented. Plan to implement eventually
			Question 4: Fallback from MPTCP
			-------------------------------------------------------------
			4.1 Yes
			4.2 The subflow PCBs remain allocated, however the subflow is not
			used to send data.
			4.3 No, tbd
			4.4 No, tbd
			4.5 No, checksumming not implemented
			4.6 Yes
			4.8 Fallback hasn't really been put through any structured tests yet
			Question 5: Heuristics
			-------------------------------------------------------------
			5.1 We use "TCP_MAXWIN << tp->rcv_scale". This is temporary and we
			will use a call into the "multipath" control layer to determine this
			value in future releases (we need to investigate a suitable way of
			calculating this).
			5.2 cwnd determines the amount of data to send (given that rcv window
			is always very large). Sendbuffer is scaled based on occupancy.
			5.3 We currently don't have Data-level retransmits enabled. However
			our policy is to retransmit on the next subflow that requests data to
			send that is suitable. There is no intelligence in the packet
			schedular currently,
			5.4 The same port numbers are re-used for additional subflows.
			Question 6: Security
			-------------------------------------------------------------
			6.1 Yes
			6.2 No
			Question 7: IANA
			-------------------------------------------------------------
			7.1 Yes
			Question 8: Congestion Control and subflow policy
			-------------------------------------------------------------
			8.1 A simple algorithm is used to divide the send buffer between
			subflows, so that traffic is spread across the subflows.
			8.3 No. (to be added)
			8.4 No
			8.5 No
			Question 9: API
			-------------------------------------------------------------
			9.1 Legacy applications are able to use MPTCP. MPTCP is set globally
			via a sysctl variable.
			9.2 No
			9.3 No
			Question 10: API
			-------------------------------------------------------------
			10.1 Some current project work is based on MPTCPs use in vehicle to
			infrastructure (V2I) connectivity (to provide a persistent connection
			using 3G and roadside wifi). Other interests are in multi-homed
			"home-user" environments, high throughput data transfers.... We are
			also interested in experimenting with congestion control across paths
			with different path-cost metrics.
			10.2 So far only within controlled testbeds
			10.3 End hosts
			10.4 Depending on the scenario, connection persistence, throughput...
			10.5 Still an alpha-quality build, so limited testing so far.
			9. Full survey response for Implementation 3
			Survey 3.1. Question 1: Your details Question 1 gathers some
			information about the team that has implemented MPTCP.
			1. Your institution: anonymized.
			2. Name(s) of people in your implementation and test teams: There
			were several folks involved in the implementation and testing.
			3. Do you want your answers to Question 1.1 and 1.2 above to be
			anonymised? Yes.
			3.2. Question 2: Preliminary information about your implementation
			Question 2 gathers some preliminary information.
			1. What OS is your implementation for? (or is it application layer?)
			anonymized (commercial OS)
			2. Do you support IPv4 or IPv6 addresses or both? Both.
			3. Is it publicly available (or will it be?) (for free or a fee?)
			No.
			4. Overall, what are you implementation and testing plans? (details
			can be given against individual items later) We plan to use it in a
			mobile environment.
			5. Is it an independent implementation? Or does it build on another
			MPTCP implementation -which one? It is an independent
			implementation.
			6. Have you already done some interop tests, for example with
			UCLouvain's "reference" Linux implementation? Most MPTCP option
			formats were tested with the reference Linux implementation.
			7. Would you be prepared to take part in an interop event, for
			example adjacent to IETF-87 in Berlin? Unsure at this point.
			3.3. Question 3: Support for MPTCP's Signalling Functionality
			Question 3 asks about support for the various signalling messages
			that the MPTCP protocol defines. *** For each message, please give a
			little information about the status of your implementation: for
			example, you may have implemented it and fully tested it; the
			implementation may be in progress; you have not yet implemented it
			but plan to soon (timescale?); you may you have no intention to
			implement it (why?); etc.
			1. Connection initiation (MP_CAPABLE) [Section 3.1 RFC6824] a. What
			is the status of your implementation? Fully implemented and tested
			against the reference Linux implementation.
			b. Any other comments or information?
			2. Starting a new subflow (MP_JOIN) [Section 3.2 RFC6824] a. What
			is the status of your implementation? Fully implemented and tested
			against the reference Linux implementation.
			b. Can either end of the connection start a new subflow (or only the
			initiator of the original subflow)? Only the initiator of the
			original sub flow can start other sub flows.
			c. What is the maximum number of subflows your implementation can
			support? There is no hard limit.
			d. Any other comments or information?
			3. Data transfer (DSS) [Section 3.3 RFC6824] a. What is the status
			of your implementation? Fully implemented and tested.
			b. The "Data ACK" field can be 4 or 8 octets. Which one(s) have you
			implemented? Both have been implemented but the use of the 4-byte
			field is the default. When an 8 byte DSS is received, an 8 byte Data
			ACK is sent in response.
			c. The "Data sequence number" field can be 4 or 8 octets. Which
			one(s) have you implemented? Both have been implemented but the use
			of the 4-byte field is the default. When a wraparound of the lower
			32-bit part of the DSS is detected, the full 8 byte DSS is sent.
			d. Does your implementation support the "DATA_FIN" operation to
			close an MPTCP connection? Yes. There are cases however where the
			sub flows are closed (TCP FIN'd) but the DATA_FIN is not sent - in
			this case the MPTCP connection must be closed through a garbage
			collector after some idle time.
			e. Does your implementation support the "Checksum" field (which is
			negotiated in the MP_CAPABLE handshake)? Yes.
			f. Any other comments or information?
			4. Address management (ADD_ADDR and REMOVE_ADDR) a. What is the
			status of your implementation? It does not support sending ADD_ADDR
			or processing ADD_ADDR as it is considered a security risk. Also, we
			only have a client side implementation at the moment which always
			initiates the sub flows. The remote end does not send ADD_ADDR in
			our configuration. The client can send REMOVE_ADDR however when one
			of the established sub flow's source address goes away. The client
			ignores incoming REMOVE_ADDR options also.
			b. Can your implementation do ADD_ADDRESS for addresses that appear
			*after* the connection has been established? No. c. Any other
			comments or information?
			5. Fast close (MP_FASTCLOSE) [Section 3.5 RFC6824] a. What is the
			status of your implementation? It is supported. Though
			Retransmission of Fast close is not supported yet.
			b. Any other comments or information?
			3.4. Question 4: Fallback from MPTCP Question 4 asks about action
			when there is a problem with MPTCP, for example due to a middlebox
			mangling MPTCP's signalling. The connection needs to fall back: if
			the problem is on the first subflow then MPTCP falls back to TCP,
			whilst if the problem is on an additional subflow then that subflow
			is closed with a TCP RST, as discussed in [Section 3.6 RFC6824].
			1. If the MP_CAPABLE option is removed by a middlebox, does your
			implementation fall back to TCP? Yes.
			2. If the MP_JOIN option does not get through on the SYNs, does your
			implementation close the additional subflow? Yes.
			3. If the DSS option does not get through on the first data
			segment(s), does your implementation fall back? (either falling back
			to MPTCP (if the issue is on the first subflow) or closing the
			additional subflow (if the issue is on an additional subflow)) Yes it
			falls back to TCP when there's one sub flow. When there are multiple
			sub flows, since MPTCP is used in active/backup mode, it is assumed
			that the sub flow transferring data is most likely to be more usable
			than any other established sub flow. So the sub flow on which
			fallback occurred is kept alive and other sub flows are closed.
			Fallback though is not guaranteed to occur safely when there are more
			than one sub flows because the infinite mapping option may be
			stripped like other DSS options and the MP_FAIL option if used in
			scenarios other than for reporting checksum failure can also be
			stripped.
			4. Similarly, if the "DATA ACK" field does not correctly acknowledge
			the first data segment(s), does your implementation fall back? No.
			Current implementation just ignores the unexpected data ack.
			5. Does your implementation protect data with the "Checksum" field
			in the DSS option [Section 3.3 RFC6824]? If the checksum fails
			(because the subflow has been affected by a middlebox), does your
			implementation immediately close the affected subflow (with a TCP
			RST) with the MP_FAIL Option? If the checksum fails and there is a
			single subflow, does your implementation handle this as a special
			case, as described in [Section 3.6 RFC6824]? Yes.
			6. Does your implementation fall back to TCP by using an "infinite
			mapping" [Section 3.3.1 RFC6824] (so that the subflow-level data is
			mapped to the connection-level data for the remainder of the
			connection)? Yes.
			7. Did you find any corner cases where MPTCP's fallback didn't
			happen properly? If the very first sub flow does not send any data
			and is disconnected right away, then the current implementation
			allows a join to occur with the addition of another sub flow which
			then becomes a fully mp capable sub flow. Thus we allow break before
			make by letting additional sub flows to be joined if the very first
			one disconnected even without sending any data. This is a very
			corner case but an instance where we do not follow the rules of
			fallback (allow second sub flow even when first sub flow did not
			send/receive data/data acks).
			8. Any other comments or information about fallback? Fallback after
			connection establishment and after a few data packets were
			transferred with MPTCP options is complicated. The spec does not
			clearly cover the cases of options being stripped by middle boxes.
			It goes into good detail about what to do when the DSS checksum
			fails, but not when DSS checksum is not in use and the MPTCP options
			are stripped. Both sender/receiver behaviors could be outlined with
			more detail.
			3.5. Question 5: Heuristics Question 5 gathers information about
			heuristics: aspects that are not required for protocol correctness
			but impact the performance. We would like to document best practice
			so that future implementers can learn from the experience of
			pioneers. The references contain some initial comments about each
			topic.
			1. Receiver considerations [S3.3.4, RFC6824]: What receiver buffer
			have you used? Does this depend on the retransmission strategy?
			What advice should we give about the receiver? We are just using
			MPTCP in active/backup mode. This mode is simpler wrt receive buffer
			utilization. The receive buffer sizes at the MPTCP and sub flow
			level is the same. Automatic buffer tuning is turned off when MPTCP
			is in use.
			2. Sender considerations [S3.3.5, RFC6824]: How do you determine how
			much data a sender is allowed to send and how big the sender buffer
			is? What advice should we give about the sender? Automatic buffer
			tuning is turned off when MPTCP is in use.
			3. Reliability and retransmissions [S3.3.6, RFC6824]: What is your
			retransmission policy? (when do you retransmit on the original
			subflow vs on another subflow or subflows?) When do you decide that
			a subflow is underperforming and should be reset, and what do you
			then do? What advice should we give about this issue?
			Retransmissions at MPTCP level do not occur on the same sub flow
			except when MP_FAIL option is received. A sub flow is said to be
			underperforming when its network connectivity goes away.
			4. Port usage [S3.3.8.1, RFC6824]: Does your implementation use the
			same port number for additional subflows as for the first subflow?
			Have you used the ability to define a specific port in the Add
			Address option? What advice should we give about this issue? The
			destination port is the same. The local port changes for additional
			sub flows so on the wire it is like two tcp connections to the same
			remote destination. We have not used Add Address option at all.
			5. Delayed subflow start [S3.3.8.2, RFC6824]: What factors does your
			implementation consider when deciding about opening additional
			subflows? What advice should we give about this issue? The client
			implementation is aware of network interfaces coming up or going down
			and establishes new sub flows or removes existing sub flows
			accordingly.
			6. Failure handling [S3.3.8.3, RFC6824]: Whilst the protocol defines
			how to handle some unexpected signals, the behaviour after other
			unexpected signals is not defined. What advice should we give about
			this issue? Fallback, post establishment is probably a case that
			needs to be more clearly defined.
			7. Use of TCP options: As discussed in [Appendix A, RFC6824], the
			TCP option space is limited, but a brief study found there was enough
			room to fit all the MPTCP options. However there are constraints on
			which MPTCP option(s) can be included in packets with other TCP
			options - do the suggestions in Appendix A need amending or
			expanding? Looks good already.
			8. What other heuristics should we give advice about? Any other
			comments or information?
			3.6. Question 6: Security Question 6 asks about Security related
			matters [Section 5 RFC6824].
			1. Does your implementation use the hash-based, HMACSHA1 security
			mechanism defined in [RFC6824]? Yes.
			2. Does your implementation support any other handshake algorithms?
			No.
			3. It has been suggested that a Standards-track MPTCP needs a more
			secure mechanism. Do you have any views about how to achieve this?
			No. But the mechanism could be tied with SSL because SSL is used
			wherever security is deemed important.
	4. Any other comments or information?		4. Any other comments or information?
	3.10. Question 10: Deployments, use cases and operational experiences		3.7. Question 7: IANA Question 7 asks about IANA related matters.
	Question 10 takes the opportunity of this survey to gather some		1. Does your implementation follow the IANA-related definitions?
	limited information about operational experiences and deployments.		[Section 8 RFC6824] defines: TCP Option Kind number (30); the sub-
	Any very brief information would be appreciated, for example:		registry for "MPTCP Option Subtypes"; and the Page 12 of 17 Survey
			6/22/13, 5:55 PM sub-registry for "MPTCP Handshake Algorithms" Yes.
	1. What deployment scenarios are you most interested in?		2. Any other comments or information? No.
			3.8. Question 8: Congestion control and subflow policy Question 8
			asks about how you share traffic across multiple subflows.
			1. How does your implementation share traffic over the available
			paths? For example: as a spare path on standby ('all-ornothing'), as
			an 'overflow', etc? Does it have the ability to send /receive
			traffic across multiple subflows simultaneously? It uses active/
			backup where one sub flow is preferred or has higher priority over
			other sub flows. When the preferred sub flow fails or begins to
			experience retransmission timeouts, the other sub flows are used.
			2. Does your implementation support "handover" from one subflow to
			another when losing an interface? Yes.
			3. Does your implementation support the coupled congestion control
			defined in [RFC6356]? No.
			4. Does your implementation support some other coupled congestion
			control (ie that balances traffic on multiple paths according to
			feedback)? No.
			5. The MP_JOIN (Starting a new subflow) Option includes the "B" bit,
			which allows the sender to indicate whether it wishes the new subflow
			to be used immediately or as a backup if other path(s) fail. The
			MP_PRIO Option is a request to change the "B" bit - either on the
			subflow on which it is sent, or (by setting the optional Address ID
			field) on other subflows. Does your implementation support the "B"
			bit and MP_PRIO mechanisms? Do you think they're useful, or have
			another suggestion? Yes the implementation uses the B bit and the
			MP_PRIO option. They are very useful for the active/backup mode of
			operation.
			6. Any other comments or information or suggestions about the advice
			we should give about congestion control [S3.3.7 RFC6824] and subflow
			policy [S3.3.8 RFC6824]?
			3.9. Question 9: API Question 9 gathers information about your API.
			[RFC6897] considers the MPTCP Application Interface.
			1. With your implementation, can legacy applications use (the
			existing sockets API to use) MPTCP? How does the implementation
			decide whether to use MPTCP? Should the advice in [Section 4,
			RFC6897] be modified or expanded? The implementation does not
			support MPTCP with existing sockets API. MPTCP is exposed through a
			private SPI today. If MPTCP becomes prolific over the next few
			years, MPTCP use shall be expanded.
			2. The "basic MPTCP API" enables MPTCP-aware applications to
			interact with the MPTCP stack via five new socket options. For each
			one, have you implemented it? has it been useful? a.
			TCP_MULTIPATH_ENABLE? b. TCP_MULTIPATH_ADD? c.
			TCP_MULTIPATH_REMOVE? d. TCP_MULTIPATH_SUBFLOWS? e.
			TCP_MULTIPATH_CONNID? This mode of API is not used. Proprietary
			methods are used for achieving these basic operations.
			3. Have you implemented any aspects of an "advanced MPTCP API"?
			([Appendix A, RFC6897] hints at what it might include.) No.
			4. Any other comments or information?
			3.10. Question 10: Deployments, use cases and operational
			experiences Question 10 takes the opportunity of this survey to
			gather some limited information about operational experiences and
			deployments. Any very brief information would be appreciated, for
			example:
			1. What deployment scenarios are you most interested in? MPTCP in
			mobile environments is very powerful when used in the active/backup
			mode. Since the network interfaces available on mobile devices have
			different cost characteristics as well as different bring up and
			power usage characteristics, it is not useful to share load across
			all available network interfaces - at least not currently. Providing
			session continuity across changing network environments is the key
			deployment scenario.
	2. Is your deployment on "the Internet" or in a controlled		2. Is your deployment on "the Internet" or in a controlled
	environment?		environment? The deployment is on the Internet.
	3. Is your deployment on end hosts or with a MPTCP-enabled proxy (at		3. Is your deployment on end hosts or with a MPTCPenabled proxy (at
	one or both ends?)?		one or both ends?)? The deployment supports MPTCP on both ends.
	4. What do you see as the most important benefits of MPTCP in your		4. What do you see as the most important benefits of MPTCP in your
	scenario(s)?		scenario(s)? Described in point 1 of this section.
	5. How extensively have you deployed and experimented with MPTCP so		5. How extensively have you deployed and experimented with MPTCP so
	far?		far? Deployment is still in early stages. We have been
			experimenting with MPTCP for about a year.
	6. MPTCP's design seeks to maximise the chances that the signalling		6. MPTCP's design seeks to maximise the chances that the signalling
	works through middleboxes. Did you find cases where middleboxes		works through middleboxes. Did you find cases where middleboxes
	blocked MPTCP signalling?		blocked MPTCP signalling? Corporate firewalls block MPTCP signaling
			by default. IETF is one venue where Cisco, and other firewall
			vendors can be asked to change their defaults to allow MPTCP signals.
	7. MPTCP's design seeks to ensure that, if there is a problem with		7. MPTCP's design seeks to ensure that, if there is a problem with
	MPTCP signalling, then the connection either falls back to TCP or		MPTCP signalling, then the connection either falls back to TCP or
	removes the problematic subflow. Did you find any corner cases		removes the problematic subflow. Did you find any corner cases where
	where this didn't happen properly?		this didn't happen properly? This has been covered a bit in the
			Fallback section. When using two sub flows in active/backup mode,
			there is a possibility that a backup sub flow that never sent data
			starts being used for retransmitting data that is not going through
			on the active path. While it is preferable to keep the initial sub
			flow that successfully sent MPTCP options and drop the backup path,
			the initial sub flow may be the failing one, and we may want to move
			to the backup path. But the backup path can be retransmitting data
			that did not get sent successfully on the active path and if there is
			a middle box in the backup sub flow's path stripping options, then we
			have a case where the MPTCP session may not be recoverable as it may
			not be evident from what point in the MPTCP sequence space, data was
			being sent. The spec does talk of retaining the initial sub flow and
			closing the failed flow. So perhaps doing the reverse is not
			recommended, however, it would certainly be advantageous to support
			MPTCP better in such a failing environment. Also, in parallel
			working with firewall vendors to allow MPTCP options always to not
			have to over-engineer these cases.
	8. Have you encountered any issues or drawbacks with MPTCP?		8. Have you encountered any issues or drawbacks with MPTCP?
	9. Any other comments or information?		9. Any other comments or information?
	3.11. Question 11: Improvements to RFC6824		3.11. Question 11: Improvements to RFC6824 1. Are there any areas
			where [RFC6824] could be improved, either in technical content or
			clarity? Discussed in the fallback section. Other areas around
			MPTCP performance such as support for sub flow level automatic buffer
			scaling, varying QoS support, varying window scaling support on each
			sub flow may be worth discussing further, although they are outside
			the scope of the current spec.
	1. Are there any areas where [RFC6824] could be improved, either in		2. Any other issues you want to raise? Some additional work on
	technical content or clarity?		option signaling that we will bring up in future discussions.
	2. Any other issues you want to raise?		10. Full survey response for Implementation 4
	4. IANA Considerations		1. Your institution: Citrix Systems, Inc.
	This document makes no request of IANA.		2. Name(s) of people in your implementation and test teams: NA
	5. Security Considerations		3. Do you want your answers to Question 1.1 and 1.2 above to be
			anonymised? No
	This survey does not impact the security of MPTCP, except to the		3.2. Question 2: Preliminary information about your implementation
	extent that it uncovers security issues that can be tackled in a		Question 2 gathers some preliminary information.
	future version of the protocol.
	6. Acknowledgements		1. What OS is your implementation for? (or is it application layer?)
			NetScaler Firmware
	7. Normative References		2. Do you support IPv4 or IPv6 addresses or both? Both
			3. Is it publicly available (or will it be?) (for free or a fee?)
			It is available for purchase
			4. Overall, what are you implementation and testing plans? (details
			can be given against individual items later)
			5. Is it an independent implementation? Or does it build on another
			MPTCP implementation -which one? It is an independent implementation
			6. Have you already done some interop tests, for example with
			UCLouvain's "reference" Linux implementation? Yes, our
			implementation is extensively tested with Linux reference
			implementation
			7. Would you be prepared to take part in an interop event, for
			example adjacent to IETF-87 in Berlin?
			3.3. Question 3: Support for MPTCP's Signalling Functionality
			Question 3 asks about support for the various signalling messages
			that the MPTCP protocol defines. *** For each message, please give a
			little information about the status of your implementation: for
			example, you may have implemented it and fully tested it; the
			implementation may be in progress; you have not yet implemented it
			but plan to soon (timescale?); you may you have no intention to
			implement it (why?); etc.
			1. Connection initiation (MP_CAPABLE) [Section 3.1 RFC6824] a. What
			is the status of your implementation? Fully implemented and tested
			b. Any other comments or information? One security concern here is
			that the keys are exchanged in plain text which is prone to attacks
			and also the key generation mechanism is highly computational
			intensive
			2. Starting a new subflow (MP_JOIN) [Section 3.2 RFC6824] a. What
			is the status of your implementation? Fully implemented and tested
			b. Can either end of the connection start a new subflow (or only the
			initiator of the original subflow)? Only the initiator of the
			original subflow can initiate additional subflows.
			c. What is the maximum number of subflows your implementation can
			support? we support maximum 6 subflows.
			d. Any other comments or information?
			3. Data transfer (DSS) [Section 3.3 RFC6824] a. What is the status
			of your implementation? Fully implemented and tested
			b. The "Data ACK" field can be 4 or 8 octets. Which one(s) have you
			implemented? Our implementation supports both 4 or 8 Octets Data Ack
			in both the directions
			c. The "Data sequence number" field can be 4 or 8 octets. Which
			one(s) have you implemented? Our implementation supports both 4 or 8
			Octets DSN in both the directions
			d. Does your implementation support the "DATA_FIN" operation to
			close an MPTCP connection? YES
			e. Does your implementation support the "Checksum" field (which is
			negotiated in the MP_CAPABLE handshake)? YES
			f. Any other comments or information?
			4. Address management (ADD_ADDR and REMOVE_ADDR) [Section 3.4
			RFC6824]
			a. What is the status of your implementation? REMOVE_ADDR is
			implemented and tested
			b. Can your implementation do ADD_ADDRESS for addresses that appear
			*after* the connection has been established? NO
			c. Any other comments or information? ADD_ADDRESS may not be much
			useful in the real environment situation given that most of the
			clients are behind the NATing devices.
			5. Fast close (MP_FASTCLOSE) [Section 3.5 RFC6824] a. What is the
			status of your implementation? Implemented and tested b. Any other
			comments or information?
			3.4. Question 4: Fallback from MPTCP Question 4 asks about action
			when there is a problem with MPTCP, for example due to a middlebox
			mangling MPTCP's signalling. The connection needs to fall back: if
			the problem is on the first subflow then MPTCP falls back to TCP,
			whilst if the problem is on an additional subflow then that subflow
			is closed with a TCP RST, as discussed in [Section 3.6 RFC6824].
			1. If the MP_CAPABLE option is removed by a middlebox, does your
			implementation fall back to TCP? YES
			2. If the MP_JOIN option does not get through on the SYNs, does your
			implementation close the additional subflow? YES
			3. If the DSS option does not get through on the first data
			segment(s), does your implementation fall back? (either falling back
			to MPTCP (if the issue is on the first subflow) or closing the
			additional subflow (if the issue is on an additional subflow)) YES
			4. Similarly, if the "DATA ACK" field does not correctly acknowledge
			the first data segment(s), does your implementation fall back? If
			the sender receives pure ack for its first DSS packet then it
			fallsback to regular TCP.
			5. Does your implementation protect data with the "Checksum" field
			in the DSS option [Section 3.3 RFC6824]? If the checksum fails
			(because the subflow has been affected by a middlebox), does your
			implementation immediately close the affected subflow (with a TCP
			RST) with the MP_FAIL Option? If the checksum fails and there is a
			single subflow, does your implementation handle this as a special
			case, as described in [Section 3.6 RFC6824]? Yes, our implementation
			supports DSS checksum and will close the subflow with RST if the
			checksum validation fails and there are more than one subflows and
			sends MP_FAIL if there is a single subflow expecting infinite map
			from the peer.
			6. Does your implementation fall back to TCP by using an "infinite
			mapping" [Section 3.3.1 RFC6824] (so that the subflow-level data is
			mapped to the connection-level data for the remainder of the
			connection)? YES.
			7. Did you find any corner cases where MPTCP's fallback didn't
			happen properly? We have found few cases where the draft is not
			clear about the recommended action and fallback strategy, like: 1.
			what is the expected behavior when pure ack or data packet without
			dss is received in middle of transaction? How the hosts should
			fallback in this case? This can happen if the routing changes and
			the new path drops mptcp options. In this case MP_FAIL/infinite map
			exchange may not be possible and so could not decide whether both
			parties are in sync to fallback to tcp. 2. whether infinite map is
			unidirectional or bidirectional? If one host is sending infinite map
			to peer, does the peer also needs to send infinite map to the host?
			Exchanging infinite map and falling back to TCP from both ends is
			easy from implementation point of view. 8. Any other comments or
			information about fallback?
			3.5. Question 5: Heuristics Question 5 gathers information about
			heuristics: aspects that are not required for protocol correctness
			but impact the performance. We would like to document best practice
			so that future implementers can learn from the experience of
			pioneers. The references contain some initial comments about each
			topic.
			1. Receiver considerations [S3.3.4, RFC6824]: What receiver buffer
			have you used? Does this depend on the retransmission strategy?
			What advice should we give about the receiver? Our implementation
			uses varying buffer size based on the services and application type.
			2. Sender considerations [S3.3.5, RFC6824]: How do you determine how
			much data a sender is allowed to send and how big the sender buffer
			is? What advice should we give about the sender? The send side flow
			control is handled at mptcp level and is independent to subflows.
			The mptcp level flow control is (almost) same as the regular TCP flow
			control.
			3. Reliability and retransmissions [S3.3.6, RFC6824]: What is your
			retransmission policy? (when do you retransmit on the original
			subflow vs on another subflow or subflows?) When do you decide that
			a subflow is underperforming and should be reset, and what do you
			then do? What advice should we give about this issue? The
			retransmission is done by the subflows as long as the subflow is
			alive and is not removed by the REM_ADDR/RST/.. . If 3 RTO happens
			on the subflow doing retransmission and multiple subflows are
			available then the mptcp starts retransmission from additional
			subflow. The original subflow continues retransmission for 7RTO and
			will be closed after that with RST.
			4. Port usage [S3.3.8.1, RFC6824]: Does your implementation use the
			same port number for additional subflows as for the first subflow?
			Have you used the ability to define a specific port in the Add
			Address option? What advice should we give about this issue? Our
			current implementation doesnot support ADD_ADDR and subflow
			initiation.
			5. Delayed subflow start [S3.3.8.2, RFC6824]: What factors does your
			implementation consider when deciding about opening additional
			subflows? What advice should we give about this issue? NA
			6. Failure handling [S3.3.8.3, RFC6824]: Whilst the protocol defines
			how to handle some unexpected signals, the behaviour after other
			unexpected signals is not defined. What advice should we give about
			this issue? RFC should clearly define failure case handling
			otherwise it creates interoperability problems among various
			implementations. Our strategy in most of the unexpected failuire
			case is to send MP_FAIL RST with expected DSN if there are multiple
			subflows and MP_FAIL if there is a single subflow expecting infinite
			map from the peer.
			7. Use of TCP options: As discussed in [Appendix A, RFC6824], the
			TCP option space is limited, but a brief study found there was enough
			room to fit all the MPTCP options. However there are constraints on
			which MPTCP option(s) can be included in packets with other TCP
			options - do the suggestions in Appendix A need amending or
			expanding? Looks fine now. Atleast timestamp can be included with
			every dss packet (28bytes for dss and 12bytes for Timestamp), but if
			there are any other options which needs to be included in data
			packets then the implementation has to choose which one to include
			among them.
			8. What other heuristics should we give advice about? Any other
			comments or information?
			3.6. Question 6: Security Question 6 asks about Security related
			matters [Section 5 RFC6824].
			1. Does your implementation use the hash-based, HMAC-SHA1 security
			mechanism defined in [RFC6824]? YES.
			2. Does your implementation support any other handshake algorithms?
			NO.
			3. It has been suggested that a Standards-track MPTCP needs a more
			secure mechanism. Do you have any views about how to achieve this?
			Yes we also feel more secure and light weight mechanism is required.
			4. Any other comments or information?
			3.7. Question 7: IANA Question 7 asks about IANA related matters.
			1. Does your implementation follow the IANA-related definitions?
			[Section 8 RFC6824] defines: TCP Option Kind number (30); the sub-
			registry for "MPTCP Option Subtypes"; and the sub-registry for "MPTCP
			Handshake Algorithms" YES. 2. Any other comments or information?
			3.8. Question 8: Congestion control and subflow policy Question 8
			asks about how you share traffic across multiple subflows.
			1. How does your implementation share traffic over the available
			paths? For example: as a spare path on standby ('all-or- nothing'),
			as an 'overflow', etc? Does it have the ability to send /receive
			traffic across multiple subflows simultaneously? We give preference
			to the path that client is currently using to send data/ack and also
			has policy based on primary/backup setup. We accept data from
			multiple subflows simultaneously but don't send it simultaneously
			out.
			2. Does your implementation support "handover" from one subflow to
			another when losing an interface? YES.
			3. Does your implementation support the coupled congestion control
			defined in [RFC6356]? NO.
			4. Does your implementation support some other coupled congestion
			control (ie that balances traffic on multiple paths according to
			feedback)? NO.
			5. The MP_JOIN (Starting a new subflow) Option includes the "B" bit,
			which allows the sender to indicate whether it wishes the new subflow
			to be used immediately or as a backup if other path(s) fail. The
			MP_PRIO Option is a request to change the "B" bit - either on the
			subflow on which it is sent, or (by setting the optional Address ID
			field) on other subflows. Does your implementation support the "B"
			bit and MP_PRIO mechanisms? Do you think they're useful, or have
			another suggestion? YES, our implementation supports both 'B' flag
			and MP_PRIO options, they are much useful to change the priority of
			the subflows and to decide which subflow to use for data transfer.
			6. Any other comments or information or suggestions about the advice
			we should give about congestion control [S3.3.7 RFC6824] and subflow
			policy [S3.3.8 RFC6824]?
			3.9. Question 9: API Question 9 gathers information about your API.
			[RFC6897] considers the MPTCP Application Interface.
			1. With your implementation, can legacy applications use (the
			existing sockets API to use) MPTCP? How does the implementation
			decide whether to use MPTCP? Should the advice in [Section 4,
			RFC6897] be modified or expanded? NA.
			2. The "basic MPTCP API" enables MPTCP-aware applications to
			interact with the MPTCP stack via five new socket options. For each
			one, have you implemented it? has it been useful? a.
			TCP_MULTIPATH_ENABLE? b. TCP_MULTIPATH_ADD? c.
			TCP_MULTIPATH_REMOVE? d. TCP_MULTIPATH_SUBFLOWS? e.
			TCP_MULTIPATH_CONNID? NA.
			3. Have you implemented any aspects of an "advanced MPTCP API"?
			([Appendix A, RFC6897] hints at what it might include.) NA. 4. Any
			other comments or information?
			3.10. Question 10: Deployments, use cases and operational
			experiences Question 10 takes the opportunity of this survey to
			gather some limited information about operational experiences and
			deployments. Any very brief information would be appreciated, for
			example:
			1. What deployment scenarios are you most interested in? MPTCP
			Proxy deployment where the mptcp connections from the clients are
			terminated and the tcp connection is established on the other side.
			2. Is your deployment on "the Internet" or in a controlled
			environment? Targeted for the Internet deployment.
			3. Is your deployment on end hosts or with a MPTCP-enabled proxy (at
			one or both ends?)? Proxy.
			4. What do you see as the most important benefits of MPTCP in your
			scenario(s)? Reliability and fault tolerance.
			5. How extensively have you deployed and experimented with MPTCP so
			far?
			6. MPTCP's design seeks to maximise the chances that the signalling
			works through middleboxes. Did you find cases where middleboxes
			blocked MPTCP signalling? Yes some firewalls seem dropping MPTCP
			options.
			7. MPTCP's design seeks to ensure that, if there is a problem with
			MPTCP signalling, then the connection either falls back to TCP or
			removes the problematic subflow. Did you find any corner cases where
			this didn't happen properly? Few cases listed above.
			8. Have you encountered any issues or drawbacks with MPTCP? 9. Any
			other comments or information?
			3.11. Question 11: Improvements to RFC6824
			1. Are there any areas where [RFC6824] could be improved, either in
			technical content or clarity? More clarity required in fallback
			cases.
			2. Any other issues you want to raise?
			11. Normative References
	[RFC6356] Raiciu, C., Handley, M., and D. Wischik, "Coupled		[RFC6356] Raiciu, C., Handley, M., and D. Wischik, "Coupled
	Congestion Control for Multipath Transport Protocols", RFC		Congestion Control for Multipath Transport Protocols",
	6356, October 2011.		RFC 6356, October 2011.
	[RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure,		[RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure,
	"TCP Extensions for Multipath Operation with Multiple		"TCP Extensions for Multipath Operation with Multiple
	Addresses", RFC 6824, January 2013.		Addresses", RFC 6824, January 2013.
	Author's Address		Author's Address
	Philip Eardley		Philip Eardley
	BT		BT
End of changes. 111 change blocks.
	250 lines changed or deleted		1547 lines changed or added
This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/