Internet-Draft Multipath sequence maintenance July 2020
Amend & von Hugo Expires 8 January 2021 [Page]
ICCRG Working Group
Intended Status:
M. Amend
D. von Hugo

Multipath sequence maintenance


This document discusses the issue of packet re-ordering which occurs as a specific problem in multi-path connections without reliable transport protocols such as TCP. The topic is relevant for devices connected via multiple accesses technologies towards the network as is foreseen e.g. within Access Traffic Selection, Switching, and Splitting (ATSSS) service of 3rd Generation Partnership Project (3GPP) enabling fixed mobile converged (FMC) scenario.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 8 January 2021.

Table of Contents

1. Introduction

Mobile end user devices nowadays are mostly equipped with multiple network interfaces allowing to connect to more than one network at a time and thus increase data throughput, reliability, coverage and so on. Ideally the user data stream originating from the application at the device is split between the available (here: N) paths at the sender side and re-assembled at an intermediate aggregation node before transmitted to the corresponding host in the network as depicted in Figure 1.

                  /            \
       +---------| Access Net 1 |--+
       |         \             /   |
       |          -------------    |
       |          ------------     |
       |         /            \    |
       | +------| Access Net 2 |-+ |
       | |       \            /  | |
       | |        ------------   | |
       | |                       | |
       | |                       | |
       | |               +-------+-+---+
    +--+-+-+             |             |   +-----+        +------+
    |End   |             | Aggregation +---|Gate-|-/.../--| Host |
    |User  |             |    Node     |   |way  |        +------+
    |Device|             |             |   +-----+
    +--+-+-+             +-------+-+---+
       | |                       | |
       | |     --------------    | |
       | |    /              \   | |
       | +---| Access Net N-1 |--+ |
       |      \              /     |
       |       --------------      |
       |                           |
       |          ------------     |
       |         /            \    |
       +--------| Access Net N |---+
                 \            /
Figure 1: Reference Architecture for multi-path re-ordering

However, when several paths are utilized concurrently to transmit user data between the sender and the receiver, different characteristics of the paths in terms of bandwidth, delay, or error proneness can impact the overall performance due to delayed packet arrival and need for re-transmit in case of lost packets. Without further arrangements the original order of packets at the sending UE side is no longer maintained at the receiving host and a re-ordering or re-arrangement has to occur before delivery to the application at the far end site. This can be performed at earliest at the aggregation node with a minimum additional delay due to re-transmission requests or at latest either by the application on the host itself or the transmission protocol.

It is a goal of the present document to collect and describe mechanisms to maintain the sequence of split traffic over multiple paths. These mechanisms are generic and not dedicated to a specific multipath network protocol, but give clear guidance on requirements and benefits to maintainers of multipath network protocols.

2. State of the Art

Regular TCP protocol [RFC0793] offers such mechanism with queues for in-order and out-of order (including damaged, lost, duplicated) arrival of packets.

This is also provided by MPTCP [RFC6824] as the first and successful Multipath protocol which however also requires new methods as sequence numbers both on (whole) data (stream) and subflow level to ensure in-order delivery to the application layer on the receiver side [RFC8684]. Moreover, careful design of buffer sizes and interpretation of sequence numbers to distinguish between (delayed) out-of-order packets and completely lost ones has to be considered.

[I-D.bonaventure-iccrg-schedulers] already reflects on proper packet scheduling schemes (at the sender side) to reduce the effort for re-assembly or even make such (time consuming) treatment unnecessary.

MP-QUIC [I-D.deconinck-quic-multipath] introduces the concept of uniflows with own IDs claiming to get rid of additional sequence numbers for re-ordering as required in Multipath TCP [RFC6824]. Although [] admits that statistical performance information should help a host in deciding on optimum packet scheduling and flow control a dedicated packet scheduling policy is out of scope of that document. A further improvement versus MPTCP can be achieved by decoupling paths used for data transmission from those for sending acknowledgments (ACKs) or claiming for re-transmission by NACKs to not introduce further latency.

[I-D.ietf-quic-recovery] specifies algorithms for QUIC Loss Detection and Congestion Control by using measurement of Round Trip Time (RTT) to determine when packets should be retransmitted. Draft [I-D.huitema-quic-ts] proposes to enable one way delay (1WD) measurements in QUIC by defining a TIME_STAMP frame to carry the time at which a packet is sent and combine the ACKs sent with a timestamp field and thus allow for more precise estimation of the (one-way) delay of each uniflow, assisting proper scheduling decisions.

Also other protocols as Multi-Access Management Services (MAMS) [RFC8743] consider the need for re-ordering on User Plane level which may be done at network and client level by introducing a new Multi-Access (MX) Convergence Layer. [I-D.zhu-intarea-mams-user-protocol] introduces accordingly Traffic Splitting Update (TSU) messages and Packet Loss Report (PLR) messages including beside others Traffic Splitting Parameters and an expected next (in-order) sequence number, respectively.

[I-D.zhu-intarea-gma] on Generic Multi-Access (GMA) Convergence Encapsulation Protocols introduces a trailer-based encapsulation which carries one or multiple IP packets or fragments thereof in a Protocol Data Unit (PDU). At receiver side PDUs with identical Sequence Numbers (in the trailer) are to be placed in the relative order indicated by a so-called Fragment Offset.

3. Problem Statement

Assuming for simplicity the minimum multipath scenario with two separate paths for transmission of a flow of packets with sequence numbers (SN) SN1 ... SM. In case the scheduling of packets is done equally to both paths and path 2 exhibits a delay of the duration of transmission time required for e.g. two packets (assuming fixed packet size and same constant data for both paths) for an exemplary App-originated sequence of packets as SN1 SN2 SN3 SN4 SN5 SN6 SN7 SN8 ... the resulting sequence of packets could look as depicted in Figure 2 which of course depends on the queue processing and buffering at the Aggregation Proxy.

APP              UE              Aggregation Proxy               Host
 |  SN1 ... SN8  |                          |                       |
 |-------------->| path 1 SN1 SN3 SN5 SN7...|                       |
 |               |------------------------->|                       |
 |               | path 2 SN2 SN4 SN6 SN8...|                       |
 |               |------------------------->|                       |
 |               |                          |SN1 SN3 SN2 SN5 SN4 SN7|
 |               |                          |======================>|
 |               |                          |                       |
Figure 2: Exemplary data transmission for a dual-path scenario

In such a case re-ordering at the Aggregation Proxy would be simple and straight forward. It even could be avoided if the scheduling would already take the expected different delays into account (e.g. by pre-delaying the traffic on path 1 thus of course not leveraging the lower delay). Different from this simplistic scenario in general the data rate on both paths will vary in time and be not equal, also different and variable latency (jitter) per path will be introduced and in addition loss of packets as well as potential duplication may occur making the situation much more complicated. In case of loss detection after a threshold waiting time a retransmission could be initiated by the Host or if possible by the Proxy. Alternatively the UE could send redundant packets in advance coded in such a way that it allows for derivation of e.g. one lost packet per M correctly received ones or by a (real-time) application able to survive single lost packets.

Holding multiple queues and a large enough buffer both at UE and at the Aggregation Proxy would be required to apply proper scheduling at UE and reordering during re-assembly at Aggregation Proxy to mitigate the sketched impact of multiple paths variable characteristics in terms of transmission performance.


4. Scheduling mechanisms

Scheduling mechanisms decides on sender side how traffic is distributed over the paths of a multipath-setup. [I-D.bonaventure-iccrg-schedulers] gives an overview of possible distribution schemes. For this document it is assumed, that schedulers are used, which simultaneously distributes traffic over more than one path and latency difference(s) exists between those multiple paths.

5. Resequencing mechanisms

Resequencing mechanism are responsible to modify the sequence of received data split over multiple paths according to a sequencing scheme. The degree of resequencing can reach from now measure up to re-generating the exact order.

Typically at least one sequencing scheme describing the order of how data was generated on sender side and is referred to as "overall sequencing". Under certain circumstances an additional sequencing scheme per path of the multi-path setup can be leveraged to optimize packet loss detection. For most multipath protocols both sequencing schemes are already available. Packet loss detection becomes important when multipath protocols are applied which does not guarantee successful transmission. For example [I-D.amend-tsvwg-multipath-dccp] or the combination of [I-D.deconinck-quic-multipath] and [I-D.ietf-quic-datagram] are unreliable in that sense.

For simplicity all the mechanism described in the following are explained based on two paths but work with an unlimited number though.

5.1. Passive

Nothing is done.

5.2. Exact

Exact order is re-generated. Only useful for network protocols which re-transmits. Risk: Head-of-Line blocking

5.3. Static Expiration

Packet gap is assumed as packet loss after certain time threshold

5.4. Adaptive Expiration

Packet gap is assumed as packet loss after dynamic time threshold derived from the latency differences between paths.

5.5. Delay Equalization

Delay data on the faster path by the latency difference to the slower path. Strictly spoken no resequencing based on sequencing information.

5.6. Fast Packet Loss Detection

5.6.1. Using overall sequencing

Compare the overall sequence number arriving. When on all path a higher number is received than the one which is waited for, packet loss can is given.

5.6.2. Using per-path sequencing

For environments where no per-path scrambling is given. Compare distance between path sequencing and overall sequencing. When mismatch then packet loss. Requires at least three packets in-order on a path to work to identify loss of the middle packet.

6. Recovery mechanisms

Recovering packets, in particular lost packets or assumed lost packets on receiver side avoids re-transmission and potentially mitigates the resequencing process in respect to detecting packet loss. Shorter latencies will be an expected outcome. Discussing the complexity, computation overhead and reachable benefit is subject of this section.

6.1. FEC

Introduce redundancy to reconstruct data. Are there two possibilities: Per-path FEC & overall FEC?

6.2. Network Coding

Linear network coding

7. Retransmission mechanisms

Re-transmission becomes interesting when it can help to reduce the time spent on waiting for outstanding packets for re-sequencing. In particular scenarios when the RTT lets expect a shorter time to re-transmit than wait for packet loss detection, a likely scenario in e.g. Figure 1. It could also avoid a potential late triggering of re-transmission by the end-to-end service.

7.1. Fast re-transmission

Signal to the sender to re-transmit a missing packet.

8. Informative References

Amend, M., Bogenfeld, E., Brunstrom, A., Kassler, A., and V. Rakocevic, "DCCP Extensions for Multipath Operation with Multiple Addresses", Work in Progress, Internet-Draft, draft-amend-tsvwg-multipath-dccp-03, , <>.
Bonaventure, O., Piraux, M., Coninck, Q., Baerts, M., Paasch, C., and M. Amend, "Multipath schedulers", Work in Progress, Internet-Draft, draft-bonaventure-iccrg-schedulers-00, , <>.
Coninck, Q. and O. Bonaventure, "Multipath Extensions for QUIC (MP-QUIC)", Work in Progress, Internet-Draft, draft-deconinck-quic-multipath-04, , <>.
Huitema, C., "Quic Timestamps For Measuring One-Way Delays", Work in Progress, Internet-Draft, draft-huitema-quic-ts-02, , <>.
Pauly, T., Kinnear, E., and D. Schinazi, "An Unreliable Datagram Extension to QUIC", Work in Progress, Internet-Draft, draft-ietf-quic-datagram-00, , <>.
Iyengar, J. and I. Swett, "QUIC Loss Detection and Congestion Control", Work in Progress, Internet-Draft, draft-ietf-quic-recovery-29, , <>.
Zhu, J. and S. Kanugovi, "Generic Multi-Access (GMA) Encapsulation Protocol", Work in Progress, Internet-Draft, draft-zhu-intarea-gma-07, , <>.
Zhu, J., Seo, S., Kanugovi, S., and S. Peng, "User-Plane Protocols for Multiple Access Management Service", Work in Progress, Internet-Draft, draft-zhu-intarea-mams-user-protocol-09, , <>.
Postel, J., "Transmission Control Protocol", STD 7, RFC 793, DOI 10.17487/RFC0793, , <>.
Ford, A., Raiciu, C., Handley, M., and O. Bonaventure, "TCP Extensions for Multipath Operation with Multiple Addresses", RFC 6824, DOI 10.17487/RFC6824, , <>.
Ford, A., Raiciu, C., Handley, M., Bonaventure, O., and C. Paasch, "TCP Extensions for Multipath Operation with Multiple Addresses", RFC 8684, DOI 10.17487/RFC8684, , <>.
Kanugovi, S., Baboescu, F., Zhu, J., and S. Seo, "Multiple Access Management Services Multi-Access Management Services (MAMS)", RFC 8743, DOI 10.17487/RFC8743, , <>.

Authors' Addresses

Markus Amend
Deutsche Telekom
Deutsche-Telekom-Allee 9
64295 Darmstadt
Dirk von Hugo
Deutsche Telekom
Deutsche-Telekom-Allee 9
64295 Darmstadt