idnits 2.17.1 draft-amend-iccrg-multipath-reordering-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (25 October 2021) is 908 days in the past. Is this intentional? Checking references for intended status: Experimental ---------------------------------------------------------------------------- == Outdated reference: A later version (-08) exists of draft-huitema-quic-ts-06 == Outdated reference: A later version (-10) exists of draft-ietf-quic-datagram-06 == Outdated reference: A later version (-14) exists of draft-zhu-intarea-gma-12 -- Obsolete informational reference (is this intentional?): RFC 793 (Obsoleted by RFC 9293) Summary: 1 error (**), 0 flaws (~~), 4 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 ICCRG Working Group M. Amend 3 Internet-Draft D. von Hugo 4 Intended status: Experimental DT 5 Expires: 28 April 2022 25 October 2021 7 Multipath sequence maintenance 8 draft-amend-iccrg-multipath-reordering-03 10 Abstract 12 This document discusses the issue of packet reordering which occurs 13 as a specific problem in multi-path connections without reliable 14 transport protocols such as TCP. The topic is relevant for devices 15 connected via multiple accesses technologies towards the network as 16 is foreseen, e.g., within Access Traffic Selection, Switching, and 17 Splitting (ATSSS) service of 3rd Generation Partnership Project 18 (3GPP) enabling fixed mobile converged (FMC) scenario. 20 Status of This Memo 22 This Internet-Draft is submitted in full conformance with the 23 provisions of BCP 78 and BCP 79. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF). Note that other groups may also distribute 27 working documents as Internet-Drafts. The list of current Internet- 28 Drafts is at https://datatracker.ietf.org/drafts/current/. 30 Internet-Drafts are draft documents valid for a maximum of six months 31 and may be updated, replaced, or obsoleted by other documents at any 32 time. It is inappropriate to use Internet-Drafts as reference 33 material or to cite them other than as "work in progress." 35 This Internet-Draft will expire on 28 April 2022. 37 Copyright Notice 39 Copyright (c) 2021 IETF Trust and the persons identified as the 40 document authors. All rights reserved. 42 This document is subject to BCP 78 and the IETF Trust's Legal 43 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 44 license-info) in effect on the date of publication of this document. 45 Please review these documents carefully, as they describe your rights 46 and restrictions with respect to this document. Code Components 47 extracted from this document must include Simplified BSD License text 48 as described in Section 4.e of the Trust Legal Provisions and are 49 provided without warranty as described in the Simplified BSD License. 51 Table of Contents 53 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 54 2. State of the Art . . . . . . . . . . . . . . . . . . . . . . 4 55 3. Problem Statement . . . . . . . . . . . . . . . . . . . . . . 5 56 4. Scheduling mechanisms . . . . . . . . . . . . . . . . . . . . 6 57 5. Resequencing mechanisms . . . . . . . . . . . . . . . . . . . 7 58 5.1. Passive . . . . . . . . . . . . . . . . . . . . . . . . . 7 59 5.2. Exact . . . . . . . . . . . . . . . . . . . . . . . . . . 8 60 5.3. Static Expiration . . . . . . . . . . . . . . . . . . . . 8 61 5.4. Adaptive Expiration . . . . . . . . . . . . . . . . . . . 8 62 5.5. Delay Equalization . . . . . . . . . . . . . . . . . . . 8 63 5.6. Fast Packet Loss Detection . . . . . . . . . . . . . . . 9 64 5.6.1. Connection sequencing . . . . . . . . . . . . . . . . 9 65 5.6.2. Per-path sequencing . . . . . . . . . . . . . . . . . 9 66 5.6.3. Combination connection and per-path sequencing . . . 10 67 6. Recovery mechanisms . . . . . . . . . . . . . . . . . . . . . 10 68 6.1. FEC (Forward Error Correction) . . . . . . . . . . . . . 10 69 6.2. Network Coding . . . . . . . . . . . . . . . . . . . . . 11 70 7. Retransmission mechanisms . . . . . . . . . . . . . . . . . . 11 71 7.1. Signaling . . . . . . . . . . . . . . . . . . . . . . . . 11 72 7.2. Anticipated . . . . . . . . . . . . . . . . . . . . . . . 12 73 7.3. Flow-selection . . . . . . . . . . . . . . . . . . . . . 12 74 7.4. Other re-transmission issues . . . . . . . . . . . . . . 12 75 8. Security Considerations . . . . . . . . . . . . . . . . . . . 13 76 9. Informative References . . . . . . . . . . . . . . . . . . . 13 77 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 15 79 1. Introduction 81 Mobile end user devices nowadays are mostly equipped with multiple 82 network interfaces allowing to connect to more than one network at a 83 time and thus increase data throughput, reliability, coverage, and so 84 on. Ideally the user data stream originating from the application at 85 the device is split between the available (here: N) paths at the 86 sender side and re-assembled at an intermediate aggregation node 87 before transmitted to the corresponding host in the network as 88 depicted in Figure 1. 90 ------------ 91 / \ 92 +---------| Access Net 1 |--+ 93 | \ / | 94 | ------------- | 95 | ------------ | 96 | / \ | 97 | +------| Access Net 2 |-+ | 98 | | \ / | | 99 | | ------------ | | 100 | | | | 101 | | | | 102 | | +-------+-+---+ 103 +--+-+-+ | | +------+ 104 |End | | Aggregation +----/.../--| Host | 105 |User | | Node | +------+ 106 |Device| | | 107 +--+-+-+ +-------+-+---+ 108 | | | | 109 | | -------------- | | 110 | | / \ | | 111 | +---| Access Net N-1 |--+ | 112 | \ / | 113 | -------------- | 114 | | 115 | ------------ | 116 | / \ | 117 +--------| Access Net N |---+ 118 \ / 119 ------------ 121 Figure 1: Reference Architecture for multi-path reordering 123 However, when several paths are utilized concurrently to transmit 124 user data between the sender and the receiver, different 125 characteristics of the paths in terms of bandwidth, delay, or error 126 proneness can impact the overall performance due to delayed packet 127 arrival and need for re-transmit in case of lost packets. Without 128 further arrangements the original order of packets at the sending UE 129 side is no longer maintained at the receiving host and a reordering 130 or re-arrangement has to occur before delivery to the application at 131 the far end site. This can be performed at earliest at the 132 aggregation node with a minimum additional delay due to re- 133 transmission requests or at latest either by the application on the 134 host itself or the transmission protocol. 136 It is a goal of the present document to collect and describe 137 mechanisms to maintain the sequence of split traffic over multiple 138 paths. These mechanisms are generic and not dedicated to a specific 139 multipath network protocol, but give clear guidance on requirements 140 and benefits to maintainers of multipath network protocols. 142 2. State of the Art 144 Regular TCP protocol [RFC0793] offers such mechanism with queues for 145 in-order and out-of order (including damaged, lost, duplicated) 146 arrival of packets. 148 This is also provided by MPTCP [RFC8684] as the first and successful 149 Multipath protocol which however also requires new methods as 150 sequence numbers both on (whole) data (stream) and subflow level to 151 ensure in-order delivery to the application layer on the receiver 152 side [RFC8684]. Moreover, careful design of buffer sizes and 153 interpretation of sequence numbers to distinguish between (delayed) 154 out-of-order packets and completely lost ones has to be considered. 156 [I-D.bonaventure-iccrg-schedulers] already reflects on proper packet 157 scheduling schemes (at the sender side) to reduce the effort for re- 158 assembly or even make such (time consuming) treatment unnecessary. 160 MP-QUIC [I-D.deconinck-quic-multipath] introduces the concept of 161 uniflows with own IDs claiming to get rid of additional sequence 162 numbers for reordering as required in Multipath TCP [RFC8684]. 163 Although [I-D.liu-multipath-quic] admits that statistical performance 164 information should help a host in deciding on optimum packet 165 scheduling and flow control a dedicated packet scheduling policy is 166 out of scope of that document. A further improvement versus MPTCP 167 can be achieved by decoupling paths used for data transmission from 168 those for sending acknowledgments (ACKs) or claiming for re- 169 transmission by NACKs to not introduce further latency. 171 [I-D.ietf-quic-recovery] specifies algorithms for QUIC Loss Detection 172 and Congestion Control by using measurement of Round Trip Time (RTT) 173 to determine when packets should be retransmitted. Draft 174 [I-D.huitema-quic-ts] proposes to enable one way delay (1WD) 175 measurements in QUIC by defining a TIME_STAMP frame to carry the time 176 at which a packet is sent and combine the ACKs sent with a timestamp 177 field and thus allow for more precise estimation of the (one-way) 178 delay of each uniflow, assisting proper scheduling decisions. 180 Also other protocols as Multi-Access Management Services (MAMS) 181 [RFC8743] consider the need for reordering on User Plane level which 182 may be done at network and client level by introducing a new Multi- 183 Access (MX) Convergence Layer. [I-D.zhu-intarea-mams-user-protocol] 184 introduces accordingly Traffic Splitting Update (TSU) messages and 185 Packet Loss Report (PLR) messages including beside others Traffic 186 Splitting Parameters and an expected next (in-order) sequence number, 187 respectively. 189 [I-D.zhu-intarea-gma] on Generic Multi-Access (GMA) Convergence 190 Encapsulation Protocols introduces a trailer-based encapsulation 191 which carries one or multiple IP packets or fragments thereof in a 192 Protocol Data Unit (PDU). At receiver side PDUs with identical 193 Sequence Numbers (in the trailer) are to be placed in the relative 194 order indicated by a so-called Fragment Offset. 196 3. Problem Statement 198 Assuming for simplicity the minimum multipath scenario with two 199 separate paths for transmission of a flow of packets with sequence 200 numbers (SN) SN1 ... SM. In case the scheduling of packets is done 201 equally to both paths and path 2 exhibits a delay of the duration of 202 transmission time required for, e.g., two packets (assuming fixed 203 packet size and same constant data for both paths) for an exemplary 204 App-originated sequence of packets as SN1 SN2 SN3 SN4 SN5 SN6 SN7 SN8 205 ... the resulting sequence of packets could look as depicted in 206 Figure 2 which of course depends on the queue processing and 207 buffering at the Aggregation Proxy. 209 APP UE Aggregation Node Host 210 | SN1 ... SN8 | | | 211 |-------------->|path 1 SN1 SN3 SN5 SN7...| | 212 | |------------------------>| | 213 | |path 2 SN2 SN4 SN6 SN8...| | 214 | |------------------------>| | 215 | | |SN1 SN3 SN2 SN5 SN4 SN7| 216 | | |======================>| 217 | | | | 219 Figure 2: Exemplary data transmission for a dual-path scenario 221 In such a case reordering at the Aggregation Node would be simple and 222 straight forward. It even could be avoided if the scheduling would 223 already take the expected different delays into account (e.g. by pre- 224 delaying the traffic on path 1 thus of course not leveraging the 225 lower delay). Different from this simplistic scenario in general the 226 data rate on both paths will vary in time and be not equal, also 227 different and variable latency (jitter) per path will be introduced 228 and in addition loss of packets as well as potential duplication may 229 occur making the situation much more complicated. In case of loss 230 detection after a threshold waiting time a retransmission could be 231 initiated by the Host or if possible already by the Aggregation Node. 233 Alternatively the UE could send redundant packets in advance coded in 234 such a way that it allows for derivation of, e.g., one lost packet 235 per M correctly received ones or by a (real-time) application able to 236 survive singular lost packets. 238 Holding multiple queues and a large enough buffer both at UE and at 239 the Aggregation Node would be required to apply proper scheduling at 240 UE and reordering during re-assembly at Aggregation Node to mitigate 241 the sketched impact of multiple paths' variable characteristics in 242 terms of transmission performance. 244 ... 246 4. Scheduling mechanisms 248 Scheduling mechanisms decide on sender side how traffic is 249 distributed over the paths of a multipath-setup. 250 [I-D.bonaventure-iccrg-schedulers] gives an overview of possible 251 distribution schemes. For this document it is assumed, that 252 schedulers are used, which simultaneously distribute traffic over 253 more than one path, whereas path characteristics differ between those 254 multiple paths (e.g. a latency difference exists). While on the one 255 hand, the traffic scheduling causes out-of-order multipath delivery 256 when simultaneously utilize heterogeneous paths, it can also be used 257 to mitigate this problem. Pre-delaying data on a fast path, 258 according to the latency difference of the slowest path is aimed, 259 e.g., by OTIAS [OTIAS], DAPS [DAPS], and BLEST [BLEST]. However, the 260 success is much dependent on the accuracy of path information like 261 path latency, throughput, and packet loss rate. In heterogeneous and 262 volatile environments most often such information have to be 263 estimated, e.g., using congestion control. That means, it takes at 264 least one RTT to gain first indications and probably several RTTs to 265 converge to a worthwhile accuracy. Changes of path characteristics 266 in sub-RTT time frames put such a system to test. Dependent on the 267 demand on in-order delivery and/or the accuracy of the relevant path 268 information, scheduling might be an exclusive alternative or can be 269 applied in conjunction with other discussed mechanisms in this 270 document. 272 [AOPS] proposes to use a predictive Adaptive Order Prediction 273 Scheduling (AOPS) mechanism considering both the anticipated time of 274 packet delivery and the reliability of each path to optimize the 275 traffic scheduling for MP-DCCP, thus coping with reordering and 276 achieving in-order delivery. 278 Scheduling will not help to overcome any degree of out-of-order 279 delivery, when the scheduling goal is different to this. For example 280 a strict cost prioritization of Wi-Fi over cellular access in a 281 mobile phone might be assumed counterproductive. 283 5. Resequencing mechanisms 285 Resequencing mechanisms are responsible to modify the sequence of 286 received data split over multiple paths according to a sequencing 287 scheme. The degree of resequencing can reach from no measure up to 288 re-generating the exact order. 290 Typically at least one sequencing scheme, describing the order of how 291 data was generated on sender side is prerequisite. This is referred 292 to as "connection sequencing". Under certain circumstances an 293 additional sequencing scheme per path of the multi-path setup can be 294 leveraged, to optimize packet loss detection and is further 295 elaborated in Section 5.6. For most multipath protocols both 296 sequencing schemes are already available. Packet loss detection 297 becomes important when multipath protocols are applied which do not 298 guarantee successful transmission as TCP achieves by acknowledgement 299 of successful reception. For example, 300 [I-D.amend-tsvwg-multipath-dccp] or the combination of 301 [I-D.deconinck-quic-multipath] and [I-D.ietf-quic-datagram] are 302 unreliable protocols in that sense. 304 For simplicity all the mechanism described in the following are 305 explained based on two paths but in principle would work with any 306 other amount though. 308 5.1. Passive 310 This approach includes no active change or reordering at the 311 transport level and purely re-combines the packet flows incoming from 312 both paths as is. All modification of the resulting sequence of 313 packets is left to the application at the end node. Here no 314 processing delay is added due to the resequencing but since no early 315 packet loss detection with subsequent re-transmission request on 316 transport level is possible the risk of a larger delay due to late 317 loss detection at the application will arise in case of lossy 318 connections. 320 5.2. Exact 322 This approach covers all mechanisms which attempt to re-generate the 323 original order of packets in the flow exactly, independent of the 324 expected or resulting delay due to waiting time for all packets on 325 all paths to arrive. In case of unreliable transport protocols this 326 may result in a large delay due to Head-of-Line blocking and for 327 actual packet loss in a remaining packet gap which causes a stand 328 still without an option to recover. For applications demanding near 329 real-time delivery of packets it should not be applied. 331 5.3. Static Expiration 333 This method to detect and decide on packet loss assumes a certain 334 fixed time threshold for the gap between packets within a sequence 335 after re-combination of both paths. A possible re-transmission - 336 either in the multipath system internally or based on the piggybacked 337 protocol/service - will possibly not be requested before this 338 threshold is exceeded. Thus an additional delay in the overall 339 latency budget will occur so that this simple approach is only 340 recommended for non-time critical applications. Every packet loss or 341 simultaneous transmission of data over the short and long latency 342 path will cause spikes in the service perceived latency. 344 5.4. Adaptive Expiration 346 Here the packet gap is assumed as packet loss after exceeding a 347 flexibly decided on time threshold which may be derived dynamically 348 from the differences between latencies both paths exhibit. As the 349 latency may vary due to propagation conditions or routing paths this 350 latency difference has to be monitored and statistically evaluated 351 (smoothed) which introduces additional effort. A possible solution 352 for this is the determination of the the one way latency as described 353 in [I-D.song-mptcp-owl] or sending available RTT information from the 354 sender from which the receiver can calculate the latency difference. 356 5.5. Delay Equalization 358 This is an ordering mechanism which delays data forwarding on the 359 faster path by the latency difference to the slower path. Ideally 360 the resequencing effort on the aggregated packet flow can be greatly 361 reduced up to no resequencing at all. Due to time variation in path 362 delays (jitter) and delay differences and the required time for 363 decision and feedback on the delay, some re-sequencing still remains 364 to be executed. Similar to Section 5.4, explicit knowledge of the 365 latency difference is required. Strictly speaking this method allows 366 to avoid resequencing based on sequencing information. However, the 367 overall delay may be larger since the advantage of the short-delay 368 path is not exploited. In combination with Section 5.3 or 369 Section 5.4 resequencing can be added with a presumably lower 370 resequencing effort to scenarios without delay equalization. The 371 essence is a in-order stream with a unified latency across the 372 multiple paths. 374 5.6. Fast Packet Loss Detection 376 The following sections describe methods to achieve unambiguous 377 detection of packet loss independent from thresholds in Section 5.3 378 or Section 5.4. Furthermore, packet loss can be differentiated from 379 delayed delivery. The benefit is a much faster decision plane based 380 on monitoring the sequence space of consecutive packets. For that, 381 the sequencing coming along with the receiver based re-sequencing is 382 further leveraged. Two sequencing schemes are considered here, the 383 connection and the per-path sequencing. 385 5.6.1. Connection sequencing 387 Connection sequencing marks the outgoing packets in the order they 388 enter the multipath system and is independent from a particular 389 selected path for transmission. After arrival at the aggregation 390 node the lowest packet sequence number at each of the multiple paths 391 is compared the that of the last correctly received packet. When the 392 numbers are not consecutive (i.e., when on all paths a higher number 393 is received than the next expected in-order packet), an overall 394 packet loss is detected. While only a single comparison of packet 395 numbers has to be performed and the out-of-order arrival on a single 396 path can be partly compensated this scheme does not allow for 397 immediate detection of where reordering happens. 399 5.6.2. Per-path sequencing 401 Per-path sequencing is a path inherent sequencing mechanism valid in 402 the particular path domain only. In this case the packets are marked 403 by path-specific sequence numbers at the sender side and at each 404 interface of the aggregation node the sequence numbers of arriving 405 packets are compared on per-path level. When a higher sequence 406 number is received than the one which is waited for (next expected 407 in-order packet), a packet loss for this specific path is declared. 408 This may prevent partly misinterpretation of out-of-order arrival as 409 packet loss and allow for path specific countermeasures towards 410 overall performance improvement, as, e.g., chosing a more robust 411 transmission technique for this path. 413 5.6.3. Combination connection and per-path sequencing 415 While the benefits from the individual sequencing schemes above can 416 be combined, a further benefit crystallizes. Since the out-of-order 417 arrival is detected on per-path basis, the path specific out-of-order 418 delivery rate can be used as a criterion to choose repair parameters 419 on a per-path basis (which thus may work more efficiently). In 420 addition the decision on the path selection and weighting can be made 421 based on this criterion. Thus an improved overall performance can be 422 achieved in this case. [to be checked/continued...] 424 6. Recovery mechanisms 426 Recovering packets, in particular lost packets or assumed lost 427 packets on receiver side avoids re-transmission and potentially 428 mitigates the resequencing process in respect to detecting packet 429 loss. Shorter latencies will be an expected outcome. Discussing the 430 complexity, computation overhead and reachable benefit is subject of 431 this section. 433 6.1. FEC (Forward Error Correction) 435 This approach is based on introduction of redundancy to user data to 436 detect errors and subsequently reconstruct data in case of a limited 437 number of bit or Byte errors. As such packet with corrupted data can 438 be recovered up to a certain degree but in case of a too high bit 439 error rate (BER) a packet is completely lost. However, in 440 combination with scrambling, i.e. the sequence of original data 441 stream is distributed over multiple packets and re-compiled 442 afterwards also data from lost packets could be recovered. As such 443 methods introduce additional delay and overhead it is mainly applied 444 in case of long re-transmission delays as, e.g., is typical for 445 satellite transmission. FEC can be applied to each path separately 446 (e.g., if they exhibit deviating performance characteristics to not 447 degrade the 'better one') or in an overall FEC fashion before split 448 and recombination which would support scrambling and facilitate 449 recovery of completely lost packets on the 'worse path'. 450 Unsuccessful application of FEC may enable quick detection of 451 unrecoverable errors in a packet and thus trigger re-transmission 452 from the sender side before time-out. 454 6.2. Network Coding 456 In linear network coding (LNC) network nodes (or interfaces of a 457 device) do not simply relay the packets of information they receive, 458 but combine several packets together for transmission. After 459 reception of combined and separate packets the maximum possible 460 information flow in a network can be detected and throughput, 461 efficiency and scalability, as well as resilience to attacks and 462 eavesdropping can be improved. The method in general improves with 463 the number of paths in excess of two. According to [COPE] drawbacks 464 of LNC are high decoding computational complexity, high transmission 465 overhead, and linear dependency among coefficients vectors for en- 466 and decoding. Triangular network coding (TNC) addresses the high 467 encoding and decoding computational complexity without degrading the 468 throughput performance, with code rate comparable to that of LNC. 469 TNC is therefore advantageous for implementation on small devices 470 mobile phones and wireless sensors with limited processing capability 471 and power supply [TNC]. 473 7. Retransmission mechanisms 475 Re-transmission becomes interesting when it can help to reduce the 476 time spent on waiting for outstanding packets for re-sequencing. In 477 particular scenarios when for example a known path RTT (Round Trip 478 Time) lets expect a shorter time to re-transmit than wait for packet 479 loss detection, a likely scenario in, e.g., Figure 1. It could also 480 avoid a potential late triggering of re-transmission by the end-to- 481 end service. On the other hand for sake of resource efficiency the 482 amount of unnecessary retransmissions should be limited to not 483 degrade the overall throughput of the connection. 485 7.1. Signaling 487 In case of detected packet loss the receiver has to send a 488 corresponding signalling message to the sender to re-transmit a 489 missing packet. This is the traditional way of negative 490 acknowledgement in case of missing the correct reception of packets 491 within a time window and sending a repeat-request. This approach 492 requires a send buffer which keeps information for a reasonable time, 493 thus allowing the beneficial use of this mechanism. On the other 494 hand the additional delay in terms of at least once the RTT until the 495 lost packet is correctly received results in performance degradation 496 for time-critical applications. ... [to be continued?] 498 7.2. Anticipated 500 To speed up the induced re-transmission delay a pro-active or 501 anticipated approach would allow to trigger the sender to re-transmit 502 data without needing to wait for notification from the receiver. 503 This method can be applied when the assumed packet loss can be 504 estimated based on other data, e.g., from lower layer, such as 505 information on path or 506 link quality degradation derived from, e.g., an increased raw BER 507 detected by FEC mechanism (see Section 6.1). 508 [to be continued/extended?] 510 7.3. Flow-selection 512 Repeating data on the same path is not always useful. In some 513 scenarios it makes sense to re-transmit data on another path, e.g., 514 when the original path is broken or another path is known to provide 515 higher throughput or lower packet loss. To apply such a selection of 516 the flow for re-transmission. 518 * Requires path independent identification of data, e.g., the 519 connection sequencing 521 * Has to consider MTU discrepancies between paths 523 Flow selection for re-transmitting data can be combined with 524 detection mechanisms as described in Section 7.1 or Section 7.2. 526 7.4. Other re-transmission issues 528 In certain scenarios data to be re-transmitted can be duplicated 529 across paths (either in advance or after loss detection) to increase 530 reliability and reduce potential overall transmission delay. 531 However, such approaches decrease the resource efficiency and reduce 532 the overall user throughput. A more pro-active measure would 533 be to encode multiple packets either on per-path or on per-connection 534 basis in a single 'repair packet' in 'XOR style' to be injected after 535 a set of packets (similarly as described in [COPE]. This would allow 536 to recreate exactly one lost packet out of the set in case the others 537 have been correctly received. Depending on the anticipated loss rate 538 the amount of packets within a set is chosen to more efficiently use 539 the transmission resources. [to be continued] 541 8. Security Considerations 543 This document does not add any additional security considerations in 544 addition to the ones introduced by multipath extensions to other 545 transmission protocols as, e.g., described for MPTCP in [RFC8684]. 546 Also the described issues for GMA [I-D.zhu-intarea-gma], MP-DCCP 547 [I-D.amend-tsvwg-multipath-dccp], and MP-QUIC 548 [I-D.liu-multipath-quic] may apply here. 550 9. Informative References 552 [AOPS] Huang, C., Chen, Y., and S. Linn, "Packet scheduling and 553 congestion control schemes for Multipath Datagram 554 Congestion Control Protocol", The Computer Journal 58, no. 555 2, pp. 188--203 ] , February 2015. 557 [BLEST] Ferlin, S., Alay, Oe., Mehani, O., and R. Boreli, "BLEST: 558 Blocking estimation-based MPTCP scheduler for 559 heterogeneous networks", IFIP Networking Conference , May 560 2014, 561 . 563 [COPE] Katti, S., Rahul, H., Katabi, W.H.D., Medard, M., and J. 564 Crowcroft, "XORs in The Air: Practical Wireless Network 565 Coding", September 2006, 566 . 568 [DAPS] Kuhn, N., Lochin, E., Mifdaoui, A., Sarwar, G., Mehani, 569 O., and R. Boreli, "DAPS: Intelligent delay-aware packet 570 scheduling for multipath transport", ICC IEEE 571 International Conference on Communications, June 2014, 572 . 574 [I-D.amend-tsvwg-multipath-dccp] 575 Amend, M., Hugo, D. V., Brunstrom, A., Kassler, A., 576 Rakocevic, V., and S. Johnson, "DCCP Extensions for 577 Multipath Operation with Multiple Addresses", Work in 578 Progress, Internet-Draft, draft-amend-tsvwg-multipath- 579 dccp-05, 12 July 2021, . 582 [I-D.bonaventure-iccrg-schedulers] 583 Bonaventure, O., Piraux, M., Coninck, Q. D., Baerts, M., 584 Paasch, C., and M. Amend, "Multipath schedulers", Work in 585 Progress, Internet-Draft, draft-bonaventure-iccrg- 586 schedulers-02, 25 October 2021, 587 . 590 [I-D.deconinck-quic-multipath] 591 Coninck, Q. D. and O. Bonaventure, "Multipath Extensions 592 for QUIC (MP-QUIC)", Work in Progress, Internet-Draft, 593 draft-deconinck-quic-multipath-07, 3 May 2021, 594 . 597 [I-D.huitema-quic-ts] 598 Huitema, C., "Quic Timestamps For Measuring One-Way 599 Delays", Work in Progress, Internet-Draft, draft-huitema- 600 quic-ts-06, 12 September 2021, 601 . 604 [I-D.ietf-quic-datagram] 605 Pauly, T., Kinnear, E., and D. Schinazi, "An Unreliable 606 Datagram Extension to QUIC", Work in Progress, Internet- 607 Draft, draft-ietf-quic-datagram-06, 5 October 2021, 608 . 611 [I-D.ietf-quic-recovery] 612 Iyengar, J. and I. Swett, "QUIC Loss Detection and 613 Congestion Control", Work in Progress, Internet-Draft, 614 draft-ietf-quic-recovery-34, 14 January 2021, 615 . 618 [I-D.liu-multipath-quic] 619 Liu, Y., Ma, Y., Huitema, C., An, Q., and Z. Li, 620 "Multipath Extension for QUIC", Work in Progress, 621 Internet-Draft, draft-liu-multipath-quic-04, 5 September 622 2021, . 625 [I-D.song-mptcp-owl] 626 Song, F., Zhang, H., Chan, H. A., and A. Wei, "One Way 627 Latency Considerations for MPTCP", Work in Progress, 628 Internet-Draft, draft-song-mptcp-owl-06, 18 June 2019, 629 . 632 [I-D.zhu-intarea-gma] 633 Zhu, J. and S. Kanugovi, "Generic Multi-Access (GMA) 634 Encapsulation Protocol", Work in Progress, Internet-Draft, 635 draft-zhu-intarea-gma-12, 21 October 2021, 636 . 639 [I-D.zhu-intarea-mams-user-protocol] 640 Zhu, J., Seo, S., Kanugovi, S., and S. Peng, "User-Plane 641 Protocols for Multiple Access Management Service", Work in 642 Progress, Internet-Draft, draft-zhu-intarea-mams-user- 643 protocol-09, 4 March 2020, 644 . 647 [OTIAS] Yang, F., Wang, Q., and P.D. Amer, "Out-of-Order 648 Transmission for In-Order Arrival Scheduling for Multipath 649 TCP", AINAW 28th International Conference on Advanced 650 Information Networking and Applications Workshops, May 651 2014, . 653 [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, 654 RFC 793, DOI 10.17487/RFC0793, September 1981, 655 . 657 [RFC8684] Ford, A., Raiciu, C., Handley, M., Bonaventure, O., and C. 658 Paasch, "TCP Extensions for Multipath Operation with 659 Multiple Addresses", RFC 8684, DOI 10.17487/RFC8684, March 660 2020, . 662 [RFC8743] Kanugovi, S., Baboescu, F., Zhu, J., and S. Seo, "Multiple 663 Access Management Services Multi-Access Management 664 Services (MAMS)", RFC 8743, DOI 10.17487/RFC8743, March 665 2020, . 667 [TNC] Qureshi, J., Foh, C.H., and J. Cai, "Optimal solution for 668 the index coding problem using network coding over GF(2)", 669 June 2012, . 671 Authors' Addresses 673 Markus Amend 674 Deutsche Telekom 675 Deutsche-Telekom-Allee 9 676 64295 Darmstadt 677 Germany 679 Email: Markus.Amend@telekom.de 680 Dirk von Hugo 681 Deutsche Telekom 682 Deutsche-Telekom-Allee 9 683 64295 Darmstadt 684 Germany 686 Email: dirk.von-Hugo@telekom.de