idnits 2.17.1 draft-tuexen-tsvwg-sctp-multipath-23.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There are 5 instances of lines with non-ascii characters in the document. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (9 February 2022) is 807 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Experimental ---------------------------------------------------------------------------- == Missing Reference: 'RFCXXXX' is mentioned on line 963, but not defined ** Obsolete normative reference: RFC 4960 (Obsoleted by RFC 9260) ** Obsolete normative reference: RFC 6096 (Obsoleted by RFC 9260) == Outdated reference: A later version (-28) exists of draft-dreibholz-tsvwg-sctpsocket-multipath-23 == Outdated reference: A later version (-28) exists of draft-dreibholz-tsvwg-sctpsocket-sqinfo-23 Summary: 2 errors (**), 0 flaws (~~), 5 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group P. D. Amer 3 Internet-Draft University of Delaware 4 Intended status: Experimental M. Becke 5 Expires: 13 August 2022 HAW Hamburg 6 T. Dreibholz 7 SimulaMet 8 N. Ekiz 9 University of Delaware 10 J. Iyengar 11 Franklin and Marshall College 12 P. Natarajan 13 Cisco Systems 14 R. R. Stewart 15 Netflix 16 M. Tuexen 17 Muenster Univ. of Appl. Sciences 18 9 February 2022 20 Load Sharing for the Stream Control Transmission Protocol (SCTP) 21 draft-tuexen-tsvwg-sctp-multipath-23 23 Abstract 25 The Stream Control Transmission Protocol (SCTP) supports multi-homing 26 for providing network fault tolerance. However, mainly one path is 27 used for data transmission. Only timer-based retransmissions are 28 carried over other paths as well. 30 This document describes how multiple paths can be used simultaneously 31 for transmitting user messages. 33 Status of This Memo 35 This Internet-Draft is submitted in full conformance with the 36 provisions of BCP 78 and BCP 79. 38 Internet-Drafts are working documents of the Internet Engineering 39 Task Force (IETF). Note that other groups may also distribute 40 working documents as Internet-Drafts. The list of current Internet- 41 Drafts is at https://datatracker.ietf.org/drafts/current/. 43 Internet-Drafts are draft documents valid for a maximum of six months 44 and may be updated, replaced, or obsoleted by other documents at any 45 time. It is inappropriate to use Internet-Drafts as reference 46 material or to cite them other than as "work in progress." 48 This Internet-Draft will expire on 13 August 2022. 50 Copyright Notice 52 Copyright (c) 2022 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 57 license-info) in effect on the date of publication of this document. 58 Please review these documents carefully, as they describe your rights 59 and restrictions with respect to this document. Code Components 60 extracted from this document must include Simplified BSD License text 61 as described in Section 4.e of the Trust Legal Provisions and are 62 provided without warranty as described in the Simplified BSD License. 64 Table of Contents 66 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 67 2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 3 68 3. Load Sharing . . . . . . . . . . . . . . . . . . . . . . . . 3 69 3.1. Split Fast Retransmissions . . . . . . . . . . . . . . . 4 70 3.2. Appropriate Congestion Window Growth . . . . . . . . . . 4 71 3.3. Appropriate Delayed Acknowledgements . . . . . . . . . . 5 72 4. Non-Renegable SACK . . . . . . . . . . . . . . . . . . . . . 6 73 4.1. Negotiation . . . . . . . . . . . . . . . . . . . . . . . 6 74 4.2. The New Chunk Type: Non-Renegable SACK (NR-SACK) . . . . 6 75 4.3. An Illustrative Example . . . . . . . . . . . . . . . . . 11 76 4.4. Procedures . . . . . . . . . . . . . . . . . . . . . . . 15 77 4.4.1. Sending an NR-SACK chunk . . . . . . . . . . . . . . 15 78 4.4.2. Receiving an NR-SACK Chunk . . . . . . . . . . . . . 17 79 5. Buffer Blocking Mitigation . . . . . . . . . . . . . . . . . 18 80 5.1. Sender Buffer Splitting . . . . . . . . . . . . . . . . . 18 81 5.2. Receiver Buffer Splitting . . . . . . . . . . . . . . . . 18 82 5.3. Chunk Rescheduling . . . . . . . . . . . . . . . . . . . 18 83 5.4. Problems during Path Failure . . . . . . . . . . . . . . 18 84 5.4.1. Problem Description . . . . . . . . . . . . . . . . . 18 85 5.4.2. Solution: Potentially-failed Destination State . . . 19 86 5.5. Non-Renegable SACK . . . . . . . . . . . . . . . . . . . 19 87 5.5.1. Problem Description . . . . . . . . . . . . . . . . . 19 88 5.5.2. Solution: Non-Renegable SACKs . . . . . . . . . . . . 20 89 6. Handling of Shared Bottlenecks . . . . . . . . . . . . . . . 20 90 6.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 20 91 6.2. Initial Values . . . . . . . . . . . . . . . . . . . . . 20 92 6.3. Congestion Window Growth . . . . . . . . . . . . . . . . 21 93 6.4. Congestion Window Decrease . . . . . . . . . . . . . . . 21 94 7. Chunk Scheduling and Rescheduling . . . . . . . . . . . . . . 21 95 8. Socket API Considerations . . . . . . . . . . . . . . . . . . 21 96 9. Testbed Platforms . . . . . . . . . . . . . . . . . . . . . . 21 97 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 98 10.1. A New Chunk Type . . . . . . . . . . . . . . . . . . . . 22 99 11. Security Considerations . . . . . . . . . . . . . . . . . . . 22 100 12. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 22 101 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 102 13.1. Normative References . . . . . . . . . . . . . . . . . . 22 103 13.2. Informative References . . . . . . . . . . . . . . . . . 23 104 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 27 106 1. Introduction 108 One of the important features of the Stream Control Transmission 109 Protocol (SCTP), which is currently specified in [RFC4960], is 110 network fault tolerance. This feature is for example required for 111 Reliable Server Pooling (RSerPool, [RFC5351]). Therefore, 112 transmitting messages over multiple paths is supported, but only for 113 redundancy. So [RFC4960] does not specify how to use multiple paths 114 simultaneously. 116 This document overcomes this limitation by specifying how multiple 117 paths can be used simultaneously. This has several benefits: 119 * Improved bandwidth usage. 121 * Better availability check with real user messages compared to 122 HEARTBEAT-based information. 124 2. Conventions 126 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 127 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 128 document are to be interpreted as described in [RFC2119]. 130 3. Load Sharing 132 Basic requirement for applying SCTP load sharing is the Concurrent 133 Multipath Transfer (CMT) extension of SCTP, which utilises multiple 134 paths simultaneously. We denote CMT-enabled SCTP as CMT-SCTP 135 throughout this document. CMT-SCTP is introduced in [IAS06] and in 136 more detail in [I06], some illustrative examples of chunk handling 137 are provided in [OMNeTWorkshop2010-SCTP]. CMT-SCTP provides three 138 modifications to standard SCTP (split Fast Retransmissions, 139 appropriate congestion window growth and delayed SACKs), which are 140 described in the following subsections. 142 3.1. Split Fast Retransmissions 144 Paths with different latencies lead to overtaking of DATA chunks. 145 This leads to gap reports, which are handled by Fast Retransmissions. 146 However, due to the fact that multiple paths are used simultaneously, 147 these Fast Retransmissions are usually useless and furthermore lead 148 to a decreased congestion window size. 150 To avoid unnecessary Fast Retransmissions, the sender has to keep 151 track of the path each DATA chunk has been sent on and consider 152 transmission paths before performing Fast Retransmissions. That is, 153 on reception of a SACK, the sender MUST identify the highest 154 acknowledged TSN on each path. A chunk SHOULD only be considered as 155 missing if its TSN is smaller than the highest acknowledged TSN on 156 its path. Section 3.1 of [OMNeTWorkshop2010-SCTP] contains an 157 illustrated example. 159 3.2. Appropriate Congestion Window Growth 161 The congestion window adaptation algorithm for SCTP [RFC4960] allows 162 increasing the congestion window only when a new cumulative ack 163 (CumAck) is received by a sender. When SACKs with unchanged CumAcks 164 are generated (due to reordering) and later arrive at a sender, the 165 sender does not modify its congestion window. Since a CMT-SCTP 166 receiver naturally observes reordering, many SACKs are sent 167 containing new gap reports but not new CumAcks. When these gaps are 168 later acked by a new CumAck, congestion window growth occurs, but 169 only for the data newly acked in the most recent SACK. Data 170 previously acked through gap reports will not contribute to 171 congestion window growth, in order to prevent sudden increases in the 172 congestion window resulting in bursts of data being sent. 174 To overcome the problems described above, the congestion window 175 growth has to be handled as follows [IAS06]: 177 * The sender SHOULD keep track of the earliest non-retransmitted 178 outstanding TSN per path. 180 * The sender SHOULD keep track of the earliest retransmitted 181 outstanding TSN per path. 183 * The in-order delivery per path SHOULD be deduced. 185 * The congestion window of a path SHOULD be increased when the 186 earliest non-retransmitted outstanding TSN of this path is 187 advanced ('Pseudo CumAck') OR when the earliest retransmitted 188 outstanding TSN of this path is advanced ('RTX Pseudo CumAck'). 190 Section 3.2 of [OMNeTWorkshop2010-SCTP] contains an illustrated 191 example of appropriate congestion window handling for CMT-SCTP. 193 3.3. Appropriate Delayed Acknowledgements 195 Standard SCTP [RFC4960] sends a SACK as soon as an out-of-sequence 196 TSN has been received. Delayed Acknowledgements are only allowed if 197 the received TSNs are in sequence. However, due to the load 198 balancing of CMT-SCTP, DATA chunks may overtake each other. This 199 leads to a high number of out-of-sequence TSNs, which have to be 200 acknowledged immediately. Clearly, this behaviour increases the 201 overhead traffic (usually nearly one SACK chunk for each received 202 packet containing a DATA chunk). 204 Delayed Acknowledgements for CMT-SCTP are handled as follows: 206 * In addition to [RFC4960], delaying of SACKs SHOULD *also* be 207 applied for out-of-sequence TSNs. 209 * A receiver MUST maintain a counter for the number of DATA chunks 210 received before sending a SACK. The value of the counter is 211 stored into each SACK chunk (FIXME: add details; needs reservation 212 of flags bits by IANA). After transmitting a SACK, the counter 213 MUST be reset to 0. Its initial value MUST be 0. 215 * The SACK handling procedure for a missing TSN M is extended as 216 follows: 218 - If all newly acknowledged TSNs have been transmitted over the 219 same path: 221 o If there are newly acknowledged TSNs L and H so that L <= M 222 <= H, the missing count of TSN M SHOULD be incremented by 223 one (like for standard SCTP according to [RFC4960]). 225 o Else if all newly acknowledged TSNs N satisfy the condition 226 M <= N, the missing count of TSN M SHOULD be incremented by 227 the number of TSNs reported in the SACK chunk. 229 - Otherwise (that is, there are newly acknowledged TSNs on 230 different paths), the missing count of TSN M SHOULD be 231 incremented by one (like for standard SCTP according to 232 [RFC4960]). 234 Section 3.3 of [OMNeTWorkshop2010-SCTP] contains an illustrated 235 example of Delayed Acknowledgements for CMT-SCTP. 237 4. Non-Renegable SACK 239 4.1. Negotiation 241 Before sending/receiving NR-SACKs (see [YEN10]), both peer endpoints 242 MUST agree on using NR-SACKs. This agreement MUST be negotiated 243 during association establishment. NR-SACK is an extension to the 244 core SCTP, and SCTP extensions that an endpoint supports are reported 245 to the peer endpoint in Supported Extensions Parameter during 246 association establishment (see Section 4.2.7 of [RFC5061].) The 247 Supported Extensions Parameter consists of a list of non-standard 248 Chunk Types that are supported by the sender. 250 An endpoint supporting the NR-SACK extension MUST list the NR-SACK 251 chunk in the Supported Extensions Parameter carried in the INIT or 252 INIT-ACK chunk, depending on whether the endpoint initiates or 253 responds to the initiation of the association. If the NR-SACK chunk 254 type ID is listed in the Chunk Types List of the Supported Extensions 255 Parameter, then the receiving endpoint MUST assume that the NR-SACK 256 chunk is supported by the sending endpoint. 258 Both endpoints MUST support NR-SACKs for either endpoint to send an 259 NR-SACK. If an endpoint establishes an association with a remote 260 endpoint that does not list NR-SACK in the Supported Extensions 261 Parameter carried in INIT chunk, then both endpoints of the 262 association MUST NOT use NR-SACKs. After association establishment, 263 an endpoint MUST NOT renegotiate the use of NR-SACKs. 265 Once both endpoints indicate during association establishment that 266 they support the NR-SACK extension, each endpoint SHOULD acknowledge 267 received DATA chunks with NR-SACK chunks, and not SACK chunks. That 268 is, throughout an SCTP association, both endpoints SHOULD send either 269 SACK chunks or NR-SACK chunks, never a mixture of the two. 271 4.2. The New Chunk Type: Non-Renegable SACK (NR-SACK) 273 Table 1 illustrates a new chunk type that will be used to transfer 274 NR-SACK information. 276 Chunk Type Chunk Name 277 -------------------------------------------------------------- 278 0x10 Non-Renegable Selective Acknowledgment (NR-SACK) 280 Table 1: NR-SACK Chunk 282 As the NR-SACK chunk replaces the SACK chunk, many SACK chunk fields 283 are preserved in the NR-SACK chunk. These preserved fields have the 284 same semantics with the corresponding SACK chunk fields, as defined 285 in [RFC4960], Section 3.3.4. The Gap Ack fields from RFC4960 have 286 been renamed as R Gap Ack to emphasize their renegable nature. Their 287 semantics are unchanged. For completeness, we describe all fields of 288 the NR-SACK chunk, including those that are identical in the SACK 289 chunk. 291 Similar to the SACK chunk, the NR-SACK chunk is sent to a peer 292 endpoint to (1) acknowledge DATA chunks received in-order, (2) 293 acknowledge DATA chunks received out-of-order, and (3) identify DATA 294 chunks received more than once (i.e., duplicate.) In addition, the 295 NR-SACK chunk (4) informs the peer endpoint of non-renegable out-of- 296 order DATA chunks. 298 0 1 2 3 299 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 300 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 301 | Type = 0x10 | Chunk Flags | Chunk Length | 302 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 303 | Cumulative TSN Ack | 304 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 305 | Advertised Receiver Window Credit (a_rwnd) | 306 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 307 |Number of R Gap Ack Blocks = N |Number of NR Gap Ack Blocks = M| 308 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 309 | Number of Duplicate TSNs = X | Reserved | 310 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 311 | R Gap Ack Block #1 Start | R Gap Ack Block #1 End | 312 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 313 / / 314 \ ... \ 315 / / 316 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 317 | R Gap Ack Block #N Start | R Gap Ack Block #N End | 318 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 319 | NR Gap Ack Block #1 Start | NR Gap Ack Block #1 End | 320 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 321 / / 322 \ ... \ 323 / / 324 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 325 | NR Gap Ack Block #M Start | NR Gap Ack Block #M End | 326 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 327 | Duplicate TSN 1 | 328 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 329 / / 330 \ ... \ 331 / / 332 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 333 | Duplicate TSN X | 334 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 336 Type: 8 bits 338 This field holds the IANA defined chunk type for NR-SACK chunk. The 339 suggested value of this field for IANA is 0x10. 341 Chunk Flags: 8 bits 343 Currently not used. It is recommended a sender set all bits to zero 344 on transmit, and a receiver ignore this field. 346 Chunk Length: 16 bits (unsigned integer) [Same as SACK chunk] 348 This value represents the size of the chunk in bytes including the 349 Chunk Type, Chunk Flags, Chunk Length, and Chunk Value fields. 351 Cumulative TSN Ack: 32 bits (unsigned integer) [Same as SACK chunk] 353 The value of the Cumulative TSN Ack is the last TSN received before a 354 break in the sequence of received TSNs occurs. The next TSN value 355 following the Cumulative TSN Ack has not yet been received at the 356 endpoint sending the NR-SACK. 358 Advertised Receiver Window Credit (a_rwnd): 32 bits (unsigned 359 integer) [Same as SACK chunk] 361 Indicates the updated receive buffer space in bytes of the sender of 362 this NR-SACK, see Section 6.2.1 of [RFC4960] for details. 364 Number of (R)enegable Gap Ack Blocks (N): 16 bits (unsigned integer) 366 Indicates the number of Renegable Gap Ack Blocks included in this NR- 367 SACK. 369 Number of (N)on(R)enegable Gap Ack Blocks (M): 16 bits (unsigned 370 integer) 372 Indicates the number of Non-Renegable Gap Ack Blocks included in this 373 NR-SACK. 375 Number of Duplicate TSNs (X): 16 bits [Same as SACK chunk] 377 Contains the number of duplicate TSNs the endpoint has received. 378 Each duplicate TSN is listed following the NR Gap Ack Block list. 380 Reserved : 16 bits 382 Currently not used. It is recommended a sender set all bits to zero 383 on transmit, and a receiver ignore this field. 385 (R)enegable Gap Ack Blocks: 387 The NR-SACK contains zero or more R Gap Ack Blocks. Each R Gap Ack 388 Block acknowledges a subsequence of renegable out-of-order TSNs. By 389 definition, all TSNs acknowledged by R Gap Ack Blocks are "greater 390 than" the value of the Cumulative TSN Ack. 392 Because of TSN numbering wraparound, comparisons and all arithmetic 393 operations discussed in this document are based on "Serial Number 394 Arithmetic" as described in Section 1.6 of [RFC4960]. 396 R Gap Ack Blocks are repeated for each R Gap Ack Block up to 'N' 397 defined in the Number of R Gap Ack Blocks field. All DATA chunks 398 with TSNs >= (Cumulative TSN Ack + R Gap Ack Block Start) and <= 399 (Cumulative TSN Ack + R Gap Ack Block End) of each R Gap Ack Block 400 are assumed to have been received correctly, and are renegable. 402 R Gap Ack Block Start: 16 bits (unsigned integer) 404 Indicates the Start offset TSN for this R Gap Ack Block. This number 405 is set relative to the cumulative TSN number defined in Cumulative 406 TSN Ack field. To calculate the actual start TSN number, the 407 Cumulative TSN Ack is added to this offset number. The calculated 408 TSN identifies the first TSN in this R Gap Ack Block that has been 409 received. 411 R Gap Ack Block End: 16 bits (unsigned integer) 413 Indicates the End offset TSN for this R Gap Ack Block. This number 414 is set relative to the cumulative TSN number defined in the 415 Cumulative TSN Ack field. To calculate the actual TSN number, the 416 Cumulative TSN Ack is added to this offset number. The calculated 417 TSN identifies the TSN of the last DATA chunk received in this R Gap 418 Ack Block. 420 N(on)R(enegable) Gap Ack Blocks: 422 The NR-SACK contains zero or more NR Gap Ack Blocks. Each NR Gap Ack 423 Block acknowledges a continuous subsequence of non-renegable out-of- 424 order DATA chunks. If a TSN is nr-gap-acked in any NR-SACK chunk, 425 then all subsequently transmitted NR-SACKs with a smaller cum-ack 426 value than that TSN SHOULD also nr-gap-ack that TSN. 428 NR Gap Ack Blocks are repeated for each NR Gap Ack Block up to 'M' 429 defined in the Number of NR Gap Ack Blocks field. All DATA chunks 430 with TSNs >= (Cumulative TSN Ack + NR Gap Ack Block Start) and <= 431 (Cumulative TSN Ack + NR Gap Ack Block End) of each NR Gap Ack Block 432 are assumed to be received correctly, and are Non-Renegable. 434 NR Gap Ack Block Start: 16 bits (unsigned integer) 436 Indicates the Start offset TSN for this NR Gap Ack Block. This 437 number is set relative to the cumulative TSN number defined in 438 Cumulative TSN Ack field. To calculate the actual TSN number, the 439 Cumulative TSN Ack is added to this offset number. The calculated 440 TSN identifies the first TSN in this NR Gap Ack Block that has been 441 received. 443 NR Gap Ack Block End: 16 bits (unsigned integer) 445 Indicates the End offset TSN for this NR Gap Ack Block. This number 446 is set relative to the cumulative TSN number defined in Cumulative 447 TSN Ack field. To calculate the actual TSN number, the Cumulative 448 TSN Ack is added to this offset number. The calculated TSN 449 identifies the TSN of the last DATA chunk received in this NR Gap Ack 450 Block. 452 Note: 454 NR Gap Ack Blocks and R Gap Ack Blocks in an NR-SACK chunk SHOULD 455 acknowledge disjoint sets of TSNs. That is, an out-of-order TSN 456 SHOULD be listed in either an R Gap Ack Block or an NR Gap Ack Block, 457 but not the both. R Gap Ack Blocks and NR Gap Ack Blocks together 458 provide the information as do the Gap Ack Block of a SACK chunk, plus 459 additional information about non-renegability. 461 If all out-of-order data acked by an NR-SACK are renegable, then the 462 Number of NR Gap Ack Blocks MUST be set to 0. If all out-of-order 463 data acked by an NR-SACK are non-renegable, then the Number of R Gap 464 Ack Blocks SHOULD be set to 0. TSNs listed in R Gap Ack Block will 465 be referred as r-gap-acked. 467 Duplicate TSN: 32 bits (unsigned integer) [Same as SACK chunk] 469 Indicates a duplicate TSN received since the last NR-SACK was sent. 470 Exactly 'X' duplicate TSNs SHOULD be reported where 'X' was defined 471 in Number of Duplicate TSNs field. 473 Each duplicate TSN is listed in this field as many times as the TSN 474 was received since the previous NR-SACK was sent. For example, if a 475 data receiver were to get the TSN 19 three times, the data receiver 476 would list 19 twice in the outbound NR-SACK. After sending the NR- 477 SACK if the receiver received one more TSN 19, the receiver would 478 list 19 as a duplicate once in the next outgoing NR-SACK. 480 4.3. An Illustrative Example 482 Assume the following DATA chunks have arrived at the receiver. 484 -------------------------------- 485 | TSN=16| SID=2 | SSN=N/A| U=1 | 486 -------------------------------- 487 | TSN=15| SID=1 | SSN= 4 | U=0 | 488 -------------------------------- 489 | TSN=14| SID=0 | SSN= 4 | U=0 | 490 -------------------------------- 491 | TSN=13| SID=2 | SSN=N/A| U=1 | 492 -------------------------------- 493 | | 494 -------------------------------- 495 | TSN=11| SID=0 | SSN= 3 | U=0 | 496 ------------------------------- 497 | | 498 -------------------------------- 499 | | 500 -------------------------------- 501 | TSN=8 | SID=2 | SSN=N/A| U=1 | 502 -------------------------------- 503 | TSN=7 | SID=1 | SSN= 2 | U=0 | 504 -------------------------------- 505 | TSN=6 | SID=1 | SSN= 1 | U=0 | 506 -------------------------------- 507 | TSN=5 | SID=0 | SSN= 1 | U=0 | 508 -------------------------------- 509 | | 510 -------------------------------- 511 | TSN=3 | SID=1 | SSN= 0 | U=0 | 512 -------------------------------- 513 | TSN=2 | SID=0 | SSN= 0 | U=0 | 514 -------------------------------- 516 The above figure shows the list of DATA chunks at the receiver. TSN 517 denotes the transmission sequence number of the DATA chunk, SID 518 denotes the stream id to which the DATA chunk belongs, SSN denotes 519 the sequence number of the DATA chunk within its stream, and the U 520 bit denotes whether the DATA chunk requires ordered(=0) or 521 unordered(=1) delivery [RFC4960]. Note that TSNs 4,9,10, and 12 have 522 not arrived. 524 This data can be viewed as three separate streams as follows (assume 525 each stream begins with SSN=0.) Note that in this example, the 526 application uses stream 2 for unordered data transfer. By 527 definition, SSN fields of unordered DATA chunks are ignored. 529 Stream-0: 531 SSN: 0 1 2 3 4 532 TSN: | 2 | 5 | | 11 | 14 | 533 U-Bit: | 0 | 0 | | 0 | 0 | 535 Stream-1: 537 SSN: 0 1 2 3 4 538 TSN: | 3 | 6 | 7 | | 15 | 539 U-Bit: | 0 | 0 | 0 | | 0 | 541 Stream-2: 543 SSN: N/A N/A N/A 544 TSN: | 8 | 13 | 16 | 545 U-Bit: | 1 | 1 | 1 | 547 The NR-SACK to acknowledge the above data SHOULD be constructed as 548 follows for each of the three cases described below (the a_rwnd is 549 arbitrarily set to 4000): 551 CASE-1: Minimal Data Receiver Responsibility - no out-of-order 552 deliverable data yet delivered 554 None of the deliverable out-of-order DATA chunks have been delivered, 555 and the receiver of the above data does not take responsibility for 556 any of the received out-of-order DATA chunks. The receiver reserves 557 the right to renege any or all of the out-of-order DATA chunks. 559 +-----------------------------+-----------------------------+ 560 | Type = 0x10 | 00000000 | Chunk Length = 32 | 561 +-----------------------------+-----------------------------+ 562 | Cumulative TSN Ack = 3 | 563 +-----------------------------+-----------------------------+ 564 | a_rwnd = 4000 | 565 +-----------------------------+-----------------------------+ 566 | Num of R Gap Ack Blocks = 3 |Num of NR Gap Ack Blocks = 0 | 567 +-----------------------------+-----------------------------+ 568 | Num of Duplicates = 0 | 0x00 | 569 +-----------------------------+-----------------------------+ 570 |R Gap Ack Block #1 Start = 2 | R Gap Ack Block #1 End = 5 | 571 +-----------------------------+-----------------------------+ 572 |R Gap Ack Block #2 Start = 8 | R Gap Ack Block #2 End = 8 | 573 +-----------------------------+-----------------------------+ 574 |R Gap Ack Block #3 Start = 10| R Gap Ack Block #3 End = 13 | 575 +-----------------------------+-----------------------------+ 577 CASE-2: Minimal Data Receiver Responsibility - all out-of-order 578 deliverable data delivered 579 In this case, the NR-SACK chunk is being sent after the data receiver 580 has delivered all deliverable out-of-order DATA chunks to its 581 receiving application(i.e., TSNs 5,6,7,8,13, and 16.) The receiver 582 reserves the right to renege on all undelivered out-of-order DATA 583 chunks(i.e., TSNs 11,14, and 15.) 585 +------------------------------+------------------------------+ 586 | Type = 0x10 | 0x00 | Chunk Length = 40 | 587 +------------------------------+------------------------------+ 588 | Cumulative TSN Ack = 3 | 589 +------------------------------+------------------------------+ 590 | a_rwnd = 4000 | 591 +------------------------------+------------------------------+ 592 | Num of R Gap Ack Blocks = 2 | Num of NR Gap Ack Blocks = 3 | 593 +------------------------------+------------------------------+ 594 | Num of Duplicates = 0 | 0x00 | 595 +------------------------------+------------------------------+ 596 | R Gap Ack Block #1 Start = 8 | R Gap Ack Block #1 End = 8 | 597 +------------------------------+------------------------------+ 598 | R Gap Ack Block #2 Start = 11| R Gap Ack Block #2 End = 12 | 599 +------------------------------+------------------------------+ 600 |NR Gap Ack Block #1 Start = 2 | NR Gap Ack Block #1 End = 5 | 601 +------------------------------+------------------------------+ 602 |NR Gap Ack Block #2 Start = 10| NR Gap Ack Block #2 End = 10 | 603 +------------------------------+------------------------------+ 604 |NR Gap Ack Block #3 Start = 13| NR Gap Ack Block #3 End = 13 | 605 +------------------------------+------------------------------+ 607 CASE-3: Maximal Data Receiver Responsibility 609 In this special case, all out-of-order data blocks acknowledged are 610 non-renegable. This case would occur when the data receiver is 611 programmed never to renege, and takes responsibility to deliver all 612 DATA chunks that arrive out-of-order. In this case Num of R Gap Ack 613 Blocks is zero indicating all reported out-of-order TSNs are nr-gap- 614 acked. 616 +--------------------------------+-------------------------------+ 617 | Type = 0x10 | 0x00 | Chunk Length = 32 | 618 +--------------------------------+-------------------------------+ 619 | Cumulative TSN Ack = 3 | 620 +--------------------------------+-------------------------------+ 621 | a_rwnd = 4000 | 622 +--------------------------------+-------------------------------+ 623 | Num of R Gap Ack Blocks = 0 | Num of NR Gap Ack Blocks = 3 | 624 +--------------------------------+-------------------------------+ 625 | Num of Duplicates = 0 | 0x00 | 626 +--------------------------------+-------------------------------+ 627 | NR Gap Ack Block #1 Start = 2 | NR Gap Ack Block #1 End = 5 | 628 +--------------------------------+-------------------------------+ 629 | NR Gap Ack Block #2 Start = 8 | NR Gap Ack Block #2 End = 8 | 630 +--------------------------------+-------------------------------+ 631 | NR Gap Ack Block #3 Start = 10 | NR Gap Ack Block #3 End = 13 | 632 +--------------------------------+-------------------------------+ 634 4.4. Procedures 636 The procedures regarding "when" to send an NR-SACK chunk are 637 identical to the procedures regarding when to send a SACK chunk, as 638 outlined in Section 6.2 of [RFC4960]. 640 4.4.1. Sending an NR-SACK chunk 642 All of the NR-SACK chunk fields identical to the SACK chunk MUST be 643 formed as described in Section 6.2 of [RFC4960]. 645 It is up to the data receiver whether or not to take responsibility 646 for delivery of each out-of-order DATA chunk. An out-of-order DATA 647 chunk that has already been delivered, or that the receiver takes 648 responsibility to deliver (i.e., guarantees not to renege) is Non 649 Renegable(NR), and SHOULD be included in an NR Gap Ack Block field of 650 the outgoing NR-SACK. All other out-of-order data is (R)enegable, 651 and SHOULD be included in R Gap Ack Block field of the outgoing NR- 652 SACK. 654 Consider three types of data receiver: 656 CASE-1: Data receiver takes no responsibility for delivery of any 657 out-of-order DATA chunks 659 CASE-2: Data receiver takes responsibility for all out-of-order DATA 660 chunks that are "deliverable" (i.e., DATA chunks in-sequence 661 within the stream they belong to, or DATA chunks whose (U)nordered 662 bit is 1) 664 CASE-3: Data receiver takes responsibility for delivery of all out- 665 of-order DATA chunks, whether deliverable or not deliverable 667 The data receiver SHOULD follow the procedures outlined below for 668 building the NR-SACK. 670 CASE-1: 672 1A) Identify the TSNs received out-of-order. 674 1B) For these out-of-order TSNs, identify the R Gap Ack Blocks. 675 Fill the Number of R Gap Ack Blocks (N) field, R Gap Ack Block #i 676 Start, and R Gap Ack Block #i End where i goes from 1 to N. 678 1C) Set the Number of NR Gap Ack Blocks (M) field to 0. 680 CASE-2: 682 2A) Identify the TSNs received out-of-order. 684 2B) For the received out-of-order TSNs, check the (U)nordered bit of 685 each TSN. Tag unordered TSNs as NR. 687 2C) For each stream, also identify the TSNs received out-of-order 688 but are in-sequence within that stream. Tag those in-sequence 689 TSNs as NR. 691 2D) Tag all out-of-order data that is not NR as (R)enegable. 693 2E) For those TSNs tagged as (R)enegable, identify the (R)enegable 694 Blocks. Fill the Number of R Gap Ack Blocks(N) field, R Gap Ack 695 Block #i Start, and R Gap Ack Block #i End where i goes from 1 to 696 N. 698 2F) For those TSNs tagged as NR, identify the NR Blocks. Fill the 699 Number of NR Gap Ack Blocks(M) field, NR Gap Ack Block #i Start, 700 and NR Gap Ack Block #i End where i goes from 1 to M. 702 CASE-3: 704 3A) Identify the TSNs received out-of-order. All of these TSNs 705 SHOULD be nr-gap-acked. 707 3B) Set the Number of R Gap Ack Blocks (N) field to 0. 709 3C) For these out-of-order TSNs, identify the NR Gap Ack Blocks. 710 Fill the Number of NR Gap Ack Blocks (M) field, NR Gap Ack Block 711 #i Start, and NR Gap Ack Block #i End where i goes from 1 to M. 713 RFC4960 states that the SCTP endpoint MUST report as many Gap Ack 714 Blocks as can fit in a single SACK chunk limited by the current path 715 MTU. When using NR-SACKs, the SCTP endpoint SHOULD fill as many R 716 Gap Ack Blocks and NR Gap Ack Blocks starting from the Cumulative TSN 717 Ack value as can fit in a single NR-SACK chunk limited by the current 718 path MTU. If space remains, the SCTP endpoint SHOULD fill as many 719 Duplicate TSNs as possible starting from Cumulative TSN Ack value. 721 4.4.2. Receiving an NR-SACK Chunk 723 When an NR-SACK chunk is received, all of the NR-SACK fields 724 identical to a SACK chunk SHOULD be processed and handled as in SACK 725 chunk handling outlined in Section 6.2.1 of [RFC4960]. 727 The NR Gap Ack Block Start(s) and NR Gap Ack Block End(s) are offsets 728 relative to the cum-ack. To calculate the actual range of nr-gap- 729 acked TSNs, the cum-ack MUST be added to the Start and End. 731 For example, assume an incoming NR-SACK chunk's cum-ack is 12 and an 732 NR Gap Ack Block defines the NR Gap Ack Block Start=5, and the NR Gap 733 Ack Block End=7. This NR Gap Ack block nr-gap-acks TSNs 17 through 734 19 inclusive. 736 Upon reception of an NR-SACK chunk, all TSNs listed in either R Gap 737 Ack Block(s) or NR Gap Ack Block(s) SHOULD be processed as would be 738 TSNs included in Gap Ack Block(s) of a SACK chunk. All TSNs in all 739 NR Gap Ack Blocks SHOULD be removed from the data sender's 740 retransmission queue as their delivery to the receiving application 741 has either already occurred, or is guaranteed by the data receiver. 743 Although R Gap Ack Blocks and NR Gap Ack Blocks SHOULD be disjoint 744 sets, NR-SACK processing SHOULD work if an NR-SACK chunk has a TSN 745 listed in both an R Gap Ack Block and an NR Gap Ack Block. In this 746 case, the TSN SHOULD be treated as Non-Renegable. 748 Implementation Note: 750 Most of NR-SACK processing at the data sender can be implemented by 751 using the same routines as in SACK that process the cum ack and the 752 gap ack(s), followed by removal of nr-gap-acked DATA chunks from the 753 retransmission queue. However, with NR-SACKs, as out-of-order DATA 754 is sometimes removed from the retransmission queue, the gap ack 755 processing routine should recognize that the data sender's 756 retransmission queue has some transmitted data removed. For example, 757 while calculating missing reports, the gap ack processing routine 758 cannot assume that the highest TSN transmitted is always at the tail 759 (right edge) of the retransmission queue. 761 5. Buffer Blocking Mitigation 763 TBD. See [Dre2012], [PAMS2011], [Globecom2010]. 765 5.1. Sender Buffer Splitting 767 TBD. See [Dre2012], [PAMS2011], [Globecom2010]. 769 5.2. Receiver Buffer Splitting 771 TBD. See [Dre2012], [PAMS2011], [Globecom2010]. 773 5.3. Chunk Rescheduling 775 This algorithm ensures quick blocking resolution for ordered data. 776 TBD. See [Dre2012], [Globecom2010]. 778 5.4. Problems during Path Failure 780 This section discusses CMT's receive buffer related problems during 781 path failure, and proposes a solution for the same. 783 5.4.1. Problem Description 785 Link failures arise when a router or a link connecting two routers 786 fails due to link disconnection, hardware malfunction, or software 787 error. Overloaded links caused by flash crowds and denial-of-service 788 (DoS) attacks also degrade end-to-end communication between peer 789 hosts. Ideally, the routing system detects link failures, and in 790 response, reconfigures the routing tables and avoids routing traffic 791 via the failed link. However, existing research highlights problems 792 with Internet backbone routing that result in long route convergence 793 times. The pervasiveness of path failures motivated us to study 794 their impact on CMT, since CMT achieves better throughput via 795 simultaneous data transmission over multiple end-to-end paths. 797 CMT is an extension to SCTP, and therefore retains SCTP's failure 798 detection process. A CMT sender uses a tunable failure detection 799 threshold called Path.Max.Retrans (PMR). When a sender experiences 800 more than PMR consecutive timeouts while trying to reach an active 801 destination, the destination is marked as failed. With PMR=5, the 802 failure detection takes 6 consecutive timeouts or 63s. After every 803 timeout, the CMT sender continues to transmit new data on the failed 804 path increasing the chances of receive buffer (rbuf) blocking and 805 degrading CMT performance during permanent and short-term path 806 failures [NEA08]. 808 5.4.2. Solution: Potentially-failed Destination State 810 To mitigate the rbuf blocking, we introduce a new destination state 811 called 'potentially-failed' state in SCTP (and CMT's) failure 812 detection process [I-D.ietf-tsvwg-sctp-failover]. This solution is 813 based on the rationale that loss detected by a timeout implies either 814 severe congestion or failure en route. After a single timeout on a 815 path, a sender is unsure, and marks the corresponding destination as 816 'potentially-failed' (PF). A PF destination is not used for data 817 transmission or retransmission. CMT's retransmission policies are 818 augmented to include the PF state. Performance evaluations prove 819 that the PF state significantly reduces rbuf blocking during failure 820 detection [NEA08]. 822 5.5. Non-Renegable SACK 824 This section discusses problems with SCTP's SACK mechanism and how it 825 affects the send buffer and CMT performance. 827 5.5.1. Problem Description 829 Gap-acks acknowledge DATA chunks that arrive out-of-order to a 830 transport layer data receiver. A gap-ack in SCTP is advisory, in 831 that, while it notifies a data sender about the reception of 832 indicated DATA chunks, the data receiver is permitted to later 833 discard DATA chunks that it previously had gap-acked. Discarding a 834 previously gap-acked DATA chunk is known as 'reneging'. Because of 835 the possibility of reneging in SCTP, any gap-acked DATA chunk MUST 836 NOT be removed from the data sender's retransmission queue until the 837 DATA chunk is later CumAcked. 839 Situations exist when a data receiver knows that reneging on a 840 particular out-of-order DATA chunk will never take place, such as 841 (but not limited to) after an out-of-order DATA chunk is delivered to 842 the receiving application. With current SACKs in SCTP, it is not 843 possible for a data receiver to inform a data sender if or when a 844 particular out-of-order 'deliverable' DATA chunk has been 'delivered' 845 to the receiving application. Thus the data sender MUST keep a copy 846 of every gap-acked out-of-order DATA chunk(s) in the data sender's 847 retransmission queue until the DATA chunk is CumAcked. This use of 848 the data sender's retransmission queue is wasteful. The wasted 849 buffer often degrades CMT performance; the degradation increases when 850 a CMT flow traverses via paths with disparate end-to-end properties 851 [NEY08]. 853 5.5.2. Solution: Non-Renegable SACKs 855 Non-Renegable Selective Acknowledgments (NR-SACKs) Section 4 are a 856 new kind of acknowledgements, extending SCTP's SACK chunk 857 functionalities. The NR-SACK chunk is an extension of the existing 858 SACK chunk. Several fields are identical, including the Cumulative 859 TSN Ack, the Advertised Receiver Window Credit (a_rwnd), and 860 Duplicate TSNs. These fields have the same semantics as described in 861 [RFC4960]. 863 NR-SACKs also identify out-of-order DATA chunks that a receiver 864 either: (1) has delivered to its receiving application, or (2) takes 865 full responsibility to eventually deliver to its receiving 866 application. These out-of-order DATA chunks are 'non-renegable.' 867 Non-Renegable data are reported in the NR Gap Ack Block field of the 868 NR-SACK chunk as described Section 4. We refer to non-renegable 869 selective acknowledgements as 'nr-gap-acks.' 871 When an out-of-order DATA chunk is nr-gap-acked, the data sender no 872 longer needs to keep that particular DATA chunk in its retransmission 873 queue, thus allowing the data sender to free up its buffer space 874 sooner than if the DATA chunk were only gap-acked. NR-SACKs improve 875 send buffer utilization and throughput for CMT flows [NEY08]. 877 6. Handling of Shared Bottlenecks 879 6.1. Introduction 881 CMT-SCTP assumes all paths to be disjoint. Since each path 882 independently uses a TCP-like congestion control, an SCTP association 883 using N paths over the same bottleneck acquires N times the bandwidth 884 of a concurrent TCP flow. This is clearly unfair. A reliable 885 detection of shared bottlenecks is impossible in arbitrary networks 886 like the Internet. Therefore, [ICC2012] [ConTEL2011], [AINA2010] 887 apply the idea of Resource Pooling to CMT-SCTP. Resource Pooling 888 (RP) denotes 'making a collection of resources behave like a single 889 pooled resource' [WHB09]. The modifications of RP-enabled CMT-SCTP, 890 further denoted as CMT/RP-SCTP, are described in the following 891 subsections. A detailed description of CMT/RP-SCTP, including 892 congestion control examples, can be found in [ICC2012], [ConTEL2011], 893 [AINA2010]. 895 6.2. Initial Values 897 TDB. 899 6.3. Congestion Window Growth 901 TDB. See [Dre2012], [ICC2012], [ConTEL2011]. 903 6.4. Congestion Window Decrease 905 TDB. See [Dre2012], [ICC2012], [ConTEL2011]. 907 7. Chunk Scheduling and Rescheduling 909 TDB. See [Dre2012], [PFLDNeT2010]. 911 8. Socket API Considerations 913 See [I-D.dreibholz-tsvwg-sctpsocket-multipath] and 914 [I-D.dreibholz-tsvwg-sctpsocket-sqinfo]. 916 9. Testbed Platforms 918 A large-scale and realistic Internet testbed platform with support 919 for the multi-homing feature of the underlying SCTP protocol is 920 NorNet. Particularly, it is also a platform for multi-path transport 921 experiments with CMT-SCTP. A description of and introduction to 922 NorNet is provided in [ComNets2013-Core], [PAMS2013-NorNet], 923 [Haikou2017-2-MultiPath], [Haikou2017-2-NorNet-Tutorial]. Further 924 information can be found on the project website [NorNet-Website] at 925 https://www.nntb.no. 927 An Open Source simulation model of CMT-SCTP is available for OMNeT++ 928 within the INET Framework. See [INET-Framework] for the Git 929 repository. For documentation on the model, together with 930 performance evaluations, see [Dre2012]. Some interesting performance 931 evaluations for delay-sensitive traffic with CMT-SCTP can be found in 932 [ComNets2016-MultipathSurvey]. 934 10. IANA Considerations 936 NOTE to RFC-Editor: 938 "RFCXXXX" is to be replaced by the RFC number you assign this 939 document. 941 NOTE to RFC-Editor: 943 The suggested values for the chunk type and the chunk parameter 944 types are tentative and to be confirmed by IANA. 946 This document (RFCXXXX) is the reference for all registrations 947 described in this section. The suggested changes are described 948 below. 950 10.1. A New Chunk Type 952 A chunk type has to be assigned by IANA. It is suggested to use the 953 values given in Section 4. IANA should assign this value from the 954 pool of chunks with the upper two bits set to '00'. 956 This requires an additional line in the "Chunk Types" registry for 957 SCTP: 959 Chunk Types 961 ID Value Chunk Type Reference 962 ----- ---------- --------- 963 16 Non-Renegable SACK (NR-SACK) [RFCXXXX] 965 The registration table as defined in [RFC6096] for the chunk flags of 966 this chunk type is empty. 968 11. Security Considerations 970 This document does not add any additional security considerations in 971 addition to the ones given in [RFC4960]. 973 12. Acknowledgments 975 The authors wish to thank Hakim Adhari, Phillip Conrad, Jonathan 976 Leighton, Ertugrul Yilmaz and Xing Zhou for their invaluable comments 977 and support. 979 13. References 981 13.1. Normative References 983 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 984 Requirement Levels", BCP 14, RFC 2119, 985 DOI 10.17487/RFC2119, March 1997, 986 . 988 [RFC4960] Stewart, R., Ed., "Stream Control Transmission Protocol", 989 RFC 4960, DOI 10.17487/RFC4960, September 2007, 990 . 992 [RFC5061] Stewart, R., Xie, Q., Tuexen, M., Maruyama, S., and M. 993 Kozuka, "Stream Control Transmission Protocol (SCTP) 994 Dynamic Address Reconfiguration", RFC 5061, 995 DOI 10.17487/RFC5061, September 2007, 996 . 998 [RFC5351] Lei, P., Ong, L., Tuexen, M., and T. Dreibholz, "An 999 Overview of Reliable Server Pooling Protocols", RFC 5351, 1000 DOI 10.17487/RFC5351, September 2008, 1001 . 1003 [RFC6096] Tuexen, M. and R. Stewart, "Stream Control Transmission 1004 Protocol (SCTP) Chunk Flags Registration", RFC 6096, 1005 DOI 10.17487/RFC6096, January 2011, 1006 . 1008 [I-D.ietf-tsvwg-sctp-failover] 1009 Nishida, Y., Natarajan, P., Caro, A., Amer, P. D., and K. 1010 E. E. Nielsen, "SCTP-PF: A Quick Failover Algorithm for 1011 the Stream Control Transmission Protocol", Work in 1012 Progress, Internet-Draft, draft-ietf-tsvwg-sctp-failover- 1013 16, 17 February 2016, . 1016 [I-D.dreibholz-tsvwg-sctpsocket-multipath] 1017 Dreibholz, T., Becke, M., and H. Adhari, "SCTP Socket API 1018 Extensions for Concurrent Multipath Transfer", Work in 1019 Progress, Internet-Draft, draft-dreibholz-tsvwg- 1020 sctpsocket-multipath-23, 6 September 2021, 1021 . 1024 [I-D.dreibholz-tsvwg-sctpsocket-sqinfo] 1025 Dreibholz, T., Seggelmann, R., and M. Becke, "Sender Queue 1026 Info Option for the SCTP Socket API", Work in Progress, 1027 Internet-Draft, draft-dreibholz-tsvwg-sctpsocket-sqinfo- 1028 23, 6 September 2021, . 1031 13.2. Informative References 1033 [I06] Iyengar, J., "End-to-End Concurrent Multipath Transfer 1034 Using Transport Layer Multihoming", PhD 1035 Dissertation Computer Science Dept., University of 1036 Delaware, April 2006, 1037 . 1040 [IAS06] Iyengar, J., Amer, P. D., and R. R. Stewart, "Concurrent 1041 Multipath Transfer Using SCTP Multihoming Over Independent 1042 End-to-End Paths", Journal IEEE/ACM Transactions on 1043 Networking, October 2006, 1044 . 1047 [NEA08] Natarajan, P., Ekiz, N., Iyengar, J., Amer, P., and R. 1048 Stewart, "Concurrent Multipath Transfer Using Transport 1049 Layer Multihoming: Introducing the Potentially-failed 1050 Destination State", Proceedings of the IFIP Networking, 1051 May 2008, 1052 . 1055 [NEY08] Natarajan, P., Ekiz, N., Yilmaz, E., Amer, P., Iyengar, 1056 J., and R. Stewart, "Non-Renegable Selective 1057 Acknowledgments (NR-SACKs) for SCTP", Proceedings of the 1058 16th IEEE International Conference on Network Protocols 1059 (ICNP) , October 2008, 1060 . 1062 [WHB09] Wischik, D., Handley, M., and M. B. Braun, "The Resource 1063 Pooling Principle", Journal ACM SIGCOMM Computer 1064 Communication Review, October 2009, 1065 . 1068 [OMNeTWorkshop2010-SCTP] 1069 Dreibholz, T., Becke, M., Pulinthanath, J., and E. P. 1070 Rathgeb, "Implementation and Evaluation of Concurrent 1071 Multipath Transfer for SCTP in the INET Framework", 1072 Proceedings of the 3rd ACM/ICST International Workshop on 1073 OMNeT++ ISBN 978-963-9799-87-5, 1074 DOI 10.4108/ICST.SIMUTOOLS2010.8673, 19 March 2010, 1075 . 1078 [AINA2010] Dreibholz, T., Becke, M., Pulinthanath, J., and E. P. 1079 Rathgeb, "Applying TCP-Friendly Congestion Control to 1080 Concurrent Multipath Transfer", Proceedings of the 24th 1081 IEEE International Conference on Advanced Information 1082 Networking and Applications (AINA) Pages 312-319, 1083 ISBN 978-0-7695-4018-4, DOI 10.1109/AINA.2010.117, 21 1084 April 2010, . 1088 [YEN10] Yilmaz, E., Ekiz, N., Natarajan, P., Amer, P., Leighton, 1089 J., Baker, F., and R. Stewart, "Throughput Analysis of 1090 Non-Renegable Selective Acknowledgments (NR-SACKs) for 1091 SCTP", Computer 1092 Communications, doi:10.1016/j.comcom.2010.06.028, 2010. 1094 [PFLDNeT2010] 1095 Dreibholz, T., Seggelmann, R., Tüxen, M., and E. P. 1096 Rathgeb, "Transmission Scheduling Optimizations for 1097 Concurrent Multipath Transfer", Proceedings of the 8th 1098 International Workshop on Protocols for Future, Large- 1099 Scale and Diverse Network Transports (PFLDNeT) Volume 8, 1100 ISSN 2074-5168, 29 November 2010, . 1104 [Globecom2010] 1105 Dreibholz, T., Becke, M., Rathgeb, E. P., and M. Tüxen, 1106 "On the Use of Concurrent Multipath Transfer over 1107 Asymmetric Paths", Proceedings of the IEEE Global 1108 Communications 1109 Conference (GLOBECOM) ISBN 978-1-4244-5637-6, 1110 DOI 10.1109/GLOCOM.2010.5683579, 7 December 2010, 1111 . 1114 [PAMS2011] Adhari, H., Dreibholz, T., Becke, M., Rathgeb, E. P., and 1115 M. Tüxen, "Evaluation of Concurrent Multipath Transfer 1116 over Dissimilar Paths", Proceedings of the 1st 1117 International Workshop on Protocols and Applications with 1118 Multi-Homing Support (PAMS) Pages 708-714, 1119 ISBN 978-0-7695-4338-3, DOI 10.1109/WAINA.2011.92, 22 1120 March 2011, . 1124 [ConTEL2011] 1125 Dreibholz, T., Becke, M., Adhari, H., and E. P. Rathgeb, 1126 "On the Impact of Congestion Control for Concurrent 1127 Multipath Transfer on the Transport Layer", Proceedings of 1128 the 11th IEEE International Conference on 1129 Telecommunications (ConTEL) Pages 397-404, 1130 ISBN 978-953-184-152-8, 16 June 2011, 1131 . 1134 [ICC2012] Becke, M., Dreibholz, T., Adhari, H., and E. P. Rathgeb, 1135 "On the Fairness of Transport Protocols in a Multi-Path 1136 Environment", Proceedings of the IEEE International 1137 Conference on Communications (ICC) Pages 2666-2672, 1138 ISBN 978-1-4577-2052-9, DOI 10.1109/ICC.2012.6363695, 12 1139 June 2012, . 1142 [ComNets2016-MultipathSurvey] 1143 Yedugundla, K. V., Ferlin, S., Dreibholz, T., Alay, Ö., 1144 Kuhn, N., Hurtig, P., and A. Brunström, "Is Multi-Path 1145 Transport Suitable for Latency Sensitive Traffic?", 1146 Computer Networks Volume 105, Pages 1-21, ISSN 1389-1286, 1147 DOI 10.1016/j.comnet.2016.05.008, 4 August 2016, 1148 . 1151 [Dre2012] Dreibholz, T., "Evaluation and Optimisation of Multi-Path 1152 Transport using the Stream Control Transmission Protocol", 1153 Habilitation Treatise, 13 March 2012, 1154 . 1158 [NorNet-Website] 1159 Dreibholz, T., "NorNet -- A Real-World, Large-Scale Multi- 1160 Homing Testbed", Online: https://www.nntb.no/, 2016, 1161 . 1163 [PAMS2013-NorNet] 1164 Dreibholz, T. and E. G. Gran, "Design and Implementation 1165 of the NorNet Core Research Testbed for Multi-Homed 1166 Systems", Proceedings of the 3nd International Workshop on 1167 Protocols and Applications with Multi-Homing 1168 Support (PAMS) Pages 1094-1100, ISBN 978-0-7695-4952-1, 1169 DOI 10.1109/WAINA.2013.71, 27 March 2013, 1170 . 1174 [ComNets2013-Core] 1175 Gran, E. G., Dreibholz, T., and A. Kvalbein, "NorNet Core 1176 - A Multi-Homed Research Testbed", Computer Networks, 1177 Special Issue on Future Internet Testbeds Volume 61, Pages 1178 75-87, ISSN 1389-1286, DOI 10.1016/j.bjp.2013.12.035, 14 1179 March 2014, 1180 . 1182 [INET-Framework] 1183 Hornig, R. and A. Varga, "INET Framework Git Repository", 1184 Online: https://github.com/inet-framework/inet, 2016, 1185 . 1187 [Haikou2017-2-MultiPath] 1188 Dreibholz, T., "An Introduction to Multi-Path Transport at 1189 Hainan University", Keynote Talk at Hainan University, 1190 College of Information Science and Technology (CIST), 14 1191 December 2017, . 1194 [Haikou2017-2-NorNet-Tutorial] 1195 Dreibholz, T., "NorNet Core Beginner Tutorial at Hainan 1196 University", Tutorial at Hainan University, College of 1197 Information Science and Technology (CIST), 15 December 1198 2017, . 1201 Authors' Addresses 1203 Paul D. Amer 1204 University of Delaware, Computer and Information Sciences Department 1205 Newark, Delaware 19716 1206 United States of America 1208 Phone: +1-302-831-1944 1209 Email: amer@cis.udel.edu 1210 URI: https://www.eecis.udel.edu/~amer/ 1212 Martin Becke 1213 HAW Hamburg, Informatics Department 1214 Berliner Tor 7 1215 20099 Hamburg 1216 Germany 1218 Phone: +49-40-42875-8104 1219 Email: martin.becke@haw-hamburg.de 1220 URI: http://www.scimbe.de/about.html 1222 Thomas Dreibholz 1223 Simula Metropolitan Centre for Digital Engineering 1224 Pilestredet 52 1225 0167 Oslo 1226 Norway 1228 Phone: +47-6782-8200 1229 Email: dreibh@simula.no 1230 URI: https://www.simula.no/people/dreibh 1231 Nasif Ekiz 1232 University of Delaware, Computer and Information Sciences Department 1233 Newark, Delaware 19716 1234 United States of America 1236 Email: nekiz@udel.edu 1238 Janardhan Iyengar 1239 Franklin and Marshall College, Mathematics and Computer Science 1240 PO Box 3003 1241 Lancaster, Pennsylvania 17604-3003 1242 United States of America 1244 Phone: +1-717-358-4774 1245 Email: jiyengar@fandm.edu 1247 Preethi Natarajan 1248 Cisco Systems 1249 425 East Tasman Drive 1250 San Jose, California 95134 1251 United States of America 1253 Email: prenatar@cisco.com 1255 Randall R. Stewart 1256 Netflix 1257 Chapin, South Carolina 29036 1258 United States of America 1260 Email: randall@lakerest.net 1262 Michael Tuexen 1263 Muenster University of Applied Sciences 1264 Stegerwaldstrasse 39 1265 48565 Steinfurt 1266 Germany 1268 Email: tuexen@fh-muenster.de 1269 URI: https://www.fh-muenster.de/fb2/personen/professoren/tuexen/