idnits 2.17.1 draft-tuexen-tsvwg-sctp-multipath-22.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There are 6 instances of lines with non-ascii characters in the document. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (9 August 2021) is 990 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Experimental ---------------------------------------------------------------------------- == Missing Reference: 'RFCXXXX' is mentioned on line 961, but not defined ** Obsolete normative reference: RFC 4960 (Obsoleted by RFC 9260) ** Obsolete normative reference: RFC 6096 (Obsoleted by RFC 9260) == Outdated reference: A later version (-28) exists of draft-dreibholz-tsvwg-sctpsocket-multipath-22 == Outdated reference: A later version (-28) exists of draft-dreibholz-tsvwg-sctpsocket-sqinfo-22 Summary: 2 errors (**), 0 flaws (~~), 5 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group P. D. Amer 3 Internet-Draft University of Delaware 4 Intended status: Experimental M. Becke 5 Expires: 10 February 2022 HAW Hamburg 6 T. Dreibholz 7 SimulaMet 8 N. Ekiz 9 University of Delaware 10 J. Iyengar 11 Franklin and Marshall College 12 P. Natarajan 13 Cisco Systems 14 R. R. Stewart 15 Netflix 16 M. Tuexen 17 Muenster Univ. of Appl. Sciences 18 9 August 2021 20 Load Sharing for the Stream Control Transmission Protocol (SCTP) 21 draft-tuexen-tsvwg-sctp-multipath-22 23 Abstract 25 The Stream Control Transmission Protocol (SCTP) supports multi-homing 26 for providing network fault tolerance. However, mainly one path is 27 used for data transmission. Only timer-based retransmissions are 28 carried over other paths as well. 30 This document describes how multiple paths can be used simultaneously 31 for transmitting user messages. 33 Status of This Memo 35 This Internet-Draft is submitted in full conformance with the 36 provisions of BCP 78 and BCP 79. 38 Internet-Drafts are working documents of the Internet Engineering 39 Task Force (IETF). Note that other groups may also distribute 40 working documents as Internet-Drafts. The list of current Internet- 41 Drafts is at https://datatracker.ietf.org/drafts/current/. 43 Internet-Drafts are draft documents valid for a maximum of six months 44 and may be updated, replaced, or obsoleted by other documents at any 45 time. It is inappropriate to use Internet-Drafts as reference 46 material or to cite them other than as "work in progress." 48 This Internet-Draft will expire on 10 February 2022. 50 Copyright Notice 52 Copyright (c) 2021 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 57 license-info) in effect on the date of publication of this document. 58 Please review these documents carefully, as they describe your rights 59 and restrictions with respect to this document. Code Components 60 extracted from this document must include Simplified BSD License text 61 as described in Section 4.e of the Trust Legal Provisions and are 62 provided without warranty as described in the Simplified BSD License. 64 Table of Contents 66 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 67 2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 3 68 3. Load Sharing . . . . . . . . . . . . . . . . . . . . . . . . 3 69 3.1. Split Fast Retransmissions . . . . . . . . . . . . . . . 4 70 3.2. Appropriate Congestion Window Growth . . . . . . . . . . 4 71 3.3. Appropriate Delayed Acknowledgements . . . . . . . . . . 5 72 4. Non-Renegable SACK . . . . . . . . . . . . . . . . . . . . . 6 73 4.1. Negotiation . . . . . . . . . . . . . . . . . . . . . . . 6 74 4.2. The New Chunk Type: Non-Renegable SACK (NR-SACK) . . . . 6 75 4.3. An Illustrative Example . . . . . . . . . . . . . . . . . 11 76 4.4. Procedures . . . . . . . . . . . . . . . . . . . . . . . 15 77 4.4.1. Sending an NR-SACK chunk . . . . . . . . . . . . . . 15 78 4.4.2. Receiving an NR-SACK Chunk . . . . . . . . . . . . . 17 79 5. Buffer Blocking Mitigation . . . . . . . . . . . . . . . . . 18 80 5.1. Sender Buffer Splitting . . . . . . . . . . . . . . . . . 18 81 5.2. Receiver Buffer Splitting . . . . . . . . . . . . . . . . 18 82 5.3. Chunk Rescheduling . . . . . . . . . . . . . . . . . . . 18 83 5.4. Problems during Path Failure . . . . . . . . . . . . . . 18 84 5.4.1. Problem Description . . . . . . . . . . . . . . . . . 18 85 5.4.2. Solution: Potentially-failed Destination State . . . 19 86 5.5. Non-Renegable SACK . . . . . . . . . . . . . . . . . . . 19 87 5.5.1. Problem Description . . . . . . . . . . . . . . . . . 19 88 5.5.2. Solution: Non-Renegable SACKs . . . . . . . . . . . . 20 89 6. Handling of Shared Bottlenecks . . . . . . . . . . . . . . . 20 90 6.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 21 91 6.2. Initial Values . . . . . . . . . . . . . . . . . . . . . 21 92 6.3. Congestion Window Growth . . . . . . . . . . . . . . . . 21 93 6.4. Congestion Window Decrease . . . . . . . . . . . . . . . 21 94 7. Chunk Scheduling and Rescheduling . . . . . . . . . . . . . . 21 95 8. Socket API Considerations . . . . . . . . . . . . . . . . . . 21 96 9. Testbed Platforms . . . . . . . . . . . . . . . . . . . . . . 21 97 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 98 10.1. A New Chunk Type . . . . . . . . . . . . . . . . . . . . 22 99 11. Security Considerations . . . . . . . . . . . . . . . . . . . 22 100 12. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 23 101 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 23 102 13.1. Normative References . . . . . . . . . . . . . . . . . . 23 103 13.2. Informative References . . . . . . . . . . . . . . . . . 24 104 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 27 106 1. Introduction 108 One of the important features of the Stream Control Transmission 109 Protocol (SCTP), which is currently specified in [RFC4960], is 110 network fault tolerance. This feature is for example required for 111 Reliable Server Pooling (RSerPool, [RFC5351]). Therefore, 112 transmitting messages over multiple paths is supported, but only for 113 redundancy. So [RFC4960] does not specify how to use multiple paths 114 simultaneously. 116 This document overcomes this limitation by specifying how multiple 117 paths can be used simultaneously. This has several benefits: 119 * Improved bandwidth usage. 121 * Better availability check with real user messages compared to 122 HEARTBEAT-based information. 124 2. Conventions 126 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 127 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 128 document are to be interpreted as described in [RFC2119]. 130 3. Load Sharing 132 Basic requirement for applying SCTP load sharing is the Concurrent 133 Multipath Transfer (CMT) extension of SCTP, which utilises multiple 134 paths simultaneously. We denote CMT-enabled SCTP as CMT-SCTP 135 throughout this document. CMT-SCTP is introduced in [IAS06] and in 136 more detail in [I06], some illustrative examples of chunk handling 137 are provided in [OMNeTWorkshop2010-SCTP]. CMT-SCTP provides three 138 modifications to standard SCTP (split Fast Retransmissions, 139 appropriate congestion window growth and delayed SACKs), which are 140 described in the following subsections. 142 3.1. Split Fast Retransmissions 144 Paths with different latencies lead to overtaking of DATA chunks. 145 This leads to gap reports, which are handled by Fast Retransmissions. 146 However, due to the fact that multiple paths are used simultaneously, 147 these Fast Retransmissions are usually useless and furthermore lead 148 to a decreased congestion window size. 150 To avoid unnecessary Fast Retransmissions, the sender has to keep 151 track of the path each DATA chunk has been sent on and consider 152 transmission paths before performing Fast Retransmissions. That is, 153 on reception of a SACK, the sender MUST identify the highest 154 acknowledged TSN on each path. A chunk SHOULD only be considered as 155 missing if its TSN is smaller than the highest acknowledged TSN on 156 its path. Section 3.1 of [OMNeTWorkshop2010-SCTP] contains an 157 illustrated example. 159 3.2. Appropriate Congestion Window Growth 161 The congestion window adaptation algorithm for SCTP [RFC4960] allows 162 increasing the congestion window only when a new cumulative ack 163 (CumAck) is received by a sender. When SACKs with unchanged CumAcks 164 are generated (due to reordering) and later arrive at a sender, the 165 sender does not modify its congestion window. Since a CMT-SCTP 166 receiver naturally observes reordering, many SACKs are sent 167 containing new gap reports but not new CumAcks. When these gaps are 168 later acked by a new CumAck, congestion window growth occurs, but 169 only for the data newly acked in the most recent SACK. Data 170 previously acked through gap reports will not contribute to 171 congestion window growth, in order to prevent sudden increases in the 172 congestion window resulting in bursts of data being sent. 174 To overcome the problems described above, the congestion window 175 growth has to be handled as follows [IAS06]: 177 * The sender SHOULD keep track of the earliest non-retransmitted 178 outstanding TSN per path. 180 * The sender SHOULD keep track of the earliest retransmitted 181 outstanding TSN per path. 183 * The in-order delivery per path SHOULD be deduced. 185 * The congestion window of a path SHOULD be increased when the 186 earliest non-retransmitted outstanding TSN of this path is 187 advanced ('Pseudo CumAck') OR when the earliest retransmitted 188 outstanding TSN of this path is advanced ('RTX Pseudo CumAck'). 190 Section 3.2 of [OMNeTWorkshop2010-SCTP] contains an illustrated 191 example of appropriate congestion window handling for CMT-SCTP. 193 3.3. Appropriate Delayed Acknowledgements 195 Standard SCTP [RFC4960] sends a SACK as soon as an out-of-sequence 196 TSN has been received. Delayed Acknowledgements are only allowed if 197 the received TSNs are in sequence. However, due to the load 198 balancing of CMT-SCTP, DATA chunks may overtake each other. This 199 leads to a high number of out-of-sequence TSNs, which have to be 200 acknowledged immediately. Clearly, this behaviour increases the 201 overhead traffic (usually nearly one SACK chunk for each received 202 packet containing a DATA chunk). 204 Delayed Acknowledgements for CMT-SCTP are handled as follows: 206 * In addition to [RFC4960], delaying of SACKs SHOULD *also* be 207 applied for out-of-sequence TSNs. 209 * A receiver MUST maintain a counter for the number of DATA chunks 210 received before sending a SACK. The value of the counter is 211 stored into each SACK chunk (FIXME: add details; needs reservation 212 of flags bits by IANA). After transmitting a SACK, the counter 213 MUST be reset to 0. Its initial value MUST be 0. 215 * The SACK handling procedure for a missing TSN M is extended as 216 follows: 218 - If all newly acknowledged TSNs have been transmitted over the 219 same path: 221 o If there are newly acknowledged TSNs L and H so that L <= M 222 <= H, the missing count of TSN M SHOULD be incremented by 223 one (like for standard SCTP according to [RFC4960]). 225 o Else if all newly acknowledged TSNs N satisfy the condition 226 M <= N, the missing count of TSN M SHOULD be incremented by 227 the number of TSNs reported in the SACK chunk. 229 - Otherwise (that is, there are newly acknowledged TSNs on 230 different paths), the missing count of TSN M SHOULD be 231 incremented by one (like for standard SCTP according to 232 [RFC4960]). 234 Section 3.3 of [OMNeTWorkshop2010-SCTP] contains an illustrated 235 example of Delayed Acknowledgements for CMT-SCTP. 237 4. Non-Renegable SACK 239 4.1. Negotiation 241 Before sending/receiving NR-SACKs (see [YEN10]), both peer endpoints 242 MUST agree on using NR-SACKs. This agreement MUST be negotiated 243 during association establishment. NR-SACK is an extension to the 244 core SCTP, and SCTP extensions that an endpoint supports are reported 245 to the peer endpoint in Supported Extensions Parameter during 246 association establishment (see Section 4.2.7 of [RFC5061].) The 247 Supported Extensions Parameter consists of a list of non-standard 248 Chunk Types that are supported by the sender. 250 An endpoint supporting the NR-SACK extension MUST list the NR-SACK 251 chunk in the Supported Extensions Parameter carried in the INIT or 252 INIT-ACK chunk, depending on whether the endpoint initiates or 253 responds to the initiation of the association. If the NR-SACK chunk 254 type ID is listed in the Chunk Types List of the Supported Extensions 255 Parameter, then the receiving endpoint MUST assume that the NR-SACK 256 chunk is supported by the sending endpoint. 258 Both endpoints MUST support NR-SACKs for either endpoint to send an 259 NR-SACK. If an endpoint establishes an association with a remote 260 endpoint that does not list NR-SACK in the Supported Extensions 261 Parameter carried in INIT chunk, then both endpoints of the 262 association MUST NOT use NR-SACKs. After association establishment, 263 an endpoint MUST NOT renegotiate the use of NR-SACKs. 265 Once both endpoints indicate during association establishment that 266 they support the NR-SACK extension, each endpoint SHOULD acknowledge 267 received DATA chunks with NR-SACK chunks, and not SACK chunks. That 268 is, throughout an SCTP association, both endpoints SHOULD send either 269 SACK chunks or NR-SACK chunks, never a mixture of the two. 271 4.2. The New Chunk Type: Non-Renegable SACK (NR-SACK) 273 Table 1 illustrates a new chunk type that will be used to transfer 274 NR-SACK information. 276 Chunk Type Chunk Name 277 -------------------------------------------------------------- 278 0x10 Non-Renegable Selective Acknowledgment (NR-SACK) 280 Table 1: NR-SACK Chunk 282 As the NR-SACK chunk replaces the SACK chunk, many SACK chunk fields 283 are preserved in the NR-SACK chunk. These preserved fields have the 284 same semantics with the corresponding SACK chunk fields, as defined 285 in [RFC4960], Section 3.3.4. The Gap Ack fields from RFC4960 have 286 been renamed as R Gap Ack to emphasize their renegable nature. Their 287 semantics are unchanged. For completeness, we describe all fields of 288 the NR-SACK chunk, including those that are identical in the SACK 289 chunk. 291 Similar to the SACK chunk, the NR-SACK chunk is sent to a peer 292 endpoint to (1) acknowledge DATA chunks received in-order, (2) 293 acknowledge DATA chunks received out-of-order, and (3) identify DATA 294 chunks received more than once (i.e., duplicate.) In addition, the 295 NR-SACK chunk (4) informs the peer endpoint of non-renegable out-of- 296 order DATA chunks. 298 0 1 2 3 299 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 300 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 301 | Type = 0x10 | Chunk Flags | Chunk Length | 302 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 303 | Cumulative TSN Ack | 304 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 305 | Advertised Receiver Window Credit (a_rwnd) | 306 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 307 |Number of R Gap Ack Blocks = N |Number of NR Gap Ack Blocks = M| 308 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 309 | Number of Duplicate TSNs = X | Reserved | 310 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 311 | R Gap Ack Block #1 Start | R Gap Ack Block #1 End | 312 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 313 / / 314 \ ... \ 315 / / 316 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 317 | R Gap Ack Block #N Start | R Gap Ack Block #N End | 318 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 319 | NR Gap Ack Block #1 Start | NR Gap Ack Block #1 End | 320 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 321 / / 322 \ ... \ 323 / / 324 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 325 | NR Gap Ack Block #M Start | NR Gap Ack Block #M End | 326 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 327 | Duplicate TSN 1 | 328 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 329 / / 330 \ ... \ 331 / / 332 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 333 | Duplicate TSN X | 334 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 336 Type: 8 bits 338 This field holds the IANA defined chunk type for NR-SACK chunk. The 339 suggested value of this field for IANA is 0x10. 341 Chunk Flags: 8 bits 343 Currently not used. It is recommended a sender set all bits to zero 344 on transmit, and a receiver ignore this field. 346 Chunk Length: 16 bits (unsigned integer) [Same as SACK chunk] 348 This value represents the size of the chunk in bytes including the 349 Chunk Type, Chunk Flags, Chunk Length, and Chunk Value fields. 351 Cumulative TSN Ack: 32 bits (unsigned integer) [Same as SACK chunk] 353 The value of the Cumulative TSN Ack is the last TSN received before a 354 break in the sequence of received TSNs occurs. The next TSN value 355 following the Cumulative TSN Ack has not yet been received at the 356 endpoint sending the NR-SACK. 358 Advertised Receiver Window Credit (a_rwnd): 32 bits (unsigned 359 integer) [Same as SACK chunk] 361 Indicates the updated receive buffer space in bytes of the sender of 362 this NR-SACK, see Section 6.2.1 of [RFC4960] for details. 364 Number of (R)enegable Gap Ack Blocks (N): 16 bits (unsigned integer) 366 Indicates the number of Renegable Gap Ack Blocks included in this NR- 367 SACK. 369 Number of (N)on(R)enegable Gap Ack Blocks (M): 16 bits (unsigned 370 integer) 372 Indicates the number of Non-Renegable Gap Ack Blocks included in this 373 NR-SACK. 375 Number of Duplicate TSNs (X): 16 bits [Same as SACK chunk] 377 Contains the number of duplicate TSNs the endpoint has received. 378 Each duplicate TSN is listed following the NR Gap Ack Block list. 380 Reserved : 16 bits 382 Currently not used. It is recommended a sender set all bits to zero 383 on transmit, and a receiver ignore this field. 385 (R)enegable Gap Ack Blocks: 387 The NR-SACK contains zero or more R Gap Ack Blocks. Each R Gap Ack 388 Block acknowledges a subsequence of renegable out-of-order TSNs. By 389 definition, all TSNs acknowledged by R Gap Ack Blocks are "greater 390 than" the value of the Cumulative TSN Ack. 392 Because of TSN numbering wraparound, comparisons and all arithmetic 393 operations discussed in this document are based on "Serial Number 394 Arithmetic" as described in Section 1.6 of [RFC4960]. 396 R Gap Ack Blocks are repeated for each R Gap Ack Block up to 'N' 397 defined in the Number of R Gap Ack Blocks field. All DATA chunks 398 with TSNs >= (Cumulative TSN Ack + R Gap Ack Block Start) and <= 399 (Cumulative TSN Ack + R Gap Ack Block End) of each R Gap Ack Block 400 are assumed to have been received correctly, and are renegable. 402 R Gap Ack Block Start: 16 bits (unsigned integer) 404 Indicates the Start offset TSN for this R Gap Ack Block. This number 405 is set relative to the cumulative TSN number defined in Cumulative 406 TSN Ack field. To calculate the actual start TSN number, the 407 Cumulative TSN Ack is added to this offset number. The calculated 408 TSN identifies the first TSN in this R Gap Ack Block that has been 409 received. 411 R Gap Ack Block End: 16 bits (unsigned integer) 413 Indicates the End offset TSN for this R Gap Ack Block. This number 414 is set relative to the cumulative TSN number defined in the 415 Cumulative TSN Ack field. To calculate the actual TSN number, the 416 Cumulative TSN Ack is added to this offset number. The calculated 417 TSN identifies the TSN of the last DATA chunk received in this R Gap 418 Ack Block. 420 N(on)R(enegable) Gap Ack Blocks: 422 The NR-SACK contains zero or more NR Gap Ack Blocks. Each NR Gap Ack 423 Block acknowledges a continuous subsequence of non-renegable out-of- 424 order DATA chunks. If a TSN is nr-gap-acked in any NR-SACK chunk, 425 then all subsequently transmitted NR-SACKs with a smaller cum-ack 426 value than that TSN SHOULD also nr-gap-ack that TSN. 428 NR Gap Ack Blocks are repeated for each NR Gap Ack Block up to 'M' 429 defined in the Number of NR Gap Ack Blocks field. All DATA chunks 430 with TSNs >= (Cumulative TSN Ack + NR Gap Ack Block Start) and <= 431 (Cumulative TSN Ack + NR Gap Ack Block End) of each NR Gap Ack Block 432 are assumed to be received correctly, and are Non-Renegable. 434 NR Gap Ack Block Start: 16 bits (unsigned integer) 435 Indicates the Start offset TSN for this NR Gap Ack Block. This 436 number is set relative to the cumulative TSN number defined in 437 Cumulative TSN Ack field. To calculate the actual TSN number, the 438 Cumulative TSN Ack is added to this offset number. The calculated 439 TSN identifies the first TSN in this NR Gap Ack Block that has been 440 received. 442 NR Gap Ack Block End: 16 bits (unsigned integer) 444 Indicates the End offset TSN for this NR Gap Ack Block. This number 445 is set relative to the cumulative TSN number defined in Cumulative 446 TSN Ack field. To calculate the actual TSN number, the Cumulative 447 TSN Ack is added to this offset number. The calculated TSN 448 identifies the TSN of the last DATA chunk received in this NR Gap Ack 449 Block. 451 Note: 453 NR Gap Ack Blocks and R Gap Ack Blocks in an NR-SACK chunk SHOULD 454 acknowledge disjoint sets of TSNs. That is, an out-of-order TSN 455 SHOULD be listed in either an R Gap Ack Block or an NR Gap Ack Block, 456 but not the both. R Gap Ack Blocks and NR Gap Ack Blocks together 457 provide the information as do the Gap Ack Block of a SACK chunk, plus 458 additional information about non-renegability. 460 If all out-of-order data acked by an NR-SACK are renegable, then the 461 Number of NR Gap Ack Blocks MUST be set to 0. If all out-of-order 462 data acked by an NR-SACK are non-renegable, then the Number of R Gap 463 Ack Blocks SHOULD be set to 0. TSNs listed in R Gap Ack Block will 464 be referred as r-gap-acked. 466 Duplicate TSN: 32 bits (unsigned integer) [Same as SACK chunk] 468 Indicates a duplicate TSN received since the last NR-SACK was sent. 469 Exactly 'X' duplicate TSNs SHOULD be reported where 'X' was defined 470 in Number of Duplicate TSNs field. 472 Each duplicate TSN is listed in this field as many times as the TSN 473 was received since the previous NR-SACK was sent. For example, if a 474 data receiver were to get the TSN 19 three times, the data receiver 475 would list 19 twice in the outbound NR-SACK. After sending the NR- 476 SACK if the receiver received one more TSN 19, the receiver would 477 list 19 as a duplicate once in the next outgoing NR-SACK. 479 4.3. An Illustrative Example 481 Assume the following DATA chunks have arrived at the receiver. 483 -------------------------------- 484 | TSN=16| SID=2 | SSN=N/A| U=1 | 485 -------------------------------- 486 | TSN=15| SID=1 | SSN= 4 | U=0 | 487 -------------------------------- 488 | TSN=14| SID=0 | SSN= 4 | U=0 | 489 -------------------------------- 490 | TSN=13| SID=2 | SSN=N/A| U=1 | 491 -------------------------------- 492 | | 493 -------------------------------- 494 | TSN=11| SID=0 | SSN= 3 | U=0 | 495 ------------------------------- 496 | | 497 -------------------------------- 498 | | 499 -------------------------------- 500 | TSN=8 | SID=2 | SSN=N/A| U=1 | 501 -------------------------------- 502 | TSN=7 | SID=1 | SSN= 2 | U=0 | 503 -------------------------------- 504 | TSN=6 | SID=1 | SSN= 1 | U=0 | 505 -------------------------------- 506 | TSN=5 | SID=0 | SSN= 1 | U=0 | 507 -------------------------------- 508 | | 509 -------------------------------- 510 | TSN=3 | SID=1 | SSN= 0 | U=0 | 511 -------------------------------- 512 | TSN=2 | SID=0 | SSN= 0 | U=0 | 513 -------------------------------- 515 The above figure shows the list of DATA chunks at the receiver. TSN 516 denotes the transmission sequence number of the DATA chunk, SID 517 denotes the stream id to which the DATA chunk belongs, SSN denotes 518 the sequence number of the DATA chunk within its stream, and the U 519 bit denotes whether the DATA chunk requires ordered(=0) or 520 unordered(=1) delivery [RFC4960]. Note that TSNs 4,9,10, and 12 have 521 not arrived. 523 This data can be viewed as three separate streams as follows (assume 524 each stream begins with SSN=0.) Note that in this example, the 525 application uses stream 2 for unordered data transfer. By 526 definition, SSN fields of unordered DATA chunks are ignored. 528 Stream-0: 530 SSN: 0 1 2 3 4 531 TSN: | 2 | 5 | | 11 | 14 | 532 U-Bit: | 0 | 0 | | 0 | 0 | 534 Stream-1: 536 SSN: 0 1 2 3 4 537 TSN: | 3 | 6 | 7 | | 15 | 538 U-Bit: | 0 | 0 | 0 | | 0 | 540 Stream-2: 542 SSN: N/A N/A N/A 543 TSN: | 8 | 13 | 16 | 544 U-Bit: | 1 | 1 | 1 | 546 The NR-SACK to acknowledge the above data SHOULD be constructed as 547 follows for each of the three cases described below (the a_rwnd is 548 arbitrarily set to 4000): 550 CASE-1: Minimal Data Receiver Responsibility - no out-of-order 551 deliverable data yet delivered 553 None of the deliverable out-of-order DATA chunks have been delivered, 554 and the receiver of the above data does not take responsibility for 555 any of the received out-of-order DATA chunks. The receiver reserves 556 the right to renege any or all of the out-of-order DATA chunks. 558 +-----------------------------+-----------------------------+ 559 | Type = 0x10 | 00000000 | Chunk Length = 32 | 560 +-----------------------------+-----------------------------+ 561 | Cumulative TSN Ack = 3 | 562 +-----------------------------+-----------------------------+ 563 | a_rwnd = 4000 | 564 +-----------------------------+-----------------------------+ 565 | Num of R Gap Ack Blocks = 3 |Num of NR Gap Ack Blocks = 0 | 566 +-----------------------------+-----------------------------+ 567 | Num of Duplicates = 0 | 0x00 | 568 +-----------------------------+-----------------------------+ 569 |R Gap Ack Block #1 Start = 2 | R Gap Ack Block #1 End = 5 | 570 +-----------------------------+-----------------------------+ 571 |R Gap Ack Block #2 Start = 8 | R Gap Ack Block #2 End = 8 | 572 +-----------------------------+-----------------------------+ 573 |R Gap Ack Block #3 Start = 10| R Gap Ack Block #3 End = 13 | 574 +-----------------------------+-----------------------------+ 576 CASE-2: Minimal Data Receiver Responsibility - all out-of-order 577 deliverable data delivered 578 In this case, the NR-SACK chunk is being sent after the data receiver 579 has delivered all deliverable out-of-order DATA chunks to its 580 receiving application(i.e., TSNs 5,6,7,8,13, and 16.) The receiver 581 reserves the right to renege on all undelivered out-of-order DATA 582 chunks(i.e., TSNs 11,14, and 15.) 584 +------------------------------+------------------------------+ 585 | Type = 0x10 | 0x00 | Chunk Length = 40 | 586 +------------------------------+------------------------------+ 587 | Cumulative TSN Ack = 3 | 588 +------------------------------+------------------------------+ 589 | a_rwnd = 4000 | 590 +------------------------------+------------------------------+ 591 | Num of R Gap Ack Blocks = 2 | Num of NR Gap Ack Blocks = 3 | 592 +------------------------------+------------------------------+ 593 | Num of Duplicates = 0 | 0x00 | 594 +------------------------------+------------------------------+ 595 | R Gap Ack Block #1 Start = 8 | R Gap Ack Block #1 End = 8 | 596 +------------------------------+------------------------------+ 597 | R Gap Ack Block #2 Start = 11| R Gap Ack Block #2 End = 12 | 598 +------------------------------+------------------------------+ 599 |NR Gap Ack Block #1 Start = 2 | NR Gap Ack Block #1 End = 5 | 600 +------------------------------+------------------------------+ 601 |NR Gap Ack Block #2 Start = 10| NR Gap Ack Block #2 End = 10 | 602 +------------------------------+------------------------------+ 603 |NR Gap Ack Block #3 Start = 13| NR Gap Ack Block #3 End = 13 | 604 +------------------------------+------------------------------+ 606 CASE-3: Maximal Data Receiver Responsibility 608 In this special case, all out-of-order data blocks acknowledged are 609 non-renegable. This case would occur when the data receiver is 610 programmed never to renege, and takes responsibility to deliver all 611 DATA chunks that arrive out-of-order. In this case Num of R Gap Ack 612 Blocks is zero indicating all reported out-of-order TSNs are nr-gap- 613 acked. 615 +--------------------------------+-------------------------------+ 616 | Type = 0x10 | 0x00 | Chunk Length = 32 | 617 +--------------------------------+-------------------------------+ 618 | Cumulative TSN Ack = 3 | 619 +--------------------------------+-------------------------------+ 620 | a_rwnd = 4000 | 621 +--------------------------------+-------------------------------+ 622 | Num of R Gap Ack Blocks = 0 | Num of NR Gap Ack Blocks = 3 | 623 +--------------------------------+-------------------------------+ 624 | Num of Duplicates = 0 | 0x00 | 625 +--------------------------------+-------------------------------+ 626 | NR Gap Ack Block #1 Start = 2 | NR Gap Ack Block #1 End = 5 | 627 +--------------------------------+-------------------------------+ 628 | NR Gap Ack Block #2 Start = 8 | NR Gap Ack Block #2 End = 8 | 629 +--------------------------------+-------------------------------+ 630 | NR Gap Ack Block #3 Start = 10 | NR Gap Ack Block #3 End = 13 | 631 +--------------------------------+-------------------------------+ 633 4.4. Procedures 635 The procedures regarding "when" to send an NR-SACK chunk are 636 identical to the procedures regarding when to send a SACK chunk, as 637 outlined in Section 6.2 of [RFC4960]. 639 4.4.1. Sending an NR-SACK chunk 641 All of the NR-SACK chunk fields identical to the SACK chunk MUST be 642 formed as described in Section 6.2 of [RFC4960]. 644 It is up to the data receiver whether or not to take responsibility 645 for delivery of each out-of-order DATA chunk. An out-of-order DATA 646 chunk that has already been delivered, or that the receiver takes 647 responsibility to deliver (i.e., guarantees not to renege) is Non 648 Renegable(NR), and SHOULD be included in an NR Gap Ack Block field of 649 the outgoing NR-SACK. All other out-of-order data is (R)enegable, 650 and SHOULD be included in R Gap Ack Block field of the outgoing NR- 651 SACK. 653 Consider three types of data receiver: 655 CASE-1: Data receiver takes no responsibility for delivery of any 656 out-of-order DATA chunks 658 CASE-2: Data receiver takes responsibility for all out-of-order DATA 659 chunks that are "deliverable" (i.e., DATA chunks in-sequence 660 within the stream they belong to, or DATA chunks whose (U)nordered 661 bit is 1) 663 CASE-3: Data receiver takes responsibility for delivery of all out- 664 of-order DATA chunks, whether deliverable or not deliverable 666 The data receiver SHOULD follow the procedures outlined below for 667 building the NR-SACK. 669 CASE-1: 671 1A) Identify the TSNs received out-of-order. 673 1B) For these out-of-order TSNs, identify the R Gap Ack Blocks. 674 Fill the Number of R Gap Ack Blocks (N) field, R Gap Ack Block #i 675 Start, and R Gap Ack Block #i End where i goes from 1 to N. 677 1C) Set the Number of NR Gap Ack Blocks (M) field to 0. 679 CASE-2: 681 2A) Identify the TSNs received out-of-order. 683 2B) For the received out-of-order TSNs, check the (U)nordered bit of 684 each TSN. Tag unordered TSNs as NR. 686 2C) For each stream, also identify the TSNs received out-of-order 687 but are in-sequence within that stream. Tag those in-sequence 688 TSNs as NR. 690 2D) Tag all out-of-order data that is not NR as (R)enegable. 692 2E) For those TSNs tagged as (R)enegable, identify the (R)enegable 693 Blocks. Fill the Number of R Gap Ack Blocks(N) field, R Gap Ack 694 Block #i Start, and R Gap Ack Block #i End where i goes from 1 to 695 N. 697 2F) For those TSNs tagged as NR, identify the NR Blocks. Fill the 698 Number of NR Gap Ack Blocks(M) field, NR Gap Ack Block #i Start, 699 and NR Gap Ack Block #i End where i goes from 1 to M. 701 CASE-3: 703 3A) Identify the TSNs received out-of-order. All of these TSNs 704 SHOULD be nr-gap-acked. 706 3B) Set the Number of R Gap Ack Blocks (N) field to 0. 708 3C) For these out-of-order TSNs, identify the NR Gap Ack Blocks. 709 Fill the Number of NR Gap Ack Blocks (M) field, NR Gap Ack Block 710 #i Start, and NR Gap Ack Block #i End where i goes from 1 to M. 712 RFC4960 states that the SCTP endpoint MUST report as many Gap Ack 713 Blocks as can fit in a single SACK chunk limited by the current path 714 MTU. When using NR-SACKs, the SCTP endpoint SHOULD fill as many R 715 Gap Ack Blocks and NR Gap Ack Blocks starting from the Cumulative TSN 716 Ack value as can fit in a single NR-SACK chunk limited by the current 717 path MTU. If space remains, the SCTP endpoint SHOULD fill as many 718 Duplicate TSNs as possible starting from Cumulative TSN Ack value. 720 4.4.2. Receiving an NR-SACK Chunk 722 When an NR-SACK chunk is received, all of the NR-SACK fields 723 identical to a SACK chunk SHOULD be processed and handled as in SACK 724 chunk handling outlined in Section 6.2.1 of [RFC4960]. 726 The NR Gap Ack Block Start(s) and NR Gap Ack Block End(s) are offsets 727 relative to the cum-ack. To calculate the actual range of nr-gap- 728 acked TSNs, the cum-ack MUST be added to the Start and End. 730 For example, assume an incoming NR-SACK chunk's cum-ack is 12 and an 731 NR Gap Ack Block defines the NR Gap Ack Block Start=5, and the NR Gap 732 Ack Block End=7. This NR Gap Ack block nr-gap-acks TSNs 17 through 733 19 inclusive. 735 Upon reception of an NR-SACK chunk, all TSNs listed in either R Gap 736 Ack Block(s) or NR Gap Ack Block(s) SHOULD be processed as would be 737 TSNs included in Gap Ack Block(s) of a SACK chunk. All TSNs in all 738 NR Gap Ack Blocks SHOULD be removed from the data sender's 739 retransmission queue as their delivery to the receiving application 740 has either already occurred, or is guaranteed by the data receiver. 742 Although R Gap Ack Blocks and NR Gap Ack Blocks SHOULD be disjoint 743 sets, NR-SACK processing SHOULD work if an NR-SACK chunk has a TSN 744 listed in both an R Gap Ack Block and an NR Gap Ack Block. In this 745 case, the TSN SHOULD be treated as Non-Renegable. 747 Implementation Note: 749 Most of NR-SACK processing at the data sender can be implemented by 750 using the same routines as in SACK that process the cum ack and the 751 gap ack(s), followed by removal of nr-gap-acked DATA chunks from the 752 retransmission queue. However, with NR-SACKs, as out-of-order DATA 753 is sometimes removed from the retransmission queue, the gap ack 754 processing routine should recognize that the data sender's 755 retransmission queue has some transmitted data removed. For example, 756 while calculating missing reports, the gap ack processing routine 757 cannot assume that the highest TSN transmitted is always at the tail 758 (right edge) of the retransmission queue. 760 5. Buffer Blocking Mitigation 762 TBD. See [Dre2012], [PAMS2011], [Globecom2010]. 764 5.1. Sender Buffer Splitting 766 TBD. See [Dre2012], [PAMS2011], [Globecom2010]. 768 5.2. Receiver Buffer Splitting 770 TBD. See [Dre2012], [PAMS2011], [Globecom2010]. 772 5.3. Chunk Rescheduling 774 This algorithm ensures quick blocking resolution for ordered data. 775 TBD. See [Dre2012], [Globecom2010]. 777 5.4. Problems during Path Failure 779 This section discusses CMT's receive buffer related problems during 780 path failure, and proposes a solution for the same. 782 5.4.1. Problem Description 784 Link failures arise when a router or a link connecting two routers 785 fails due to link disconnection, hardware malfunction, or software 786 error. Overloaded links caused by flash crowds and denial-of-service 787 (DoS) attacks also degrade end-to-end communication between peer 788 hosts. Ideally, the routing system detects link failures, and in 789 response, reconfigures the routing tables and avoids routing traffic 790 via the failed link. However, existing research highlights problems 791 with Internet backbone routing that result in long route convergence 792 times. The pervasiveness of path failures motivated us to study 793 their impact on CMT, since CMT achieves better throughput via 794 simultaneous data transmission over multiple end-to-end paths. 796 CMT is an extension to SCTP, and therefore retains SCTP's failure 797 detection process. A CMT sender uses a tunable failure detection 798 threshold called Path.Max.Retrans (PMR). When a sender experiences 799 more than PMR consecutive timeouts while trying to reach an active 800 destination, the destination is marked as failed. With PMR=5, the 801 failure detection takes 6 consecutive timeouts or 63s. After every 802 timeout, the CMT sender continues to transmit new data on the failed 803 path increasing the chances of receive buffer (rbuf) blocking and 804 degrading CMT performance during permanent and short-term path 805 failures [NEA08]. 807 5.4.2. Solution: Potentially-failed Destination State 809 To mitigate the rbuf blocking, we introduce a new destination state 810 called 'potentially-failed' state in SCTP (and CMT's) failure 811 detection process [I-D.ietf-tsvwg-sctp-failover]. This solution is 812 based on the rationale that loss detected by a timeout implies either 813 severe congestion or failure en route. After a single timeout on a 814 path, a sender is unsure, and marks the corresponding destination as 815 'potentially-failed' (PF). A PF destination is not used for data 816 transmission or retransmission. CMT's retransmission policies are 817 augmented to include the PF state. Performance evaluations prove 818 that the PF state significantly reduces rbuf blocking during failure 819 detection [NEA08]. 821 5.5. Non-Renegable SACK 823 This section discusses problems with SCTP's SACK mechanism and how it 824 affects the send buffer and CMT performance. 826 5.5.1. Problem Description 828 Gap-acks acknowledge DATA chunks that arrive out-of-order to a 829 transport layer data receiver. A gap-ack in SCTP is advisory, in 830 that, while it notifies a data sender about the reception of 831 indicated DATA chunks, the data receiver is permitted to later 832 discard DATA chunks that it previously had gap-acked. Discarding a 833 previously gap-acked DATA chunk is known as 'reneging'. Because of 834 the possibility of reneging in SCTP, any gap-acked DATA chunk MUST 835 NOT be removed from the data sender's retransmission queue until the 836 DATA chunk is later CumAcked. 838 Situations exist when a data receiver knows that reneging on a 839 particular out-of-order DATA chunk will never take place, such as 840 (but not limited to) after an out-of-order DATA chunk is delivered to 841 the receiving application. With current SACKs in SCTP, it is not 842 possible for a data receiver to inform a data sender if or when a 843 particular out-of-order 'deliverable' DATA chunk has been 'delivered' 844 to the receiving application. Thus the data sender MUST keep a copy 845 of every gap-acked out-of-order DATA chunk(s) in the data sender's 846 retransmission queue until the DATA chunk is CumAcked. This use of 847 the data sender's retransmission queue is wasteful. The wasted 848 buffer often degrades CMT performance; the degradation increases when 849 a CMT flow traverses via paths with disparate end-to-end properties 850 [NEY08]. 852 5.5.2. Solution: Non-Renegable SACKs 854 Non-Renegable Selective Acknowledgments (NR-SACKs) Section 4 are a 855 new kind of acknowledgements, extending SCTP's SACK chunk 856 functionalities. The NR-SACK chunk is an extension of the existing 857 SACK chunk. Several fields are identical, including the Cumulative 858 TSN Ack, the Advertised Receiver Window Credit (a_rwnd), and 859 Duplicate TSNs. These fields have the same semantics as described in 860 [RFC4960]. 862 NR-SACKs also identify out-of-order DATA chunks that a receiver 863 either: (1) has delivered to its receiving application, or (2) takes 864 full responsibility to eventually deliver to its receiving 865 application. These out-of-order DATA chunks are 'non-renegable.' 866 Non-Renegable data are reported in the NR Gap Ack Block field of the 867 NR-SACK chunk as described Section 4. We refer to non-renegable 868 selective acknowledgements as 'nr-gap-acks.' 870 When an out-of-order DATA chunk is nr-gap-acked, the data sender no 871 longer needs to keep that particular DATA chunk in its retransmission 872 queue, thus allowing the data sender to free up its buffer space 873 sooner than if the DATA chunk were only gap-acked. NR-SACKs improve 874 send buffer utilization and throughput for CMT flows [NEY08]. 876 6. Handling of Shared Bottlenecks 877 6.1. Introduction 879 CMT-SCTP assumes all paths to be disjoint. Since each path 880 independently uses a TCP-like congestion control, an SCTP association 881 using N paths over the same bottleneck acquires N times the bandwidth 882 of a concurrent TCP flow. This is clearly unfair. A reliable 883 detection of shared bottlenecks is impossible in arbitrary networks 884 like the Internet. Therefore, [ICC2012] [ConTEL2011], [AINA2010] 885 apply the idea of Resource Pooling to CMT-SCTP. Resource Pooling 886 (RP) denotes 'making a collection of resources behave like a single 887 pooled resource' [WHB09]. The modifications of RP-enabled CMT-SCTP, 888 further denoted as CMT/RP-SCTP, are described in the following 889 subsections. A detailed description of CMT/RP-SCTP, including 890 congestion control examples, can be found in [ICC2012], [ConTEL2011], 891 [AINA2010]. 893 6.2. Initial Values 895 TDB. 897 6.3. Congestion Window Growth 899 TDB. See [Dre2012], [ICC2012], [ConTEL2011]. 901 6.4. Congestion Window Decrease 903 TDB. See [Dre2012], [ICC2012], [ConTEL2011]. 905 7. Chunk Scheduling and Rescheduling 907 TDB. See [Dre2012], [PFLDNeT2010]. 909 8. Socket API Considerations 911 See [I-D.dreibholz-tsvwg-sctpsocket-multipath] and 912 [I-D.dreibholz-tsvwg-sctpsocket-sqinfo]. 914 9. Testbed Platforms 916 A large-scale and realistic Internet testbed platform with support 917 for the multi-homing feature of the underlying SCTP protocol is 918 NorNet. Particularly, it is also a platform for multi-path transport 919 experiments with CMT-SCTP. A description of and introduction to 920 NorNet is provided in [ComNets2013-Core], [PAMS2013-NorNet], 921 [Haikou2017-2-MultiPath], [Haikou2017-2-NorNet-Tutorial]. Further 922 information can be found on the project website [NorNet-Website] at 923 https://www.nntb.no. 925 An Open Source simulation model of CMT-SCTP is available for OMNeT++ 926 within the INET Framework. See [INET-Framework] for the Git 927 repository. For documentation on the model, together with 928 performance evaluations, see [Dre2012]. Some interesting performance 929 evaluations for delay-sensitive traffic with CMT-SCTP can be found in 930 [ComNets2016-MultipathSurvey]. 932 10. IANA Considerations 934 NOTE to RFC-Editor: 936 "RFCXXXX" is to be replaced by the RFC number you assign this 937 document. 939 NOTE to RFC-Editor: 941 The suggested values for the chunk type and the chunk parameter 942 types are tentative and to be confirmed by IANA. 944 This document (RFCXXXX) is the reference for all registrations 945 described in this section. The suggested changes are described 946 below. 948 10.1. A New Chunk Type 950 A chunk type has to be assigned by IANA. It is suggested to use the 951 values given in Section 4. IANA should assign this value from the 952 pool of chunks with the upper two bits set to '00'. 954 This requires an additional line in the "Chunk Types" registry for 955 SCTP: 957 Chunk Types 959 ID Value Chunk Type Reference 960 ----- ---------- --------- 961 16 Non-Renegable SACK (NR-SACK) [RFCXXXX] 963 The registration table as defined in [RFC6096] for the chunk flags of 964 this chunk type is empty. 966 11. Security Considerations 968 This document does not add any additional security considerations in 969 addition to the ones given in [RFC4960]. 971 12. Acknowledgments 973 The authors wish to thank Hakim Adhari, Phillip Conrad, Jonathan 974 Leighton, Ertugrul Yilmaz and Xing Zhou for their invaluable comments 975 and support. 977 13. References 979 13.1. Normative References 981 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 982 Requirement Levels", BCP 14, RFC 2119, 983 DOI 10.17487/RFC2119, March 1997, 984 . 986 [RFC4960] Stewart, R., Ed., "Stream Control Transmission Protocol", 987 RFC 4960, DOI 10.17487/RFC4960, September 2007, 988 . 990 [RFC5061] Stewart, R., Xie, Q., Tuexen, M., Maruyama, S., and M. 991 Kozuka, "Stream Control Transmission Protocol (SCTP) 992 Dynamic Address Reconfiguration", RFC 5061, 993 DOI 10.17487/RFC5061, September 2007, 994 . 996 [RFC5351] Lei, P., Ong, L., Tuexen, M., and T. Dreibholz, "An 997 Overview of Reliable Server Pooling Protocols", RFC 5351, 998 DOI 10.17487/RFC5351, September 2008, 999 . 1001 [RFC6096] Tuexen, M. and R. Stewart, "Stream Control Transmission 1002 Protocol (SCTP) Chunk Flags Registration", RFC 6096, 1003 DOI 10.17487/RFC6096, January 2011, 1004 . 1006 [I-D.ietf-tsvwg-sctp-failover] 1007 Nishida, Y., Natarajan, P., Caro, A., Amer, P. D., and K. 1008 E. E. Nielsen, "SCTP-PF: A Quick Failover Algorithm for 1009 the Stream Control Transmission Protocol", Work in 1010 Progress, Internet-Draft, draft-ietf-tsvwg-sctp-failover- 1011 16, 17 February 2016, . 1014 [I-D.dreibholz-tsvwg-sctpsocket-multipath] 1015 Dreibholz, T., Becke, M., and H. Adhari, "SCTP Socket API 1016 Extensions for Concurrent Multipath Transfer", Work in 1017 Progress, Internet-Draft, draft-dreibholz-tsvwg- 1018 sctpsocket-multipath-22, 13 March 2021, 1019 . 1022 [I-D.dreibholz-tsvwg-sctpsocket-sqinfo] 1023 Dreibholz, T., Seggelmann, R., and M. Becke, "Sender Queue 1024 Info Option for the SCTP Socket API", Work in Progress, 1025 Internet-Draft, draft-dreibholz-tsvwg-sctpsocket-sqinfo- 1026 22, 13 March 2021, . 1029 13.2. Informative References 1031 [I06] Iyengar, J., "End-to-End Concurrent Multipath Transfer 1032 Using Transport Layer Multihoming", PhD 1033 Dissertation Computer Science Dept., University of 1034 Delaware, April 2006, 1035 . 1038 [IAS06] Iyengar, J., Amer, P. D., and R. R. Stewart, "Concurrent 1039 Multipath Transfer Using SCTP Multihoming Over Independent 1040 End-to-End Paths", Journal IEEE/ACM Transactions on 1041 Networking, October 2006, 1042 . 1045 [NEA08] Natarajan, P., Ekiz, N., Iyengar, J., Amer, P., and R. 1046 Stewart, "Concurrent Multipath Transfer Using Transport 1047 Layer Multihoming: Introducing the Potentially-failed 1048 Destination State", Proceedings of the IFIP Networking, 1049 May 2008, 1050 . 1053 [NEY08] Natarajan, P., Ekiz, N., Yilmaz, E., Amer, P., Iyengar, 1054 J., and R. Stewart, "Non-Renegable Selective 1055 Acknowledgments (NR-SACKs) for SCTP", Proceedings of the 1056 16th IEEE International Conference on Network Protocols 1057 (ICNP) , October 2008, 1058 . 1060 [WHB09] Wischik, D., Handley, M., and M. B. Braun, "The Resource 1061 Pooling Principle", Journal ACM SIGCOMM Computer 1062 Communication Review, October 2009, 1063 . 1066 [OMNeTWorkshop2010-SCTP] 1067 Dreibholz, T., Becke, M., Pulinthanath, J., and E. P. 1068 Rathgeb, "Implementation and Evaluation of Concurrent 1069 Multipath Transfer for SCTP in the INET Framework", 1070 Proceedings of the 3rd ACM/ICST International Workshop on 1071 OMNeT++ ISBN 978-963-9799-87-5, 1072 DOI 10.4108/ICST.SIMUTOOLS2010.8673, 19 March 2010, 1073 . 1076 [AINA2010] Dreibholz, T., Becke, M., Pulinthanath, J., and E. P. 1077 Rathgeb, "Applying TCP-Friendly Congestion Control to 1078 Concurrent Multipath Transfer", Proceedings of the 24th 1079 IEEE International Conference on Advanced Information 1080 Networking and Applications (AINA) Pages 312-319, 1081 ISBN 978-0-7695-4018-4, DOI 10.1109/AINA.2010.117, 21 1082 April 2010, . 1086 [YEN10] Yilmaz, E., Ekiz, N., Natarajan, P., Amer, P., Leighton, 1087 J., Baker, F., and R. Stewart, "Throughput Analysis of 1088 Non-Renegable Selective Acknowledgments (NR-SACKs) for 1089 SCTP", Computer 1090 Communications, doi:10.1016/j.comcom.2010.06.028, 2010. 1092 [PFLDNeT2010] 1093 Dreibholz, T., Seggelmann, R., Tüxen, M., and E. P. 1094 Rathgeb, "Transmission Scheduling Optimizations for 1095 Concurrent Multipath Transfer", Proceedings of the 8th 1096 International Workshop on Protocols for Future, Large- 1097 Scale and Diverse Network Transports (PFLDNeT) Volume 8, 1098 ISSN 2074-5168, 29 November 2010, . 1102 [Globecom2010] 1103 Dreibholz, T., Becke, M., Rathgeb, E. P., and M. Tüxen, 1104 "On the Use of Concurrent Multipath Transfer over 1105 Asymmetric Paths", Proceedings of the IEEE Global 1106 Communications 1107 Conference (GLOBECOM) ISBN 978-1-4244-5637-6, 1108 DOI 10.1109/GLOCOM.2010.5683579, 7 December 2010, 1109 . 1112 [PAMS2011] Adhari, H., Dreibholz, T., Becke, M., Rathgeb, E. P., and 1113 M. Tüxen, "Evaluation of Concurrent Multipath Transfer 1114 over Dissimilar Paths", Proceedings of the 1st 1115 International Workshop on Protocols and Applications with 1116 Multi-Homing Support (PAMS) Pages 708-714, 1117 ISBN 978-0-7695-4338-3, DOI 10.1109/WAINA.2011.92, 22 1118 March 2011, . 1122 [ConTEL2011] 1123 Dreibholz, T., Becke, M., Adhari, H., and E. P. Rathgeb, 1124 "On the Impact of Congestion Control for Concurrent 1125 Multipath Transfer on the Transport Layer", Proceedings of 1126 the 11th IEEE International Conference on 1127 Telecommunications (ConTEL) Pages 397-404, 1128 ISBN 978-953-184-152-8, 16 June 2011, 1129 . 1132 [ICC2012] Becke, M., Dreibholz, T., Adhari, H., and E. P. Rathgeb, 1133 "On the Fairness of Transport Protocols in a Multi-Path 1134 Environment", Proceedings of the IEEE International 1135 Conference on Communications (ICC) Pages 2666-2672, 1136 ISBN 978-1-4577-2052-9, DOI 10.1109/ICC.2012.6363695, 12 1137 June 2012, . 1140 [ComNets2016-MultipathSurvey] 1141 Yedugundla, K. V., Ferlin, S., Dreibholz, T., Alay, Ö., 1142 Kuhn, N., Hurtig, P., and A. Brunström, "Is Multi-Path 1143 Transport Suitable for Latency Sensitive Traffic?", 1144 Computer Networks Volume 105, Pages 1-21, ISSN 1389-1286, 1145 DOI 10.1016/j.comnet.2016.05.008, 4 August 2016, 1146 . 1149 [Dre2012] Dreibholz, T., "Evaluation and Optimisation of Multi-Path 1150 Transport using the Stream Control Transmission 1151 Protocol", Habilitation Treatise, 13 March 2012, 1152 . 1156 [NorNet-Website] 1157 Dreibholz, T., "NorNet -- A Real-World, Large-Scale Multi- 1158 Homing Testbed", Online: https://www.nntb.no/, 2016, 1159 . 1161 [PAMS2013-NorNet] 1162 Dreibholz, T. and E. G. Gran, "Design and Implementation 1163 of the NorNet Core Research Testbed for Multi-Homed 1164 Systems", Proceedings of the 3nd International Workshop on 1165 Protocols and Applications with Multi-Homing 1166 Support (PAMS) Pages 1094-1100, ISBN 978-0-7695-4952-1, 1167 DOI 10.1109/WAINA.2013.71, 27 March 2013, 1168 . 1172 [ComNets2013-Core] 1173 Gran, E. G., Dreibholz, T., and A. Kvalbein, "NorNet Core 1174 – A Multi-Homed Research Testbed", Computer Networks, 1175 Special Issue on Future Internet Testbeds Volume 61, Pages 1176 75-87, ISSN 1389-1286, DOI 10.1016/j.bjp.2013.12.035, 14 1177 March 2014, 1178 . 1180 [INET-Framework] 1181 Hornig, R. and A. Varga, "INET Framework Git Repository", 1182 Online: https://github.com/inet-framework/inet, 2016, 1183 . 1185 [Haikou2017-2-MultiPath] 1186 Dreibholz, T., "An Introduction to Multi-Path Transport at 1187 Hainan University", Keynote Talk at Hainan University, 1188 College of Information Science and Technology (CIST), 14 1189 December 2017, . 1192 [Haikou2017-2-NorNet-Tutorial] 1193 Dreibholz, T., "NorNet Core Beginner Tutorial at Hainan 1194 University", Tutorial at Hainan University, College of 1195 Information Science and Technology (CIST), 15 December 1196 2017, . 1199 Authors' Addresses 1200 Paul D. Amer 1201 University of Delaware, Computer and Information Sciences Department 1202 Newark, Delaware 19716 1203 United States of America 1205 Phone: +1-302-831-1944 1206 Email: amer@cis.udel.edu 1207 URI: https://www.eecis.udel.edu/~amer/ 1209 Martin Becke 1210 HAW Hamburg, Informatics Department 1211 Berliner Tor 7 1212 20099 Hamburg 1213 Germany 1215 Phone: +49-40-42875-8104 1216 Email: martin.becke@haw-hamburg.de 1217 URI: http://www.scimbe.de/about.html 1219 Thomas Dreibholz 1220 Simula Metropolitan Centre for Digital Engineering 1221 Pilestredet 52 1222 0167 Oslo 1223 Norway 1225 Phone: +47-6782-8200 1226 Email: dreibh@simula.no 1227 URI: https://www.simula.no/people/dreibh 1229 Nasif Ekiz 1230 University of Delaware, Computer and Information Sciences Department 1231 Newark, Delaware 19716 1232 United States of America 1234 Email: nekiz@udel.edu 1236 Janardhan Iyengar 1237 Franklin and Marshall College, Mathematics and Computer Science 1238 PO Box 3003 1239 Lancaster, Pennsylvania 17604-3003 1240 United States of America 1242 Phone: +1-717-358-4774 1243 Email: jiyengar@fandm.edu 1244 Preethi Natarajan 1245 Cisco Systems 1246 425 East Tasman Drive 1247 San Jose, California 95134 1248 United States of America 1250 Email: prenatar@cisco.com 1252 Randall R. Stewart 1253 Netflix 1254 Chapin, South Carolina 29036 1255 United States of America 1257 Email: randall@lakerest.net 1259 Michael Tuexen 1260 Muenster University of Applied Sciences 1261 Stegerwaldstrasse 39 1262 48565 Steinfurt 1263 Germany 1265 Email: tuexen@fh-muenster.de 1266 URI: https://www.fh-muenster.de/fb2/personen/professoren/tuexen/