idnits 2.17.1 draft-an-multipath-quic-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a Security Considerations section. ** The abstract seems to contain references ([I-D.ietf-quic-transport]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (October 22, 2020) is 1282 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-07) exists of draft-deconinck-quic-multipath-05 == Outdated reference: A later version (-19) exists of draft-ietf-quic-load-balancers-04 == Outdated reference: A later version (-34) exists of draft-ietf-quic-transport-32 Summary: 2 errors (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 QUIC Q. An 3 Internet-Draft Y. Liu 4 Intended status: Standards Track Y. Ma 5 Expires: April 25, 2021 Alibaba Inc. 6 Z. Li 7 ICT-CAS 8 October 22, 2020 10 Multipath Extension for QUIC 11 draft-an-multipath-quic-00 13 Abstract 15 This document specifies multipath extension for the QUIC protocol to 16 enable the simultaneous usage of multiple paths for a single 17 connection. 19 The extension is compliant with the single-path QUIC design. The 20 design principle is to support multipath by adding limited extension 21 to QUIC-Transport [I-D.ietf-quic-transport]. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at https://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on April 25, 2021. 40 Copyright Notice 42 Copyright (c) 2020 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (https://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 58 2. Notational Conventions . . . . . . . . . . . . . . . . . . . 3 59 3. Sub-Connection . . . . . . . . . . . . . . . . . . . . . . . 4 60 4. Enable Multipath QUIC - Handshake . . . . . . . . . . . . . . 5 61 5. Sub-connection Management . . . . . . . . . . . . . . . . . . 5 62 5.1. Multipath QUIC Interaction . . . . . . . . . . . . . . . 5 63 5.2. Path validation and sub-connection ID negotiation . . . . 8 64 5.3. New sub-connection establishment . . . . . . . . . . . . 9 65 5.4. Close sub-connection . . . . . . . . . . . . . . . . . . 9 66 5.5. Sub-connection Lookup . . . . . . . . . . . . . . . . . . 10 67 5.6. Sub-connection migration . . . . . . . . . . . . . . . . 10 68 5.7. Sub-connection state machine management . . . . . . . . . 10 69 5.8. Sender and Receiver Connection (Sub-connection) States . 11 70 5.9. Use load balancer in Multipath QUIC . . . . . . . . . . . 11 71 6. Packet scheduling . . . . . . . . . . . . . . . . . . . . . . 11 72 6.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 11 73 6.2. Basic Static Scheduling Strategy . . . . . . . . . . . . 12 74 6.3. Dynamic (feedback-based) Scheduling Strategy . . . . . . 13 75 6.4. Application Policy-Awareness . . . . . . . . . . . . . . 14 76 6.4.1. Per-connection Policy . . . . . . . . . . . . . . . . 15 77 6.4.2. Per-stream Policy . . . . . . . . . . . . . . . . . . 17 78 7. Congestion control and loss detect . . . . . . . . . . . . . 18 79 7.1. Congestion control . . . . . . . . . . . . . . . . . . . 18 80 7.2. Packet number space and acknowledgements . . . . . . . . 18 81 7.3. Flow control . . . . . . . . . . . . . . . . . . . . . . 18 82 8. New frames . . . . . . . . . . . . . . . . . . . . . . . . . 18 83 8.1. MP_SUB_CONN_NEW frame . . . . . . . . . . . . . . . . . . 19 84 8.2. MP_SUB_CONN_ACCEPT frame . . . . . . . . . . . . . . . . 19 85 8.3. MP_SUB_CONN_CLOSE frame . . . . . . . . . . . . . . . . . 19 86 8.4. MP_ACK frame . . . . . . . . . . . . . . . . . . . . . . 20 87 8.5. MP_ADD_ADDRESS frame . . . . . . . . . . . . . . . . . . 21 88 8.6. MP_REMOVE_ADDRESS frame . . . . . . . . . . . . . . . . . 21 89 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 90 10. Informative References . . . . . . . . . . . . . . . . . . . 21 91 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 93 1. Introduction 95 In this document, we propose an extension to the current QUIC design 96 to enable the simultaneous usage of multiple paths for a single 97 connection. 99 This proposal differs from past proposals 100 [I-D.deconinck-quic-multipath] in two fundamental perspectives: 102 o The multi-path QUIC is built on top of the concept of the 103 bidirectional sub-connection, which readily fits into the nature 104 of both cellular and wifi links that cover the majority of multi- 105 path applications in QUIC while keeping the design simple and easy 106 to implement. In doing so, we are able to re-use most of the 107 current QUIC transport design with the sole addition of six new 108 frames. 110 o The multi-path QUIC design enables feedback-based dynamic 111 scheduling strategy. As the major goal of multi-path QUIC is to 112 enhance performance in mobile applications, where the sender and 113 receiver may have different viewpoints about the fast-changing 114 wireless connectivity, especially in high-mobility scenarios, the 115 proposed design allows the sender and receiver to synchronize 116 their viewpoints via message exchange in ACK packet in order to 117 maximize performance. 119 This document is organized as follows. It first provides definition 120 of sub-connection in Section 3. It then specifies how to enable 121 multipath QUIC during handshake in Section 4, and sub-connection 122 management in Section 5. It discusses packet scheduling in 123 Section 6, and congestion control in Section 7. It specifies the new 124 frames in Section 8. 126 2. Notational Conventions 128 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 129 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 130 document are to be interpreted as described in [RFC2119]. 132 We assume that the reader is familiar with the terminology used in 133 [I-D.ietf-quic-transport]. In addition, we define the following 134 terms: 136 o Path: A sequence of links between a sender and a receiver, defined 137 in this context by a 4-tuple of source and destination address/ 138 port pairs[RFC8684]. 140 o Sub-Connection: Sub-connection is bidirectional and provides 141 reliable transmission between client and server. A connection can 142 contain one or multiple sub-connections. A sub-connection is 143 identified by an internal identifier, called Sub-Connection Index 144 (SCI). Each sub-connection has its unique Source Connection ID 145 and Destination Connection ID. The Connection ID is mapped with 146 the 2-tuple of IP address and port. 148 o (Multipath QUIC) Connection: A set of one or more sub-connections, 149 over which an application can communicate between two host. 151 3. Sub-Connection 153 A connection can contain one or multiple sub-connections which are 154 bidirectional and provides reliable transmission between client and 155 server. Sub-connection is identified by Sub-Connection Index (SCI). 157 If a connection contains at least 2 sub-connections, then the first 158 established sub-connection is called Initial sub-connection. The 159 rest sub-connections are called supplementary sub-connections. 161 Every sub-connection has its own unique CID pair that is associated 162 with the 4-tuple (source IP, source port, destination IP, destination 163 port) of the underlying network path currently used by the sub- 164 connection. The Connection ID negotiation process is specified in 165 Section 5.1. In case of sub-connection migration, the CID pair will 166 be renegotiated following the connection migration procedure 167 specified in [I-D.ietf-quic-transport]. 169 Endpoints can find which sub-connection a received packet belongs to 170 according to the CID pair of the packet. Endpoints can find the 171 context of a sub-connection by its' CID pair or SCI. In the context 172 of a sub-connection, a reference pointer MUST be provided to access 173 the context of the multipath QUIC connection that the sub-connection 174 belongs to. 176 Each sub-connection has its independent Packet Number Space. And all 177 sub-connections in the same connection share the same 1-RTT 178 encryption key which is generated during the connection's 179 cryptographic handshake. 181 Note: The reason of using SCI to identify a Sub-connection: 182 acknowledgements may not be transferred via the same sub-connection 183 where the packets were sent, therefore the MP_ACK frame SHOULD 184 contain field that can uniquely identify the sub-connection, and the 185 same logic applies to other new MP frames. If we use Connection ID 186 to identify a sub-connection in MP frames, the length of Connection 187 ID is too long and will add more overhead in the frames. 189 4. Enable Multipath QUIC - Handshake 191 The connection handshake flow follows QUIC-Transport 192 [I-D.ietf-quic-transport], using the transport parameter to negotiate 193 multipath feature. The negotiation mechanism is similar to the 194 negotiation in [I-D.deconinck-quic-multipath] Section 5.1, while the 195 semantic of the transport parameter is different. 197 A new transport parameter is defined: 199 o name: max_sub_conn_index (0x40) 201 o value: a variable-length integer (1 to 8 bytes) 203 The value range and definition: 205 o 0: MP feature disabled 207 o [1, 2^31-1]: the maximum number of sub-connections 209 The value of SCI(sub-connection index) starts from 1 and increases by 210 1 when a new sub-connection is created. The value range of SCI is 211 [1, max_sub_conn_index]. The SCI of initial sub-connection is 1. A 212 multipath QUIC connection MUST NOT reuse any used SCI for new sub- 213 connections in its' lifetime. 215 If the peer does not carry the max_sub_conn_index(0x40) transport 216 parameter, which means the peer does NOT support multipath, endpoint 217 MUST fallback to QUIC-Transport [I-D.ietf-quic-transport] with single 218 path, and MUST NOT send any MP frames in the following packets. 220 5. Sub-connection Management 222 This section describes the details of sub-connection management. 224 5.1. Multipath QUIC Interaction 226 Figure 1 illustrates the Multipath QUIC interaction process. 228 Server Client Server 229 Path 2 Path 1 231 Initial (DCID=A, SCID=B, 232 ClientHello(max_sub_conn_index=4)) -> 234 Handshake(DCID=B, SCID=C, 235 <- EE(max_sub_conn_index=4)) 236 Pkt(DCID=B, SCID=C, 237 <- frames=[New_connection_id(E)]) 239 Pkt(DCID=C, SCID=B, 240 frames=[New_connection_id(E)]) -> 242 (Detect new path 2) 244 Pkt(DCID=E, 245 frames=[MP_SUB_CONN_NEW( 246 <- SCI=2, X)]) 248 Pkt(DCID=F, 249 frames=[MP_SUB_CONN_ACCEPT( 250 SCI=2, X)], 251 PATH_CHALLENGE(Y)) -> 253 Pkt(DCID=E, 254 <- frames=[PATH_CHALLENGE(Y)]) 256 (Data transmission) 258 Pkt(DCID=E, pktnum=N1, 259 <- frames=[STREAM("Request 2")]) 261 Pkt(DCID=F, pktnum=N2, 262 frames=[MP_ACK(SCI=2, N1), 263 STREAM("Response 2")]) -> 265 Pkt(DCID=C, pktnum=M1, 266 frames=[STREAM("Request 1")]) -> 268 Pkt(DCOD=E, pktnum=N3, 269 frames=[MP_ACK(SCI=2, N2), 270 <- STREAM("Request 3")]) 272 Pkt(DCID=B, pktnum=M2, 273 frames=[MP_ACK(SCI=1, M1), 274 <- STREAM("Response 1")]) 276 Pkt(DCID=F, pktnum=N4, 277 frames=[MP_ACK(SCI=2, N3), 278 STREAM("Response 3")]) -> 280 Pkt(DCID=C, pktnum=M3, 281 frames=[MP_ACK(SCI=1, M2), 282 MP_ACK(SCI=2, N4)]) -> 284 Figure 1: Multipath QUIC interaction process 286 The process is composed of four phases. 288 A. Handshake negotiation 290 During the QUIC-Transport [I-D.ietf-quic-transport] handshake, 291 endpoints negotiate whether multipath feature is supported. The 292 negotiation parameter (see Section 4) is carried within the transport 293 parameters of TLS cryptographic handshake. After the handshake 294 finished, the connection contains the initial sub-connection with SCI 295 equals 1. In Figure 1, the maximum sub-connection index is four. 297 B. Exchange unused Connection ID in advance 299 After the two endpoints complete the connection establishment, they 300 can exchange unused Connection IDs by NEW_CONNECTION_ID frame. 301 Before an endpoint starts to create a new sub-connection, it SHOULD 302 check if there are unused Connection IDs for both endpoints. 304 Note: QUIC-Transport [I-D.ietf-quic-transport] requires Connection ID 305 is uniquely mapped with 2-tuple of IP address and port. If client 306 attempts to use a new 2-tuple as source address to establish a new 307 sub-connection, a new Connection ID is required for client, and also 308 a new Connection ID is required for server. 310 C. New sub-connection establishment 312 During this phase, a new sub-connection is established between client 313 and server, and address validation is needed. 315 When client detects a new network path, it MAY attempts to establish 316 a new sub-connection by sending MP_NEW_SUB_CONN frame which carries a 317 64-bit random value and claims the new sub-connection's SCI (which is 318 2 in the example flow in Figure 1). The establishment of sub- 319 connection is always initiated by client. 321 After the server receives the MP_NEW_SUB_CONN frame from the client, 322 it responds with MP_SUB_CONN_ACCEPT frame which carries the identical 323 64-bit random value from the received MP_NEW_SUB_CONN frame and 324 agrees with the sub-connection's SCI (2 in the example). Server MUST 325 also perform path validation following the procedure specified in 326 QUIC-Transport [I-D.ietf-quic-transport]. Once the server 327 successfully validates its' peers' address, the new sub-connection is 328 established. 330 D. Data transmission on new sub-connection 331 As soon as sub-connections are established, endpoints can communicate 332 with each others over the newly established sub-connections. All 333 valid short header packets defined in QUIC-Transport 334 [I-D.ietf-quic-transport] can be carried on these sub-connections. 335 Every sub-connection has its' independent PNS. Thus, standard QUIC 336 ACK frames defined in QUIC-Transport [I-D.ietf-quic-transport] only 337 acknowledge packets that belong to the same PNS of the sub-connection 338 on which the ACK frames were received. 340 To enable endpoints reply acknowledgements on different sub- 341 connections rather than the sub-connection where the corresponding 342 packets were received, a new type of frame, MP_ACK, is defined. 343 MP_ACK frames can also be replied over the same sub-connection on 344 which data packets were received. In this case, MP_ACK frames serves 345 very similar purposes as QUIC ACK frames do. 347 MP_ACK frame contains the sub-connection index of the packets to be 348 acknowledged. For example, in Figure 1, the packet (packet number is 349 N4) is sent via the second sub-connection (SCI is 2), and its 350 corresponding acknowledgement MP_ACK(Sub-Connection Index=2, N4) is 351 sent via the initial sub-connection. 353 5.2. Path validation and sub-connection ID negotiation 355 Before clients initiate new sub-connections by sending 356 MP_SUB_CONN_NEW frames to servers through their additional network 357 addresses, they MAY want to validate the reachability between their 358 new network addresses and servers' addresses. In this case, clients 359 can initiate a path validation procedure as specified in QUIC- 360 Transport [I-D.ietf-quic-transport] per address pair. 362 Path validation uses the PATH_CHALLENGE and PATH_RESPONSE frame 363 defined in QUIC-Transport [I-D.ietf-quic-transport]. 365 Each sub-connection MUST has a unique pair of SCID and DCID within a 366 multipath QUIC connection. Thus, endpoints MUST NOT initiate or 367 accept new sub-connections if they currently have no free CIDs 368 supplied by their peers. In this case, endpoints SHOULD announce new 369 free CIDs to their peers by exchanging NEW_CONNECTION_ID frames. 371 To ensure that endpoints have free CIDs to create new sub-connections 372 as soon as they get new network addresses, an endpoint SHOULD 373 announce a least one free CID to its peer by sending 374 NEW_CONNECTION_ID frame [I-D.ietf-quic-transport] over its initial 375 sub-connection as soon as the handshake on the initial sub-connection 376 is completed. Endpoints MAY also track the number of free CIDs that 377 their peers can use and announce more free CIDs if needed. 379 Sub-connection ID negotiation follows the Connection ID negotiation 380 method in Connection Migration defined in QUIC-Transport 381 [I-D.ietf-quic-transport], which is to let client and server claim 382 its own unused Connection ID in advance by NEW_CONNECTION_ID frame. 383 If there is no available unused Connection ID, then establishment of 384 new sub-connection is not allowed. 386 5.3. New sub-connection establishment 388 New sub-connection establishment is always initiated by client, by 389 sending MP_NEW_SUB_CONN frame. 391 Because source address(2-tuple of IP address and port) is usually 392 different in the new network path, client needs to generate and claim 393 new Source Connection IDs prior to the new sub-connection 394 establishment. 396 Client that sends the MP_SUB_CONN_NEW frame in 1-RTT packets with 397 short headers, MUST use the unused Connection ID claimed in advance 398 by server as Destination Connection ID. MP_SUB_CONN_NEW frame 399 carries a 64-bit random value, and a SCI (increased progressively). 401 After receiving the MP_SUB_CONN_NEW frame, server responds with 402 MP_SUB_CONN_ACCEPT frame carrying the identical SCI and identical 403 64-bit random value from the received MP_NEW_SUB_CONN frame. Then, 404 server sends PATH_CHALLENGE to verify the client address. 406 After client receives the PATH_CHALLENGE frame, it replies with 407 PATH_RESPONSE frame In the following 1-RTT packet (short header) to 408 complete the address validation. After the address validation is 409 completed, client and server can send and receive data unrestrictedly 410 on the established sub-connection. 412 Before the client's address validation is completed, server needs to 413 limit the cumulative size of packets it sends to an unvalidated 414 address to three times the size of packets it receives from that 415 address in the new sub-connection (to prevent amplification attack). 417 5.4. Close sub-connection 419 Both client and server can terminate a sub-connection, by sending 420 MP_SUB_CONN_CLOSE frame that carries a SCI. In scenarios such as 421 client detects the network environment change (client's 4G/Wi-Fi is 422 turned off, Wi-Fi signal is fading to a threshold), or endpoints 423 detect that the quality of RTT or loss rate is becoming worse, client 424 or server can terminate a sub-connection immediately. 426 MP_SUB_CONN_CLOSE frame can be sent via a different sub-connection 427 instead of the sub-connection to be closed. 429 5.5. Sub-connection Lookup 431 Endpoints use Connection IDs to find the context of a connection. 432 Figure 2 illustrates the Connection context. Each sub-connection's 433 Connection IDs can be mapped to the connection. 435 Connection 436 +--------------------------------------------------------------+ 437 | | 438 | Sub-Conn 1 | 439 | | 440 +------+ IP 1, Port 1 SCI 1 - SCID A, DCID B IP 3, Port 3 +------+ 441 | |<----------------------------------------------------->| | 442 |Client| |Server| 443 | |<----------------------------------------------------->| | 444 +------+ IP 2, Port 2 SCI 2 - SCID C, DCID D IP 3, Port 3 +------+ 445 | | 446 | Sub-Conn 2 | 447 | | 448 +--------------------------------------------------------------+ 450 Figure 2: Connection context 452 In the connection context, client and server can use SCI or 453 Connection ID to find a sub-connection. Note that if sub-connection 454 migration happens, sub-connection's Connection ID need to be 455 renegotiated (See Section 5.6), while the SCI of sub-connection could 456 remain unchanged. 458 5.6. Sub-connection migration 460 Sub-connection migration follows the Connection Migration defined in 461 QUIC-Transport [I-D.ietf-quic-transport]. When client experiences 462 NAT rebinding (source address is changed), server needs to revalidate 463 the client address. 465 5.7. Sub-connection state machine management 467 TODO 469 5.8. Sender and Receiver Connection (Sub-connection) States 471 For each sender and receiver, the sub-connection states include: 473 +---------+----------------------+----------+--------------+--------+ 474 | Sender | SubConnectionIndex(S | CIDs(SCI | 4-tuple(sIP, | packet | 475 | | CI) | D, DCID) | dIP, sPort, | number | 476 | | | | dPort) | space | 477 +---------+----------------------+----------+--------------+--------+ 478 | Receive | SubConnectionIndex(S | CIDs(SCI | 4-tuple(sIP' | packet | 479 | r | CI) | D, DCID) | , dIP', | number | 480 | | | | sPort', | space | 481 | | | | dPort') | | 482 +---------+----------------------+----------+--------------+--------+ 484 Table 1: Sender and Receiver Connection (Sub-connection) States 486 5.9. Use load balancer in Multipath QUIC 488 This specification follows the Connection ID negotiation defined in 489 QUIC-Transport [I-D.ietf-quic-transport]. For stateless or low-state 490 load balancers supporting Multipath QUIC, implementations SHOULD use 491 the specification of Connection ID generation and Load balancer 492 routing defined in QUIC-LB [I-D.ietf-quic-load-balancers], guarantee 493 that packets with Connection IDs belonging to the same connection, 494 can be routed to same server. 496 6. Packet scheduling 498 6.1. Overview 500 For an outgoing packet, the packet scheduler decides which sub- 501 connection the packet shall be transmitted. The concept of packet 502 scheduler in Multipath QUIC is similar to that in MPTCP. As long as 503 more than one path's congestion controller allows for a new packet 504 transmission, the packet scheduler is enabled. However, the proposed 505 packet scheduler in this draft differs from past MPTCP proposals in 506 the following aspects: 508 o We enable dynamic (feedback-based) scheduling strategy with 509 feedback in this proposal to further enhance quality of user 510 experience(QoE) and to facilitate the expression of the 511 application policy-awareness. Such a capability is made available 512 by adding the QoE control signal length field and QoE control 513 signal field in MP_ACK frame (see Section 8.4). With the help of 514 such extension, a receiver is able to interact with a sender's 515 scheduling strategy in real time. 517 o Unlike MPTCP which send ACK packet over the same path, multipath 518 QUIC allows a packet to be acknowledged over a different path, 519 which allows multipath QUIC to better handle the uplink-downlink 520 heterogeneity in wireless networks. 522 o We support application policy-awareness in multipath QUIC. An 523 application can implement both per-connection policies and per- 524 stream policies. For example, a live streaming application is 525 allowed to choose a different policy from a web application. The 526 per-connection policy includes path mode, path preference, 527 scheduling algorithm and packet redundancy strategy. A per-stream 528 policy is associated with user-defined stream priorities to 529 express the applications's intent. 531 6.2. Basic Static Scheduling Strategy 533 A basic static scheduling strategy consists of four major components: 535 o Path mode: A scheduler may want to decide which sub-connections 536 shall be activated to transmit data. For instance, a scheduler 537 can choose to use only one of the two sub-connections and 538 completely ignore the other one. A scheduler marks the selected 539 sub-connections to be in the "active" state and the un-selected 540 ones in the "inactive" state. 542 o Path preference: Due to the fact that costs of transmitting data 543 over different sub-connections are not always equal. For example, 544 the energy (battery) cost over a 5G sub-connection and a Wi-Fi 545 sub-connection are very different, so a user may prefer the Wi-Fi 546 sub-connection when his/her cell phone's battery is low. In 547 another example, transmissions over a Wi-Fi sub-connection and a 548 cellular sub-connection may incur different service charges per 549 packet such that a user prefers to use the Wi-Fi sub-connection 550 over the LTE one. Note that a user's preference may change over 551 time. For instance, certain mobile carriers offer unlimited free 552 data for a particular streaming app. Therefore, the sub- 553 connection priority should be made available in the scheduler. 555 o Sub-connection selection algorithm：A selection algorithm 556 splits packets across different sub-connections and determines the 557 order of sub-connections to be selected. The selection algorithm 558 takes congestion controller states as inputs, such as smoothed 559 RTTs (sRTTs), estimated bandwidths (eBWs) and congestion window 560 sizes (CWNDs) as well as application-defined information such as 561 sub-connection priorities and path states. The outputs of the 562 algorithm is an ordered list of sub-connections to put a packet 563 on. To name a few, some of the commonly used algorithms are: 565 o 567 * Round-Robin: There is no priority. it selects sub-connections 568 one by one in order to transmit data. 570 * Lowest-RTT: It first chooses the sub-connection with the lowest 571 RTT and feeds packets to it until that sub-connection's 572 congestion window is full. Then it chooses the sub-connection 573 with the second lowest RTT. 575 * Highest-Sending-Rate: It first chooses the sub-connection with 576 the highest bandwidth and feeds packets to it until that sub- 577 connection's congestion window is full. Then it chooses sub- 578 connection with the second largest bandwidth. 580 o Packet redundancy strategy: One major challenge in multi-path 581 transmission is that a packet loss on the slow sub-connection 582 might block the overall transmission when packets are split across 583 fast-changing sub-connections. As the sub-connection selection 584 algorithm takes inputs from congestion controllers which are 585 basically rough predictions of the network and may not be accurate 586 enough for fast-changing wireless channels, such an imprecise 587 estimation could lead to network overuse/underuse. A solution to 588 this problem is to implement packet redundancy strategy. A 589 redundancy strategy can be applied to only ACK packets (partial 590 redundancy) or all data packets (full redundancy). The multipath 591 QUIC in this draft is designed to enable such flexible redundancy 592 strategies. It is up to the application to determine whether, 593 when, and on which packets to activate transmission redundancy. 595 6.3. Dynamic (feedback-based) Scheduling Strategy 597 An important feature of this proposal is the capability of dynamic 598 (feedback-based) scheduling. In a dynamic scheduling strategy, a 599 receiver notifies its currently preferred scheduling strategy to a 600 sender. Such feedback information is carried by QoE control signal 601 in MP_ACK frames. The frequency of such feedback can be controlled 602 to limit the amount of extra information. To do so, four types of 603 MP_ACK frames are designed (Figure 8): 605 o Type(i) = 0x22 , with no ECN Counts and no QoE Control Signals 607 o Type(i) = 0x23 , with ECN Counts and no QoE Control Signals 609 o Type(i) = 0x24 , with no ECN Counts and QoE Control Signals 611 o Type(i) = 0x25 , with ECN Counts and QoE Control Signals 612 The type 0x24 and 0x25 give the flexibility of carrying QoE control 613 signals. Given that the sender and the receiver may have different 614 views of the wireless environments, especially in high-mobility 615 scenarios, the QoE control signal allows a synchronization between 616 their viewpoints dynamically. It is up to the application to 617 determine the interpretation of QoE control signal and its encoding 618 method. 620 6.4. Application Policy-Awareness 622 Applications may have completely different QoE requirements---the 623 interactive applications are delay sensitive, while the video 624 streaming applications are more throughput sensitive. There is thus 625 a trend of cross-layer design that tries to take applications' 626 demands into account when managing paths or scheduling packets. The 627 static scheduling strategy and the dynamic scheduling strategy are 628 used together to fully support application policy-awareness in 629 multipath scheduling. To be more specifically, a 'control plane' is 630 separated from a `data plane' as in software-defined networking. The 631 'control plane' takes applications' high-level demands (a.k.a intent) 632 as input to generate the corresponding policies, which later are 633 deployed on the 'data plane'. The 'data plane' maps users policies 634 to the 'actions', which control the packet scheduler and other 635 functionalities that the transport implements. To allow maximum 636 design flexibility, the proposed multipath QUIC let applications to 637 access/change every single logic of the packet scheduling and path 638 management. The application policy consists of two layers: per- 639 connection policy and per-stream policy. 641 per-stream intent per-conn policy 642 Control e.g. stream prioirity e.g. path preference, path mode, 643 Plane deadline-aware etc 644 | | 645 -----------|----------------------------------------|------------- 646 V V 647 Data +--------+ - +------------------------+ 648 Plane | | | | | 649 +--------+ | | +-----+ | 650 | | +->| ... | | 651 +--------+ | | | +-----+ | 652 | | | | | +-----+ | 653 +--------+ | | +->| ... | | 654 | | | +-----+ | 655 +--------+ | +----------+ chunks |+----------+ | | 656 | | ->| stream |------->|| packet |-+ ... | 657 +--------+ | |scheduling| ||scheduling| | | 658 ... | +----------+ |+----------+ | +-----+ | 659 +--------+ | | +->| ... | | 660 | | | | | +-----+ | 661 +--------+ | | | +-----+ | 662 | | +->| ... | | 663 +--------+ | | +-----+ | 664 | | | | path manager | 665 +--------+ - +------------------------+ 666 streams 668 Figure 3: Application Policy Awareness in Multi-path QUIC framework 670 6.4.1. Per-connection Policy 672 An application imposes per-connection policy through the primitives 673 provided by the control plane. 675 As described above, the policy is translated into indications on sub- 676 connection states, sub-connection priorities, sub-connection 677 selection algorithms and packet redundant strategies. The packet 678 scheduler at the data plane will act based on these indications. We 679 assume the policies are 'soft'---the policies are not a must. 680 Instead, the data plane will follow the policies as much as possible. 682 +-----+-----------------+-------------+------------+----------------+ 683 | No. | Application | Application | Underlying | Underlying | 684 | | defined policy: | defined | action: | action: Path | 685 | | Path mode | policy: | Packet | mngm. | 686 | | | Path | Scheduling | | 687 | | | Preference | | | 688 +-----+-----------------+-------------+------------+----------------+ 689 | 1 | Wi-Fi=full, | Wi-Fi=1, | full | / | 690 | | Cellular=full | Cellular=1 | redundant | | 691 | | | | | | 692 | 2 | Wi-Fi=full, | Wi-Fi=1, | full | activate | 693 | | Cellular=backup | Cellular=1 | redundant | backup | 694 | | | | | interfaces | 695 | | | | | when the | 696 | | | | | active one's | 697 | | | | | performance is | 698 | | | | | lower than X | 699 | | | | | for 5s | 700 | | | | | | 701 | 3 | Wi-Fi=full, | Wi-Fi=2, | partially | / | 702 | | Cellular=full | Cellular=1 | redundant | | 703 +-----+-----------------+-------------+------------+----------------+ 705 Table 2: Example policies of a real-time interaction application 707 Let us take real-time interaction applications as an example to 708 illustrate the basic idea. The applications are indeed delay 709 sensitive but data volume is often low. 3 types of policies may be 710 used by different applications, as shown in Table 2 where we assume 711 only two paths are available (Wi-Fi and Cellular) 713 The first type of policies would like to use two paths equally, and 714 because the applications are delay sensitive, the actions will be to 715 active 'full redundancy' for the packet redundancy strategy---two 716 paths send the same data. The second type of policies, on the other 717 hand, would like to use the Wi-Fi interface (possibly because of data 718 charge) as much as possible, hence giving the Wi-Fi sub-connection a 719 higher priority. But if two paths have to be activated at the same 720 time due to the lower performance of Wi-Fi, then the two paths are 721 set with same the priority which can be configured dynamically 722 through QoE control signal in MP_ACK feedbacks. The third type of 723 policies would like to use the two interfaces at the same time, but 724 Wi-Fi is preferred twice as the cellular one. The actions will take 725 this into consideration, by implementing a weighed round-robin sub- 726 connection selection algorithm. 728 Likewise, we can define a mapping between the policies of different 729 types of applications and the actions in the data plane. We leave 730 the design of such a mapping to the designers. 732 6.4.2. Per-stream Policy 734 Per-stream intent is a unique feature provided by (MP)QUIC---it is 735 implemented through the multiple streams in QUIC. Streams can be 736 associated with priorities to implement applications intent. For 737 instance, objects in a web page may be dependent on others and thus 738 have different priorities [MPQUIC-Scheduler]. A stream priority- 739 aware packet scheduling algorithm will improve the performance 740 notably. 742 High priority /\ +---------+ 743 || | | 744 || +---------+ 745 || +---------+ 746 || | | 747 || +---------+ 748 || ... User-defined priority 749 || +---------+ 750 Low priority || | | 751 || +---------+ 752 ----------------------------------------------------------- 753 High priority /\ +---------+ 754 || | | 755 || +---------+ 756 || +---------+ 757 || | | 758 || +---------+ 759 || ... Default priority 760 || +---------+ 761 Low priority || | | 762 || +---------+ 764 Figure 4: Stream priority 766 We envision a priority management scheme of two separated priority 767 ranges (see Figure 4). The user-defined priority ranges are those 768 streams that the applications explicitly designate the priorities, 769 where the default priority ranges include the streams with no 770 priority values set by the applications. Only when the streams in 771 the user-defined ranges have no data sent, the data in the streams in 772 the default priority ranges can be sent. In the same range, one can 773 use the weighted round robin for scheduling---the higher-priority 774 streams get more quantum for data sending in each round. One can 775 also dynamically set/change the priorities of the streams in the 776 default priority ranges to enable short stream first if needed. 778 7. Congestion control and loss detect 780 7.1. Congestion control 782 Implementations MAY support coupled congestion controllers such as 783 LIA [MPTCP-LIA], OLIA [MPTCP-OLIA]s, and etc., or support decoupled 784 congestion controllers in environments using disjoint network paths. 786 In decoupled congestion control, every sub-connection runs its own 787 congestion controller without interacting with the congestion 788 controllers of other sub-connections. That is to say, in the aspect 789 of congestion control, a sub-connection behaves exactly the same as a 790 normal QUIC connection over the same network path. 792 Each sub-connection MAY choose congestion control algorithm 793 independently. 795 7.2. Packet number space and acknowledgements 797 Every sub-connection has its' own packet number space for 798 transmitting 1-RTT packets. 800 ACK frame [I-D.ietf-quic-transport] MUST be returned via the same 801 sub-connection on which the corresponding packets were sent. 803 MP_ACK frame can be returned via either a different sub-connection, 804 or the same sub-connection, based on different strategies of sending 805 MP_ACK frames. 807 Note: Only MP_ACK frame returned via the same sub-connection can be 808 used to calculate RTT(round trip time). 810 7.3. Flow control 812 TODO 814 8. New frames 816 All the MP frames MUST be sent in 1-RTT packet, and MUST NOT use 817 other encryption levels. 819 If an endpoint receives MP frames from packets of other encryption 820 levels, it MAY return MP_PROTOCOL_VIOLATION as a connection error and 821 close the connection. 823 8.1. MP_SUB_CONN_NEW frame 825 MP_SUB_CONN_NEW frame(type=0x2a) is used to establish a new sub- 826 connection. The MP_SUB_CONN_NEW frame will specify a SCI and include 827 a 64-bit random value. 829 MP_SUB_CONN_NEW frames are formatted as shown in Figure 5. 831 MP_SUB_CONN_NEW Frame { 832 Type (i) = 0x2a, 833 Sub_Connection_Index (i), 834 Data (64), 835 } 837 Figure 5: MP_SUB_CONN_NEW Frame Format 839 8.2. MP_SUB_CONN_ACCEPT frame 841 MP_SUB_CONN_ACCEPT frame (type=0x2b) is used by endto accept a new 842 sub-connection, as a response to MP_NEW_SUB_CONN frame. 844 MP_SUB_CONN_ACCEPT frames are formatted as shown in Figure 6, which 845 is identical to the MP_NEW_SUB_CONN frame (Section 8.1). 847 MP_SUB_CONN_ACCEPT Frame { 848 Type (i) = 0x2b, 849 Sub_Connection_Index (i), 850 Data (64), 851 } 853 Figure 6: MP_SUB_CONN_ACCEPT Frame Format 855 8.3. MP_SUB_CONN_CLOSE frame 857 MP_SUB_CONN_CLOSE frame(type=0x2c..0x2d) is used to close a sub- 858 connection, which is formatted by adding a SCI field to QUIC- 859 Transport [I-D.ietf-quic-transport] CONNECTION_CLOSE frame. The SCI 860 is used to distinguish sub-connections, so each sub-connection can be 861 closed independently. 863 MP_SUB_CONN_CLOSE frames are formatted as shown in Figure 7. 865 MP_SUB_CONN_CLOSE Frame { 866 Type (i) = 0x2c..0x2d, 867 Sub_Connection_Index (i), 868 Error Code (i), 869 [Frame Type (i)], 870 Reason Phrase Length (i), 871 Reason Phrase (..), 872 } 874 Figure 7: MP_SUB_CONN_CLOSE Frame Format 876 8.4. MP_ACK frame 878 MP_ACK frame allows for acknowledgements on different sub- 879 connections. 881 MP_ACK frame is formatted by adding a SCI field and QoE signal fields 882 to QUIC-Transport [I-D.ietf-quic-transport] ACK frame. 884 MP_ACK frames are formatted as shown in Figure 8. 886 MP_ACK Frame { 887 Type (i) = 0x22..0x23..0x24..0x25, 888 Sub_Connection_Index(i), 889 Largest Acknowledged (i), 890 ACK Delay (i), 891 ACK Range Count (i), 892 First ACK Range (i), 893 ACK Range (..) ..., 894 [ECN Counts (..)], 895 [QoE Control Signals Length(8)], 896 [QoE Control Signals (..)], 897 } 899 Figure 8: MP_ACK Frame Format 901 Type(i) = 0x22 , with no ECN Counts and no QoE Control Signals 903 Type(i) = 0x23 , with ECN Counts and no QoE Control Signals 905 Type(i) = 0x24 , with no ECN Counts and QoE Control Signals 907 Type(i) = 0x25 , with ECN Counts and QoE Control Signals 909 8.5. MP_ADD_ADDRESS frame 911 TODO 913 8.6. MP_REMOVE_ADDRESS frame 915 TODO 917 9. IANA Considerations 919 This document makes no request of IANA. 921 10. Informative References 923 [I-D.deconinck-quic-multipath] 924 De Coninck, Q. and O. Bonaventure, "Multipath Extensions 925 for QUIC (MP-QUIC)", draft-deconinck-quic-multipath-05 926 (work in progress), August 2020. 928 [I-D.ietf-quic-load-balancers] 929 Duke, M. and N. Banks, "QUIC-LB: Generating Routable QUIC 930 Connection IDs", draft-ietf-quic-load-balancers-04 (work 931 in progress), August 2020. 933 [I-D.ietf-quic-transport] 934 Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed 935 and Secure Transport", draft-ietf-quic-transport-32 (work 936 in progress), October 2020. 938 [MPQUIC-Scheduler] 939 Wang, J., Gao, Y., and C. Xu, "A Multipath QUIC Scheduler 940 for Mobile HTTP/2", Proceedings of the 3rd Asia-Pacific 941 Workshop on Networking 2019 (APNet '19). Association for 942 Computing Machinery, New York, NY, USA, 43-49., 2019, 943 . 945 [MPTCP-LIA] 946 Raiciu, C., Handly, M., and D. Wischik, "Coupled 947 Congestion Control for Multipath Transport Protocols", 948 RFC 6365, October 2011, 949 . 951 [MPTCP-OLIA] 952 Khalili, R., Gast, N., and J. Boudec, "Opportunistic 953 Linked-Increases Congestion Control Algorithm for MPTCP", 954 July 2014, . 957 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 958 Requirement Levels", BCP 14, RFC 2119, 959 DOI 10.17487/RFC2119, March 1997, 960 . 962 [RFC8684] Ford, A., Raiciu, C., Handley, M., Bonaventure, O., and C. 963 Paasch, "TCP Extensions for Multipath Operation with 964 Multiple Addresses", RFC 8684, March 2020, 965 . 967 Authors' Addresses 969 Qing An 970 Alibaba Inc. 972 Email: anqing.aq@alibaba-inc.com 974 Yanmei Liu 975 Alibaba Inc. 977 Email: miaoji.lym@alibaba-inc.com 979 Yunfei Ma 980 Alibaba Inc. 982 Email: yunfei.ma@alibaba-inc.com 984 Zhenyu Li 985 ICT-CAS 987 Email: zyli@ict.ac.cn