idnits 2.17.1 draft-welzl-loops-gen-info-04.txt: -(5): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There are 2 instances of lines with non-ascii characters in the document. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The abstract seems to contain references ([I-D.li-tsvwg-loops-problem-opportunities], [I-D.ietf-nvo3-geneve]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 1036 has weird spacing: '...packets for...' -- The document date (13 July 2020) is 1355 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Obsolete informational reference (is this intentional?): RFC 793 (Obsoleted by RFC 9293) == Outdated reference: A later version (-15) exists of draft-ietf-tcpm-rack-08 == Outdated reference: A later version (-06) exists of draft-li-tsvwg-loops-problem-opportunities-05 Summary: 1 error (**), 0 flaws (~~), 5 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TSVWG M. Welzl 3 Internet-Draft University of Oslo 4 Intended status: Standards Track C. Bormann, Ed. 5 Expires: 14 January 2021 Universität Bremen TZI 6 13 July 2020 8 LOOPS Generic Information Set 9 draft-welzl-loops-gen-info-04 11 Abstract 13 LOOPS (Local Optimizations on Path Segments) aims to provide local 14 (not end-to-end but in-network) recovery of lost packets to achieve 15 better data delivery in the presence of losses. 16 [I-D.li-tsvwg-loops-problem-opportunities] provides an overview over 17 the problems and optimization opportunities that LOOPS could address. 19 The present document is a strawman for the set of information that 20 would be interchanged in a LOOPS protocol, without already defining a 21 specific data packet format. 23 The generic information set needs to be mapped to a specific 24 encapsulation protocol to actually run the LOOPS optimizations. A 25 companion document contains a sketch of a binding to the tunnel 26 encapsulation protocol Geneve [I-D.ietf-nvo3-geneve]. 28 Status of This Memo 30 This Internet-Draft is submitted in full conformance with the 31 provisions of BCP 78 and BCP 79. 33 Internet-Drafts are working documents of the Internet Engineering 34 Task Force (IETF). Note that other groups may also distribute 35 working documents as Internet-Drafts. The list of current Internet- 36 Drafts is at https://datatracker.ietf.org/drafts/current/. 38 Internet-Drafts are draft documents valid for a maximum of six months 39 and may be updated, replaced, or obsoleted by other documents at any 40 time. It is inappropriate to use Internet-Drafts as reference 41 material or to cite them other than as "work in progress." 43 This Internet-Draft will expire on 14 January 2021. 45 Copyright Notice 47 Copyright (c) 2020 IETF Trust and the persons identified as the 48 document authors. All rights reserved. 50 This document is subject to BCP 78 and the IETF Trust's Legal 51 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 52 license-info) in effect on the date of publication of this document. 53 Please review these documents carefully, as they describe your rights 54 and restrictions with respect to this document. Code Components 55 extracted from this document must include Simplified BSD License text 56 as described in Section 4.e of the Trust Legal Provisions and are 57 provided without warranty as described in the Simplified BSD License. 59 Table of Contents 61 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 62 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 5 63 2. Challenges . . . . . . . . . . . . . . . . . . . . . . . . . 6 64 2.1. No Access to End-to-End Transport Information . . . . . . 6 65 2.2. Path Asymmetry . . . . . . . . . . . . . . . . . . . . . 6 66 2.3. Reordering vs. Spurious Retransmission . . . . . . . . . 6 67 2.4. Informing the End-to-End Transport . . . . . . . . . . . 7 68 3. Simplifying assumptions . . . . . . . . . . . . . . . . . . . 8 69 4. LOOPS Architecture . . . . . . . . . . . . . . . . . . . . . 9 70 5. LOOPS Generic Information Set . . . . . . . . . . . . . . . . 10 71 5.1. Setup Information . . . . . . . . . . . . . . . . . . . . 10 72 5.2. Forward Information . . . . . . . . . . . . . . . . . . . 10 73 5.3. Reverse Information . . . . . . . . . . . . . . . . . . . 11 74 6. LOOPS General Operation . . . . . . . . . . . . . . . . . . . 12 75 6.1. Initial Packet Sequence Number . . . . . . . . . . . . . 12 76 6.1.1. Minimizing collisions . . . . . . . . . . . . . . . . 12 77 6.1.2. Optional Initial PSN procedure . . . . . . . . . . . 12 78 6.2. Acknowledgement Generation . . . . . . . . . . . . . . . 13 79 6.3. Measurement . . . . . . . . . . . . . . . . . . . . . . . 14 80 6.3.1. Ingress-relative timestamps . . . . . . . . . . . . . 14 81 6.3.2. ACK generation . . . . . . . . . . . . . . . . . . . 15 82 6.4. Loss detection and Recovery . . . . . . . . . . . . . . . 15 83 6.4.1. Local Retransmission . . . . . . . . . . . . . . . . 15 84 6.4.2. FEC . . . . . . . . . . . . . . . . . . . . . . . . . 16 85 6.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . 16 86 7. Sketches of Bindings to Tunnel Protocols . . . . . . . . . . 16 87 7.1. Embedding LOOPS in Geneve . . . . . . . . . . . . . . . . 17 88 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17 89 9. Security Considerations . . . . . . . . . . . . . . . . . . . 17 90 9.1. Threat model . . . . . . . . . . . . . . . . . . . . . . 17 91 9.2. Discussion . . . . . . . . . . . . . . . . . . . . . . . 18 92 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 18 93 10.1. Normative References . . . . . . . . . . . . . . . . . . 18 94 10.2. Informative References . . . . . . . . . . . . . . . . . 19 95 Appendix A. Protocol used in Prototype Implementation . . . . . 21 96 A.1. Block Code FEC . . . . . . . . . . . . . . . . . . . . . 22 97 Appendix B. Transparent mode . . . . . . . . . . . . . . . . . . 22 98 B.1. Packet identification . . . . . . . . . . . . . . . . . . 24 99 B.2. Generic information and protocol operation . . . . . . . 25 100 B.3. A hybrid mode . . . . . . . . . . . . . . . . . . . . . . 25 101 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 27 102 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 27 104 1. Introduction 106 Today's networks exhibit a wide variety of data rates and, relative 107 to those, processing power and memory capacities of nodes acting as 108 routers. For instance, networks that employ tunneling to build 109 overlay networks may position powerful virtual router nodes in the 110 network to act as tunnel endpoints. The capabilities available in 111 the more powerful cases provide new opportunities for optimizations. 113 LOOPS (Local Optimizations on Path Segments) aims to provide local 114 (not end-to-end but in-network) recovery of lost packets to achieve 115 better data delivery. [I-D.li-tsvwg-loops-problem-opportunities] 116 provides an overview over the problems and optimization opportunities 117 that LOOPS could address. One simplifying assumption (Section 3) in 118 the present document is that LOOPS segments operate independently 119 from each other, each as a pair of a LOOPS Ingress and a LOOPS Egress 120 node. 122 The present document is a strawman for the set of information that 123 would be interchanged in a LOOPS protocol between these nodes, 124 without already defining a specific data packet format. The main 125 body of the document defines a mode of the LOOPS protocol that is 126 based on traditional tunneling, the "tunnel mode". Appendix B is an 127 even rougher strawman of a radically different, alternative mode that 128 we call "transparent mode", as well as a slightly more conventional 129 "hybrid mode" (Appendix B.3). These different modes may be 130 applicable to different usage scenarios and will be developed in 131 parallel, with a view of ultimately standardizing one or more of 132 them. 134 For tunnel mode, the generic information set needs to be mapped to a 135 specific encapsulation protocol to actually run the LOOPS 136 optimizations. LOOPS is not tied to any specific overlay protocol, 137 but is meant to run embedded into a variety of tunnel protocols. 138 LOOPS information is added as part of a tunnel protocol header at the 139 LOOPS ingress as shown in Figure 1. A companion document 140 [I-D.bormann-loops-geneve-binding] contains a sketch of a binding to 141 the tunnel encapsulation protocol Geneve [I-D.ietf-nvo3-geneve]. 143 +------------------------------------+ 144 | Outer header | 145 +------------------------------------+ 146 / | Tunnel Base Header | 147 / +------------------------------------+\ 148 Tunnel | +-------------------------+ | \ 149 Header ~ | LOOPS Information | ~ Tunnel Header 150 \ | +-------------------------+ | Extensions 151 \ +------------------------------------+ / 152 | Data packet | 153 +------------------------------------+ 155 Figure 1: Packet in Tunnel with LOOPS Information 157 Figure 2 is extracted from the LOOPS problems and opportunities 158 document [I-D.li-tsvwg-loops-problem-opportunities]. It illustrates 159 the basic architecture and terms of the applicable scenario of LOOPS. 160 Not all of the concepts introduced in the problems and opportunities 161 document are actually used in the current strawman specification; 162 Section 3 lays out some simplifying assumptions that the present 163 proposal makes. 165 ON=overlay node 166 UN=underlay node 168 +---------+ +---------+ 169 | App | <---------------- end-to-end ---------------> | App | 170 +---------+ +---------+ 171 |Transport| <---------------- end-to-end ---------------> |Transport| 172 +---------+ +---------+ 173 | | | | 174 | | +--+ path +--+ path segment2 +--+ | | 175 | | | |<-seg1->| |<--------------> | | | | 176 | Network | +--+ |ON| +--+ |ON| +--+ +----+ |ON| | Network | 177 | |--|UN|--| |--|UN|--| |--|UN|---| UN |--| |--| | 178 +---------+ +--+ +--+ +--+ +--+ +--+ +----+ +--+ +---------+ 179 End Host End Host 180 <---------------------------------> 181 LOOPS domain: path segments enabling 182 optimization for local in-network recovery 184 Figure 2: LOOPS Usage Scenario 186 1.1. Terminology 188 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 189 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 190 "OPTIONAL" in this document are to be interpreted as described in 191 BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all 192 capitals, as shown here. 194 This document makes use of the terminology defined in 195 [I-D.li-tsvwg-loops-problem-opportunities]. This section defines 196 additional terminology used by this document. 198 Data packets: The payload packets that enter and exit a LOOPS 199 segment. 201 LOOPS Segment: A part of an end-to-end path covered by a single 202 instance of the LOOPS protocol, the sub-path between the LOOPS 203 Ingress and the LOOPS Egress. Several LOOPS segments may be 204 encountered on an end-to-end path, with or without intervening 205 routers. 207 LOOPS Ingress: The node that forwards data packets and forward 208 information into the LOOPS segment, potentially performing 209 retransmission and forward error correction based on 210 acknowledgements and measurements received from the LOOPS Egress. 212 LOOPS Egress: The node that receives the data packets and forward 213 information from the LOOPS ingress, sends acknowledgements and 214 measurements back to the LOOPS ingress (reverse information), 215 potentially recovers data packets from forward error correction 216 information received. 218 LOOPS Nodes: Collective term for LOOPS Ingress and LOOPS Egress in a 219 LOOPS Segment. 221 Forward Information: Information that is added to the stream of data 222 packets in the forward direction by the LOOPS Ingress. 224 Reverse Information: Information that flows in the reverse 225 direction, from the LOOPS Egress back to the LOOPS Ingress. 227 Setup Information: Information that is not transferred as part of 228 the Forward or Reverse Information, but is part of the setup of 229 the LOOPS Nodes. 231 PSN: Packet Sequence Number, a sequence number identifying a data 232 packet between the LOOPS Ingress and Egress. 234 Sender: Original sender of a packet on an end-to-end path that 235 includes one or more LOOPS segment(s). 237 Receiver: Ultimate receiver of a packet on an end-to-end path that 238 includes one or more LOOPS segment(s). 240 2. Challenges 242 LOOPS has to perform well in the presence of some challenges, which 243 are discussed in this section. 245 2.1. No Access to End-to-End Transport Information 247 LOOPS is defined to be independent of the content of the packets 248 being forwarded: there is no dependency on transport-layer or higher 249 information. The intention is to keep LOOPS useful with a traffic 250 mix that may contain encrypted transport protocols such as QUIC as 251 well as encrypted VPN traffic. 253 2.2. Path Asymmetry 255 A LOOPS segment is defined as a unidirectional forwarding path. The 256 tunnel might be shared with a LOOPS segment in the inverse direction; 257 this then allows to piggyback Reverse Information on encapsulated 258 packets on that segment. But there is no guarantee that the inverse 259 direction of any end-to-end-path crosses that segment, so the LOOPS 260 optimizations have to be useful on their own in each direction. 262 2.3. Reordering vs. Spurious Retransmission 264 The end-to-end transport layer protocol may have its own 265 retransmission mechanism to recover lost packets. When LOOPS 266 recovers a loss, ideally this local recovery would replace the 267 triggering of a retransmission at the end-to-end sender. 269 Whether this is possible depends on the specific end-to-end mechanism 270 used for triggering retransmission. When end-to-end retransmission 271 is triggered by receiving a sequence of duplicate acknowledgements 272 (DUPACKs), and with more than a few packets in flight, the recovered 273 packet is likely to be too late to fill the hole in the sequence 274 number space that triggers the DUPACK detection. 276 (Given a reasonable setting of parameters, the local retransmission 277 will still arrive earlier than the end-to-end retransmission and will 278 possibly unblock application processing earlier; with spurious 279 retransmission detection, there also will be little long-term effect 280 on the send rate.) 281 While LOOPS makes no requirements on end-to-end protocols, it is 282 worth noting that the waste of bandwidth caused by a DUPACK-based 283 end-to-end retransmission can be avoided when the end-to-end loss 284 detection is based on time instead of sequence numbers, e.g., with 285 RACK [I-D.ietf-tcpm-rack]. This requires a limit on the additional 286 latency that LOOPS will incur in its attempt to recover the loss 287 locally. In the present version of this document, opportunity to set 288 such a limit is provided in the Setup Information. The limit can be 289 used to compute a deadline for retransmission, but also can be used 290 to choose FEC parameters that keep extra latency low. 292 2.4. Informing the End-to-End Transport 294 Congestion control at the end-to-end sender is used to adapt its 295 sending rate to the network congestion status. In typical TCP 296 senders, packet loss implies congestion and leads to a reduction in 297 sending rate. With LOOPS operating, packet loss can be masked from 298 the sender as the loss may have been locally recovered. In this 299 case, rate reduction may not be invoked at the sender. This is a 300 desirable performance improvement if the loss was a random loss, but 301 it is hard to ascertain that. 303 If LOOPS successfully conceals congestion losses from the end-to-end 304 transport protocol, that might increase the rate to a level that 305 congests the LOOPS segment, or that causes excessive queueing at the 306 LOOPS ingress. What LOOPS should be able to achieve is to let the 307 end host sender invoke the rate reduction mechanism when there is a 308 congestion loss no matter if the lost packet was recovered locally. 310 As with any tunneling protocol, information about congestion events 311 inside the tunnel needs to be exported to the end-to-end path the 312 tunnel is part of. See e.g., [RFC6040] for a discussion of how to do 313 this in the presence of ECN. A more recent draft, 314 [I-D.ietf-tsvwg-tunnel-congestion-feedback], proposes to activate ECN 315 for the tunnel regardless of whether the end-to-end protocol signals 316 the use of an ECN-capable transport (ECT), which requires more 317 complicated action at the tunnel egress. 319 A sender that interprets reordering as a signal of packet loss 320 (DUPACKs) initiates a retransmission and reduces the sending rate. 321 When spurious retransmission detection (e.g., via F-RTO [RFC5862] or 322 DSACK [RFC3708]) is enabled by the TCP sender, it will often be able 323 undo the unnecessary window reduction shortly afterwards. As LOOPS 324 recovers lost packets locally, in most cases the end host sender will 325 eventually find out its reordering-based retransmission (if any) is 326 spurious. This is an appropriate performance improvement if the loss 327 was a random loss. For congestion losses, a congestion event needs 328 to be signaled to the end-to-end transport. 330 The present version of LOOPS requires the end-to-end transport to be 331 ECN-capable (which is visible at the IP level). Congestion loss 332 events can easily be signaled to them by setting the CE (congestion 333 experienced) mark. Effectively, LOOPS converts a packet loss (which 334 would be a congestion indication) to a CE mark (which also is a 335 congestion indication). 337 In effect, LOOPS can be used to convert a path segment that does not 338 yet use CE marks for congestion indication, and drops packets 339 instead, into a segment that marks for congestion and does not drop 340 packets except in extreme cases, incurring the benefits of Using 341 Explicit Congestion Notification (ECN) [RFC8087]. We speak about the 342 "drop-to-mark" function of LOOPS. 344 3. Simplifying assumptions 346 The above notwithstanding, Implementations may want to make use of 347 indicators such as transport layer port numbers to partition a tunnel 348 flow into separate application flows, e.g., for active queue 349 management (AQM). Any such functionality is orthogonal to the LOOPS 350 protocol itself and thus out of scope for the present document. 352 One observation that simplifies the design of LOOPS in comparison to 353 that of a reliable transport protocol is that LOOPS does not _have_ 354 to recover every packet loss. Therefore, probabilistic approaches, 355 and simply giving up after some time has elapsed, can simplify the 356 protocol significantly. 358 For now, we assume that LOOPS segments that may line up on an end-to- 359 end path operate independently of each other. Since the objective of 360 LOOPS ultimately is to assist the end-to-end protocol, it is likely 361 that some cooperation between them would be beneficial, e.g., to 362 obtain some measurements that cover a larger part of the end-to-end 363 path. For instance, cooperating LOOPS segments could try to divide 364 up permissible increases to end-to-end latency between them. This is 365 out of scope for the present version. 367 Another simplifying assumption is that LOOPS nodes have reasonably 368 precise absolute time available to them, so there is no need to 369 burden the LOOPS protocol with time synchronization. How this is 370 achieved is out of scope. 372 LOOPS nodes are created and set up (information about their peers, 373 parameters) by some control plane mechanism that is out of scope for 374 this specification. This means there is no need in the LOOPS 375 protocol itself to manage setup information. 377 4. LOOPS Architecture 379 From the above, the following architecture is derived for LOOPS. 381 LOOPS governs the segment from an ingress node to an egress node, 382 which is part of one or more end-to-end paths. Often, a LOOPS 383 segment will operate on aggregate traffic from many such end-to-end 384 paths. 386 The LOOPS protocol itself does not define how a LOOPS segment and the 387 protocol entities in the ingress and egress node are set up. We 388 expect that a _setup protocol_ on the control plane will provide some 389 _setup information_ to the two nodes, including when to start and to 390 tear down processing. 392 Each LOOPS segment governs traffic on one direction in the segment. 393 The LOOPS ingress adds _forward information_ to that traffic; the 394 LOOPS egress removes the forward information and sends some _reverse 395 information_ to inform the behavior of the ingress. 397 Hence, in the data plane, forward information is added to each data 398 packet. Reverse information can be sent in separate packets (e.g., 399 Geneve control-only packets [I-D.ietf-nvo3-geneve]) and/or 400 piggybacked on a related, reverse-direction LOOPS flow, similar to 401 the way the forward information for that flow is carried. The setup 402 protocol is used to provide the relationship between the LOOPS 403 segments in the two directions that is used for piggybacking reverse 404 information. 406 The above describes the "tunnel mode". A transparent mode is 407 described in Appendix B, which does not modify the data packets and 408 therefore needs to send any forward information (if needed, e.g., for 409 FEC) in separate packets, usually aggregated. 411 The LOOPS _generic information set_ defines what information is 412 provided as setup information, forward information, and reverse 413 information. _Bindings_ map this information set to specific control 414 plane and data plane protocols, including defining the specific 415 encoding being used. Where separate packets (outside the data plane 416 protocols being used) need to be sent, a special UDP-based protocol 417 needs to be defined as well. The various bindings aim for some 418 commonality, so that an implementation for multiple bindings does not 419 need to support gratuitous variety between them. 421 5. LOOPS Generic Information Set 423 This section sketches a generic information set for the LOOPS 424 protocol. Entries marked with (*) are items that may not be 425 necessary and probably should be left out of an initial 426 specification. 428 5.1. Setup Information 430 Setup Information might include: 432 * encapsulation protocol in use, and its vital parameters 434 * identity of LOOPS ingress and LOOPS egress; information relevant 435 for running the encapsulation protocol such as port numbers 437 * target maximum latency increase caused by the operation of LOOPS 438 on this segment 440 * maximum retransmission count (*) 442 5.2. Forward Information 444 In the forward information, we have identified: 446 * tunnel type (a few bits, meaning agreed between Ingress and 447 Egress) 449 * packet sequence number PSN (20+ bits), counting the data packets 450 forwarded transmitted by the LOOPS ingress (i.e., retransmissions 451 re-use the PSN) 453 * an "ACK desirable" flag (one bit, usually set for a certain 454 percentage of the data packets only) 456 * an optional blob, to be echoed by the egress 458 * anything that the FEC scheme needs. 460 The first four together (say, 3+24+4+1) might even fit into 32 bits, 461 but probably need up to 48 bits total. FEC info of course often 462 needs more space. 464 (Note that in this proposal there is no timestamp in the forward 465 information; see Section 6.3.) 466 24 bits of PSN, minus one bit for sequence number arithmetic, gives 8 467 million packets (or 2.4 GB at typical packet sizes) per worst-case 468 RTT. So if that is, say, 30 seconds, this would be enough to fill 469 640 Mbit/s. 471 5.3. Reverse Information 473 For the reverse information, we have identified: 475 * one optional block 1, possibly repeated: 477 - PSN being acknowledged 479 - absolute time of reception for the packet acknowledged (PSN) 481 - the blob, if present, echoed back 483 * one optional block 2, possibly repeated: 485 - an ACK bitmap (based on PSN), always starting at a multiple of 486 8 488 - a delta indicating the end PSN of the bitmap (actually the 489 first PSN that is beyond it), using (Acked-PSN & ~7) + 490 8*(delta+1) as the end of the bitmap. Acked-PSN in that 491 formula is the previous block 1 PSN seen in this packet, or 0 492 if none so far. 494 Block 1 and Block 2 can be interspersed and repeated. They can be 495 piggybacked on a reverse direction data packet or sent separately if 496 none occurs within some timeout. They will usually be aggregated in 497 some useful form. Block 1 information sets are only returned for 498 packets that have "ACK desirable" set. Block 2 information is sent 499 by the receiver based on some saturation scheme (e.g., at least three 500 copies for each PSN span over time). Still, it might be possible to 501 go down to 1 or 2 amortized bytes per forward packet spent for all 502 this. 504 The latency calculation is done by the sender, who occasionally sets 505 "ACK desirable", and notes down the absolute time of transmission for 506 this data packet (the timekeeping can be done quite efficiently as 507 deltas). Upon reception of a block 1 ACK, it can then subtract that 508 from the absolute time of reception indicated. This assumes time 509 synchronization between the nodes is at least as good as the 510 precision of latency measurement needed, which should be no problem 511 with IEEE 1588 PTP synchronization (but could be if using NTP-based 512 synchronization only). A sender can freely garbage collect noted 513 down transmission time information; doing this too early just means 514 that the quality of the RTT sampling will reduce. 516 6. LOOPS General Operation 518 In the Tunnel Mode described in the main body of this document, LOOPS 519 information is carried by some tunnel encapsulation. 521 6.1. Initial Packet Sequence Number 523 There is no connection establishment procedure in LOOPS. The initial 524 PSN is assigned unilaterally by the LOOPS Ingress. 526 Because of the short time that is usually set in the maximum latency 527 increase, there is little damage from a collision of PSNs with 528 packets still in flight from previous instances of LOOPS. 530 6.1.1. Minimizing collisions 532 If desired, collisions can be minimized by assigning initial PSNs 533 randomly, or using stable storage. Random assignment is more useful 534 for longer PSNs, where the likelihood of overlap will be low. The 535 specific way a LOOPS ingress uses stable storage is a local matter 536 and thus out of scope. (Implementation note: this can be made to 537 work similar to secure nonce generation with write attenuation: Say, 538 every 10000 packets, the sender notes down the PSN into stable 539 storage. After a reboot, it reloads the PSN and adds 10000 in 540 sequence number arithmetic [RFC1982], plus maybe another 10000 so the 541 sender does not have to wait for the store operation to succeed 542 before sending more packets.) 544 6.1.2. Optional Initial PSN procedure 546 As a potential option (to be discussed), an initial packet sequence 547 number could be communicated using a simple two-bit protocol, based 548 on an I flag (Initial PSN) carried in the forward information and an 549 R flag (Initial PSN Received) in the reverse information. This 550 procedure essentially clears the egress of any previous state, 551 however, the benefits of this procedure are limited. 553 The initial PSN is assigned unilaterally by the LOOPS ingress, 554 selected randomly. The ingress will keep setting the I flag to one 555 when it starts to send packets from a new beginning or whenever it 556 believes there is a need to notify the egress about a new initial 557 PSN. The ingress will stop setting the I flag when it receives an 558 acknowledgement with the R flag set from the egress. 560 When the LOOPS egress receives a packets with the I flag set, it 561 stops performing services that assume a sequential PSN. The egress 562 will no longer provide acknowledgement information for the packets 563 with PSN smaller than this new initial PSN (per sequence number 564 arithmetic [IEN74]). The egress sends acknowledgement information 565 back without any delay by echoing the value of the I flag in the R 566 flag. This also means the egress unsets the R flag in subsequent 567 acknowledgements for packets with the I flag unset. 569 It may happen that the first few packets are lost in an initial PSN 570 assignment process. In this case, the loss of these packets is not 571 detectable by the LOOPS ingress since the first received PSN will be 572 treated as an initial PSN at the egress. This is an acceptable 573 temporary performance degradation: LOOPS does not intend to provide 574 perfect reliability, and LOOPS usually applies to the aggregated 575 traffic over a tunnel so that the initial PSN assignment happens 576 infrequently. 578 6.2. Acknowledgement Generation 580 A data packet forwarded by the LOOPS ingress always carries PSN 581 information. The LOOPS egress uses the largest newly received PSN 582 with the "ACK desired" bit as the ACK number in the block 1 part of 583 the acknowledgement. This means that the LOOPS ingress gets to 584 modulate the number of acknowledgement sent by the LOOPS egress. 585 However, whenever an out-of-order packet arrives while there still 586 are "holes" in the PSNs received, the LOOPS receiver should generate 587 a block 2 acknowledgement immediately that the LOOPS sender can use 588 as an ACK list. 590 Reverse information can be piggybacked in a reverse direction data 591 packet. When the reverse direction has no user data to be sent, a 592 pure reverse information packet needs to be generated. This may be 593 based on a short delay during which the LOOPS egress waits for a data 594 packet to piggyback on. (To reduce MTU considerations, the egress 595 could wait for less-than-full data packets.) 597 6.3. Measurement 599 When sending a block 1 acknowledgement, the LOOPS egress indicates 600 the absolute time of reception of the packet. The LOOPS ingress can 601 subtract the absolute time of transmission that it still has 602 available, resulting in one high quality latency sample. (In an 603 alternative design, the forward information could include the 604 absolute time of transmission as well, and block1 information would 605 echo it back. This trades memory management at the ingress for 606 increased bandwidth and MTU reduction.) 608 When a data packet has been transmitted, it may not be clear which 609 specific copy is acknowledged in a block 1 acknowledgement: the 610 acknowledgement for the initial (or, more generally, an earlier) copy 611 may have been delayed (ACK ambiguity)). The LOOPS ingress therefore 612 SHOULD NOT base its measurements on acknowledgements for 613 retransmitted data packets. One way to achieve this is by not 614 setting the "ACK desired" bit on retransmissions in the first place. 616 The LOOPS ingress can also use the time of reception of the block 1 617 acknowledgement to obtain a segment RTT sample. Note that this will 618 include any wait time the LOOPS egress incurs while waiting for a 619 piggybacking opportunity -- this is appropriate, as all uses of an 620 RTT will be for keeping a retransmission timeout. 622 To maintain quality of information during idle times, the LOOPS 623 ingress may send keepalive packets, which are discarded at the LOOPS 624 egress after sending acknowledgements. The indication that a packet 625 is a keepalive packet is dependent on the encapsulation protocol. 627 6.3.1. Ingress-relative timestamps 629 As an optional procedure, the ingress node can attach a small blob of 630 data to a forward packet that carries an ACK desired flag; this blob 631 is then echoed by the egress in its block 1 acknowledgement. This is 632 typically used to attach a timestamp on a time scale defined by the 633 ingress; we speak of an ingress-relative timestamp. Alternatively, 634 the ingress can keep a timestamp in its local storage, associated 635 with the PSN of the packet that carries an ACK desired flag; it can 636 then retrieve this timestamp when the block 1 acknowledgement 637 arrives. 639 In either case, the LOOPS ingress keeps track of the local segment 640 round trip time (LRTT) based on the (saved or received) timestamp and 641 the arrival time of the block 1 acknowledgement, by setting the ACK 642 Desired flag (D flag) occasionally (several times per RTT) and 643 saving/including a sending timestamp for/in the packet. 645 As the egress will send block 1 acknowledgement information right 646 away when it receives a packet with the D flag set, the measurement 647 of LRTT is more accurate for such packets. A smoothed local segment 648 round trip time S_LRTT can be computed in a similar way as defined by 649 [RFC0793]. A recent minimum value of LRTT is also kept as min_LRTT. 650 S_LRTT is used as a basis for the overall timing of retransmission 651 and state management. 653 Retransmitted packets MUST NOT be used for local segment round trip 654 time (LRTT) calculation. 656 6.3.2. ACK generation 658 A block 1 acknowledgement is generated based on receiving a forward 659 packet with a D flag. 661 The way block 2 acknowledgement information is sent is more subject 662 to control by the egress. Generally, the egress will aggregate ACK 663 bits for at least K packets before sending a block 2; this can be 664 used to amortize the overhead to close to a couple of bits per ACK. 665 In order to counter loss of reverse information packets, an egress 666 will also want to send an ACK bit more than once -- a saturation 667 value of 3 or more may be chosen based on setup information. 668 Typically, ACK bits already sent will be prepended to ACK bits that 669 are new in this block 2 information set. If K packets do not 670 accumulate for a while, the egress will send one or more packets with 671 block 2 information that covers the unsent ACK bits it has so far. 673 (Discussion: This works best if the egress has information both about 674 the S_RTT and min_RTT that the ingress uses and the reverse packet 675 loss rate.) 677 6.4. Loss detection and Recovery 679 There are two ways for LOOPS local recovery, retransmission and FEC. 681 6.4.1. Local Retransmission 683 When retransmission is used as recovery mechanism, the LOOPS ingress 684 detects a packet loss by not receiving an ACK for the packet within 685 the time expected based on an RTO value (which might be calculated as 686 in [RFC6298]). Packet retransmission should then not be performed 687 more than once within an LRTT. 689 When a retransmission is desired, the LOOPS ingress performs the 690 local in-network recovery by retransmitting the packet. Further 691 retransmissions may be desirable if the lack of ACK is persistent 692 beyond an RTO, as long as the maximum latency increase is not 693 reached. 695 6.4.2. FEC 697 FEC is another way to perform local recovery. When FEC is in use, a 698 FEC header is sent with data packets as well as with special repair 699 packets added to the flow. The specific FEC scheme used could be 700 defined in the Setup Information, using a mechanism like [RFC5052]. 701 The FEC rate (amount of redundancy added) and possibly the FEC scheme 702 could be unilaterally adjusted by the LOOPS ingress in an adaptive 703 mechanism based on the measurement information. 705 6.5. Discussion 707 Without progress in the way that end-host transport protocols handle 708 reordering, LOOPS will be unable to prevent end-to-end 709 retransmissions that duplicate effort that is spent in local 710 retransmissions. It depends on parameters of the path segment 711 whether this wasted effort is significant or not. 713 One remedy against this waste could be the introduction of 714 resequencing at the LOOPS Egress node. This increases overall mean 715 packet latency, but does not always increase actual end-to-end data 716 stream latency if a head-of-line blocking transport such as TCP is in 717 use. For applications with a large percentage of legacy TCP end- 718 hosts and sufficient processing capabilities at the LOOPS Egress 719 node, resequencing may be a viable choice. Note that resequencing 720 could be switched off and on depending on some measurement 721 information. 723 The packet numbering scheme chosen by LOOPS already provides the 724 necessary information for the LOOPS Egress to reconstruct the 725 sequence of data packets at the LOOPS ingress. 727 7. Sketches of Bindings to Tunnel Protocols 729 The LOOPS information defined above in a generic way can be mapped to 730 specific tunnel encapsulation protocols. A sketch for the tunnel 731 protocol Geneve is given below (Section 7.1). The actual 732 encapsulation can be designed in a "native" way by putting each of 733 the various elements into the TLV format of the encapsulation 734 protocol, or it can be achieved by providing single TLVs for forward 735 and reverse information and using some generic encoding of both kinds 736 of information as shown in Appendix B.3. 738 7.1. Embedding LOOPS in Geneve 740 Geneve [I-D.ietf-nvo3-geneve] is an extensible overlay protocol which 741 can embed LOOPS functions. Geneve uses TLVs to carry optional 742 information between NVEs. NVE is logically the same entity as the 743 LOOPS node. 745 The Geneve header has a mandatory Virtual Network Identifier (VNI) 746 field. The specific VNI value to be used is part of the setup 747 information for the LOOPS tunnel. 749 More details for a Geneve binding for LOOPS can be found in 750 [I-D.bormann-loops-geneve-binding]. 752 8. IANA Considerations 754 No IANA action is required at this stage. When a LOOPS 755 representation is designed for a specific tunneling protocol, new 756 codepoints will be required in the registries that pertain to that 757 protocol. 759 9. Security Considerations 761 The security of a specific LOOPS segment will depend both on the 762 properties of the generic information set described here and those of 763 the encapsulation protocol employed. The security considerations of 764 the encapsulation protocol will apply, as will the protection 765 afforded by any security measures provided by the encapsulation 766 protocol. Any LOOPS encapsulation specification is expected to 767 provide information about preferred configurations of the 768 encapsulation protocol employed, including security mechanisms, and 769 to provide a security considerations section discussing the 770 combination. The following discussion aims at discussing security 771 considerations that will be common between different encapsulations. 773 9.1. Threat model 775 Attackers might attempt to perturb the operation of a LOOPS segment 776 for a number of purposes: 778 * Denial of Service: Damaging the ability of LOOPS to recover 779 packets, or damaging packet forwarding through the LOOPS segment 780 in general. 782 * Attacks on Confidentiality or Integrity: Obtaining the content of 783 data packets, modifying them, injecting new or suppressing 784 specific data packets. 786 For the purposes of these security considerations, we can distinguish 787 three classes of attackers: 789 1. on-path read-write: The attacker sees packets under way on the 790 segment and can modify, inject, or suppress them. 792 In this case there is really nothing LOOPS can do, except for 793 acting as a full security protocol on its own, which would be the 794 task of the encapsulation protocol. Without that, attackers 795 already can manipulate the packet stream as they wish. This 796 class of attackers is considered out of scope for these security 797 considerations. 799 2. on-path read + inject: The attacker sees packets under way on the 800 segment and can inject new packets. 802 For this case, LOOPS itself similarly cannot add to the 803 confidentiality of the data stream. However, LOOPS could protect 804 against denial of service against its own protocol operation and, 805 in a limited fashion, against attacks on integrity that wouldn't 806 already have been possible by packet injection without LOOPS. 808 3. off-path inject: The attacker can inject new packets, but cannot 809 see existing packets under way on the segment. 811 Similar considerations apply as for class 2, except that the 812 "blind" class 3 attacker might need to guess information it could 813 have extracted from the packet stream in class 2. 815 9.2. Discussion 817 Class 2 attackers can see e.g. sequence numbers and can inject, but 818 not modify traffic. Attacks might include injecting false ACKs, 819 initial PSN flags, ... (TBD) 821 Class 3 ("blind") attackers might still be able to fake initial PSN 822 bits + false ACKs, but will have a harder time otherwise as it would 823 need to guess the PSN range in which it can wreak havoc. Even random 824 guesses will sometimes hit, though, so the protocol needs to be 825 robust to such injection attacks. ... (TBD) 827 10. References 829 10.1. Normative References 831 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 832 Requirement Levels", BCP 14, RFC 2119, 833 DOI 10.17487/RFC2119, March 1997, 834 . 836 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 837 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 838 May 2017, . 840 10.2. Informative References 842 [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, 843 RFC 793, DOI 10.17487/RFC0793, September 1981, 844 . 846 [RFC1982] Elz, R. and R. Bush, "Serial Number Arithmetic", RFC 1982, 847 DOI 10.17487/RFC1982, August 1996, 848 . 850 [RFC3708] Blanton, E. and M. Allman, "Using TCP Duplicate Selective 851 Acknowledgement (DSACKs) and Stream Control Transmission 852 Protocol (SCTP) Duplicate Transmission Sequence Numbers 853 (TSNs) to Detect Spurious Retransmissions", RFC 3708, 854 DOI 10.17487/RFC3708, February 2004, 855 . 857 [RFC5052] Watson, M., Luby, M., and L. Vicisano, "Forward Error 858 Correction (FEC) Building Block", RFC 5052, 859 DOI 10.17487/RFC5052, August 2007, 860 . 862 [RFC5862] Yasukawa, S. and A. Farrel, "Path Computation Clients 863 (PCC) - Path Computation Element (PCE) Requirements for 864 Point-to-Multipoint MPLS-TE", RFC 5862, 865 DOI 10.17487/RFC5862, June 2010, 866 . 868 [RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion 869 Notification", RFC 6040, DOI 10.17487/RFC6040, November 870 2010, . 872 [RFC6298] Paxson, V., Allman, M., Chu, J., and M. Sargent, 873 "Computing TCP's Retransmission Timer", RFC 6298, 874 DOI 10.17487/RFC6298, June 2011, 875 . 877 [RFC6330] Luby, M., Shokrollahi, A., Watson, M., Stockhammer, T., 878 and L. Minder, "RaptorQ Forward Error Correction Scheme 879 for Object Delivery", RFC 6330, DOI 10.17487/RFC6330, 880 August 2011, . 882 [RFC8610] Birkholz, H., Vigano, C., and C. Bormann, "Concise Data 883 Definition Language (CDDL): A Notational Convention to 884 Express Concise Binary Object Representation (CBOR) and 885 JSON Data Structures", RFC 8610, DOI 10.17487/RFC8610, 886 June 2019, . 888 [RFC8087] Fairhurst, G. and M. Welzl, "The Benefits of Using 889 Explicit Congestion Notification (ECN)", RFC 8087, 890 DOI 10.17487/RFC8087, March 2017, 891 . 893 [I-D.ietf-tcpm-rack] 894 Cheng, Y., Cardwell, N., Dukkipati, N., and P. Jha, "RACK: 895 a time-based fast loss detection algorithm for TCP", Work 896 in Progress, Internet-Draft, draft-ietf-tcpm-rack-08, 9 897 March 2020, . 900 [I-D.ietf-nvo3-geneve] 901 Gross, J., Ganga, I., and T. Sridhar, "Geneve: Generic 902 Network Virtualization Encapsulation", Work in Progress, 903 Internet-Draft, draft-ietf-nvo3-geneve-16, 7 March 2020, 904 . 907 [I-D.li-tsvwg-loops-problem-opportunities] 908 Yizhou, L., Zhou, X., Boucadair, M., Wang, J., and F. Qin, 909 "LOOPS (Localized Optimizations on Path Segments) Problem 910 Statement and Opportunities for Network-Assisted 911 Performance Enhancement", Work in Progress, Internet- 912 Draft, draft-li-tsvwg-loops-problem-opportunities-05, 6 913 July 2020, . 916 [I-D.ietf-tsvwg-tunnel-congestion-feedback] 917 Wei, X., Yizhou, L., Boutros, S., and L. Geng, "Tunnel 918 Congestion Feedback", Work in Progress, Internet-Draft, 919 draft-ietf-tsvwg-tunnel-congestion-feedback-07, 5 May 920 2019, . 923 [I-D.bormann-loops-geneve-binding] 924 Bormann, C., "Embedding LOOPS in Geneve", Work in 925 Progress, Internet-Draft, draft-bormann-loops-geneve- 926 binding-01, 12 June 2020, . 929 [IEN74] Plummer, W.W., "Sequence Number Arithmetic", Internet 930 Experiment Note 74, September 1978. 932 Appendix A. Protocol used in Prototype Implementation 934 This appendix describes, in a somewhat abstracted form, the protocol 935 as used in a prototype implementation, as described by Yizhou Li, and 936 Xingwang Zhou. 938 The prototype protocol can be run in one of two modes (defined by 939 preconfiguration): 941 * Retransmission mode 943 * Forward Error Correction (FEC) mode 945 Forward information is piggybacked in data packets. 947 Reverse information can be carried in a pure acknowledgement packet 948 or piggybacked when carrying packets for the inverse direction. 950 The forward information includes: 952 * Packet Sequence Number (PSN) (32 bits): This identifies a packet 953 over a specific overlay segment from a specific LOOPS Ingress. If 954 a packet is retransmitted by LOOPS, the retransmission uses the 955 original PSN. 957 * Timestamp (32 bits): Information, in a format local to the LOOPS 958 ingress, that provides the time when the packet was sent. In the 959 current implementation, a 32-bit unsigned value specifying the 960 time delta in some granularity from the epoch time to the sending 961 time of the packet carrying this timestamp. The granularity can 962 be from 1 ms to 1 second. The epoch time follows the current TCP 963 practice which is 1 January 1970 00:00:00 UTC. Note that a 964 retransmitted packet uses its own Timestamp. 966 * FEC Info for Block Code (56 bits): This header is used in FEC 967 mode. It currently only provides for a block code FEC scheme. It 968 includes the Source Block Number (SBN), Encoding Symbol ID (ESI), 969 number of symbols in a single source block and symbol size. 970 Appendix A.1 gives more details on FEC. 972 The reverse information includes: 974 * ACK Number (32 bits): The largest (in sequence number arithmetic 975 [RFC1982]) PSN received so far. 977 * ACK List (variable): This indicates an array of PSN numbers to 978 describe the PSN "holes" preceding the ACK number. It 979 conceptually lists the PSNs of every packet perceived as lost by 980 the LOOPS egress. In actual use, it is truncated. 982 * Echoed Timestamp (32 bits): The timestamp received with the packet 983 being acknowledged. 985 A.1. Block Code FEC 987 The prototype currently uses a block code FEC scheme (RaptorQ 988 [RFC6330]). The fields in the FEC Info forward information are: 990 * Source Block Number (SBN): 16 bits. An integer identifier for the 991 source block that the encoding symbols within the packet relate 992 to. 994 * Encoding Symbol ID (ESI): 16 bits. An integer identifier for the 995 encoding symbols within the packet. 997 * K: 8 bits. Number of symbols in a single source block. 999 * T: 16 bits. Symbol size in bytes. 1001 The LOOPS Ingress uses the data packet in Figure 1 to generate the 1002 encoding packet. Both source packets and repair packets carry the 1003 FEC header information; the LOOPS Egress reconstructs the data 1004 packets from both kinds of packets. The LOOPS Egress currently 1005 resequences the forwarded and reconstructed packets, so they are 1006 passed on in-order when the lost packets are recoverable within the 1007 source block. 1009 The LOOPS Nodes need to agree on the use of FEC block mode and on the 1010 specific FEC Encoding ID to use; this is currently done by 1011 configuration. 1013 Appendix B. Transparent mode 1015 This appendix defines a very different way to provide the LOOPS 1016 services, "transparent mode". (We call the protocol described in the 1017 main body of the document "encapsulated mode".) 1018 In transparent mode, the idea is that LOOPS does not meddle with the 1019 forward transmission of data packets, but runs on the side exchanging 1020 additional information. 1022 An implementation could be based on conventional forwarding switches 1023 that just provide a copy of the ingress and egress packet stream to 1024 the LOOPS implementations. The LOOPS process would occasionally 1025 inject recovered packets back into the LOOPS egress node's forwarding 1026 switch, see Figure 3. 1028 | 1029 +-------+-------------------------------------------+ 1030 | | | 1031 | +----+--------+ +-------------------+ | 1032 | | | copy | | | | 1033 | | |----------------> LOOPS ingress | | 1034 | | | | | | ^ | | 1035 | +----+--------+ +-----|-----|-------+ | 1036 | data|packets forward| |reverse | 1037 | | info| |info | 1038 +-------+------------------|-----|------------------+ 1039 | | | 1040 +-------+------------------|-----|------------------+ 1041 | | | | | 1042 | +----+---------+ +----|-----|----------+ | 1043 | | | copy | | v | | | 1044 | | |---------|---|---> LOOPS egress | | 1045 | | | | | | | 1046 | | |<--------|---|---- inject | | 1047 | +----+---------+ +---------------------+ | 1048 | | | 1049 +-------+-------------------------------------------+ 1050 | 1051 v 1053 Figure 3: LOOPS Transparent Mode 1055 The obvious advantage of transparent mode is that no encapsulation is 1056 needed, reducing processing requirements and keeping the MTU 1057 unchanged. The obvious disadvantage is that no forward information 1058 can be provided with each data packet, so a replacement needs to be 1059 found for the PSN (packet sequence number) employed in encapsulated 1060 mode. Any forward information beyond the data packets is sent in 1061 separate packets exchanged directly between the LOOPS nodes. 1063 B.1. Packet identification 1065 Retransmission mode and FEC mode differ in their needs for packet 1066 identification. For retransmission mode, a somewhat probabilistic 1067 accuracy of the packet identification is sufficient, for FEC mode, 1068 packet identification should not make mistakes (as these would lead 1069 to faultily reconstructed packets). 1071 In Retransmission mode, misidentification of a packet could lead to 1072 measurement errors as well as missed retransmission opportunities. 1073 The latter will be fixed end-to-end. The tolerance for measurement 1074 errors would influence the degree of accuracy that is aimed for. 1076 Packet identification can be based on a cryptographic hash of the 1077 packet, computed in LOOPS ingress and egress using the same algorithm 1078 (excluding fields that can change in transit, such as TTL/hop limit). 1079 The hash can directly be used as a packet number, or it can be sent 1080 in the forward information together with a packet sequence number, 1081 establishing a mapping. 1083 For probabilistic packet identification, it is almost always 1084 sufficient to hash the first few (say, 64) bytes of the packet; all 1085 known transport protocols keep sufficient identifying information in 1086 that part (and, for encrypted protocols, the entropy will be 1087 sufficient). Any collisions of the hash could be used to disqualify 1088 the packet for measurement purposes, minimizing the measurement 1089 errors; this could allow rather short packet identifiers in 1090 retransmission mode. 1092 For FEC mode, the packet identification together with the per-packet 1093 FEC information needs to be sent in the (separate) forward 1094 information, so that a systematic code can be reconstructed. For 1095 retransmission mode, there is no need to send any forward information 1096 for most packets, or a mapping from packet identifiers to packet 1097 sequence numbers could be sent in the forward information (probably 1098 in some aggregated form). The latter would allow keeping the 1099 acknowledgement form described in the main body (with aggregate 1100 acknowledgement); otherwise, packet identifiers need to be 1101 acknowledged. With this change, the LOOPS egress will send reverse 1102 information as in the encapsulating LOOPS protocol. 1104 B.2. Generic information and protocol operation 1106 With the changes outlined above, transparent mode operates just as 1107 encapsulated mode. If packet sequence numbers are not used, there is 1108 no use for block2 reverse information; if they are used, a new block3 1109 needs to be defined that provides the mapping from packet identifiers 1110 to packet sequence numbers in the forward information. To avoid MTU 1111 reduction, some mechanism will be needed to encapsulate the actual 1112 FEC information (additional packets) in the forward information. 1114 B.3. A hybrid mode 1116 Figure 3 can be modified by including a GRE encapsulator into the top 1117 left corner and a GRE decapsulator in the bottom left corner. This 1118 provides more defined ingress and egress points, but it also provides 1119 an opportunity to add a packet sequence number at the ingress. The 1120 copies to the top right and bottom right corners are the encapsulated 1121 form, i.e., include the sequence number. 1123 The GRE packet header then has the form: 1125 0 1 2 3 1126 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1127 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1128 |0|0|0|1| 000000000 | 000 | Protocol Type | 1129 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1130 | Sequence Number | 1131 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1133 The forward and reverse information can be designed closer to the 1134 approach in the main body of the document, to be exchanged using UDP 1135 packets between top right ingress and bottom right egress using a 1136 port number allocated for this purpose. 1138 Rough ideas for both directions are given below in CDDL [RFC8610]. 1139 This information set could be encoded in CBOR or in a bespoke 1140 encoding; details such as this can be defined later. 1142 forward-information = [ 1143 [rel-psn, ack-desired, ? fec-info] / 1144 fec-repair-data 1145 ] 1147 rel-psn = uint; relative packet sequence number 1148 ; always given as a delta from the previous one in the array 1149 ; starting out with a "previous value" of 0 1151 ack-desired = bool 1153 fec-info = [ 1154 sbn: uint, ; Source Block Number 1155 esi: uint, ; Encoding Symbol ID 1156 ? ( 1157 nsssb: uint; number of symbols in a single source block 1158 ss: uint; symbol size 1159 ) 1160 ] 1162 fec-repair-data = [ 1163 repair-data: bytes 1164 ? ( 1165 sbn: uint, ; Source Block Number 1166 esi: uint, ; Encoding Symbol ID 1167 ) 1168 ] 1170 If left out for a sequence number, the fec-info block is constructed 1171 by adding one to the previous one. fec-repair-data contain repair 1172 symbols for the sbn/esi given (which, again, are reconstructed from 1173 context if not given). 1175 reverse-information = [ 1176 block1 / block2 1177 ] 1179 block1 = [rel-psn, timestamp] 1180 block2 = [end-psn-delta: uint, acked-bits: bytes] 1182 The acked-bits in a block2 is a bitmap that gives acknowledgments for 1183 received data packets. The bitmap always comes as a multiple of 8 1184 bits (all bytes are filled in with 8 bits, each identifying a PSN). 1185 The end PSN of the bitmap (actually the first PSN that would be 1186 beyond it) is computed from the current PSN as set by rel-psn, 1187 rounded down to a multiple of 8, and adding 8*(end-psn-delta+1) to 1188 that value. 1190 Acknowledgements 1192 Sami Boutros helped with sketching the use of Geneve (Section 7.1). 1194 Michael Welzl has been supported by the Research Council of Norway 1195 under its "Toppforsk" programme through the "OCARINA" project. 1197 Authors' Addresses 1199 Michael Welzl 1200 University of Oslo 1201 PO Box 1080 Blindern 1202 N-0316 Oslo 1203 Norway 1205 Phone: +47 22 85 24 20 1206 Email: michawe@ifi.uio.no 1208 Carsten Bormann (editor) 1209 Universität Bremen TZI 1210 Postfach 330440 1211 D-28359 Bremen 1212 Germany 1214 Phone: +49-421-218-63921 1215 Email: cabo@tzi.org