idnits 2.17.1 draft-li-tsvwg-loops-problem-opportunities-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 13, 2020) is 1354 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Obsolete informational reference (is this intentional?): RFC 793 (Obsoleted by RFC 9293) -- Obsolete informational reference (is this intentional?): RFC 6830 (Obsoleted by RFC 9300, RFC 9301) == Outdated reference: A later version (-15) exists of draft-ietf-tcpm-rack-08 == Outdated reference: A later version (-04) exists of draft-welzl-loops-gen-info-03 Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TSVWG Y. Li 3 Internet-Draft X. Zhou 4 Intended status: Informational Huawei 5 Expires: January 14, 2021 M. Boucadair 6 Orange 7 J. Wang 8 China Telecom 9 F. Qin 10 China Mobile 11 July 13, 2020 13 LOOPS (Localized Optimizations on Path Segments) Problem Statement and 14 Opportunities for Network-Assisted Performance Enhancement 15 draft-li-tsvwg-loops-problem-opportunities-06 17 Abstract 19 In various network deployments, end to end forwarding paths are 20 partitioned into multiple segments. For example, in some cloud-based 21 WAN communications, stitching multiple overlay tunnels are used for 22 traffic policy enforcement matters such as to optimize traffic 23 distribution or to select paths exposing a lower latency. Likewise, 24 in satellite communications, the communication path is decomposed 25 into two terrestrial segments and a satellite segment. Such long- 26 haul paths are naturally composed of multiple network segments with 27 various encapsulation schemes. Packet loss may show different 28 characteristics on different segments. 30 Traditional transport protocols (e.g., TCP) respond to packet loss 31 slowly especially in long-haul networks: they either wait for some 32 signal from the receiver to indicate a loss and then retransmit from 33 the sender or rely on sender's timeout which is often quite long. 34 With the increase of end-to-end transport encryption (e.g., QUIC), 35 traditional PEP (performance enhancing proxy) techniques such as TCP 36 splitting are no longer applicable. 38 LOOPS (Local Optimizations on Path Segments) is a network-assisted 39 performance enhancement over path segment and it aims to provide 40 local in-network recovery to achieve better data delivery by making 41 packet loss recovery faster. In an overlay network scenario, LOOPS 42 can be performed over a variety of the existing, or purposely 43 created, tunnel-based path segments. 45 Status of This Memo 47 This Internet-Draft is submitted in full conformance with the 48 provisions of BCP 78 and BCP 79. 50 Internet-Drafts are working documents of the Internet Engineering 51 Task Force (IETF). Note that other groups may also distribute 52 working documents as Internet-Drafts. The list of current Internet- 53 Drafts is at https://datatracker.ietf.org/drafts/current/. 55 Internet-Drafts are draft documents valid for a maximum of six months 56 and may be updated, replaced, or obsoleted by other documents at any 57 time. It is inappropriate to use Internet-Drafts as reference 58 material or to cite them other than as "work in progress." 60 This Internet-Draft will expire on January 14, 2021. 62 Copyright Notice 64 Copyright (c) 2020 IETF Trust and the persons identified as the 65 document authors. All rights reserved. 67 This document is subject to BCP 78 and the IETF Trust's Legal 68 Provisions Relating to IETF Documents 69 (https://trustee.ietf.org/license-info) in effect on the date of 70 publication of this document. Please review these documents 71 carefully, as they describe your rights and restrictions with respect 72 to this document. Code Components extracted from this document must 73 include Simplified BSD License text as described in Section 4.e of 74 the Trust Legal Provisions and are provided without warranty as 75 described in the Simplified BSD License. 77 Table of Contents 79 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 80 1.1. The Problem and Opportunity Overview . . . . . . . . . . 3 81 1.2. Sketching a Work Direction: Rationale & Goals . . . . . . 4 82 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 83 3. Usage Scenarios . . . . . . . . . . . . . . . . . . . . . . . 6 84 3.1. Cloud-Internet Overlay Network . . . . . . . . . . . . . 6 85 3.2. Satellite Communication . . . . . . . . . . . . . . . . . 8 86 3.3. Branch Office WAN Connection . . . . . . . . . . . . . . 9 87 4. Impact of Packet loss . . . . . . . . . . . . . . . . . . . . 10 88 4.1. Tail Loss or Loss in Short Flows . . . . . . . . . . . . 10 89 4.2. Packet Loss in Real Time Media Streams . . . . . . . . . 11 90 5. Features to be Considered for LOOPS . . . . . . . . . . . . . 11 91 5.1. Local Recovery . . . . . . . . . . . . . . . . . . . . . 11 92 5.2. Congestion Control Interaction . . . . . . . . . . . . . 12 93 5.3. Overlay Protocol Extensions . . . . . . . . . . . . . . . 12 94 6. Local in-network Recovery and End-to-end Retransmission . . . 13 95 7. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 96 8. Security Considerations . . . . . . . . . . . . . . . . . . . 15 97 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 15 98 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 15 99 11. Informative References . . . . . . . . . . . . . . . . . . . 15 100 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 18 102 1. Introduction 104 1.1. The Problem and Opportunity Overview 106 Packet loss is ubiquitous in Internet. A reliable transport layer 107 normally employs some end-to-end retransmission mechanisms which also 108 address congestion control [RFC0793] [RFC5681]. The sender either 109 waits for the receiver to send some signals on a packet loss or sets 110 some form of timeout for retransmission. For unreliable transport 111 protocols such as RTP [RFC3550], optional and limited usage of end- 112 to-end retransmission is employed to recover from packet loss 113 [RFC4585] [RFC4588]. End-to-end retransmission to recover lost 114 packets is slow especially when the network is long-haul. For short- 115 lived flows and transactional flows, latency suffers a lot from tail 116 loss. 118 Tunnels are widely deployed within many networks to achieve various 119 engineering goals, including long-haul WAN interconnection or 120 enterprise wireless access networks. A connection between two 121 endpoints can be decomposed into many connection legs. As such, the 122 corresponding forwarding path can be partitioned into multiple path 123 segments that some of them are using network overlays by means of 124 tunnels. This design serves a number of purposes such as steering 125 the traffic, optimizing egress/ingress link utilization, optimizing 126 traffic performance metrics (such as delay, delay variation, or 127 loss), optimizing resource utilization by invoking resource bonding, 128 provide high-availability, etc. 130 When a path is partitioned into multiple path segments that are 131 realized typically as overlay tunnels, LOOPS (Local Optimizations on 132 Path Segments) aims to provide in-network recovery over segments to 133 achieve better data delivery by making packet loss recovery faster. 134 In an overlay network scenario, LOOPS can be performed over the 135 existing, or purposely created, overlay tunnel based path segments. 136 Figure 1 show an overall usage scenarios of LOOPS. 138 ON=overlay node 139 UN=underlay node 141 +---------+ +---------+ 142 | App | <---------------- end-to-end ---------------> | App | 143 +---------+ +---------+ 144 |Transport| <---------------- end-to-end ---------------> |Transport| 145 +---------+ +---------+ 146 | | | | 147 | | +--+ path +--+ path segment2 +--+ | | 148 | | | |<-seg1->| |<--------------> | | | | 149 | Network | +--+ |ON| +--+ |ON| +--+ +----+ |ON| | Network | 150 | |--|UN|--| |--|UN|--| |--|UN|---| UN |--| |--| | 151 +---------+ +--+ +--+ +--+ +--+ +--+ +----+ +--+ +---------+ 152 End Host End Host 153 <---------------------------------> 154 LOOPS domain: path segment enables 155 local optimizations for better experience 157 Figure 1: LOOPS Usage Scenario 159 1.2. Sketching a Work Direction: Rationale & Goals 161 This document sketches a proposal that is meant to experimentally 162 investigate to what extent a network-assisted approach can contribute 163 to increase the overall perceived quality of experience in specific 164 situations (e.g., Sections 3.5 and 3.6 of [RFC8517]) without 165 requiring access to internal transport primitives. The rationale 166 beneath this approach is that some information (loss detection and 167 segment characteristics, etc.) can be used to trigger local in- 168 network recovery actions which have a faster effect while not 169 impacting the end-to-end congestion control loop. 171 To that aim, the work is structured into two (2) phased stages: 173 o Stage 1: Network-assisted optimization. This one assumes that 174 optimizations can be implemented at the network without requiring 175 defining new interaction with the endpoint. Existing tools such 176 as ECN will be used. Loss signal would be converted to CE 177 (congestion experienced) signal to interact with the end-to-end 178 control loop. 180 o Stage 2: Collaborative networking optimization. This one requires 181 more interaction between the network and an endpoint to implement 182 coordinated and more surgical network-assisted optimizations based 183 on information/instructions shared by an endpoint or sharing 184 locally-visible information with endpoint for better and faster 185 recovery. 187 The document focuses on the first stage. Effort related to the 188 second stage is out of scope of the initial planned work. 190 The proposed mechanism is not meant to be applied to all traffic, but 191 only to a subset which is particularly benefits from, and has been 192 selected for the network-assisted optimization service. 194 Which traffic is selected is deployment-specific and policy-based. 195 For example, techniques for dynamic information about optimization 196 function (e.g., SFC) may be leveraged to unambiguously identify the 197 aggregate of traffic that is eligible to the service. Such 198 identification may be triggered by subscription actions made by 199 customers or be provided by a network provider (e.g., specific 200 applications, during specific events such as during severe DDoS 201 attack or flash crowds events). 203 Likewise, whether the optimization function is permanently 204 instantiated or on-demand is deployment-specific. 206 This document does not intend to provide a comprehensive list of 207 target deployment cases. Sample scenarios are described to 208 illustrate some LOOPS potentials. Similar issues and optimizations 209 may be helpful in other deployments such as enhancing the reliability 210 of data transfer when a fleet of drones are used for specific 211 missions (e.g., site inspection, live streaming, and emergency 212 service). Captured data should be reliably transmitted via paths 213 involving radio connections. 215 It is not required that all segments are LOOPS-aware to benefit from 216 LOOPS advantages. 218 Section 3 presents the issues and opportunities found in some 219 multiple path segments scenarios. Section 3 describes the impact of 220 packet loss for different traffic. Section 5 describes the LOOPS 221 desired features and their impact on existing network technologies. 222 Section 6 shows the analysis on local retransmission and end-to-end 223 retransmission. Section 7 summarizes LOOPS key elements. 225 2. Terminology 227 This document makes use of the following terms: 229 LOOPS: Local Optimizations on Path Segments. LOOPS includes the 230 local in-network (i.e., non end-to-end) recovery functions and 231 other supporting features such as local measurement, loss 232 detection, and congestion feedback. 234 LOOPS Node: A node supporting LOOPS functions. 236 Overlay Node (ON): A node having overlay functions (e.g., overlay 237 protocol encapsulation/decapsulation, header modification, TLV 238 inspection) and LOOPS functions in the LOOPS overlay network usage 239 scenario. 241 Overlay Tunnel: A tunnel with designated ingress and egress nodes 242 using some network overlay protocol as encapsulation, optionally 243 with a specific traffic type. 245 Path segment: A LOOPS enabled tunnel-based network subpath. It is 246 used interchangeably with overlay segment in this document when 247 the context wants to emphasize on its overlay encapsulated nature. 248 It is also called segment for simplicity in this document. 250 Overlay segment: Refers to path segment. 252 Underlay Node (UN): A node not participating in the overlay network. 254 3. Usage Scenarios 256 3.1. Cloud-Internet Overlay Network 258 CSPs (Cloud Service Providers) are connecting their data centers 259 using the Internet or via self-constructed networks/links. This 260 expands the traditional Internet's infrastructure and, together with 261 the original ISP's infrastructure, forms the Internet underlay. 263 Automation techniques and NFV (Network Function Virtualization) make 264 it easier to dynamically provision a new virtual node/function as a 265 workload in a cloud for CPU/storage intensive functions. Virtual 266 nodes can be in form of virtual machines or containers hosting the 267 workloads sharing a physical node's infrastructure. With the aid of 268 various mechanisms such as kernel bypassing and Virtual IO, 269 forwarding based on virtual nodes is becoming more and more 270 effective. The interconnection among the purposely positioned 271 virtual nodes and/or the existing nodes with virtualization functions 272 potentially form an overlay infrastructure. It is called the Cloud- 273 Internet Overlay Network (CION) in this document for short. 275 This architecture scenario makes use of overlay technologies to 276 direct the traffic going through the specific overlay path in order 277 to achieve better service delivery. It purposely creates or selects 278 overlay nodes (ON) from providers. By continuously measuring the 279 delay of path segments and use them as metrics for path selection, 280 when the number of overlay nodes is sufficiently large, there is a 281 high chance that a better path could be found 282 [DOI_10.1109_ICDCS.2016.49] [DOI_10.1145_3038912.3052560]. 283 [DOI_10.1145_3038912.3052560] further shows all cloud providers 284 experience random loss episodes and random loss accounts for more 285 than 35% of total loss. 287 Figure 2 shows an example of an overlay path over large geographic 288 distances. An overlay node (ON) is usually a virtual node, though it 289 does not have to be. Three path segments, i.e., ON1-ON2, ON2-ON3, 290 ON3-ON4 are shown. Each segment transmits packets using some form of 291 network overlay protocol encapsulation. ON has the computing and 292 memory resources that can be used for some functions like packet loss 293 detection, network measurement and feedback, packet retransmission 294 and FEC (Forward Error Correction) computation. ONs here are managed 295 by a single administrator though they can be workloads created from 296 different CSPs. 298 _____________ 299 / domain 1 \ 300 / \ 301 ___/ -------------\ 302 / \ 303 PoP1 ->--ON1 \ 304 | | ON4------>-- PoP2 305 | | ON2 ___|__/ 306 \__|_ |->| _____ / | 307 | \|__|__ / \ / | 308 | | | \____/ \__/ | 309 \|/ | | _____ | 310 | | | ___/ \ | 311 | | \|/ / \_____ | 312 | | | / domain 2 \ /|\ 313 | | | | ON3 | | 314 | | | \ |->| | | 315 | | | \_____|__|_______/ | 316 | /|\ | | \|/ | 317 | | | | | | 318 | | | /|\ | | 319 +--------------------------------------------------+ 320 | | | | | | | Internet | 321 | o--o o---o->---o o---o->--o--o underlay | 322 +--------------------------------------------------+ 324 Figure 2: Cloud-Internet Overlay Network (CION) 326 We tested based on 37 overlay nodes from multiple cloud providers 327 globally. Each pair of the overlay nodes are used as sender and 328 receiver. When the traffic is not intentionally directed to go 329 through any intermediate virtual nodes, we call the path followed by 330 the traffic in the test the default path. When any of the virtual 331 nodes is intentionally used as an intermediate node to forward the 332 traffic, the path that the traffic takes is called an overlay path. 333 The preliminary experiments showed that the delay of a specifically 334 selected overlay path has lower latency than the one of the default 335 path in 69% of cases at 99% percentile and improvement is 17.5% at 336 99% percentile when we probe Ping packets every second for a week. 337 The average number of hops for an overlay path is 3.02. More 338 experimental information can be found in 339 [DOI_10.1109_INFCOMW.2019.8845208]. 341 Lower average delay does not necessarily mean less or no packet loss. 342 Different path segments have different packet loss rates. Loss rate 343 is another major factor impacting the user experience, espcially for 344 the short-lived or transactional flows. From some customer 345 requirements, the target loss rate is set in the test to be less than 346 1% at 99% percentile and 99.9% percentile, respectively. The loss 347 was measured between any two overlay nodes, i.e., any potential path 348 segment. Two thousand Ping packets were sent every 20 seconds 349 between two overlay nodes for 55 hours. This preliminary experiment 350 showed that the packet loss rate satisfaction are only 44.27% and 351 29.51% at the 99% and 99.9% percentiles, respectively. 353 As CION naturally consists of multiple overlay segments, LOOPS can 354 leverage this to perform local optimizations on a single hop between 355 two overlay nodes. ("Local" here is a concept relative to end-to- 356 end, it does not mean such optimization is limited to LAN networks.) 358 3.2. Satellite Communication 360 Traditionally, satellite communications deploy PEP (performance 361 enhancing proxy [RFC3135]) nodes around the satellite link to enhance 362 end-to-end performance. TCP splitting is a common approach employed 363 by such PEPs, where the TCP connection is split into three: the 364 segment before the satellite hop, the satellite section (uplink, 365 downlink), and the segment behind the satellite hop. This requires 366 heavy interactions with the end-to-end transport protocols, usually 367 without the explicit consent of the end hosts. Unfortunately, this 368 is indistinguishable from a man-in-the-middle attack on TCP. With 369 end-to-end encryption moving under the transport (QUIC), this 370 approach is no longer useful. 372 Geosynchronous Earth Orbit (GEO) satellites have a one-way delay (up 373 to the satellite and back) on the order of 250 milliseconds. This 374 does not include queueing, coding and other delays in the satellite 375 ground equipment. The Round Trip Time for a TCP or QUIC connection 376 going over a satellite hop in both directions, in the best case, will 377 be on the order of 600 milliseconds. And, it may be considerably 378 longer. RTTs on this order of magnitude have significant performance 379 implications. 381 Packet loss recovery is an area where splitting the TCP connection 382 into different parts helps. Packets lost on the terrestrial links 383 can be recovered at terrestrial latencies. Packet loss on the 384 satellite link can be recovered more quickly by an optimized 385 satellite protocol between the PEPs and/or link layer FEC than they 386 could be end to end. Again, encryption makes TCP splitting no longer 387 applicable. Enhanced error recovery at the satellite link layer 388 helps for the loss on the satellite link but doesn't help for the 389 terrestrial links. Even when the terrestrial segments are short, any 390 loss must be recovered across the satellite link delay. And, there 391 are cases when a satellite ground station connects to the general 392 Internet with a potentially larger terrestrial segment (e.g., to a 393 correspondent host in another country). Faster recovery over such 394 long terrestrial segments is desirable. 396 There are two high level classes of solutions for making encrypted 397 transport traffic like QUIC work well over satellite: 399 o Hooks in the transport protocol which can adapt to large BDPs 400 where both the bandwidth and the latency are large. This would 401 require end to end enhancement. 403 o Capabilities (such as LOOPS) under the transport protocol to 404 improve performance over specific segments of the path. In 405 particular, separating the terrestrial from the satellite losses. 406 Fixing the terrestrial loss quickly. 408 This document focuses on the latter. 410 3.3. Branch Office WAN Connection 412 Enterprises usually require network connections between the branch 413 offices, or between branch office and cloud data center over 414 geographic distances. With the increasing deployment of vCPE 415 (virtual CPE), services hosted on the CPE are moved to the provider 416 network from the customer site. Such vCPE approach enables some 417 value added service to be provided such as WAN optimization and 418 traffic steering. 420 Figure 3 shows a branch office access to public cloud via a selected 421 PoP (point of presence) for service access or reaching another branch 422 office via vPC (Virtual Private Cloud) interconnect. vCPE connects to 423 the PoP which can be hundreds of kilometers away via Internet. From 424 vCPE1 to vCPE2, it can consist of three segments, vCPE1-PoP1, 425 PoP1-PoP2 and PoP2-vCPE2. Packet loss can happen on any of them. 426 Segment based in-network recovery can be employed here to improve the 427 WAN connection quality. 429 +-------------+ 430 | public cloud| 431 | +------+ | 432 +------+ +-----+ | | vPC1 | | 433 | GW1 |----------|vCPE1| | +------+ | 434 +------+ +-----+ | | | 435 | | | | 436 Site A | | +------+ | 437 _____ | | PoP1 | | 438 ___/ \ | +------+ | 439 / \_____ +-------------+ 440 / \ | 441 | Internet | ---------+ 442 \ | 443 \___ ______/ ----------+ 444 \ / | 445 \ / +----+--------+ 446 \__/ | +------+ | 447 | | | PoP2 | | 448 | | +------+ | 449 +------+ +--+--+ | | | 450 | GW2 |------------|vCPE2| | | | 451 +------+ +-----+ | +------+ | 452 | | vPC2 | | 453 Site B | +------+ | 454 |public cloud | 455 +-------------+ 457 Figure 3: Enterprise Cloud Access 459 4. Impact of Packet loss 461 4.1. Tail Loss or Loss in Short Flows 463 When the lost segments are at the end of a transaction, TCP's fast 464 retransmit algorithm does not work as there are no ACKs to trigger 465 it. When a sender does not receive an ACK for a given segment within 466 a certain amount of time called retransmission timeout (RTO), it re- 467 sends the segment [RFC6298]. RTO can be as long as several seconds. 468 Hence the recovery of lost segments triggered by RTO is lengthy. 469 [I-D.dukkipati-tcpm-tcp-loss-probe] indicates that large RTOs make a 470 significant contribution to the long tail on the latency statistics 471 of short flows such as loading web pages. 473 The short-lived flows often complete in one or two RTTs. Even when 474 the lost packet is not an exact tail, it can possibly add another RTT 475 because there may not be enough packets in flight to trigger the fast 476 retransmit). In long-haul networks, it can result in extra time of 477 tens or hundreds of milliseconds. For ant short lived or 478 transactional flows, it affects the latency greatly. 480 An overlay segment transmits the aggregated flows from ON to ON. As 481 short-lived flows are aggregated, the probability of tail loss over 482 this specific overlay segment decreases compared to an individual 483 flow. The overlay segment is much shorter than the end-to-end path, 484 hence loss recovery over an overlay segment helps to obtain low 485 latency. 487 4.2. Packet Loss in Real Time Media Streams 489 The Real-time transport protocol (RTP) is widely used in interactive 490 audio and video. Packet loss degrades the quality of the received 491 media. When the latency tolerance of the application is sufficiently 492 large, the RTP sender may use RTCP NACK feedback from the receiver 493 [RFC4585] to trigger the retransmission of the lost packets before 494 the playout time is reached at the receiver. 496 The end-to-end path over WAN can be hundreds of milliseconds, so the 497 end-to-end feedback based retransmission may be not be very useful 498 when applications can not tolerate one more RTT. Loss recovery over 499 an overlay segment can then be used for the scenarios in which a 500 shorter delayed retransmission can catch up with the playout time. 502 5. Features to be Considered for LOOPS 504 This section provides an overview of the LOOPS features. This 505 section is not meant to document a detailed specification, but it is 506 meant to highlight some design choices that may be followed during 507 the solution design phase. 509 5.1. Local Recovery 511 LOOPS (Local Optimizations on Path Segments) aims to provide in- 512 network recovery over segments to achieve better data delivery by 513 making packet loss recovery faster. This is viable because LOOPS 514 nodes will be instantiated to partition the path into segments. At 515 the same time, LOOPS does not replace the end-to-end loss recovery 516 (if any). With the advent of automation and technologies like NFV 517 and virtual IO, it is possible to dynamically instantiate functions 518 to nodes. The enabling of LOOPS is expected to be dynamic. When to 519 enable this function is out of scope. The operator or administrator 520 can make the decision based on their historical experience or real- 521 time monitoring. 523 There are two ways to recover packet, retransmission and Forward 524 Error Correction (FEC). A document to specify the generic elements 525 for loss detection, sequence number space, acknowledgment generation 526 and state transition is available in [I-D.welzl-loops-gen-info]. 528 5.2. Congestion Control Interaction 530 When a TCP-like transport layer protocol is used, local recovery in 531 LOOPS has to interact with the upper layer transport congestion 532 control. Classic TCP adjusts the congestion window when a loss is 533 detected and then fast retransmit is invoked. LOOPS performs in- 534 network recovery which may cause a loss event not being observed by 535 the TCP sender. Then TCP sender may overshoot then. 537 To solve this issue, LOOPS needs to report the loss to end-to-end 538 congestion control LOOPS. LOOPS can CE(Congestion Experienced) marks 539 its recovered packets as the loss signal to end-to-end. Converting a 540 packet loss signal to CE marking signal brings the benefits of 541 reducing Head-of-Line blocking and probability of RTO expiry 542 [RFC8087] without affecting TCP sender's loss based congestion 543 control behaviour while enjoying the faster local recovery. ECN 544 based indication is equivalent to a loss event at the TCP sender 545 [RFC3168]. In this way, a requirement is set for applying LOOPS. 546 Only ECT (ECN-Capable Transport) flows should be directed to an LOOPS 547 enabled path segment. 549 5.3. Overlay Protocol Extensions 551 Some tunnel protocols such as VXLAN [RFC7348], GENEVE 552 [I-D.ietf-nvo3-geneve], LISP [RFC6830] or CAPWAP [RFC5415] are 553 employed in overlay network. They are used in various ways. A path 554 can have single overlay tunnel as a sub-path or stitch multiple 555 segments together, like VXLAN [RFC7348] or GENEVE 556 [I-D.ietf-nvo3-geneve], or specify a sequence of intermediate nodes, 557 as in SRv6 [RFC8754]. 559 LOOPS does not look into the inner packet. LOOPS information is 560 required to be embedded in the overlay protocol header. An example 561 shown in Figure 4. The current protocol focus is GENEVE 562 [I-D.ietf-nvo3-geneve]. The specific information is to be defined in 563 separate documents. 565 +------------+------------+-----------------+---------+---------+ 566 |Outer IP hdr|Overlay hdr |LOOPS information|Inner hdr|payload | 567 +------------+------------+-----------------+---------+---------+ 569 Figure 4: LOOPS Extension Header Example 571 6. Local in-network Recovery and End-to-end Retransmission 573 Most transport layer protocols have their own end-to-end 574 retransmission to recover the lost packet. When LOOPS is in use, its 575 local recovery can affect the end-to-end one. This section talks 576 about such impacts. 578 There are two ways to perform local recovery, retransmission and FEC 579 (Forward Error Correction). They are possibly used together in some 580 cases. Such approaches between two overlay nodes recover the lost 581 packet in relatively shorter distance and thus shorter latency. 582 Therefore the local recovery is generally faster compared to end-to- 583 end. 585 End-to-end retransmission is normally triggered by a NACK as in RTCP, 586 multiple duplicate ACKs as in traditional TCP or time based detection 587 as in RACK [I-D.ietf-tcpm-rack]. 589 When FEC is used for local recovery, it may come with a buffer to 590 make sure the recovered packets delivered are in order subsequently. 591 Therefore the receiver side is unlikely to see the out-of-order 592 packets and then send a NACK or multiple duplicate ACKs. The side 593 effect to unnecessarily trigger end-to-end retransmit is minimum. 594 When FEC is used in this way, if redundancy and block size are 595 determined, extra latency required to recover lost packets is also 596 bounded. Then RTT variation caused by it is predictable. In some 597 extreme case like a large number of packet loss caused by persistent 598 burst, FEC may not be able to recover it. Then end-to-end retransmit 599 will work as a last resort. In summary, when FEC is used as local 600 recovery, the impact on end-to-end retransmission is limited. 602 When local retransmission is used, it has the following impacts on 603 the end-to-end retransmission. 605 For packet loss in RTP streaming, local retransmission can recover 606 those packets which would not be retransmitted end-to-end otherwise 607 due to long RTT. Therefore when the segment(s) being retransmitted 608 on is a small portion of the whole end to end path, the 609 retransmission will have a significant effect of improving the 610 quality at receiver. 612 When the sender also re-transmits the packet based on a NACK 613 received, the receiver may receive the duplicated retransmitted 614 packets. 616 For packet loss in TCP flows, TCP RENO and CUBIC use duplicate ACKs 617 as a loss signal to trigger the fast retransmit. Though we are not 618 standardize the buffering feature of a LOOPS egress, an introductory 619 analysis is given as follows. 621 o The egress overlay node can buffer the out-of-order packets for a 622 while, giving a limited time for a packet being retransmitted 623 somewhere in the overlay path to reach it. The retransmitted 624 packet and the buffered packets caused by it may increase the RTT 625 variation at the sender. When the retransmitted latency is a 626 small portion of RTT or the loss is rare, such RTT variation will 627 be smoothed without much impact. 629 The buffer management is nontrivial in this case. It has to be 630 determined how many out-of-order packets can be buffered at the 631 egress overlay node before it gives up waiting for a successful 632 local retransmission. In some extreme case the lost packet is not 633 recovered successfully locally, the sender may invoke end-to-end 634 fast retransmit slower than it would be in classic TCP. 636 o If LOOPS network does not buffer the out-of-order packets caused 637 by packet loss, TCP sender which uses a time based loss detection 638 like RACK [I-D.ietf-tcpm-rack] will perform well here. It uses 639 the notion of time to replace the conventional DUPACK threshold 640 approach to detect losses. Hence it prevents the TCP sender from 641 invoking fast retransmit too early. Local retransmission will not 642 interfere the sender's retransmission generally in this case. If 643 time based loss detection is not supported at the sender, end to 644 end retransmission may be invoked as usual. It consumes extra 645 bandwidth Because the lost packets (i.e. recovered packet) is 646 normally a very small percentage of the total packets. Then extra 647 bandwidth cost is not significant. 649 7. Summary 651 LOOPS will extend the existing overlay protocols in data plane, 652 potential starting from GENEVE [I-D.ietf-nvo3-geneve] which has good 653 extensibility. Path or segment selection can be feature provided by 654 the overlay protocols via SDN techniques [RFC7149] or other 655 approaches and is not a part of LOOPS. LOOPS is a set of functions 656 to be implemented on Overlay Nodes as a tunnel transport with best 657 effort reliability. LOOPS targets the following features. 659 1. Local recovery: Local recovery: Retransmission, FEC, or 660 combination thereof can be used as local recovery method. Such 661 recovery mechanism is in-network. It is performed by two network 662 nodes with computing and memory resources. 664 2. Local measurement: Ingress/Egress overlay nodes measure the local 665 segment RTT, loss and/or throughput to immediately get the 666 overlay segment status for loss detection. 668 3. Interact with end-to-end congestion control: Convert a packet 669 loss signal to an ECN-marking signal to notify the end host 670 sender. 672 8. Security Considerations 674 LOOPS does not require access to the traffic payload in clear, so 675 encrypted payload does not affect functionality of LOOPS. 677 The use of LOOPS introduces some issues which impact security. ON 678 with LOOPS function represents a point in the network where the 679 traffic can be potentially manipulated and intercepted by malicious 680 nodes. Means to ensure that only legitimate nodes are involved 681 should be considered. 683 Denial of service attack can be launched from an ON. A rogue ON 684 might be able to spoof packets as if it come from a legitimate ON. 685 It may also modify the ECN CE marking in packets to influence the 686 sender's rate. In order to protected from such attacks, the overlay 687 protocol itself should have some built-in security protection which 688 is used by LOOPS. The operator should use some authentication 689 mechanism to make sure ONs are valid and non-compromised. 691 9. IANA Considerations 693 No IANA action is required. 695 10. Acknowledgements 697 Thanks to etosat mailing list about the discussion about the SatCom 698 and LOOPS use case. 700 11. Informative References 702 [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, 703 RFC 793, DOI 10.17487/RFC0793, September 1981, 704 . 706 [RFC3135] Border, J., Kojo, M., Griner, J., Montenegro, G., and Z. 707 Shelby, "Performance Enhancing Proxies Intended to 708 Mitigate Link-Related Degradations", RFC 3135, 709 DOI 10.17487/RFC3135, June 2001, 710 . 712 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 713 of Explicit Congestion Notification (ECN) to IP", 714 RFC 3168, DOI 10.17487/RFC3168, September 2001, 715 . 717 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 718 Jacobson, "RTP: A Transport Protocol for Real-Time 719 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 720 July 2003, . 722 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 723 "Extended RTP Profile for Real-time Transport Control 724 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, 725 DOI 10.17487/RFC4585, July 2006, 726 . 728 [RFC4588] Rey, J., Leon, D., Miyazaki, A., Varsa, V., and R. 729 Hakenberg, "RTP Retransmission Payload Format", RFC 4588, 730 DOI 10.17487/RFC4588, July 2006, 731 . 733 [RFC5415] Calhoun, P., Ed., Montemurro, M., Ed., and D. Stanley, 734 Ed., "Control And Provisioning of Wireless Access Points 735 (CAPWAP) Protocol Specification", RFC 5415, 736 DOI 10.17487/RFC5415, March 2009, 737 . 739 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 740 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 741 . 743 [RFC6298] Paxson, V., Allman, M., Chu, J., and M. Sargent, 744 "Computing TCP's Retransmission Timer", RFC 6298, 745 DOI 10.17487/RFC6298, June 2011, 746 . 748 [RFC6830] Farinacci, D., Fuller, V., Meyer, D., and D. Lewis, "The 749 Locator/ID Separation Protocol (LISP)", RFC 6830, 750 DOI 10.17487/RFC6830, January 2013, 751 . 753 [RFC7149] Boucadair, M. and C. Jacquenet, "Software-Defined 754 Networking: A Perspective from within a Service Provider 755 Environment", RFC 7149, DOI 10.17487/RFC7149, March 2014, 756 . 758 [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, 759 L., Sridhar, T., Bursell, M., and C. Wright, "Virtual 760 eXtensible Local Area Network (VXLAN): A Framework for 761 Overlaying Virtualized Layer 2 Networks over Layer 3 762 Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014, 763 . 765 [RFC8087] Fairhurst, G. and M. Welzl, "The Benefits of Using 766 Explicit Congestion Notification (ECN)", RFC 8087, 767 DOI 10.17487/RFC8087, March 2017, 768 . 770 [RFC8517] Dolson, D., Ed., Snellman, J., Boucadair, M., Ed., and C. 771 Jacquenet, "An Inventory of Transport-Centric Functions 772 Provided by Middleboxes: An Operator Perspective", 773 RFC 8517, DOI 10.17487/RFC8517, February 2019, 774 . 776 [RFC8754] Filsfils, C., Ed., Dukes, D., Ed., Previdi, S., Leddy, J., 777 Matsushima, S., and D. Voyer, "IPv6 Segment Routing Header 778 (SRH)", RFC 8754, DOI 10.17487/RFC8754, March 2020, 779 . 781 [I-D.dukkipati-tcpm-tcp-loss-probe] 782 Dukkipati, N., Cardwell, N., Cheng, Y., and M. Mathis, 783 "Tail Loss Probe (TLP): An Algorithm for Fast Recovery of 784 Tail Losses", draft-dukkipati-tcpm-tcp-loss-probe-01 (work 785 in progress), February 2013. 787 [I-D.ietf-nvo3-geneve] 788 Gross, J., Ganga, I., and T. Sridhar, "Geneve: Generic 789 Network Virtualization Encapsulation", draft-ietf- 790 nvo3-geneve-16 (work in progress), March 2020. 792 [I-D.ietf-tcpm-rack] 793 Cheng, Y., Cardwell, N., Dukkipati, N., and P. Jha, "RACK: 794 a time-based fast loss detection algorithm for TCP", 795 draft-ietf-tcpm-rack-08 (work in progress), March 2020. 797 [I-D.welzl-loops-gen-info] 798 Welzl, M. and C. Bormann, "LOOPS Generic Information Set", 799 draft-welzl-loops-gen-info-03 (work in progress), March 800 2020. 802 [DOI_10.1109_ICDCS.2016.49] 803 Cai, C., Le, F., Sun, X., Xie, G., Jamjoom, H., and R. 804 Campbell, "CRONets: Cloud-Routed Overlay Networks", 2016 805 IEEE 36th International Conference on Distributed 806 Computing Systems (ICDCS), DOI 10.1109/icdcs.2016.49, June 807 2016. 809 [DOI_10.1145_3038912.3052560] 810 Haq, O., Raja, M., and F. Dogar, "Measuring and Improving 811 the Reliability of Wide-Area Cloud Paths", Proceedings of 812 the 26th International Conference on World Wide Web, 813 DOI 10.1145/3038912.3052560, April 2017. 815 [DOI_10.1109_INFCOMW.2019.8845208] 816 Xu, Z., Ju, R., Gu, L., Wang, W., Li, J., Li, F., and L. 817 Han, "Using Overlay Cloud Network to Accelerate Global 818 Communications", IEEE INFOCOM 2019 - IEEE Conference on 819 Computer Communications Workshops (INFOCOM WKSHPS), 820 DOI 10.1109/infcomw.2019.8845208, April 2019. 822 Authors' Addresses 824 Yizhou Li 825 Huawei Technologies 827 Email: liyizhou@huawei.com 829 Xingwang Zhou 830 Huawei Technologies 832 Email: zhouxingwang@huawei.com 834 Mohamed Boucadair 835 Orange 837 Email: mohamed.boucadair@orange.com 839 Jianglong Wang 840 China Telecom 842 Email: wangjl1.bri@chinatelecom.cn 843 Fengwei Qin 844 China Mobile 846 Email: qinfengwei@chinamobile.com