idnits 2.17.1 draft-irtf-panrg-what-not-to-do-17.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == The page length should not exceed 58 lines per page, but there was 1 longer page, the longest (page 1) being 1847 lines Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (10 February 2021) is 1170 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-04) exists of draft-iab-use-it-or-lose-it-00 == Outdated reference: A later version (-08) exists of draft-irtf-panrg-path-properties-01 == Outdated reference: A later version (-12) exists of draft-irtf-panrg-questions-08 -- Obsolete informational reference (is this intentional?): RFC 793 (Obsoleted by RFC 9293) -- Obsolete informational reference (is this intentional?): RFC 1190 (Obsoleted by RFC 1819) -- Obsolete informational reference (is this intentional?): RFC 2001 (Obsoleted by RFC 2581) -- Obsolete informational reference (is this intentional?): RFC 3697 (Obsoleted by RFC 6437) -- Obsolete informational reference (is this intentional?): RFC 5575 (Obsoleted by RFC 8955) Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 6 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 PANRG S. Dawkins, Ed. 3 Internet-Draft Tencent America 4 Intended status: Informational 10 February 2021 5 Expires: 14 August 2021 7 Path Aware Networking: Obstacles to Deployment (A Bestiary of Roads Not 8 Taken) 9 draft-irtf-panrg-what-not-to-do-17 11 Abstract 13 At the first meeting of the Path Aware Networking Research Group, the 14 research group agreed to catalog and analyze past efforts to develop 15 and deploy Path Aware techniques, most of which were unsuccessful or 16 at most partially successful, in order to extract insights and 17 lessons for path-aware networking researchers. 19 This document contains that catalog and analysis. 21 Status of This Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at https://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on 14 August 2021. 38 Copyright Notice 40 Copyright (c) 2021 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 45 license-info) in effect on the date of publication of this document. 46 Please review these documents carefully, as they describe your rights 47 and restrictions with respect to this document. 49 Table of Contents 51 1. Introduction 52 1.1. What Do "Path" and "Path Awareness" Mean in this Document? 53 2. A Perspective On This Document 54 2.1. Notes for the Reader 55 2.2. A Note About Path-Aware Techniques Included In This 56 Document 57 2.3. Venue for Discussion of this Document 58 2.4. Architectural Guidance 59 2.5. Terminology Used in this Document 60 2.6. Methodology for Contributions 61 3. Applying the Lessons We've Learned 62 4. Summary of Lessons Learned 63 4.1. Justifying Deployment 64 4.2. Providing Benefits for Early Adopters 65 4.3. Providing Benefits During Partial Deployment 66 4.4. Outperforming End-to-end Protocol Mechanisms 67 4.5. Paying for Path Aware Techniques 68 4.6. Impact on Operational Practices 69 4.7. Per-connection State 70 4.8. Keeping Traffic on Fast-paths 71 4.9. Endpoints Trusting Intermediate Nodes 72 4.10. Intermediate Nodes Trusting Endpoints 73 4.11. Reacting to Distant Signals 74 4.12. Support in Endpoint Protocol Stacks 75 5. Future Work 76 6. Contributions 77 6.1. Stream Transport (ST, ST2, ST2+) 78 6.1.1. Reasons for Non-deployment 79 6.1.2. Lessons Learned. 80 6.2. Integrated Services (IntServ) 81 6.2.1. Reasons for Non-deployment 82 6.2.2. Lessons Learned. 83 6.3. Quick-Start TCP 84 6.3.1. Reasons for Non-deployment 85 6.3.2. Lessons Learned 86 6.4. ICMP Source Quench 87 6.4.1. Reasons for Non-deployment 88 6.4.2. Lessons Learned 89 6.5. Triggers for Transport (TRIGTRAN) 90 6.5.1. Reasons for Non-deployment 91 6.5.2. Lessons Learned. 92 6.6. Shim6 93 6.6.1. Reasons for Non-deployment 94 6.6.2. Lessons Learned 95 6.6.3. Addendum on MultiPath TCP 96 6.7. Next Steps in Signaling (NSIS) 97 6.7.1. Reasons for Non-deployment 98 6.7.2. Lessons Learned 99 6.8. IPv6 Flow Label 100 6.8.1. Reasons for Non-deployment 101 6.8.2. Lessons Learned 102 7. Security Considerations 103 8. IANA Considerations 104 9. Acknowledgments 105 10. Informative References 106 Author's Address 108 1. Introduction 110 This document describes the lessons that IETF participants have 111 learned (and learned the hard way) about Path Aware Networking over a 112 period of several decades, and provides an analysis of reasons why 113 various Path Aware Networking techniques have seen limited or no 114 deployment. 116 1.1. What Do "Path" and "Path Awareness" Mean in this Document? 118 One of the first questions reviewers of this document have asked is 119 "what's the definition of a path, and what's the definition of path 120 awareness?" That is not an easy question to answer for this 121 document. 123 These terms have definitions in other [PANRG] documents, and are 124 still the subject of some discussion in the research group, as of the 125 date of this document. But because this document reflects work 126 performed over several decades, the technologies described in 127 Section 6 significantly predate the current definitions of "path" and 128 "path aware" in use in the Path Aware Networking Research Group, and 129 it is unlikely that all the contributors to Section 6 would have had 130 the same understanding of these terms. Those technologies were 131 considered "path aware" in early PANRG discussions, and so are 132 included in this retrospective document. 134 It is worth noting that the definitions of "path" and "path aware" in 135 [I-D.irtf-panrg-path-properties] would apply to path aware networking 136 techniques at a number of levels of the Internet protocol 137 architecture ([RFC1122], plus several decades of refinements), but 138 the contributions received for this document tended to target the 139 Transport Layer, and to treat a "path" constructed by routers as a 140 "black box". It would be useful to consider how applicable the 141 Lessons Learned cataloged in this document are, at other layers, and 142 that would be a fine topic for follow-on research. 144 The current definition of "Path" in the Path Aware Networking 145 Research Group appears in Section 2 ("Terminology") in 146 [I-D.irtf-panrg-path-properties]. That definition is included here 147 as a convenience to the reader. 149 | Path: A sequence of adjacent path elements over which a packet can 150 | be transmitted, starting and ending with a node. A path is 151 | unidirectional. Paths are time-dependent, i.e., the sequence of 152 | path elements over which packets are sent from one node to another 153 | may change. A path is defined between two nodes. For multicast 154 | or broadcast, a packet may be sent by one node and received by 155 | multiple nodes. In this case, the packet is sent over multiple 156 | paths at once, one path for each combination of sending and 157 | receiving node; these paths do not have to be disjoint. Note that 158 | an entity may have only partial visibility of the path elements 159 | that comprise a path and visibility may change over time. 160 | Different entities may have different visibility of a path and/or 161 | treat path elements at different levels of abstraction. 163 The current definition of "Path Awareness", used by the Path Aware 164 Networking Research Group, appears in Section 1.1 ("Definition") in 165 [I-D.irtf-panrg-questions]. That definition is included here as a 166 convenience to the reader. 168 | For purposes of this document, "path aware networking" describes 169 | endpoint discovery of the properties of paths they use for 170 | communication, and endpoint reaction to these properties that 171 | affects routing and/or transmission; note that this can and 172 | already does happen to some extent in the current Internet 173 | architecture. Expanding on this definition, a "path aware 174 | internetwork" is one in which endpoint discovery of path 175 | properties and endpoint selection of paths used by traffic 176 | exchanged by the endpoint are explicitly supported, regardless of 177 | the specific design of the protocol features which enable this 178 | discovery and selection. 180 2. A Perspective On This Document 182 At the first meeting of the Path Aware Networking Research Group 183 [PANRG], at IETF 99 [PANRG-99], Olivier Bonaventure led a discussion 184 of "A Decade of Path Awareness" [PATH-Decade], on attempts, which 185 were mostly unsuccessful for a variety of reasons, to exploit Path 186 Aware techniques and achieve a variety of goals over the past decade. 187 At the end of that discussion, two things were abundantly clear. 189 * The Internet community has accumulated considerable experience 190 with many Path Aware techniques over a long period of time, and 192 * Although some path aware techniques have been deployed (for 193 example, Differentiated Services, or DiffServ [RFC2475]), most of 194 these techniques haven't seen widespread adoption and deplyment. 195 Even "successful" techniques like DiffServ can face obstacles that 196 prevents wider usage. The reasons for non-adoption and limited 197 adoption and deployment are many, and are worthy of study. 199 The meta-lessons from that experience were 201 * Path aware networking has been more Research than Engineering, so 202 establishing an IRTF Research Group for Path Aware Networking is 203 the right thing to do [RFC7418]. 205 * Analyzing a catalog of past experience to learn the reasons for 206 non-adoption would be a great first step for the Research Group. 208 Allison Mankin, as IRTF Chair, officially chartered the Path Aware 209 Networking Research Group in July, 2018. 211 This document contains the analysis performed by that research group 212 (Section 4), based on that catalog (Section 6). 214 This document represents the consensus of the Path Aware Networking 215 Research Group. 217 2.1. Notes for the Reader 219 This Informational document discusses Path Aware protocol mechanisms 220 considered, and in some cases standardized, by the Internet 221 Engineering Task Force (IETF), and considers Lessons Learned from 222 those mechanisms. The intention is to inform the work of protocol 223 designers, whether in the IRTF, the IETF, or elsewhere in the 224 Internet ecosystem. 226 As an Informational document published in the IRTF stream, this 227 document has no authority beyond the quality of the analysis it 228 contains. 230 2.2. A Note About Path-Aware Techniques Included In This Document 232 This document does not catalog every proposed path aware networking 233 technique that was not adopted and deployed. Instead, we limited our 234 focus to technologies that passed through the IETF community, and 235 still identified enough techniques to provide background for the 236 lessons included in Section 4 to inform researchers and protocol 237 engineers in their work. 239 No shame is intended for the techniques included in this document. 240 As shown in Section 4, the quality of specific techniques had little 241 to do with whether they were deployed or not. Based on the 242 techniques cataloged in this document, it is likely that when these 243 techniques were put forward, the proponents were trying to engineer 244 something that could not be engineered without first carrying out 245 research. Actual shame would be failing to learn from experience, 246 and failing to share that experience with other networking 247 researchers and engineers. 249 2.3. Venue for Discussion of this Document 251 (RFC Editor: please remove this section before publication) 253 Discussion of specific contributed experiences and this document in 254 general should take place on the PANRG mailing list. 256 2.4. Architectural Guidance 258 As background for understanding the Lessons Learned contained in this 259 document, the reader is encouraged to become familiar with the 260 Internet Architecture Board's documents on "What Makes for a 261 Successful Protocol?" [RFC5218] and "Planning for Protocol Adoption 262 and Subsequent Transitions" [RFC8170]. 264 Although these two documents do not specifically target path-aware 265 networking protocols, they are helpful resources for readers seeking 266 to improve their understanding of considerations for successful 267 adoption and deployment of any protocol. For example, the Basic 268 Success Factors described in Setion 2.1 of [RFC5218] are helpful for 269 readers of this document. 271 Because there is an economic aspect to decisions about deployment, 272 the IAB Workshop on Internet Technology Adoption and Transition 273 [ITAT] report [RFC7305] also provides food for thought. 275 Several of the Lessons Learned in Section 4 reflect considerations 276 described in [RFC5218], [RFC7305], and [RFC8170]. 278 2.5. Terminology Used in this Document 280 The terms Node and Element in this document have the meaning defined 281 in [I-D.irtf-panrg-path-properties]. 283 2.6. Methodology for Contributions 285 This document grew out of contributions by various IETF participants 286 with experience with one or more Path Aware Networking techniques. 288 There are many things that could be said about the Path Aware 289 networking techniques that have been developed. For the purposes of 290 this document, contributors were requested to provide 292 * the name of a technique, including an abbreviation if one was used 294 * if available, a long-term pointer to the best reference describing 295 the technique 297 * a short description of the problem the technique was intended to 298 solve 300 * a short description of the reasons why the technique wasn't 301 adopted 303 * a short statement of the lessons that researchers can learn from 304 our experience with this technique. 306 3. Applying the Lessons We've Learned 308 The initial scope for this document was roughly "what mistakes have 309 we made in the decade prior to [PANRG-99], that we shouldn't make 310 again". Some of the contributions in Section 6 predate the initial 311 scope. The earliest Path-Aware Networking technique referred to in 312 Section 6 is Section 6.1, published in the late 1970s. Given that 313 the networking ecosystem has evolved continuously, it seems 314 reasonable to consider how to apply these lessons. 316 The PANRG Research Group reviewed the Lessons Learned (Section 4) 317 contained in the May 23, 2019 version of this document at IETF 105 318 [PANRG-105-Min], and carried out additional discussion at IETF 106 319 [PANRG-106-Min]. Table 1 provides the "sense of the room" about each 320 lesson after those discussions. The intention is to capture whether 321 a specific lesson seems to be 323 * "Invariant" - well-understood and is likely to be applicable for 324 any proposed Path Aware Networking solution. 326 * "Variable" - has impeded deployment in the past, but might not be 327 applicable in a specific technique. Engineering analysis to 328 understand whether the lesson is applicable is prudent. 330 * "Not Now" - this characteristic tends to turn up a minefield full 331 of dragons, and prudent network engineers will wish to avoid 332 gambling on a technique that relies on this, until something 333 significant changes 335 +-----------------------------------------------------+-----------+ 336 | Lesson | Category | 337 +=====================================================+===========+ 338 | Justifying Deployment (Section 4.1) | Invariant | 339 +-----------------------------------------------------+-----------+ 340 | Providing Benefits for Early Adopters (Section 4.2) | Invariant | 341 +-----------------------------------------------------+-----------+ 342 | Providing Benefits during Partial Deployment | Invariant | 343 | (Section 4.3) | | 344 +-----------------------------------------------------+-----------+ 345 | Outperforming End-to-end Protocol Mechanisms | Variable | 346 | (Section 4.4) | | 347 +-----------------------------------------------------+-----------+ 348 | Paying for Path Aware Techniques (Section 4.5) | Invariant | 349 +-----------------------------------------------------+-----------+ 350 | Impact on Operational Practices (Section 4.6) | Invariant | 351 +-----------------------------------------------------+-----------+ 352 | Per-connection State (Section 4.7) | Variable | 353 +-----------------------------------------------------+-----------+ 354 | Keeping Traffic on Fast-paths (Section 4.8) | Variable | 355 +-----------------------------------------------------+-----------+ 356 | Endpoints Trusting Intermediate Nodes (Section 4.9) | Not Now | 357 +-----------------------------------------------------+-----------+ 358 | Intermediate Nodes Trusting Endpoints | Not Now | 359 | (Section 4.10) | | 360 +-----------------------------------------------------+-----------+ 361 | Reacting to Distant Signals (Section 4.11) | Variable | 362 +-----------------------------------------------------+-----------+ 363 | Support in Endpoint Protocol Stacks (Section 4.12) | Variable | 364 +-----------------------------------------------------+-----------+ 366 Table 1 368 "Justifying Deployment", "Providing Benefits for Early Adopters", 369 "Paying for Path Aware Techniques", and "Impact on Operational 370 Practice" were considered to be invariant - the sense of the room was 371 that these would always be considerations for any proposed Path Aware 372 Technique. 374 "Providing Benefits During Partial Deployment" was added after IETF 375 105, during research group last call, and is also considered to be 376 invariant. 378 For "Outperforming End-to-end Protocol Mechanisms", there is a trade- 379 off between improved performance from Path Aware Techniques and 380 additional complexity required by some Path Aware Techniques. 382 * For example, if you can obtain the same understanding of path 383 characteristics from measurements obtained over a few more round 384 trips, endpoint implementers are unlikely to be eager to add 385 complexity, and many attributes can be measured from an endpoint, 386 without assistance from intermediate nodes. 388 For "Per-connection State", the key questions discussed in the 389 research group were "how much state" and "where state is maintained". 391 * IntServ (Section 6.2) required state at every intermediate node 392 for every connection between two endpoints. As the Internet 393 ecosystem has evolved, carrying many connections in a tunnel that 394 appears to intermediate nodes as a single connection has become 395 more common, so that additional end-to-end connections don't add 396 additional state to intermediate nodes between tunnel endpoints. 397 If these tunnels are encrypted, intermediate nodes between tunnel 398 endpoints can't distinguish between connections, even if that were 399 desirable. 401 For "Keeping Traffic on Fast-paths", we noted that this was true for 402 many platforms, but not for all. 404 * For backbone routers, this is likely an invariant, but for 405 platforms that rely more on general-purpose computers to make 406 forwarding decisions, this may not be a fatal flaw for Path Aware 407 Networking techniques. 409 For "Endpoints Trusting Intermediate Nodes" and "Intermediate Nodes 410 Trusting Endpoints", these lessons point to the broader need to 411 revisit the Internet Threat Model. 413 * We noted with relief that discussions about this were already 414 underway in the IETF community at IETF 105 (see the Security Area 415 Open Meeting minutes [SAAG-105-Min] for discussion of 416 [I-D.arkko-arch-internet-threat-model] and [I-D.farrell-etm]), and 417 the Internet Architecture Board has created a mailing list for 418 continued discussions ([model-t]), but we recognize that there are 419 Path Aware Networking aspects of this effort, requiring research. 421 For "Reacting to Distant Signals", we noted that not all attributes 422 are equal. 424 * If an attribute is stable over an extended period of time, is 425 difficult to observe via end-to-end mechanisms, and is valuable, 426 Path Aware Techniques that rely on that attribute to provide a 427 significant benefit become more attractive. 429 * Analysis to help identify attributes that are useful enough to 430 justify deployment of Path Aware techniques that make use of those 431 attributes would be helpful. 433 For "Support in Endpoint Protocol Stacks", we noted that Path Aware 434 applications must be able to identify and communicate requirements 435 about path characteristics. 437 * The de-facto sockets API has no way of signaling application 438 expectations for the network path to the protocol stack. 440 4. Summary of Lessons Learned 442 This section summarizes the Lessons Learned from the contributed 443 subsections in Section 6. 445 Each Lesson Learned is tagged with one or more contributions that 446 encountered this obstacle as a significant impediment to deployment. 447 Other contributed techniques may have also encountered this obstacle, 448 but this obstacle may not have been the biggest impediment to 449 deployment for those techniques. 451 It is useful to notice that sometimes an obstacle might impede 452 deployment, while at other times, the same obstacle might prevent 453 adoption and deployment entirely. The research group discussed 454 distinguishing between obstacles that impede and obstacles that 455 prevent, but it appears that the boundary between "impede" and 456 "prevent" can shift over time - some of the Lessons Learned are based 457 on both Path Aware techniques that were not deployed, and Path Aware 458 techniques that were deployed, but were not deployed widely or 459 quickly. See Section 6.6 and Section 6.6.3 as one example of this 460 shifting boundary. 462 4.1. Justifying Deployment 464 The benefit of Path Awareness must be great enough to justify making 465 changes in an operational network. The colloquial U.S. American 466 English expression, "If it ain't broke, don't fix it" is a "best 467 current practice" on today's Internet. (See Section 6.3, 468 Section 6.5, and Section 6.4, in addition to [RFC5218]). 470 4.2. Providing Benefits for Early Adopters 472 Providing benefits for early adopters can be key - if everyone must 473 deploy a technique in order for the technique to provide benefits, or 474 even to work at all, the technique is unlikely to be adopted widely 475 or quickly. (See Section 6.2 and Section 6.3, in addition to 476 [RFC5218]). 478 4.3. Providing Benefits During Partial Deployment 480 Some proposals require that all path elements along the full length 481 of the path must be upgraded to support a new technique, before any 482 benefits can be seen. This is likely to require coordination between 483 operators who control a subset of path elements, and between 484 operators and end users if endpoint upgrades are required. If a 485 technique provides benefits when only a part of the path has been 486 upgraded, this is likely to encourage adoption and deployment. (See 487 Section 6.2 and Section 6.3, in addition to [RFC5218]). 489 4.4. Outperforming End-to-end Protocol Mechanisms 491 Adaptive end-to-end protocol mechanisms may respond to feedback 492 quickly enough that the additional realizable benefit from a new Path 493 Aware mechanism that tries to manipulate nodes along a path, or 494 observe the attributes of nodes along a path, may be much smaller 495 than anticipated (Section 6.3 and Section 6.5). 497 4.5. Paying for Path Aware Techniques 499 "Follow the money." If operators can't charge for a Path Aware 500 technique to recover the costs of deploying it, the benefits to the 501 operator must be really significant. Corollary: If operators charge 502 for a Path Aware technique, the benefits to users of that Path Aware 503 technique must be significant enough to justify the cost. (See 504 Section 6.1, Section 6.2, and Section 6.5). 506 4.6. Impact on Operational Practices 508 Impact of a Path Aware technique requiring changes to operational 509 practices can affect how quickly or widely a promising technique is 510 deployed. The impacts of these changes may make deployment more 511 likely, but often discourage deployment. (See Section 6.6, including 512 Section 6.6.3). 514 4.7. Per-connection State 516 Per-connection state in intermediate nodes has been an impediment to 517 adoption and deployment in the past, because of added cost and 518 complexity. Often, similar benefits can be achieved with much less 519 finely-grained state. This is especially true as we move from the 520 edge of the network, further into the routing core (See Section 6.1 521 and Section 6.2). 523 4.8. Keeping Traffic on Fast-paths 525 Many modern platforms, especially high-end routers, have been 526 designed with hardware that can make simple per-packet forwarding 527 decisions ("fast-paths"), but have not been designed to make heavy 528 use of in-band mechanisms such as IPv4 and IPv6 Router Alert Options 529 (RAO) that require more processing to make forwarding decisions. 530 Packets carrying in-band mechanisms are diverted to other processors 531 in the router with much lower packet processing rates. Operators can 532 be reluctant to deploy techniques that rely heavily on in-band 533 mechanisms because they may significantly reduce packet throughput. 534 (See Section 6.7). 536 4.9. Endpoints Trusting Intermediate Nodes 538 If intermediate nodes along the path can't be trusted, it's unlikely 539 that endpoints will rely on signals from intermediate nodes to drive 540 changes to endpoint behaviors. (See Section 6.5, Section 6.4). We 541 note that "trust" is not binary - one, low, level of trust applies 542 when a node issuing a message can confirm that it has visibility of 543 the packets on the path it is seeking to control [RFC8085] (e.g., an 544 ICMP message included a quoted packet from the source). A higher 545 level of trust can arise when an endpoint has established a short 546 term, or even long term, trust relationship with network nodes. 548 4.10. Intermediate Nodes Trusting Endpoints 550 If the endpoints do not have any trust relationship with the 551 intermediate nodes along a path, operators have been reluctant to 552 deploy techniques that rely on endpoints sending unauthenticated 553 control signals to routers. (See Section 6.2 and Section 6.7. We 554 also note this still remains a factor hindering deployment of 555 DiffServ). 557 4.11. Reacting to Distant Signals 559 Because the Internet is a distributed system, if the distance that 560 information from distant path elements travels to a Path Aware host 561 is sufficiently large, the information may no longer accurately 562 represent the state and situation at the distant host or elements 563 along the path when it is received locally. In this case, the 564 benefit that a Path Aware technique provides will be inconsistent, 565 and may not always be beneficial. (See Section 6.3). 567 4.12. Support in Endpoint Protocol Stacks 569 Just because a protocol stack provides a new feature/signal does not 570 mean that applications will use the feature/signal. Protocol stacks 571 may not know how to effectively utilize Path-Aware techniques, 572 because the protocol stack may require information from applications 573 to permit the technique to work effectively, but applications may not 574 a-priori know that information. Even if the application does know 575 that information, the de-facto sockets API has no way of signaling 576 application expectations for the network path to the protocol stack. 577 In order for applications to provide these expectations to protocol 578 stacks, we need an API that signals more than the packets to be sent. 579 (See Section 6.1 and Section 6.2). 581 5. Future Work 583 By its nature, this document has been retrospective. In addition to 584 considering how the Lessons Learned to date apply to current and 585 future Path Aware networking proposals, it's also worth considering 586 whether there is deeper investigation left to do. 588 * We note that this work was based on contributions from experts on 589 various Path Aware networking techniques, and all of the 590 contributed techniques involved unicast protocols. We didn't 591 consider how these lessons might apply to multicast, and, given 592 anecdotal reports at the IETF 109 MOPS working group meeting of IP 593 multicast offerings within data centers at one or more cloud 594 providers ([MOPS-109-Min]), it might be useful to think about path 595 awareness in multicast, before we have a history of unsuccessful 596 deployments to document. 598 * The question of whether a mechanism supports admission control, 599 based on either endpoints or applications, is associated with Path 600 Awareness. One of the motivations of IntServ and a number of 601 other architectures (e.g. Deterministic Networking, [RFC8655]) is 602 the ability to "say no" to an application based on resource 603 availability on a path, before the application tries to inject 604 traffic onto that path and discovers the path does not have the 605 capacity to sustain enough utility to meet the application's 606 minimum needs. The question of whether admission control is 607 needed comes up repeatedly, but we have learned a few useful 608 lessons that, while covered implicitly in some of the lessons 609 learned of the document, might be explained explicitly: 611 - We have gained a lot of experience with application-based 612 adaptation since the days where applications just injected 613 traffic in-elastically into the network. Such adaptations seem 614 to work well enough that admission control is of less value to 615 these applications 617 - There are end-to-end measurement techniques that can steer 618 traffic at the application layer (Content Distribution 619 Networks, multi-CDNs like Conviva [Conviva], etc.) 621 - We noted in Section 4.12 that applications often don't know how 622 to utilize Path Aware techniques. This includes not knowing 623 enough about their admission control threshold to be able to 624 ask accurately for the resources they need, whether this is 625 because the application itself doesn't know, or because the 626 application has no way to signal its expectations to the 627 underlying protocol stack. To date, attempts to help them 628 haven't gotten anywhere (e.g. the multiple-TSPEC additions to 629 RSVP to attempt to mirror codec selection by applications 630 [I-D.ietf-tsvwg-intserv-multiple-tspec] expired in 2013). 632 * We note that this work took the then-current IP network 633 architecture as given, at least at the time each technique was 634 proposed. It might be useful to consider aspects of the now- 635 current IP network architecture that ease, or impede, Path Aware 636 networking techniques. For example, there is limited ability in 637 IP to constrain bidirectional paths to be symmetric, and 638 information-centric networking protocols such as Named Data 639 Networking (NDN) and Content-Centric Networking (CCNx) ([RFC8793]) 640 must force bidirectional path symmetry using protocol-specific 641 mechanisms. 643 6. Contributions 645 Contributions on these Path Aware networking techniques were analyzed 646 to arrive at the Lessons Learned captured in Section 4. 648 Our expectation is that most readers will not need to read through 649 this section carefully, but we wanted to record these hard-fought 650 lessons as a service to others who may revisit this document, so 651 they'll have the details close at hand. 653 6.1. Stream Transport (ST, ST2, ST2+) 655 The suggested references for Stream Transport are: 657 * ST - A Proposed Internet Stream Protocol [IEN-119] 659 * Experimental Internet Stream Protocol, Version 2 (ST-II) [RFC1190] 661 * Internet Stream Protocol Version 2 (ST2) Protocol Specification - 662 Version ST2+ [RFC1819] 664 The first version of Stream Transport, ST [IEN-119], was published in 665 the late 1970's and was implemented and deployed on the ARPANET at 666 small scale. It was used throughout the 1980's for experimental 667 transmission of voice, video, and distributed simulation. 669 The second version of the ST specification (ST2) [RFC1190] [RFC1819] 670 was an experimental connection-oriented internetworking protocol that 671 operated at the same layer as connectionless IP. ST2 packets could 672 be distinguished by their IP header version numbers (IP, at that 673 time, used version number 4, while ST2 used version number 5). 675 ST2 used a control plane layered over IP to select routes and reserve 676 capacity for real-time streams across a network path, based on a flow 677 specification communicated by a separate protocol. The flow 678 specification could be associated with QoS state in routers, 679 producing an experimental resource reservation protocol. This 680 allowed ST2 routers along a path to offer end-to-end guarantees, 681 primarily to satisfy the QoS requirements for realtime services over 682 the Internet. 684 6.1.1. Reasons for Non-deployment 686 Although implemented in a range of equipment, ST2 was not widely used 687 after completion of the experiments. It did not offer the 688 scalability and fate-sharing properties that have come to be desired 689 by the Internet community. 691 The ST2 protocol is no longer in use. 693 6.1.2. Lessons Learned. 695 As time passed, the trade-off between router processing and link 696 capacity changed. Links became faster and the cost of router 697 processing became comparatively more expensive. 699 The ST2 control protocol used "hard state" - once a route was 700 established, and resources were reserved, routes and resources 701 existing until they were explicitly released via signaling. A soft- 702 state approach was thought superior to this hard-state approach, and 703 led to development of the IntServ model described in Section 6.2. 705 6.2. Integrated Services (IntServ) 707 The suggested references for IntServ are: 709 * RFC 1633 Integrated Services in the Internet Architecture: an 710 Overview [RFC1633] 712 * RFC 2211 Specification of the Controlled-Load Network Element 713 Service [RFC2211] 715 * RFC 2212 Specification of Guaranteed Quality of Service [RFC2212] 717 * RFC 2215 General Characterization Parameters for Integrated 718 Service Network Elements [RFC2215] 720 * RFC 2205 Resource ReSerVation Protocol (RSVP) [RFC2205] 722 In 1994, when the IntServ architecture document [RFC1633] was 723 published, real-time traffic was first appearing on the Internet. At 724 that time, bandwidth was still a scarce commodity. Internet Service 725 Providers built networks over DS3 (45 Mbps) infrastructure, and sub- 726 rate (< 1 Mpbs) access was common. Therefore, the IETF anticipated a 727 need for a fine-grained QoS mechanism. 729 In the IntServ architecture, some applications can require service 730 guarantees. Therefore, those applications use the Resource 731 Reservation Protocol (RSVP) [RFC2205] to signal QoS reservations 732 across network paths. Every router in the network that participates 733 in IntServ maintains per-flow soft-state to a) perform call admission 734 control and b) deliver guaranteed service. 736 Applications use Flow Specification (Flow Specs) [RFC2210] to 737 describe the traffic that they emit. RSVP reserves capacity for 738 traffic on a per Flow Spec basis. 740 6.2.1. Reasons for Non-deployment 742 Although IntServ has been used in enterprise and government networks, 743 IntServ was never widely deployed on the Internet because of its 744 cost. The following factors contributed to operational cost: 746 * IntServ must be deployed on every router that is on a path where 747 IntServ is to be used. Although it is possible to include a 748 router that does not participate in IntServ along the path being 749 controlled, if that router is likely to become a bottleneck, 750 IntServ cannot be used to avoid that bottleneck along the path 752 * IntServ maintained per flow state 754 As IntServ was being discussed, the following occurred: 756 * For many expected uses, it became more cost effective to solve the 757 QoS problem by adding bandwidth. Between 1994 and 2000, Internet 758 Service Providers upgraded their infrastructures from DS3 (45 759 Mbps) to OC-48 (2.4 Gbps). This meant that even if an endpoint 760 was using IntServ in an IntServ-enabled network, its requests 761 would rarely, if ever, be denied, so endpoints and Internet 762 Service Providers had little reason to enable IntServ. 764 * DiffServ [RFC2475] offered a more cost-effective, albeit less 765 fine-grained, solution to the QoS problem. 767 6.2.2. Lessons Learned. 769 The following lessons were learned: 771 * Any mechanism that requires every participating onpath router to 772 maintain per-flow state is not likely to succeed, unless the 773 additional cost for offering the feature can be recovered from the 774 user. 776 * Any mechanism that requires an operator to upgrade all of its 777 routers is not likely to succeed, unless the additional cost for 778 offering the feature can be recovered from the user. 780 In environments where IntServ has been deployed, trust relationships 781 with endpoints are very different from trust relationships on the 782 Internet itself, and there are often clearly-defined hierarchies in 783 Service Level Agreements (SLAs), and well-defined transport flows 784 operating with pre-determined capacity and latency requirements over 785 paths where capacity or other attributes are constrained. 787 IntServ was never widely deployed to manage capacity across the 788 Internet. However, the technique that it produced was deployed for 789 reasons other than bandwidth management. RSVP is widely deployed as 790 an MPLS signaling mechanism. BGP reuses the RSVP concept of Filter 791 Specs to distribute firewall filters, although they are called Flow 792 Spec Component Types in BGP [RFC5575]. 794 6.3. Quick-Start TCP 796 The suggested references for Quick-Start TCP are: 798 * Quick-Start for TCP and IP [RFC4782] 800 * Determining an appropriate initial sending rate over an 801 underutilized network path [SAF07] 803 * Fast Startup Internet Congestion Control for Broadband Interactive 804 Applications [Sch11] 806 * Using Quick-Start to enhance TCP-friendly rate control performance 807 in bidirectional satellite networks [QS-SAT] 809 Quick-Start [RFC4782] is an Experimental TCP extension that leverages 810 support from the routers on the path to determine an allowed initial 811 sending rate for a path through the Internet, either at the start of 812 data transfers or after idle periods. Without information about the 813 path, a sender cannot easily determine an appropriate initial sending 814 rate. The default TCP congestion control therefore uses the safe but 815 time-consuming slow-start algorithm [RFC5681]. With Quick-Start, 816 connections are allowed to use higher initial sending rates if there 817 is significant unused bandwidth along the path, and if the sender and 818 all of the routers along the path approve the request. 820 By examining the Time To Live (TTL) field in Quick-Start packets, a 821 sender can determine if routers on the path have approved the Quick- 822 Start request. However, this method is unable to take into account 823 the routers hidden by tunnels or other network nodes invisible at the 824 IP layer. 826 The protocol also includes a nonce that provides protection against 827 cheating routers and receivers. If the Quick-Start request is 828 explicitly approved by all routers along the path, the TCP host can 829 send at up to the approved rate; otherwise TCP would use the default 830 congestion control. Quick-Start requires modifications in the 831 involved end-systems as well in routers. Due to the resulting 832 deployment challenges, Quick-Start was only proposed in [RFC4782] for 833 controlled environments. 835 The Quick-Start mechanism is a lightweight, coarse-grained, in-band, 836 network-assisted fast startup mechanism. The benefits are studied by 837 simulation in a research paper [SAF07] that complements the protocol 838 specification. The study confirms that Quick-Start can significantly 839 speed up mid-sized data transfers. That paper also presents router 840 algorithms that do not require keeping per-flow state. Later studies 841 [Sch11] comprehensively analyzes Quick-Start with a full Linux 842 implementation and with a router fast path prototype using a network 843 processor. In both cases, Quick-Start could be implemented with 844 limited additional complexity. 846 6.3.1. Reasons for Non-deployment 848 However, experiments with Quick-Start in [Sch11] revealed several 849 challenges: 851 * Having information from the routers along the path can reduce the 852 risk of congestion, but cannot avoid it entirely. Determining 853 whether there is unused capacity is not trivial in actual router 854 and host implementations. Data about available capacity visible 855 at the IP layer may be imprecise, and due to the propagation 856 delay, information can already be outdated when it reaches a 857 sender. There is a trade-off between the speedup of data 858 transfers and the risk of congestion even with Quick-Start. This 859 could be mitigated by only allowing Quick-Start to access a 860 proportion of the unused capacity along a path. 862 * For scalable router fast path implementation, it is important to 863 enable parallel processing of packets, as this is a widely used 864 method e.g. in network processors. One challenge is 865 synchronization of information between packets that are processed 866 in parallel, which should be avoided as much as possible. 868 * Only some types of application traffic can benefit from Quick- 869 Start. Capacity needs to be requested and discovered. The 870 discovered capacity needs to be utilized by the flow, or it 871 implicitly becomes available for other flows. Failing to use the 872 requested capacity may have already reduced the pool of Quick- 873 Start capacity that was made available to other competing Quick- 874 Start requests. The benefit is greatest when senders use this 875 only for bulk flows and avoid sending unnecessary Quick-Start 876 requests, e.g. for flows that only send a small amount of data. 877 Choosing an appropriate request size requires application-internal 878 knowledge that is not commonly expressed by the transport API. 879 How a sender can determine the rate for an initial Quick-Start 880 request is still a largely unsolved problem. 882 There is no known deployment of Quick-Start for TCP or other IETF 883 transports. 885 6.3.2. Lessons Learned 887 Some lessons can be learned from Quick-Start. Despite being a very 888 light-weight protocol, Quick-Start suffers from poor incremental 889 deployment properties, both regarding the required modifications in 890 network infrastructure as well as its interactions with applications. 891 Except for corner cases, congestion control can be quite efficiently 892 performed end-to-end in the Internet, and in modern stacks there is 893 not much room for significant improvement by additional network 894 support. 896 After publication of the Quick-Start specification, there have been 897 large-scale experiments with an initial window of up to 10 MSS 898 [RFC6928]. This alternative "IW10" approach can also ramp-up data 899 transfers faster than the standard congestion control, but it only 900 requires sender-side modifications. As a result, this approach can 901 be easier and incrementally deployed in the Internet. While 902 theoretically Quick-Start can outperform "IW10", the improvement in 903 completion time for data transfer times can, in many cases, be small. 904 After publication of [RFC6928], most modern TCP stacks have increased 905 their default initial window. 907 6.4. ICMP Source Quench 909 The suggested references for ICMP Source Quench are: 911 * INTERNET CONTROL MESSAGE PROTOCOL [RFC0792] 913 The ICMP Source Quench message [RFC0792] allowed an on-path router to 914 request the source of a flow to reduce its sending rate. This method 915 allowed a router to provide an early indication of impending 916 congestion on a path to the sources that contribute to that 917 congestion. 919 6.4.1. Reasons for Non-deployment 921 This method was deployed in Internet routers over a period of time, 922 the reaction of endpoints to receiving this signal has varied. For 923 low speed links, with low multiplexing of flows the method could be 924 used to regulate (momentarily reduce) the transmission rate. 925 However, the simple signal does not scale with link speed, or the 926 number of flows sharing a link. 928 The approach was overtaken by the evolution of congestion control 929 methods in TCP [RFC2001], and later also by other IETF transports. 930 Because these methods were based upon measurement of the end-to-end 931 path and an algorithm in the endpoint, they were able to evolve and 932 mature more rapidly than methods relying on interactions between 933 operational routers and endpoint stacks. 935 After ICMP Source Quench was specified, the IETF began to recommend 936 that transports provide end-to-end congestion control [RFC2001]. The 937 Source Quench method has been obsoleted by the IETF [RFC6633], and 938 both hosts and routers must now silently discard this message. 940 6.4.2. Lessons Learned 942 This method had several problems: 944 First, [RFC0792] did not sufficiently specify how the sender would 945 react to the ICMP Source Quench signal from the path (e.g., 946 [RFC1016]). There was ambiguity in how the sender should utilize 947 this additional information. This could lead to unfairness in the 948 way that receivers (or routers) responded to this message. 950 Second, while the message did provide additional information, the 951 Explicit Congestion Notification (ECN) mechanism [RFC3168] provided a 952 more robust and informative signal for network nodes to provide early 953 indication that a path has become congested. 955 The mechanism originated at a time when the Internet trust model was 956 very different. Most endpoint implementations did not attempt to 957 verify that the message originated from an on-path node before they 958 utilized the message. This made it vulnerable to denial of service 959 attacks. In theory, routers might have chosen to use the quoted 960 packet contained in the ICMP payload to validate that the message 961 originated from an on-path node, but this would have increased per- 962 packet processing overhead for each router along the path, would have 963 required transport functionality in the router to verify whether the 964 quoted packet header corresponded to a packet the router had sent. 965 In addition, section 5.2 of [RFC4443] noted ICMPv6-based attacks on 966 hosts that would also have threatened routers processing ICMPv6 967 Source Quench payloads. As time passed, it became increasingly 968 obvious that the lack of validation of the messages exposed receivers 969 to a security vulnerability where the messages could be forged to 970 create a tangible denial of service opportunity. 972 6.5. Triggers for Transport (TRIGTRAN) 974 The suggested references for TRIGTRAN are: 976 * TRIGTRAN BOF at IETF 55 [TRIGTRAN-55] 978 * TRIGTRAN BOF at IETF 56 [TRIGTRAN-56] 980 TCP [RFC0793] has a well-known weakness - the end-to-end flow control 981 mechanism has only a single signal, the loss of a segment, and TCP 982 implementations since the late 1980s have interpreted the loss of a 983 segment as evidence that the path between two endpoints may have 984 become congested enough to exhaust buffers on intermediate hops, so 985 that the TCP sender should "back off" - reduce its sending rate until 986 it knows that its segments are now being delivered without loss 987 [RFC5681]. More modern TCP stacks have added a growing array of 988 strategies about how to establish the sending rate [RFC5681], but 989 when a path is no longer operational, TCP would continue to retry 990 transmissions, which would fail, again, and double their 991 Retransmission Time Out (RTO) timers with each failed transmission, 992 with the result that TCP would wait many seconds before retrying a 993 segment, even if the path becomes operational while the sender is 994 waiting for its next retry. 996 The thinking behind TRIGTRAN was that if a path completely stopped 997 working because a link along the path was "down", somehow something 998 along the path could signal TCP when that link returned to service, 999 and the sending TCP could retry immediately, without waiting for a 1000 full retransmission timeout (RTO) period. 1002 6.5.1. Reasons for Non-deployment 1004 The early dreams for TRIGTRAN were dashed because of an assumption 1005 that TRIGTRAN triggers would be unauthenticated. This meant that any 1006 "safe" TRIGTRAN mechanism would have relied on a mechanism such as 1007 setting the IPv4 TTL or IPv6 Hop Count to 255 at a sender and testing 1008 that it was 254 upon receipt, so that a receiver could verify that a 1009 signal was generated by an adjacent sender known to be on the path 1010 being used, and not some unknown sender which might not even be on 1011 the path (e.g., "The Generalized TTL Security Mechanism (GTSM)" 1012 [RFC5082]). This situation is very similar to the case for ICMP 1013 Source Quench messages as described in Section 6.4, which were also 1014 unauthenticated, and could be sent by an off-path attacker, resulting 1015 in deprecation of ICMP Source Quench message processing [RFC6633]. 1017 TRIGTRAN's scope shrunk from "the path is down" to "the first-hop 1018 link is down". 1020 But things got worse. 1022 Because TRIGTRAN triggers would only be provided when the first-hop 1023 link was "down", TRIGTRAN triggers couldn't replace normal TCP 1024 retransmission behavior if the path failed because some link further 1025 along the network path was "down". So TRIGTRAN triggers added 1026 complexity to an already complex TCP state machine, and did not allow 1027 any existing complexity to be removed. 1029 There was also an issue that the TRIGTRAN signal was not sent in 1030 response to a specific host that had been sending packets, and was 1031 instead a signal that stimulated a response by any sender on the 1032 link. This needs to scale when there are multiple flows trying to 1033 use the same resource, yet the sender of a trigger has no 1034 understanding how many of the potential traffic sources will respond 1035 by sending packets - if recipients of the signal back-off their 1036 responses to a trigger to improve scaling, then that immediately 1037 mitigates the benefit of the signal. 1039 Finally, intermediate forwarding nodes required modification to 1040 provide TRIGTRAN triggers, but operators couldn't charge for TRIGTRAN 1041 triggers, so there was no way to recover the cost of modifying, 1042 testing, and deploying updated intermediate nodes. 1044 Two TRIGTRAN BOFs were held, at IETF 55 [TRIGTRAN-55] and IETF 56 1045 [TRIGTRAN-56], but this work was not chartered, and there was no 1046 interest in deploying TRIGTRAN unless it was chartered and 1047 standardized in the IETF. 1049 6.5.2. Lessons Learned. 1051 The reasons why this work was not chartered, much less deployed, 1052 provide several useful lessons for researchers. 1054 * TRIGTRAN started with a plausible value proposition, but 1055 networking realities in the early 2000s forced reductions in scope 1056 that led directly to reductions in potential benefits, but no 1057 corresponding reductions in costs and complexity. 1059 * These reductions in scope were the direct result of an inability 1060 for hosts to trust or authenticate TRIGTRAN signals they received 1061 from the network. 1063 * Operators did not believe they could charge for TRIGTRAN 1064 signaling, because first-hop links didn't fail frequently, and 1065 TRIGTRAN provided no reduction in operating expenses, so there was 1066 little incentive to purchase and deploy TRIGTRAN-capable network 1067 equipment. 1069 It is also worth noting that the targeted environment for TRIGTRAN in 1070 the late 1990s contained links with a relatively small number of 1071 directly-connected hosts - for instance, cellular or satellite links. 1072 The transport community was well aware of the dangers of sender 1073 synchronization based on multiple senders receiving the same stimulus 1074 at the same time, but the working assumption for TRIGTRAN was that 1075 there wouldn't be enough senders for this to be a meaningful problem. 1076 In the 2010s, it is common for a single "link" to support many 1077 senders and receivers on a single link, likely requiring TRIGTRAN 1078 senders to wait some random amount of time before sending after 1079 receiving a TRIGTRAN signal, which would have reduced the benefits of 1080 TRIGTRAN even more. 1082 6.6. Shim6 1084 The suggested references for Shim6 are: 1086 * Shim6: Level 3 Multihoming Shim Protocol for IPv6 [RFC5533] 1088 The IPv6 routing architecture [RFC1887] assumed that most sites on 1089 the Internet would be identified by Provider Assigned IPv6 prefixes, 1090 so that Default-Free Zone routers only contained routes to other 1091 providers, resulting in a very small IPv6 global routing table. 1093 For a single-homed site, this could work well. A multihomed site 1094 with only one upstream provider could also work well, although BGP 1095 multihoming from a single upstream provider was often a premium 1096 service (costing more than twice as much as two single-homed sites), 1097 and if the single upstream provider went out of service, all of the 1098 multihomed paths could fail simultaneously. 1100 IPv4 sites often multihomed by obtaining Provider Independent 1101 prefixes, and advertising these prefixes through multiple upstream 1102 providers. With the assumption that any multihomed IPv4 site would 1103 also multihome in IPv6, it seemed likely that IPv6 routing would be 1104 subject to the same pressures to announce Provider Independent 1105 prefixes, resulting in a global IPv6 routing table that exhibited the 1106 same explosive growth as the global IPv4 routing table. During the 1107 early 2000s, work began on a protocol that would provide multihoming 1108 for IPv6 sites without requiring sites to advertise Provider 1109 Independent prefixes into the IPv6 global routing table. 1111 This protocol, called Shim6, allowed two endpoints to exchange 1112 multiple addresses ("Locators") that all mapped to the same endpoint 1113 ("Identity"). After an endpoint learned multiple Locators for the 1114 other endpoint, it could send to any of those Locators with the 1115 expectation that those packets would all be delivered to the endpoint 1116 with the same Identity. Shim6 was an example of an "Identity/Locator 1117 Split" protocol. 1119 Shim6, as defined in [RFC5533] and related RFCs, provided a workable 1120 solution for IPv6 multihoming using Provider Assigned prefixes, 1121 including capability discovery and negotiation, and allowing end-to- 1122 end application communication to continue even in the face of path 1123 failure, because applications don't see Locator failures, and 1124 continue to communicate with the same Identity using a different 1125 Locator. 1127 6.6.1. Reasons for Non-deployment 1129 Note that the problem being addressed was "site multihoming", but 1130 Shim6 was providing "host multihoming". That meant that the decision 1131 about what path would be used was under host control, not under edge 1132 router control. 1134 Although more work could have been done to provide a better technical 1135 solution, the biggest impediments to Shim6 deployment were 1136 operational and business considerations. These impediments were 1137 discussed at multiple network operator group meetings, including 1138 [Shim6-35] at [NANOG-35]. 1140 The technical issues centered around concerns that Shim6 relied on 1141 the host to track all the connections, while also tracking Identity/ 1142 Locator mappings in the kernel, and tracking failures to recognize 1143 that an available path has failed. 1145 The operational issues centered around concerns that operators were 1146 performing traffic engineering on traffic aggregates. With Shim6, 1147 these operator traffic engineering policies must be pushed down to 1148 individual hosts. 1150 In addition, operators would have no visibility or control over the 1151 decision of hosts choosing to switch to another path. They expressed 1152 concerns that relying on hosts to steer traffic exposed operator 1153 networks to oscillation based on feedback loops, if hosts moved from 1154 path to path frequently. Given that Shim6 was intended to support 1155 multihoming across operators, operators providing only one of the 1156 paths would have even less visibility as traffic suddenly appeared 1157 and disappeared on their networks. 1159 In addition, firewalls that expected to find a TCP or UDP transport- 1160 level protocol header in the IP payload would see a Shim6 Identity 1161 header instead, and would not perform transport-protocol-based 1162 firewalling functions because the firewall's normal processing logic 1163 would not look past the Identity header. 1165 The business issues centered on reducing or removing the ability to 1166 sell BGP multihoming service to their own customers, which is often 1167 more expensive than two single-homed connectivity services. 1169 6.6.2. Lessons Learned 1171 It is extremely important to take operational concerns into account 1172 when a path-aware protocol is making decisions about path selection 1173 that may conflict with existing operational practices and business 1174 considerations. 1176 6.6.3. Addendum on MultiPath TCP 1178 During discussions in the PANRG session at IETF 103 [PANRG-103-Min], 1179 Lars Eggert, past Transport Area Director, pointed out that during 1180 charter discussions for the Multipath TCP working group [MP-TCP], 1181 operators expressed concerns that customers could use Multipath TCP 1182 to loadshare TCP connections across operators simultaneously and 1183 compare passive performance measurements across network paths in real 1184 time, changing the balance of power in those business relationships. 1185 Although the Multipath TCP working group was chartered, this concern 1186 could have acted as an obstacle to deployment. 1188 Operator objections to Shim6 were focused on technical concerns, but 1189 this concern could have also been an obstacle to Shim6 deployment if 1190 the technical concerns had been overcome. 1192 6.7. Next Steps in Signaling (NSIS) 1194 The suggested references for Next Steps in Signaling (NSIS) are: 1196 * the concluded working group charter [NSIS-CHARTER-2001] 1198 * GIST: General Internet Signalling Transport [RFC5971] 1200 * NAT/Firewall NSIS Signaling Layer Protocol (NSLP) [RFC5973] 1202 * NSIS Signaling Layer Protocol (NSLP) for Quality-of-Service 1203 Signaling [RFC5974] 1205 * Authorization for NSIS Signaling Layer Protocols [RFC5981] 1207 The NSIS Working Group worked on signaling techniques for network 1208 layer resources (e.g., QoS resource reservations, Firewall and NAT 1209 traversal). 1211 When RSVP [RFC2205] was used in deployments, a number of questions 1212 came up about its perceived limitations and potential missing 1213 features. The issues noted in the NSIS Working Group charter 1214 [NSIS-CHARTER-2001] include interworking between domains with 1215 different QoS architectures, mobility and roaming for IP interfaces, 1216 and complexity. Later, the lack of security in RSVP was also 1217 recognized ([RFC4094]). 1219 The NSIS Working Group was chartered to tackle those issues and 1220 initially focused on QoS signaling as its primary use case. However, 1221 over time a new approach evolved that introduced a modular 1222 architecture using application-specific signaling protocols (the NSIS 1223 Signaling Layer Protocol (NSLP)) on top of a generic signaling 1224 transport protocol (the NSIS Transport Layer Protocol (NTLP)). 1226 The NTLP is defined in [RFC5971]. Two NSLPs are defined: the NSIS 1227 Signaling Layer Protocol (NSLP) for Quality-of-Service Signaling 1228 [RFC5974] as well as the NAT/Firewall NSIS Signaling Layer Protocol 1229 (NSLP) [RFC5973]. 1231 6.7.1. Reasons for Non-deployment 1233 The obstacles for deployment can be grouped into implementation- 1234 related aspects and operational aspects. 1236 * Implementation-related aspects: 1238 Although NSIS provides benefits with respect to flexibility, 1239 mobility, and security compared to other network signaling 1240 techniques, hardware vendors were reluctant to deploy this solution, 1241 because it would require additional implementation effort and would 1242 result in additional complexity for router implementations. 1244 The NTLP mainly operates as path-coupled signaling protocol, i.e, its 1245 messages are processed at the intermediate node's control plane that 1246 are also forwarding the data flows. This requires a mechanism to 1247 intercept signaling packets while they are forwarded in the same 1248 manner (especially along the same path) as data packets. NSIS uses 1249 the IPv4 and IPv6 Router Alert Option (RAO) to allow for interception 1250 of those path-coupled signaling messages, and this technique requires 1251 router implementations to correctly understand and implement the 1252 handling of RAOs, e.g., to only process packet with RAOs of interest 1253 and to leave packets with irrelevant RAOs in the fast forwarding 1254 processing path (a comprehensive discussion of these issues can be 1255 found in [RFC6398]). The latter was an issue with some router 1256 implementations at the time of standardization. 1258 Another reason is that path-coupled signaling protocols that interact 1259 with routers and request manipulation of state at these routers (or 1260 any other network element in general) are under scrutiny: a packet 1261 (or sequence of packets) out of the mainly untrusted data path is 1262 requesting creation and manipulation of network state. This is seen 1263 as potentially dangerous (e.g., opens up a Denial of Service (DoS) 1264 threat to a router's control plane) and difficult for an operator to 1265 control. Path-coupled signaling approaches were considered 1266 problematic (see also section 3 of [RFC6398]). There are 1267 recommendations on how to secure NSIS nodes and deployments (e.g., 1268 [RFC5981]). 1270 * Operational Aspects: 1272 NSIS not only required trust between customers and their provider, 1273 but also among different providers. Especially, QoS signaling 1274 techniques would require some kind of dynamic service level agreement 1275 support that would imply (potentially quite complex) bilateral 1276 negotiations between different Internet service providers. This 1277 complexity was not considered to be justified and increasing the 1278 bandwidth (and thus avoiding bottlenecks) was cheaper than actively 1279 managing network resource bottlenecks by using path-coupled QoS 1280 signaling techniques. Furthermore, an end-to-end path typically 1281 involves several provider domains and these providers need to closely 1282 cooperate in cases of failures. 1284 6.7.2. Lessons Learned 1286 One goal of NSIS was to decrease the complexity of the signaling 1287 protocol, but a path-coupled signaling protocol comes with the 1288 intrinsic complexity of IP-based networks, beyond the complexity of 1289 the signaling protocol itself. Sources of intrinsic complexity 1290 include: 1292 * the presence of asymmetric routes between endpoints and routers 1294 * the lack of security and trust at large in the Internet 1295 infrastructure 1297 * the presence of different trust boundaries 1299 * the effects of best-effort networks (e.g., robustness to packet 1300 loss) 1302 * divergence from the fate sharing principle (e.g., state within the 1303 network). 1305 Any path-coupled signaling protocol has to deal with these realities. 1307 Operators view the use of IPv4 and IPv6 Router Alert Option (RAO) to 1308 signal routers along the path from end systems with suspicion, 1309 because these end systems are usually not authenticated and heavy use 1310 of RAOs can easily increase the CPU load on routers that are designed 1311 to process most packets using a hardware "fast path" and diverting 1312 packets containing RAO to a slower, more capable processor. 1314 6.8. IPv6 Flow Label 1316 The suggested references for IPv6 Flow Label are: 1318 * IPv6 Flow Label Specification [RFC6437] 1320 IPv6 specifies a 20-bit field Flow Label field [RFC6437], included in 1321 the fixed part of the IPv6 header and hence present in every IPv6 1322 packet. An endpoint sets the value in this field to one of a set of 1323 pseudo-randomly assigned values. If a packet is not part of any 1324 flow, the flow label value is set to zero [RFC3697]. A number of 1325 Standards Track and Best Current Practice RFCs (e.g., [RFC8085], 1326 [RFC6437], [RFC6438]) encourage IPv6 endpoints to set a non-zero 1327 value in this field. A multiplexing transport could choose to use 1328 multiple flow labels to allow the network to independently forward 1329 its subflows, or to use one common value for the traffic aggregate. 1330 The flow label is present in all fragments. IPsec was originally put 1331 forward as one important use-case for this mechanism and does encrypt 1332 the field [RFC6438]. 1334 Once set, the flow label can provide information that can help inform 1335 network nodes about subflows present at the transport layer, without 1336 needing to interpret the setting of upper layer protocol fields 1337 [RFC6294]. This information can also be used to coordinate how 1338 aggregates of transport subflows are grouped when queued in the 1339 network and to select appropriate per-flow forwarding when choosing 1340 between alternate paths [RFC6438] (e.g. for Equal Cost Multipath 1341 Routing (ECMP) and Link Aggregation (LAG)). 1343 6.8.1. Reasons for Non-deployment 1345 Despite the field being present in every IPv6 packet, the mechanism 1346 did not receive as much use as originally envisioned. One reason is 1347 that to be useful it requires engagement by two different 1348 stakeholders: 1350 * Endpoint Implementation: 1352 For network nodes along a path to utilize the flow label there needs 1353 to be a non-zero value inserted in the field [RFC6437] at the sending 1354 endpoint. There needs to be an incentive for an endpoint to set an 1355 appropriate non-zero value. The value should appropriately reflect 1356 the level of aggregation the traffic expects to be provided by the 1357 network. However, this requires the stack to know granularity at 1358 which flows should be identified (or conversely which flows should 1359 receive aggregated treatment), i.e., which packets carry the same 1360 flow label. Therefore, setting a non-zero value may result in 1361 additional choices that need to be made by an application developer. 1363 Although the standard [RFC3697] forbids any encoding of meaning into 1364 the flow label value, the opportunity to use the flow label as a 1365 covert channel or to signal other meta-information may have raised 1366 concerns about setting a non-zero value [RFC6437]. 1368 Before methods are widely deployed to use this method, there could be 1369 no incentive for an endpoint to set the field. 1371 * Operational support in network nodes: 1373 A benefit can only be realized when a network node along the path 1374 also uses this information to inform its decisions. Network 1375 equipment (routers and/or middleboxes) need to include appropriate 1376 support so they can utilize the field when making decisions about how 1377 to classify flows, or to inform forwarding choices. Use of any 1378 optional feature in a network node also requires corresponding 1379 updates to operational procedures, and therefore is normally only 1380 introduced when the cost can be justified. 1382 A benefit from utilizing the flow label is expected to be increased 1383 quality of experience for applications - but this comes at some 1384 operational cost to an operator, and requires endpoints to set the 1385 field. 1387 6.8.2. Lessons Learned 1389 The flow label is a general purpose header field for use by the path. 1390 Multiple uses have been proposed. One candidate use was to reduce 1391 the complexity of forwarding decisions. However, modern routers can 1392 use a "fast path", often taking advantage of hardware to accelerate 1393 processing. The method can assist in more complex forwarding, such 1394 as ECMP and load balancing. 1396 Although [RFC6437] recommended that endpoints should by default 1397 choose uniformly-distributed labels for their traffic, the 1398 specification permitted an endpoint to choose to set a zero value. 1399 This ability of endpoints to choose to set a flow label of zero has 1400 had consequences on deployability: 1402 * Before wide-scale support by endpoints, it would be impossible to 1403 rely on a non-zero flow label being set. Network nodes therefore 1404 would need to also employ other techniques to realize equivalent 1405 functions. An example of a method is one assuming semantics of 1406 the source port field to provide entropy input to a network-layer 1407 hash. This use of a 5-tuple to classify a packet represents a 1408 layering violation [RFC6294]. When other methods have been 1409 deployed, they increase the cost of deploying standards-based 1410 methods, even though they may offer less control to endpoints and 1411 result in potential interaction with other uses/interpretation of 1412 the field. 1414 * Even though the flow label is specified as an end-to-end field, 1415 some network paths have been observed to not transparently forward 1416 the flow label. This could result from non-conformant equipment, 1417 or could indicate that some operational networks have chosen to 1418 re-use the protocol field for other (e.g. internal purposes). 1419 This results in lack of transparency, and a deployment hurdle to 1420 endpoints expecting that they can set a flow label that is 1421 utilized by the network. The more recent practice of "greasing" 1422 [GREASE] would suggest that a different outcome could have been 1423 achieved if endpoints were always required to set a non-zero 1424 value. 1426 * [RFC1809] noted that setting the choice of the flow label value 1427 can depend on the expectations of the traffic generated by an 1428 application, which suggests an API should be presented to control 1429 the setting or policy that is used. However, many currently 1430 available APIs do not have this support. 1432 A growth in the use of encrypted transports, (e.g. QUIC [QUIC-WG]) 1433 seems likely to raise similar issues to those discussed above and 1434 could motivate renewed interest in utilizing the flow label. 1436 7. Security Considerations 1438 This document describes Path Aware techniques that were not adopted 1439 and widely deployed on the Internet, so it doesn't affect the 1440 security of the Internet. 1442 If this document meets its goals, we may develop new techniques for 1443 Path Aware Networking that would affect the security of the Internet, 1444 but security considerations for those techniques will be described in 1445 the corresponding RFCs that specify them. 1447 8. IANA Considerations 1449 This document makes no requests of IANA. 1451 9. Acknowledgments 1453 Initial material for Section 6.1 on ST2 was provided by Gorry 1454 Fairhurst. 1456 Initial material for Section 6.2 on IntServ was provided by Ron 1457 Bonica. 1459 Initial material for Section 6.3 on Quick-Start TCP was provided by 1460 Michael Scharf, who also provided suggestions to improve this section 1461 after it was edited. 1463 Initial material for Section 6.4 on ICMP Source Quench was provided 1464 by Gorry Fairhurst. 1466 Initial material for Section 6.5 on Triggers for Transport (TRIGTRAN) 1467 was provided by Spencer Dawkins. 1469 Section 6.6 on Shim6 builds on initial material describing obstacles 1470 provided by Erik Nordmark, with background added by Spencer Dawkins. 1472 Initial material for Section 6.7 on Next Steps In Signaling (NSIS) 1473 was provided by Roland Bless and Martin Stiemerling. 1475 Initial material for Section 6.8 on IPv6 Flow Labels was provided by 1476 Gorry Fairhurst. 1478 Our thanks to Adrian Farrel, C.M. Heard, David Black, Erik 1479 Auerswald, Gorry Fairhurst, Joe Touch, Joeri de Ruiter, Mohamed 1480 Boucadair, Roland Bless, Ruediger Geib, Theresa Enghardt, and Wes 1481 Eddy, who provided review comments on this document as a "work in 1482 process". 1484 Mallory Knodel reviewed this document for the Internet Research 1485 Steering Group, and provided many helpful suggestions. 1487 David Oran also provided helpful comments and text suggestions on 1488 this document during Internet Research Steering Group balloting. In 1489 particular, Section 5 reflects his review. 1491 Benjamin Kaduk and Rob Wilton provided helpful comments during 1492 Internet Engineering Steering Group conflict review. 1494 Special thanks to Adrian Farrel for helping Spencer navigate the 1495 twisty little passages of Flow Specs and Filter Specs in IntServ, 1496 RSVP, MPLS, and BGP. They are all alike, except when they are 1497 different [Colossal-Cave]. 1499 10. Informative References 1501 [Colossal-Cave] 1502 "Wikipedia Page for Colossal Cave Adventure", January 1503 2019, 1504 . 1506 [Conviva] "Conviva Precision : Data Sheet", December 2020, 1507 . 1510 [GREASE] Thomson, M., "Long-term Viability of Protocol Extension 1511 Mechanisms", July 2019, . 1514 [I-D.arkko-arch-internet-threat-model] 1515 Arkko, J., "Changes in the Internet Threat Model", Work in 1516 Progress, Internet-Draft, draft-arkko-arch-internet- 1517 threat-model-01, 8 July 2019, . 1521 [I-D.farrell-etm] 1522 Farrell, S., "We're gonna need a bigger threat model", 1523 Work in Progress, Internet-Draft, draft-farrell-etm-03, 6 1524 July 2019, . 1527 [I-D.ietf-tsvwg-intserv-multiple-tspec] 1528 Polk, J. and S. Dhesikan, "Integrated Services (IntServ) 1529 Extension to Allow Signaling of Multiple Traffic 1530 Specifications and Multiple Flow Specifications in 1531 RSVPv1", Work in Progress, Internet-Draft, draft-ietf- 1532 tsvwg-intserv-multiple-tspec-02, 25 February 2013, 1533 . 1536 [I-D.irtf-panrg-path-properties] 1537 Enghardt, T. and C. Krahenbuhl, "A Vocabulary of Path 1538 Properties", Work in Progress, Internet-Draft, draft-irtf- 1539 panrg-path-properties-01, 7 September 2020, 1540 . 1543 [I-D.irtf-panrg-questions] 1544 Trammell, B., "Current Open Questions in Path Aware 1545 Networking", Work in Progress, Internet-Draft, draft-irtf- 1546 panrg-questions-08, 23 December 2020, 1547 . 1550 [IEN-119] Forgie, J., "ST - A Proposed Internet Stream Protocol", 1551 September 1979, 1552 . 1554 [ITAT] "IAB Workshop on Internet Technology Adoption and 1555 Transition (ITAT)", December 2013, 1556 . 1558 [model-t] "Model-t -- Discussions of changes in Internet deployment 1559 patterns and their impact on the Internet threat model", 1560 n.d., . 1562 [MOPS-109-Min] 1563 "Media Operations Working Group - IETF-109 Minutes", 1564 November 2020, 1565 . 1568 [MP-TCP] "Multipath TCP Working Group Home Page", n.d., 1569 . 1571 [NANOG-35] "North American Network Operators Group NANOG-35 Agenda", 1572 October 2005, 1573 . 1575 [NSIS-CHARTER-2001] 1576 "Next Steps In Signaling Working Group Charter", March 1577 2011, 1578 . 1580 [PANRG] "Path Aware Networking Research Group (Home Page)", n.d., 1581 . 1583 [PANRG-103-Min] 1584 "Path Aware Networking Research Group - IETF-103 Minutes", 1585 November 2018, 1586 . 1588 [PANRG-105-Min] 1589 "Path Aware Networking Research Group - IETF-105 Minutes", 1590 July 2019, 1591 . 1593 [PANRG-106-Min] 1594 "Path Aware Networking Research Group - IETF-106 Minutes", 1595 November 2019, 1596 . 1598 [PANRG-99] "Path Aware Networking Research Group - IETF-99", July 1599 2017, 1600 . 1602 [PATH-Decade] 1603 Bonaventure, O., "A Decade of Path Awareness", July 2017, 1604 . 1607 [QS-SAT] Secchi, R., Sathiaseelan, A., Potorti, F., Gotta, A., and 1608 G. Fairhurst, "Using Quick-Start to enhance TCP-friendly 1609 rate control performance in bidirectional satellite 1610 networks", 2009, 1611 . 1613 [QUIC-WG] "QUIC Working Group Home Page", n.d., 1614 . 1616 [RFC0792] Postel, J., "Internet Control Message Protocol", STD 5, 1617 RFC 792, DOI 10.17487/RFC0792, September 1981, 1618 . 1620 [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, 1621 RFC 793, DOI 10.17487/RFC0793, September 1981, 1622 . 1624 [RFC1016] Prue, W. and J. Postel, "Something a Host Could Do with 1625 Source Quench: The Source Quench Introduced Delay 1626 (SQuID)", RFC 1016, DOI 10.17487/RFC1016, July 1987, 1627 . 1629 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - 1630 Communication Layers", STD 3, RFC 1122, 1631 DOI 10.17487/RFC1122, October 1989, 1632 . 1634 [RFC1190] Topolcic, C., "Experimental Internet Stream Protocol: 1635 Version 2 (ST-II)", RFC 1190, DOI 10.17487/RFC1190, 1636 October 1990, . 1638 [RFC1633] Braden, R., Clark, D., and S. Shenker, "Integrated 1639 Services in the Internet Architecture: an Overview", 1640 RFC 1633, DOI 10.17487/RFC1633, June 1994, 1641 . 1643 [RFC1809] Partridge, C., "Using the Flow Label Field in IPv6", 1644 RFC 1809, DOI 10.17487/RFC1809, June 1995, 1645 . 1647 [RFC1819] Delgrossi, L., Ed. and L. Berger, Ed., "Internet Stream 1648 Protocol Version 2 (ST2) Protocol Specification - Version 1649 ST2+", RFC 1819, DOI 10.17487/RFC1819, August 1995, 1650 . 1652 [RFC1887] Rekhter, Y., Ed. and T. Li, Ed., "An Architecture for IPv6 1653 Unicast Address Allocation", RFC 1887, 1654 DOI 10.17487/RFC1887, December 1995, 1655 . 1657 [RFC2001] Stevens, W., "TCP Slow Start, Congestion Avoidance, Fast 1658 Retransmit, and Fast Recovery Algorithms", RFC 2001, 1659 DOI 10.17487/RFC2001, January 1997, 1660 . 1662 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1663 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1664 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1665 September 1997, . 1667 [RFC2210] Wroclawski, J., "The Use of RSVP with IETF Integrated 1668 Services", RFC 2210, DOI 10.17487/RFC2210, September 1997, 1669 . 1671 [RFC2211] Wroclawski, J., "Specification of the Controlled-Load 1672 Network Element Service", RFC 2211, DOI 10.17487/RFC2211, 1673 September 1997, . 1675 [RFC2212] Shenker, S., Partridge, C., and R. Guerin, "Specification 1676 of Guaranteed Quality of Service", RFC 2212, 1677 DOI 10.17487/RFC2212, September 1997, 1678 . 1680 [RFC2215] Shenker, S. and J. Wroclawski, "General Characterization 1681 Parameters for Integrated Service Network Elements", 1682 RFC 2215, DOI 10.17487/RFC2215, September 1997, 1683 . 1685 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1686 and W. Weiss, "An Architecture for Differentiated 1687 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1688 . 1690 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1691 of Explicit Congestion Notification (ECN) to IP", 1692 RFC 3168, DOI 10.17487/RFC3168, September 2001, 1693 . 1695 [RFC3697] Rajahalme, J., Conta, A., Carpenter, B., and S. Deering, 1696 "IPv6 Flow Label Specification", RFC 3697, 1697 DOI 10.17487/RFC3697, March 2004, 1698 . 1700 [RFC4094] Manner, J. and X. Fu, "Analysis of Existing Quality-of- 1701 Service Signaling Protocols", RFC 4094, 1702 DOI 10.17487/RFC4094, May 2005, 1703 . 1705 [RFC4443] Conta, A., Deering, S., and M. Gupta, Ed., "Internet 1706 Control Message Protocol (ICMPv6) for the Internet 1707 Protocol Version 6 (IPv6) Specification", STD 89, 1708 RFC 4443, DOI 10.17487/RFC4443, March 2006, 1709 . 1711 [RFC4782] Floyd, S., Allman, M., Jain, A., and P. Sarolahti, "Quick- 1712 Start for TCP and IP", RFC 4782, DOI 10.17487/RFC4782, 1713 January 2007, . 1715 [RFC5082] Gill, V., Heasley, J., Meyer, D., Savola, P., Ed., and C. 1716 Pignataro, "The Generalized TTL Security Mechanism 1717 (GTSM)", RFC 5082, DOI 10.17487/RFC5082, October 2007, 1718 . 1720 [RFC5218] Thaler, D. and B. Aboba, "What Makes for a Successful 1721 Protocol?", RFC 5218, DOI 10.17487/RFC5218, July 2008, 1722 . 1724 [RFC5533] Nordmark, E. and M. Bagnulo, "Shim6: Level 3 Multihoming 1725 Shim Protocol for IPv6", RFC 5533, DOI 10.17487/RFC5533, 1726 June 2009, . 1728 [RFC5575] Marques, P., Sheth, N., Raszuk, R., Greene, B., Mauch, J., 1729 and D. McPherson, "Dissemination of Flow Specification 1730 Rules", RFC 5575, DOI 10.17487/RFC5575, August 2009, 1731 . 1733 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 1734 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 1735 . 1737 [RFC5971] Schulzrinne, H. and R. Hancock, "GIST: General Internet 1738 Signalling Transport", RFC 5971, DOI 10.17487/RFC5971, 1739 October 2010, . 1741 [RFC5973] Stiemerling, M., Tschofenig, H., Aoun, C., and E. Davies, 1742 "NAT/Firewall NSIS Signaling Layer Protocol (NSLP)", 1743 RFC 5973, DOI 10.17487/RFC5973, October 2010, 1744 . 1746 [RFC5974] Manner, J., Karagiannis, G., and A. McDonald, "NSIS 1747 Signaling Layer Protocol (NSLP) for Quality-of-Service 1748 Signaling", RFC 5974, DOI 10.17487/RFC5974, October 2010, 1749 . 1751 [RFC5981] Manner, J., Stiemerling, M., Tschofenig, H., and R. Bless, 1752 Ed., "Authorization for NSIS Signaling Layer Protocols", 1753 RFC 5981, DOI 10.17487/RFC5981, February 2011, 1754 . 1756 [RFC6294] Hu, Q. and B. Carpenter, "Survey of Proposed Use Cases for 1757 the IPv6 Flow Label", RFC 6294, DOI 10.17487/RFC6294, June 1758 2011, . 1760 [RFC6398] Le Faucheur, F., Ed., "IP Router Alert Considerations and 1761 Usage", BCP 168, RFC 6398, DOI 10.17487/RFC6398, October 1762 2011, . 1764 [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, 1765 "IPv6 Flow Label Specification", RFC 6437, 1766 DOI 10.17487/RFC6437, November 2011, 1767 . 1769 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 1770 for Equal Cost Multipath Routing and Link Aggregation in 1771 Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011, 1772 . 1774 [RFC6633] Gont, F., "Deprecation of ICMP Source Quench Messages", 1775 RFC 6633, DOI 10.17487/RFC6633, May 2012, 1776 . 1778 [RFC6928] Chu, J., Dukkipati, N., Cheng, Y., and M. Mathis, 1779 "Increasing TCP's Initial Window", RFC 6928, 1780 DOI 10.17487/RFC6928, April 2013, 1781 . 1783 [RFC7305] Lear, E., Ed., "Report from the IAB Workshop on Internet 1784 Technology Adoption and Transition (ITAT)", RFC 7305, 1785 DOI 10.17487/RFC7305, July 2014, 1786 . 1788 [RFC7418] Dawkins, S., Ed., "An IRTF Primer for IETF Participants", 1789 RFC 7418, DOI 10.17487/RFC7418, December 2014, 1790 . 1792 [RFC8085] Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage 1793 Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085, 1794 March 2017, . 1796 [RFC8170] Thaler, D., Ed., "Planning for Protocol Adoption and 1797 Subsequent Transitions", RFC 8170, DOI 10.17487/RFC8170, 1798 May 2017, . 1800 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 1801 "Deterministic Networking Architecture", RFC 8655, 1802 DOI 10.17487/RFC8655, October 2019, 1803 . 1805 [RFC8793] Wissingh, B., Wood, C., Afanasyev, A., Zhang, L., Oran, 1806 D., and C. Tschudin, "Information-Centric Networking 1807 (ICN): Content-Centric Networking (CCNx) and Named Data 1808 Networking (NDN) Terminology", RFC 8793, 1809 DOI 10.17487/RFC8793, June 2020, 1810 . 1812 [SAAG-105-Min] 1813 "Security Area Open Meeting - IETF-105 Minutes", July 1814 2019, . 1817 [SAF07] Sarolahti, P., Allman, M., and S. Floyd, "Determining an 1818 appropriate sending rate over an underutilized network 1819 path", Computer Networking Volume 51, Number 7, May 2007. 1821 [Sch11] Scharf, M., "Fast Startup Internet Congestion Control for 1822 Broadband Interactive Applications", Ph.D. Thesis, 1823 University of Stuttgart, April 2011. 1825 [Shim6-35] Meyer, D., Huston, G., Schiller, J., and V. Gill, "IAB 1826 IPv6 Multihoming Panel at NANOG 35", NANOG North American 1827 Network Operator Group, October 2005, 1828 . 1830 [TRIGTRAN-55] 1831 "Triggers for Transport BOF at IETF 55", July 2003, 1832 . 1834 [TRIGTRAN-56] 1835 "Triggers for Transport BOF at IETF 56", November 2003, 1836 . 1838 Author's Address 1840 Spencer Dawkins (editor) 1841 Tencent America 1843 Email: spencerdawkins.ietf@gmail.com