idnits 2.17.1 draft-irtf-panrg-what-not-to-do-16.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (29 December 2020) is 1215 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-04) exists of draft-iab-use-it-or-lose-it-00 == Outdated reference: A later version (-12) exists of draft-irtf-panrg-questions-08 -- Obsolete informational reference (is this intentional?): RFC 793 (Obsoleted by RFC 9293) -- Obsolete informational reference (is this intentional?): RFC 1190 (Obsoleted by RFC 1819) -- Obsolete informational reference (is this intentional?): RFC 2001 (Obsoleted by RFC 2581) -- Obsolete informational reference (is this intentional?): RFC 3697 (Obsoleted by RFC 6437) -- Obsolete informational reference (is this intentional?): RFC 5575 (Obsoleted by RFC 8955) Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 6 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 PANRG S. Dawkins, Ed. 3 Internet-Draft Tencent America 4 Intended status: Informational 29 December 2020 5 Expires: 2 July 2021 7 Path Aware Networking: Obstacles to Deployment (A Bestiary of Roads Not 8 Taken) 9 draft-irtf-panrg-what-not-to-do-16 11 Abstract 13 At the first meeting of the Path Aware Networking Research Group, the 14 research group agreed to catalog and analyze past efforts to develop 15 and deploy Path Aware techniques, most of which were unsuccessful or 16 at most partially successful, in order to extract insights and 17 lessons for path-aware networking researchers. 19 This document contains that catalog and analysis. 21 Status of This Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at https://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on 2 July 2021. 38 Copyright Notice 40 Copyright (c) 2020 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 45 license-info) in effect on the date of publication of this document. 46 Please review these documents carefully, as they describe your rights 47 and restrictions with respect to this document. 49 Table of Contents 51 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 52 1.1. What Does "Path Awareness" Mean in this Document? . . . . 3 53 1.2. Note to RFC Editor . . . . . . . . . . . . . . . . . . . 4 54 2. A Perspective On This Document . . . . . . . . . . . . . . . 4 55 2.1. Notes for the Reader . . . . . . . . . . . . . . . . . . 5 56 2.2. A Note About Path-Aware Techniques Included In This 57 Document . . . . . . . . . . . . . . . . . . . . . . . . 5 58 2.3. Venue for Discussion of this Document . . . . . . . . . . 5 59 2.4. Architectural Guidance . . . . . . . . . . . . . . . . . 5 60 2.5. Terminology Used in this Document . . . . . . . . . . . . 6 61 2.6. Methodology for Contributions . . . . . . . . . . . . . . 6 62 3. Applying the Lessons We've Learned . . . . . . . . . . . . . 7 63 4. Summary of Lessons Learned . . . . . . . . . . . . . . . . . 10 64 4.1. Justifying Deployment . . . . . . . . . . . . . . . . . . 10 65 4.2. Providing Benefits for Early Adopters . . . . . . . . . . 10 66 4.3. Providing Benefits During Partial Deployment . . . . . . 11 67 4.4. Outperforming End-to-end Protocol Mechanisms . . . . . . 11 68 4.5. Paying for Path Aware Techniques . . . . . . . . . . . . 11 69 4.6. Impact on Operational Practices . . . . . . . . . . . . . 11 70 4.7. Per-connection State . . . . . . . . . . . . . . . . . . 11 71 4.8. Keeping Traffic on Fast-paths . . . . . . . . . . . . . . 12 72 4.9. Endpoints Trusting Intermediate Nodes . . . . . . . . . . 12 73 4.10. Intermediate Nodes Trusting Endpoints . . . . . . . . . . 12 74 4.11. Reacting to Distant Signals . . . . . . . . . . . . . . . 12 75 4.12. Support in Endpoint Protocol Stacks . . . . . . . . . . . 13 76 5. Future Work . . . . . . . . . . . . . . . . . . . . . . . . . 13 77 6. Contributions . . . . . . . . . . . . . . . . . . . . . . . . 14 78 6.1. Stream Transport (ST, ST2, ST2+) . . . . . . . . . . . . 14 79 6.1.1. Reasons for Non-deployment . . . . . . . . . . . . . 15 80 6.1.2. Lessons Learned. . . . . . . . . . . . . . . . . . . 15 81 6.2. Integrated Services (IntServ) . . . . . . . . . . . . . . 16 82 6.2.1. Reasons for Non-deployment . . . . . . . . . . . . . 16 83 6.2.2. Lessons Learned. . . . . . . . . . . . . . . . . . . 17 84 6.3. Quick-Start TCP . . . . . . . . . . . . . . . . . . . . . 17 85 6.3.1. Reasons for Non-deployment . . . . . . . . . . . . . 18 86 6.3.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 19 87 6.4. ICMP Source Quench . . . . . . . . . . . . . . . . . . . 20 88 6.4.1. Reasons for Non-deployment . . . . . . . . . . . . . 20 89 6.4.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 20 90 6.5. Triggers for Transport (TRIGTRAN) . . . . . . . . . . . . 21 91 6.5.1. Reasons for Non-deployment . . . . . . . . . . . . . 22 92 6.5.2. Lessons Learned. . . . . . . . . . . . . . . . . . . 23 93 6.6. Shim6 . . . . . . . . . . . . . . . . . . . . . . . . . . 23 94 6.6.1. Reasons for Non-deployment . . . . . . . . . . . . . 24 95 6.6.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 25 96 6.6.3. Addendum on MultiPath TCP . . . . . . . . . . . . . . 25 98 6.7. Next Steps in Signaling (NSIS) . . . . . . . . . . . . . 26 99 6.7.1. Reasons for Non-deployment . . . . . . . . . . . . . 26 100 6.7.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 28 101 6.8. IPv6 Flow Label . . . . . . . . . . . . . . . . . . . . . 28 102 6.8.1. Reasons for Non-deployment . . . . . . . . . . . . . 29 103 6.8.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 30 104 7. Security Considerations . . . . . . . . . . . . . . . . . . . 31 105 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31 106 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 31 107 10. Informative References . . . . . . . . . . . . . . . . . . . 32 108 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 39 110 1. Introduction 112 This document describes the lessons that IETF participants have 113 learned (and learned the hard way) about Path Aware Networking over a 114 period of several decades, and provides an analysis of reasons why 115 various Path Aware Networking techniques have seen limited or no 116 deployment. 118 1.1. What Does "Path Awareness" Mean in this Document? 120 The current definition of "Path Awareness", used by the Path Aware 121 Networking Research Group, appears in Section 1.1 ("Definition") in 122 [I-D.irtf-panrg-questions]. That definition is included here as a 123 convenience to the reader. 125 | For purposes of this document, "path aware networking" describes 126 | endpoint discovery of the properties of paths they use for 127 | communication, and endpoint reaction to these properties that 128 | affects routing and/or transmission; note that this can and 129 | already does happen to some extent in the current Internet 130 | architecture. Expanding on this definition, a "path aware 131 | internetwork" is one in which endpoint discovery of path 132 | properties and endpoint selection of paths used by traffic 133 | exchanged by the endpoint are explicitly supported, regardless of 134 | the specific design of the protocol features which enable this 135 | discovery and selection. 137 Because this document reflects work performed over several decades, 138 some technologies described in Section 6 may not reflect the current 139 definition, but these technologies were considered "path aware" by 140 their contributors, so these contributions are included in this 141 retrospective document. 143 1.2. Note to RFC Editor 145 If the "Definition" in Section 1.1 ("Definition") of 146 [I-D.irtf-panrg-questions] changes, the text in Section 1.1 of this 147 document should be should be changed as well. 149 Whether that happens or not, the RFC Editor is requested to remove 150 this section. 152 2. A Perspective On This Document 154 At the first meeting of the Path Aware Networking Research Group 155 [PANRG], at IETF 99 [PANRG-99], Olivier Bonaventure led a discussion 156 of "A Decade of Path Awareness" [PATH-Decade], on attempts, which 157 were mostly unsuccessful for a variety of reasons, to exploit Path 158 Aware techniques and achieve a variety of goals over the past decade. 159 At the end of that discussion, two things were abundantly clear. 161 * The Internet community has accumulated considerable experience 162 with many Path Aware techniques over a long period of time, and 164 * Although some path aware techniques have been deployed (for 165 example, Differentiated Services, or DiffServ [RFC2475]), most of 166 these techniques haven't seen widespread adoption and deplyment. 167 Even "successful" techniques like DiffServ can face obstacles that 168 prevents wider usage. The reasons for non-adoption and limited 169 adoption and deployment are many, and are worthy of study. 171 The meta-lessons from that experience were 173 * Path aware networking has been more Research than Engineering, so 174 establishing an IRTF Research Group for Path Aware Networking is 175 the right thing to do [RFC7418]. 177 * Analyzing a catalog of past experience to learn the reasons for 178 non-adoption would be a great first step for the Research Group. 180 Allison Mankin, as IRTF Chair, officially chartered the Path Aware 181 Networking Research Group in July, 2018. 183 This document contains the analysis performed by that research group 184 (Section 4), based on that catalog (Section 6). 186 This document represents the consensus of the Path Aware Networking 187 Research Group. 189 2.1. Notes for the Reader 191 This Informational document discusses Path Aware protocol mechanisms 192 considered, and in some cases standardized, by the Internet 193 Engineering Task Force (IETF), and considers Lessons Learned from 194 those mechanisms. The intention is to inform the work of protocol 195 designers, whether in the IRTF, the IETF, or elsewhere in the 196 Internet ecosystem. 198 As an Informational document published in the IRTF stream, this 199 document has no authority beyond the quality of the analysis it 200 contains. 202 2.2. A Note About Path-Aware Techniques Included In This Document 204 This document does not catalog every proposed path aware networking 205 technique that was not adopted and deployed. Instead, we limited our 206 focus to technologies that passed through the IETF community, and 207 still identified enough techniques to provide background for the 208 lessons included in Section 4 to inform researchers and protocol 209 engineers in their work. 211 No shame is intended for the techniques included in this document. 212 As shown in Section 4, the quality of specific techniques had little 213 to do with whether they were deployed or not. Based on the 214 techniques cataloged in this document, it is likely that when these 215 techniques were put forward, the proponents were trying to engineer 216 something that could not be engineered without first carrying out 217 research. Actual shame would be failing to learn from experience, 218 and failing to share that experience with other networking 219 researchers and engineers. 221 2.3. Venue for Discussion of this Document 223 (RFC Editor: please remove this section before publication) 225 Discussion of specific contributed experiences and this document in 226 general should take place on the PANRG mailing list. 228 2.4. Architectural Guidance 230 As background for understanding the Lessons Learned contained in this 231 document, the reader is encouraged to become familiar with the 232 Internet Architecture Board's documents on "What Makes for a 233 Successful Protocol?" [RFC5218] and "Planning for Protocol Adoption 234 and Subsequent Transitions" [RFC8170]. 236 Although these two documents do not specifically target path-aware 237 networking protocols, they are helpful resources for readers seeking 238 to improve their understanding of considerations for successful 239 adoption and deployment of any protocol. For example, the Basic 240 Success Factors described in Setion 2.1 of [RFC5218] are helpful for 241 readers of this document. 243 Because there is an economic aspect to decisions about deployment, 244 the IAB Workshop on Internet Technology Adoption and Transition 245 [ITAT] report [RFC7305] also provides food for thought. 247 Several of the Lessons Learned in Section 4 reflect considerations 248 described in [RFC5218], [RFC7305], and [RFC8170]. 250 2.5. Terminology Used in this Document 252 The terms Node and Element in this document have the meaning defined 253 in [PathProp]. 255 2.6. Methodology for Contributions 257 This document grew out of contributions by various IETF participants 258 with experience with one or more Path Aware Networking techniques. 260 There are many things that could be said about the Path Aware 261 networking techniques that have been developed. For the purposes of 262 this document, contributors were requested to provide 264 * the name of a technique, including an abbreviation if one was used 266 * if available, a long-term pointer to the best reference describing 267 the technique 269 * a short description of the problem the technique was intended to 270 solve 272 * a short description of the reasons why the technique wasn't 273 adopted 275 * a short statement of the lessons that researchers can learn from 276 our experience with this technique. 278 3. Applying the Lessons We've Learned 280 The initial scope for this document was roughly "what mistakes have 281 we made in the decade prior to [PANRG-99], that we shouldn't make 282 again". Some of the contributions in Section 6 predate the initial 283 scope. The earliest Path-Aware Networking technique referred to in 284 Section 6 is Section 6.1, published in the late 1970s. Given that 285 the networking ecosystem has evolved continuously, it seems 286 reasonable to consider how to apply these lessons. 288 The PANRG Research Group reviewed the Lessons Learned (Section 4) 289 contained in the May 23, 2019 version of this document at IETF 105 290 [PANRG-105-Min], and carried out additional discussion at IETF 106 291 [PANRG-106-Min]. Table 1 provides the "sense of the room" about each 292 lesson after those discussions. The intention is to capture whether 293 a specific lesson seems to be 295 * "Invariant" - well-understood and is likely to be applicable for 296 any proposed Path Aware Networking solution. 298 * "Variable" - has impeded deployment in the past, but might not be 299 applicable in a specific technique. Engineering analysis to 300 understand whether the lesson is applicable is prudent. 302 * "Not Now" - this characteristic tends to turn up a minefield full 303 of dragons, and prudent network engineers will wish to avoid 304 gambling on a technique that relies on this, until something 305 significant changes 307 +-----------------------------------------------------+-----------+ 308 | Lesson | Category | 309 +=====================================================+===========+ 310 | Justifying Deployment (Section 4.1) | Invariant | 311 +-----------------------------------------------------+-----------+ 312 | Providing Benefits for Early Adopters (Section 4.2) | Invariant | 313 +-----------------------------------------------------+-----------+ 314 | Providing Benefits during Partial Deployment | Invariant | 315 | (Section 4.3) | | 316 +-----------------------------------------------------+-----------+ 317 | Outperforming End-to-end Protocol Mechanisms | Variable | 318 | (Section 4.4) | | 319 +-----------------------------------------------------+-----------+ 320 | Paying for Path Aware Techniques (Section 4.5) | Invariant | 321 +-----------------------------------------------------+-----------+ 322 | Impact on Operational Practices (Section 4.6) | Invariant | 323 +-----------------------------------------------------+-----------+ 324 | Per-connection State (Section 4.7) | Variable | 325 +-----------------------------------------------------+-----------+ 326 | Keeping Traffic on Fast-paths (Section 4.8) | Variable | 327 +-----------------------------------------------------+-----------+ 328 | Endpoints Trusting Intermediate Nodes (Section 4.9) | Not Now | 329 +-----------------------------------------------------+-----------+ 330 | Intermediate Nodes Trusting Endpoints | Not Now | 331 | (Section 4.10) | | 332 +-----------------------------------------------------+-----------+ 333 | Reacting to Distant Signals (Section 4.11) | Variable | 334 +-----------------------------------------------------+-----------+ 335 | Support in Endpoint Protocol Stacks (Section 4.12) | Variable | 336 +-----------------------------------------------------+-----------+ 338 Table 1 340 "Justifying Deployment", "Providing Benefits for Early Adopters", 341 "Paying for Path Aware Techniques", and "Impact on Operational 342 Practice" were considered to be invariant - the sense of the room was 343 that these would always be considerations for any proposed Path Aware 344 Technique. 346 "Providing Benefits During Partial Deployment" was added after IETF 347 105, during research group last call, and is also considered to be 348 invariant. 350 For "Outperforming End-to-end Protocol Mechanisms", there is a trade- 351 off between improved performance from Path Aware Techniques and 352 additional complexity required by some Path Aware Techniques. 354 * For example, if you can obtain the same understanding of path 355 characteristics from measurements obtained over a few more round 356 trips, endpoint implementers are unlikely to be eager to add 357 complexity, and many attributes can be measured from an endpoint, 358 without assistance from intermediate nodes. 360 For "Per-connection State", the key questions discussed in the 361 research group were "how much state" and "where state is maintained". 363 * IntServ (Section 6.2) required state at every intermediate node 364 for every connection between two endpoints. As the Internet 365 ecosystem has evolved, carrying many connections in a tunnel that 366 appears to intermediate nodes as a single connection has become 367 more common, so that additional end-to-end connections don't add 368 additional state to intermediate nodes between tunnel endpoints. 369 If these tunnels are encrypted, intermediate nodes between tunnel 370 endpoints can't distinguish between connections, even if that were 371 desirable. 373 For "Keeping Traffic on Fast-paths", we noted that this was true for 374 many platforms, but not for all. 376 * For backbone routers, this is likely an invariant, but for 377 platforms that rely more on general-purpose computers to make 378 forwarding decisions, this may not be a fatal flaw for Path Aware 379 Networking techniques. 381 For "Endpoints Trusting Intermediate Nodes" and "Intermediate Nodes 382 Trusting Endpoints", these lessons point to the broader need to 383 revisit the Internet Threat Model. 385 * We noted with relief that discussions about this were already 386 underway in the IETF community at IETF 105 (see the Security Area 387 Open Meeting minutes [SAAG-105-Min] for discussion of 388 [I-D.arkko-arch-internet-threat-model] and [I-D.farrell-etm]), and 389 the Internet Architecture Board has created a mailing list for 390 continued discussions ([model-t]), but we recognize that there are 391 Path Aware Networking aspects of this effort, requiring research. 393 For "Reacting to Distant Signals", we noted that not all attributes 394 are equal. 396 * If an attribute is stable over an extended period of time, is 397 difficult to observe via end-to-end mechanisms, and is valuable, 398 Path Aware Techniques that rely on that attribute to provide a 399 significant benefit become more attractive. 401 * Analysis to help identify attributes that are useful enough to 402 justify deployment of Path Aware techniques that make use of those 403 attributes would be helpful. 405 For "Support in Endpoint Protocol Stacks", we noted that Path Aware 406 applications must be able to identify and communicate requirements 407 about path characteristics. 409 * The de-facto sockets API has no way of signaling application 410 expectations for the network path to the protocol stack. 412 4. Summary of Lessons Learned 414 This section summarizes the Lessons Learned from the contributed 415 subsections in Section 6. 417 Each Lesson Learned is tagged with one or more contributions that 418 encountered this obstacle as a significant impediment to deployment. 419 Other contributed techniques may have also encountered this obstacle, 420 but this obstacle may not have been the biggest impediment to 421 deployment for those techniques. 423 It is useful to notice that sometimes an obstacle might impede 424 deployment, while at other times, the same obstacle might prevent 425 adoption and deployment entirely. The research group discussed 426 distinguishing between obstacles that impede and obstacles that 427 prevent, but it appears that the boundary between "impede" and 428 "prevent" can shift over time - some of the Lessons Learned are based 429 on both Path Aware techniques that were not deployed, and Path Aware 430 techniques that were deployed, but were not deployed widely or 431 quickly. See Section 6.6 and Section 6.6.3 as one example of this 432 shifting boundary. 434 4.1. Justifying Deployment 436 The benefit of Path Awareness must be great enough to justify making 437 changes in an operational network. The colloquial U.S. American 438 English expression, "If it ain't broke, don't fix it" is a "best 439 current practice" on today's Internet. (See Section 6.3, 440 Section 6.5, and Section 6.4, in addition to [RFC5218]). 442 4.2. Providing Benefits for Early Adopters 444 Providing benefits for early adopters can be key - if everyone must 445 deploy a technique in order for the technique to provide benefits, or 446 even to work at all, the technique is unlikely to be adopted widely 447 or quickly. (See Section 6.2 and Section 6.3, in addition to 448 [RFC5218]). 450 4.3. Providing Benefits During Partial Deployment 452 Some proposals require that all path elements along the full length 453 of the path must be upgraded to support a new technique, before any 454 benefits can be seen. This is likely to require coordination between 455 operators who control a subset of path elements, and between 456 operators and end users if endpoint upgrades are required. If a 457 technique provides benefits when only a part of the path has been 458 upgraded, this is likely to encourage adoption and deployment. (See 459 Section 6.2 and Section 6.3, in addition to [RFC5218]). 461 4.4. Outperforming End-to-end Protocol Mechanisms 463 Adaptive end-to-end protocol mechanisms may respond to feedback 464 quickly enough that the additional realizable benefit from a new Path 465 Aware mechanism that tries to manipulate nodes along a path, or 466 observe the attributes of nodes along a path, may be much smaller 467 than anticipated (Section 6.3 and Section 6.5). 469 4.5. Paying for Path Aware Techniques 471 "Follow the money." If operators can't charge for a Path Aware 472 technique to recover the costs of deploying it, the benefits to the 473 operator must be really significant. Corollary: If operators charge 474 for a Path Aware technique, the benefits to users of that Path Aware 475 technique must be significant enough to justify the cost. (See 476 Section 6.1, Section 6.2, and Section 6.5). 478 4.6. Impact on Operational Practices 480 Impact of a Path Aware technique requiring changes to operational 481 practices can affect how quickly or widely a promising technique is 482 deployed. The impacts of these changes may make deployment more 483 likely, but often discourage deployment. (See Section 6.6, including 484 Section 6.6.3). 486 4.7. Per-connection State 488 Per-connection state in intermediate nodes has been an impediment to 489 adoption and deployment in the past, because of added cost and 490 complexity. Often, similar benefits can be achieved with much less 491 finely-grained state. This is especially true as we move from the 492 edge of the network, further into the routing core (See Section 6.1 493 and Section 6.2). 495 4.8. Keeping Traffic on Fast-paths 497 Many modern platforms, especially high-end routers, have been 498 designed with hardware that can make simple per-packet forwarding 499 decisions ("fast-paths"), but have not been designed to make heavy 500 use of in-band mechanisms such as IPv4 and IPv6 Router Alert Options 501 (RAO) that require more processing to make forwarding decisions. 502 Packets carrying in-band mechanisms are diverted to other processors 503 in the router with much lower packet processing rates. Operators can 504 be reluctant to deploy techniques that rely heavily on in-band 505 mechanisms because they may significantly reduce packet throughput. 506 (See Section 6.7). 508 4.9. Endpoints Trusting Intermediate Nodes 510 If intermediate nodes along the path can't be trusted, it's unlikely 511 that endpoints will rely on signals from intermediate nodes to drive 512 changes to endpoint behaviors. (See Section 6.5, Section 6.4). We 513 note that "trust" is not binary - one, low, level of trust applies 514 when a node issuing a message can confirm that it has visibility of 515 the packets on the path it is seeking to control [RFC8085] (e.g., an 516 ICMP message included a quoted packet from the source). A higher 517 level of trust can arise when an endpoint has established a short 518 term, or even long term, trust relationship with network nodes. 520 4.10. Intermediate Nodes Trusting Endpoints 522 If the endpoints do not have any trust relationship with the 523 intermediate nodes along a path, operators have been reluctant to 524 deploy techniques that rely on endpoints sending unauthenticated 525 control signals to routers. (See Section 6.2 and Section 6.7. We 526 also note this still remains a factor hindering deployment of 527 DiffServ). 529 4.11. Reacting to Distant Signals 531 Because the Internet is a distributed system, if the distance that 532 information from distant path elements travels to a Path Aware host 533 is sufficiently large, the information may no longer accurately 534 represent the state and situation at the distant host or elements 535 along the path when it is received locally. In this case, the 536 benefit that a Path Aware technique provides will be inconsistent, 537 and may not always be beneficial. (See Section 6.3). 539 4.12. Support in Endpoint Protocol Stacks 541 Just because a protocol stack provides a new feature/signal does not 542 mean that applications will use the feature/signal. Protocol stacks 543 may not know how to effectively utilize Path-Aware techniques, 544 because the protocol stack may require information from applications 545 to permit the technique to work effectively, but applications may not 546 a-priori know that information. Even if the application does know 547 that information, the de-facto sockets API has no way of signaling 548 application expectations for the network path to the protocol stack. 549 In order for applications to provide these expectations to protocol 550 stacks, we need an API that signals more than the packets to be sent. 551 (See Section 6.1 and Section 6.2). 553 5. Future Work 555 By its nature, this document has been retrospective. In addition to 556 considering how the Lessons Learned to date apply to current and 557 future Path Aware networking proposals, it's also worth considering 558 whether there is deeper investigation left to do. 560 * We note that this work was based on contributions from experts on 561 various Path Aware networking techniques, and all of the 562 contributed techniques involved unicast protocols. We didn't 563 consider how these lessons might apply to multicast, and, given 564 anecdotal reports at the IETF 109 MOPS working group meeting of IP 565 multicast offerings within data centers at one or more cloud 566 providers ([MOPS-109-Min]), it might be useful to think about path 567 awareness in multicast, before we have a history of unsuccessful 568 deployments to document. 570 * The question of whether a mechanism supports admission control, 571 based on either endpoints or applications, is associated with Path 572 Awareness. One of the motivations of IntServ and a number of 573 other architectures (e.g. Deterministic Networking, [RFC8655]) is 574 the ability to "say no" to an application based on resource 575 availability on a path, before the application tries to inject 576 traffic onto that path and discovers the path does not have the 577 capacity to sustain enough utility to meet the application's 578 minimum needs. The question of whether admission control is 579 needed comes up repeatedly, but we have learned a few useful 580 lessons that, while covered implicitly in some of the lessons 581 learned of the document, might be explained explicitly: 583 - We have gained a lot of experience with application-based 584 adaptation since the days where applications just injected 585 traffic in-elastically into the network. Such adaptations seem 586 to work well enough that admission control is of less value to 587 these applications 589 - There are end-to-end measurement techniques that can steer 590 traffic at the application layer (Content Distribution 591 Networks, multi-CDNs like Conviva [Conviva], etc.) 593 - We noted in Section 4.12 that applications often don't know how 594 to utilize Path Aware techniques. This includes not knowing 595 enough about their admission control threshold to be able to 596 ask accurately for the resources they need, whether this is 597 because the application itself doesn't know, or because the 598 application has no way to signal its expectations to the 599 underlying protocol stack. To date, attempts to help them 600 haven't gotten anywhere (e.g. the multiple-TSPEC additions to 601 RSVP to attempt to mirror codec selection by applications 602 [I-D.ietf-tsvwg-intserv-multiple-tspec] expired in 2013). 604 * We note that this work took the then-current IP network 605 architecture as given, at least at the time each technique was 606 proposed. It might be useful to consider aspects of the now- 607 current IP network architecture that ease, or impede, Path Aware 608 networking techniques. For example, there is limited ability in 609 IP to constrain bidirectional paths to be symmetric, and 610 information-centric networking protocols such as Named Data 611 Networking (NDN) and Content-Centric Networking (CCNx) ([RFC8793]) 612 must force bidirectional path symmetry using protocol-specific 613 mechanisms. 615 6. Contributions 617 Contributions on these Path Aware networking techniques were analyzed 618 to arrive at the Lessons Learned captured in Section 4. 620 Our expectation is that most readers will not need to read through 621 this section carefully, but we wanted to record these hard-fought 622 lessons as a service to others who may revisit this document, so 623 they'll have the details close at hand. 625 6.1. Stream Transport (ST, ST2, ST2+) 627 The suggested references for Stream Transport are: 629 * ST - A Proposed Internet Stream Protocol [IEN-119] 630 * Experimental Internet Stream Protocol, Version 2 (ST-II) [RFC1190] 632 * Internet Stream Protocol Version 2 (ST2) Protocol Specification - 633 Version ST2+ [RFC1819] 635 The first version of Stream Transport, ST [IEN-119], was published in 636 the late 1970's and was implemented and deployed on the ARPANET at 637 small scale. It was used throughout the 1980's for experimental 638 transmission of voice, video, and distributed simulation. 640 The second version of the ST specification (ST2) [RFC1190] [RFC1819] 641 was an experimental connection-oriented internetworking protocol that 642 operated at the same layer as connectionless IP. ST2 packets could 643 be distinguished by their IP header version numbers (IP, at that 644 time, used version number 4, while ST2 used version number 5). 646 ST2 used a control plane layered over IP to select routes and reserve 647 capacity for real-time streams across a network path, based on a flow 648 specification communicated by a separate protocol. The flow 649 specification could be associated with QoS state in routers, 650 producing an experimental resource reservation protocol. This 651 allowed ST2 routers along a path to offer end-to-end guarantees, 652 primarily to satisfy the QoS requirements for realtime services over 653 the Internet. 655 6.1.1. Reasons for Non-deployment 657 Although implemented in a range of equipment, ST2 was not widely used 658 after completion of the experiments. It did not offer the 659 scalability and fate-sharing properties that have come to be desired 660 by the Internet community. 662 The ST2 protocol is no longer in use. 664 6.1.2. Lessons Learned. 666 As time passed, the trade-off between router processing and link 667 capacity changed. Links became faster and the cost of router 668 processing became comparatively more expensive. 670 The ST2 control protocol used "hard state" - once a route was 671 established, and resources were reserved, routes and resources 672 existing until they were explicitly released via signaling. A soft- 673 state approach was thought superior to this hard-state approach, and 674 led to development of the IntServ model described in Section 6.2. 676 6.2. Integrated Services (IntServ) 678 The suggested references for IntServ are: 680 * RFC 1633 Integrated Services in the Internet Architecture: an 681 Overview [RFC1633] 683 * RFC 2211 Specification of the Controlled-Load Network Element 684 Service [RFC2211] 686 * RFC 2212 Specification of Guaranteed Quality of Service [RFC2212] 688 * RFC 2215 General Characterization Parameters for Integrated 689 Service Network Elements [RFC2215] 691 * RFC 2205 Resource ReSerVation Protocol (RSVP) [RFC2205] 693 In 1994, when the IntServ architecture document [RFC1633] was 694 published, real-time traffic was first appearing on the Internet. At 695 that time, bandwidth was still a scarce commodity. Internet Service 696 Providers built networks over DS3 (45 Mbps) infrastructure, and sub- 697 rate (< 1 Mpbs) access was common. Therefore, the IETF anticipated a 698 need for a fine-grained QoS mechanism. 700 In the IntServ architecture, some applications can require service 701 guarantees. Therefore, those applications use the Resource 702 Reservation Protocol (RSVP) [RFC2205] to signal QoS reservations 703 across network paths. Every router in the network that participates 704 in IntServ maintains per-flow soft-state to a) perform call admission 705 control and b) deliver guaranteed service. 707 Applications use Flow Specification (Flow Specs) [RFC2210] to 708 describe the traffic that they emit. RSVP reserves capacity for 709 traffic on a per Flow Spec basis. 711 6.2.1. Reasons for Non-deployment 713 Although IntServ has been used in enterprise and government networks, 714 IntServ was never widely deployed on the Internet because of its 715 cost. The following factors contributed to operational cost: 717 * IntServ must be deployed on every router that is on a path where 718 IntServ is to be used. Although it is possible to include a 719 router that does not participate in IntServ along the path being 720 controlled, if that router is likely to become a bottleneck, 721 IntServ cannot be used to avoid that bottleneck along the path 723 * IntServ maintained per flow state 724 As IntServ was being discussed, the following occurred: 726 * For many expected uses, it became more cost effective to solve the 727 QoS problem by adding bandwidth. Between 1994 and 2000, Internet 728 Service Providers upgraded their infrastructures from DS3 (45 729 Mbps) to OC-48 (2.4 Gbps). This meant that even if an endpoint 730 was using IntServ in an IntServ-enabled network, its requests 731 would rarely, if ever, be denied, so endpoints and Internet 732 Service Providers had little reason to enable IntServ. 734 * DiffServ [RFC2475] offered a more cost-effective, albeit less 735 fine-grained, solution to the QoS problem. 737 6.2.2. Lessons Learned. 739 The following lessons were learned: 741 * Any mechanism that requires every participating onpath router to 742 maintain per-flow state is not likely to succeed, unless the 743 additional cost for offering the feature can be recovered from the 744 user. 746 * Any mechanism that requires an operator to upgrade all of its 747 routers is not likely to succeed, unless the additional cost for 748 offering the feature can be recovered from the user. 750 In environments where IntServ has been deployed, trust relationships 751 with endpoints are very different from trust relationships on the 752 Internet itself, and there are often clearly-defined hierarchies in 753 Service Level Agreements (SLAs), and well-defined transport flows 754 operating with pre-determined capacity and latency requirements over 755 paths where capacity or other attributes are constrained. 757 IntServ was never widely deployed to manage capacity across the 758 Internet. However, the technique that it produced was deployed for 759 reasons other than bandwidth management. RSVP is widely deployed as 760 an MPLS signaling mechanism. BGP reuses the RSVP concept of Filter 761 Specs to distribute firewall filters, although they are called Flow 762 Spec Component Types in BGP [RFC5575]. 764 6.3. Quick-Start TCP 766 The suggested references for Quick-Start TCP are: 768 * Quick-Start for TCP and IP [RFC4782] 770 * Determining an appropriate initial sending rate over an 771 underutilized network path [SAF07] 773 * Fast Startup Internet Congestion Control for Broadband Interactive 774 Applications [Sch11] 776 * Using Quick-Start to enhance TCP-friendly rate control performance 777 in bidirectional satellite networks [QS-SAT] 779 Quick-Start [RFC4782] is an Experimental TCP extension that leverages 780 support from the routers on the path to determine an allowed initial 781 sending rate for a path through the Internet, either at the start of 782 data transfers or after idle periods. Without information about the 783 path, a sender cannot easily determine an appropriate initial sending 784 rate. The default TCP congestion control therefore uses the safe but 785 time-consuming slow-start algorithm [RFC5681]. With Quick-Start, 786 connections are allowed to use higher initial sending rates if there 787 is significant unused bandwidth along the path, and if the sender and 788 all of the routers along the path approve the request. 790 By examining the Time To Live (TTL) field in Quick-Start packets, a 791 sender can determine if routers on the path have approved the Quick- 792 Start request. However, this method is unable to take into account 793 the routers hidden by tunnels or other network nodes invisible at the 794 IP layer. 796 The protocol also includes a nonce that provides protection against 797 cheating routers and receivers. If the Quick-Start request is 798 explicitly approved by all routers along the path, the TCP host can 799 send at up to the approved rate; otherwise TCP would use the default 800 congestion control. Quick-Start requires modifications in the 801 involved end-systems as well in routers. Due to the resulting 802 deployment challenges, Quick-Start was only proposed in [RFC4782] for 803 controlled environments. 805 The Quick-Start mechanism is a lightweight, coarse-grained, in-band, 806 network-assisted fast startup mechanism. The benefits are studied by 807 simulation in a research paper [SAF07] that complements the protocol 808 specification. The study confirms that Quick-Start can significantly 809 speed up mid-sized data transfers. That paper also presents router 810 algorithms that do not require keeping per-flow state. Later studies 811 [Sch11] comprehensively analyzes Quick-Start with a full Linux 812 implementation and with a router fast path prototype using a network 813 processor. In both cases, Quick-Start could be implemented with 814 limited additional complexity. 816 6.3.1. Reasons for Non-deployment 818 However, experiments with Quick-Start in [Sch11] revealed several 819 challenges: 821 * Having information from the routers along the path can reduce the 822 risk of congestion, but cannot avoid it entirely. Determining 823 whether there is unused capacity is not trivial in actual router 824 and host implementations. Data about available capacity visible 825 at the IP layer may be imprecise, and due to the propagation 826 delay, information can already be outdated when it reaches a 827 sender. There is a trade-off between the speedup of data 828 transfers and the risk of congestion even with Quick-Start. This 829 could be mitigated by only allowing Quick-Start to access a 830 proportion of the unused capacity along a path. 832 * For scalable router fast path implementation, it is important to 833 enable parallel processing of packets, as this is a widely used 834 method e.g. in network processors. One challenge is 835 synchronization of information between packets that are processed 836 in parallel, which should be avoided as much as possible. 838 * Only some types of application traffic can benefit from Quick- 839 Start. Capacity needs to be requested and discovered. The 840 discovered capacity needs to be utilized by the flow, or it 841 implicitly becomes available for other flows. Failing to use the 842 requested capacity may have already reduced the pool of Quick- 843 Start capacity that was made available to other competing Quick- 844 Start requests. The benefit is greatest when senders use this 845 only for bulk flows and avoid sending unnecessary Quick-Start 846 requests, e.g. for flows that only send a small amount of data. 847 Choosing an appropriate request size requires application-internal 848 knowledge that is not commonly expressed by the transport API. 849 How a sender can determine the rate for an initial Quick-Start 850 request is still a largely unsolved problem. 852 There is no known deployment of Quick-Start for TCP or other IETF 853 transports. 855 6.3.2. Lessons Learned 857 Some lessons can be learned from Quick-Start. Despite being a very 858 light-weight protocol, Quick-Start suffers from poor incremental 859 deployment properties, both regarding the required modifications in 860 network infrastructure as well as its interactions with applications. 861 Except for corner cases, congestion control can be quite efficiently 862 performed end-to-end in the Internet, and in modern stacks there is 863 not much room for significant improvement by additional network 864 support. 866 After publication of the Quick-Start specification, there have been 867 large-scale experiments with an initial window of up to 10 MSS 868 [RFC6928]. This alternative "IW10" approach can also ramp-up data 869 transfers faster than the standard congestion control, but it only 870 requires sender-side modifications. As a result, this approach can 871 be easier and incrementally deployed in the Internet. While 872 theoretically Quick-Start can outperform "IW10", the improvement in 873 completion time for data transfer times can, in many cases, be small. 874 After publication of [RFC6928], most modern TCP stacks have increased 875 their default initial window. 877 6.4. ICMP Source Quench 879 The suggested references for ICMP Source Quench are: 881 * INTERNET CONTROL MESSAGE PROTOCOL [RFC0792] 883 The ICMP Source Quench message [RFC0792] allowed an on-path router to 884 request the source of a flow to reduce its sending rate. This method 885 allowed a router to provide an early indication of impending 886 congestion on a path to the sources that contribute to that 887 congestion. 889 6.4.1. Reasons for Non-deployment 891 This method was deployed in Internet routers over a period of time, 892 the reaction of endpoints to receiving this signal has varied. For 893 low speed links, with low multiplexing of flows the method could be 894 used to regulate (momentarily reduce) the transmission rate. 895 However, the simple signal does not scale with link speed, or the 896 number of flows sharing a link. 898 The approach was overtaken by the evolution of congestion control 899 methods in TCP [RFC2001], and later also by other IETF transports. 900 Because these methods were based upon measurement of the end-to-end 901 path and an algorithm in the endpoint, they were able to evolve and 902 mature more rapidly than methods relying on interactions between 903 operational routers and endpoint stacks. 905 After ICMP Source Quench was specified, the IETF began to recommend 906 that transports provide end-to-end congestion control [RFC2001]. The 907 Source Quench method has been obsoleted by the IETF [RFC6633], and 908 both hosts and routers must now silently discard this message. 910 6.4.2. Lessons Learned 912 This method had several problems: 914 First, [RFC0792] did not sufficiently specify how the sender would 915 react to the ICMP Source Quench signal from the path (e.g., 916 [RFC1016]). There was ambiguity in how the sender should utilize 917 this additional information. This could lead to unfairness in the 918 way that receivers (or routers) responded to this message. 920 Second, while the message did provide additional information, the 921 Explicit Congestion Notification (ECN) mechanism [RFC3168] provided a 922 more robust and informative signal for network nodes to provide early 923 indication that a path has become congested. 925 The mechanism originated at a time when the Internet trust model was 926 very different. Most endpoint implementations did not attempt to 927 verify that the message originated from an on-path node before they 928 utilized the message. This made it vulnerable to denial of service 929 attacks. In theory, routers might have chosen to use the quoted 930 packet contained in the ICMP payload to validate that the message 931 originated from an on-path node, but this would have increased per- 932 packet processing overhead for each router along the path, would have 933 required transport functionality in the router to verify whether the 934 quoted packet header corresponded to a packet the router had sent. 935 In addition, section 5.2 of [RFC4443] noted ICMPv6-based attacks on 936 hosts that would also have threatened routers processing ICMPv6 937 Source Quench payloads. As time passed, it became increasingly 938 obvious that the lack of validation of the messages exposed receivers 939 to a security vulnerability where the messages could be forged to 940 create a tangible denial of service opportunity. 942 6.5. Triggers for Transport (TRIGTRAN) 944 The suggested references for TRIGTRAN are: 946 * TRIGTRAN BOF at IETF 55 [TRIGTRAN-55] 948 * TRIGTRAN BOF at IETF 56 [TRIGTRAN-56] 950 TCP [RFC0793] has a well-known weakness - the end-to-end flow control 951 mechanism has only a single signal, the loss of a segment, and TCP 952 implementations since the late 1980s have interpreted the loss of a 953 segment as evidence that the path between two endpoints may have 954 become congested enough to exhaust buffers on intermediate hops, so 955 that the TCP sender should "back off" - reduce its sending rate until 956 it knows that its segments are now being delivered without loss 957 [RFC5681]. More modern TCP stacks have added a growing array of 958 strategies about how to establish the sending rate [RFC5681], but 959 when a path is no longer operational, TCP would continue to retry 960 transmissions, which would fail, again, and double their 961 Retransmission Time Out (RTO) timers with each failed transmission, 962 with the result that TCP would wait many seconds before retrying a 963 segment, even if the path becomes operational while the sender is 964 waiting for its next retry. 966 The thinking behind TRIGTRAN was that if a path completely stopped 967 working because a link along the path was "down", somehow something 968 along the path could signal TCP when that link returned to service, 969 and the sending TCP could retry immediately, without waiting for a 970 full retransmission timeout (RTO) period. 972 6.5.1. Reasons for Non-deployment 974 The early dreams for TRIGTRAN were dashed because of an assumption 975 that TRIGTRAN triggers would be unauthenticated. This meant that any 976 "safe" TRIGTRAN mechanism would have relied on a mechanism such as 977 setting the IPv4 TTL or IPv6 Hop Count to 255 at a sender and testing 978 that it was 254 upon receipt, so that a receiver could verify that a 979 signal was generated by an adjacent sender known to be on the path 980 being used, and not some unknown sender which might not even be on 981 the path (e.g., "The Generalized TTL Security Mechanism (GTSM)" 982 [RFC5082]). This situation is very similar to the case for ICMP 983 Source Quench messages as described in Section 6.4, which were also 984 unauthenticated, and could be sent by an off-path attacker, resulting 985 in deprecation of ICMP Source Quench message processing [RFC6633]. 987 TRIGTRAN's scope shrunk from "the path is down" to "the first-hop 988 link is down". 990 But things got worse. 992 Because TRIGTRAN triggers would only be provided when the first-hop 993 link was "down", TRIGTRAN triggers couldn't replace normal TCP 994 retransmission behavior if the path failed because some link further 995 along the network path was "down". So TRIGTRAN triggers added 996 complexity to an already complex TCP state machine, and did not allow 997 any existing complexity to be removed. 999 There was also an issue that the TRIGTRAN signal was not sent in 1000 response to a specific host that had been sending packets, and was 1001 instead a signal that stimulated a response by any sender on the 1002 link. This needs to scale when there are multiple flows trying to 1003 use the same resource, yet the sender of a trigger has no 1004 understanding how many of the potential traffic sources will respond 1005 by sending packets - if recipients of the signal back-off their 1006 responses to a trigger to improve scaling, then that immediately 1007 mitigates the benefit of the signal. 1009 Finally, intermediate forwarding nodes required modification to 1010 provide TRIGTRAN triggers, but operators couldn't charge for TRIGTRAN 1011 triggers, so there was no way to recover the cost of modifying, 1012 testing, and deploying updated intermediate nodes. 1014 Two TRIGTRAN BOFs were held, at IETF 55 [TRIGTRAN-55] and IETF 56 1015 [TRIGTRAN-56], but this work was not chartered, and there was no 1016 interest in deploying TRIGTRAN unless it was chartered and 1017 standardized in the IETF. 1019 6.5.2. Lessons Learned. 1021 The reasons why this work was not chartered, much less deployed, 1022 provide several useful lessons for researchers. 1024 * TRIGTRAN started with a plausible value proposition, but 1025 networking realities in the early 2000s forced reductions in scope 1026 that led directly to reductions in potential benefits, but no 1027 corresponding reductions in costs and complexity. 1029 * These reductions in scope were the direct result of an inability 1030 for hosts to trust or authenticate TRIGTRAN signals they received 1031 from the network. 1033 * Operators did not believe they could charge for TRIGTRAN 1034 signaling, because first-hop links didn't fail frequently, and 1035 TRIGTRAN provided no reduction in operating expenses, so there was 1036 little incentive to purchase and deploy TRIGTRAN-capable network 1037 equipment. 1039 It is also worth noting that the targeted environment for TRIGTRAN in 1040 the late 1990s contained links with a relatively small number of 1041 directly-connected hosts - for instance, cellular or satellite links. 1042 The transport community was well aware of the dangers of sender 1043 synchronization based on multiple senders receiving the same stimulus 1044 at the same time, but the working assumption for TRIGTRAN was that 1045 there wouldn't be enough senders for this to be a meaningful problem. 1046 In the 2010s, it is common for a single "link" to support many 1047 senders and receivers on a single link, likely requiring TRIGTRAN 1048 senders to wait some random amount of time before sending after 1049 receiving a TRIGTRAN signal, which would have reduced the benefits of 1050 TRIGTRAN even more. 1052 6.6. Shim6 1054 The suggested references for Shim6 are: 1056 * Shim6: Level 3 Multihoming Shim Protocol for IPv6 [RFC5533] 1057 The IPv6 routing architecture [RFC1887] assumed that most sites on 1058 the Internet would be identified by Provider Assigned IPv6 prefixes, 1059 so that Default-Free Zone routers only contained routes to other 1060 providers, resulting in a very small IPv6 global routing table. 1062 For a single-homed site, this could work well. A multihomed site 1063 with only one upstream provider could also work well, although BGP 1064 multihoming from a single upstream provider was often a premium 1065 service (costing more than twice as much as two single-homed sites), 1066 and if the single upstream provider went out of service, all of the 1067 multihomed paths could fail simultaneously. 1069 IPv4 sites often multihomed by obtaining Provider Independent 1070 prefixes, and advertising these prefixes through multiple upstream 1071 providers. With the assumption that any multihomed IPv4 site would 1072 also multihome in IPv6, it seemed likely that IPv6 routing would be 1073 subject to the same pressures to announce Provider Independent 1074 prefixes, resulting in a global IPv6 routing table that exhibited the 1075 same explosive growth as the global IPv4 routing table. During the 1076 early 2000s, work began on a protocol that would provide multihoming 1077 for IPv6 sites without requiring sites to advertise Provider 1078 Independent prefixes into the IPv6 global routing table. 1080 This protocol, called Shim6, allowed two endpoints to exchange 1081 multiple addresses ("Locators") that all mapped to the same endpoint 1082 ("Identity"). After an endpoint learned multiple Locators for the 1083 other endpoint, it could send to any of those Locators with the 1084 expectation that those packets would all be delivered to the endpoint 1085 with the same Identity. Shim6 was an example of an "Identity/Locator 1086 Split" protocol. 1088 Shim6, as defined in [RFC5533] and related RFCs, provided a workable 1089 solution for IPv6 multihoming using Provider Assigned prefixes, 1090 including capability discovery and negotiation, and allowing end-to- 1091 end application communication to continue even in the face of path 1092 failure, because applications don't see Locator failures, and 1093 continue to communicate with the same Identity using a different 1094 Locator. 1096 6.6.1. Reasons for Non-deployment 1098 Note that the problem being addressed was "site multihoming", but 1099 Shim6 was providing "host multihoming". That meant that the decision 1100 about what path would be used was under host control, not under 1101 router control. 1103 Although more work could have been done to provide a better technical 1104 solution, the biggest impediments to Shim6 deployment were 1105 operational and business considerations. These impediments were 1106 discussed at multiple network operator group meetings, including 1107 [Shim6-35] at [NANOG-35]. 1109 The technique issues centered around concerns that Shim6 relied on 1110 the host to track all the connections, while also tracking Identity/ 1111 Locator mappings in the kernel, and tracking failures to recognize 1112 that a backup path has failed. 1114 The operator issues centered around concerns that operators were 1115 performing traffic engineering, but would have no visibility or 1116 control over hosts when they chose to begin using another path, and 1117 relying on hosts to engineer traffic exposed their networks to 1118 oscillation based on feedback loops, as hosts move from path to path. 1119 At a minimum, traffic engineering policies must be pushed down to 1120 individual hosts. In addition, firewalls that expected to find a 1121 transport-level protocol header in the IP payload, would see a Shim6 1122 Identity header, and be unable to perform transport-protocol-based 1123 firewalling functions because its normal processing logic would not 1124 look past the Identity header. 1126 The business issues centered removing or reducing the ability to sell 1127 BGP multihoming service, which is often more expensive than single- 1128 homed connectivity. 1130 6.6.2. Lessons Learned 1132 It is extremely important to take operational concerns into account 1133 when a path-aware protocol is making decisions about path selection 1134 that may conflict with existing operational practices and business 1135 considerations. 1137 6.6.3. Addendum on MultiPath TCP 1139 During discussions in the PANRG session at IETF 103 [PANRG-103-Min], 1140 Lars Eggert, past Transport Area Director, pointed out that during 1141 charter discussions for the Multipath TCP working group [MP-TCP], 1142 operators expressed concerns that customers could use Multipath TCP 1143 to loadshare TCP connections across operators simultaneously and 1144 compare passive performance measurements across network paths in real 1145 time, changing the balance of power in those business relationships. 1146 Although the Multipath TCP working group was chartered, this concern 1147 could have acted as an obstacle to deployment. 1149 Operator objections to Shim6 were focused on technical concerns, but 1150 this concern could have also been an obstacle to Shim6 deployment if 1151 the technical concerns had been overcome. 1153 6.7. Next Steps in Signaling (NSIS) 1155 The suggested references for Next Steps in Signaling (NSIS) are: 1157 * the concluded working group charter [NSIS-CHARTER-2001] 1159 * GIST: General Internet Signalling Transport [RFC5971] 1161 * NAT/Firewall NSIS Signaling Layer Protocol (NSLP) [RFC5973] 1163 * NSIS Signaling Layer Protocol (NSLP) for Quality-of-Service 1164 Signaling [RFC5974] 1166 * Authorization for NSIS Signaling Layer Protocols [RFC5981] 1168 The NSIS Working Group worked on signaling techniques for network 1169 layer resources (e.g., QoS resource reservations, Firewall and NAT 1170 traversal). 1172 When RSVP [RFC2205] was used in deployments, a number of questions 1173 came up about its perceived limitations and potential missing 1174 features. The issues noted in the NSIS Working Group charter 1175 [NSIS-CHARTER-2001] include interworking between domains with 1176 different QoS architectures, mobility and roaming for IP interfaces, 1177 and complexity. Later, the lack of security in RSVP was also 1178 recognized ([RFC4094]). 1180 The NSIS Working Group was chartered to tackle those issues and 1181 initially focused on QoS signaling as its primary use case. However, 1182 over time a new approach evolved that introduced a modular 1183 architecture using application-specific signaling protocols (the NSIS 1184 Signaling Layer Protocol (NSLP)) on top of a generic signaling 1185 transport protocol (the NSIS Transport Layer Protocol (NTLP)). 1187 The NTLP is defined in [RFC5971]. Two NSLPs are defined: the NSIS 1188 Signaling Layer Protocol (NSLP) for Quality-of-Service Signaling 1189 [RFC5974] as well as the NAT/Firewall NSIS Signaling Layer Protocol 1190 (NSLP) [RFC5973]. 1192 6.7.1. Reasons for Non-deployment 1194 The obstacles for deployment can be grouped into implementation- 1195 related aspects and operational aspects. 1197 * Implementation-related aspects: 1199 Although NSIS provides benefits with respect to flexibility, 1200 mobility, and security compared to other network signaling 1201 techniques, hardware vendors were reluctant to deploy this solution, 1202 because it would require additional implementation effort and would 1203 result in additional complexity for router implementations. 1205 The NTLP mainly operates as path-coupled signaling protocol, i.e, its 1206 messages are processed at the intermediate node's control plane that 1207 are also forwarding the data flows. This requires a mechanism to 1208 intercept signaling packets while they are forwarded in the same 1209 manner (especially along the same path) as data packets. NSIS uses 1210 the IPv4 and IPv6 Router Alert Option (RAO) to allow for interception 1211 of those path-coupled signaling messages, and this technique requires 1212 router implementations to correctly understand and implement the 1213 handling of RAOs, e.g., to only process packet with RAOs of interest 1214 and to leave packets with irrelevant RAOs in the fast forwarding 1215 processing path (a comprehensive discussion of these issues can be 1216 found in [RFC6398]). The latter was an issue with some router 1217 implementations at the time of standardization. 1219 Another reason is that path-coupled signaling protocols that interact 1220 with routers and request manipulation of state at these routers (or 1221 any other network element in general) are under scrutiny: a packet 1222 (or sequence of packets) out of the mainly untrusted data path is 1223 requesting creation and manipulation of network state. This is seen 1224 as potentially dangerous (e.g., opens up a Denial of Service (DoS) 1225 threat to a router's control plane) and difficult for an operator to 1226 control. Path-coupled signaling approaches were considered 1227 problematic (see also section 3 of [RFC6398]). There are 1228 recommendations on how to secure NSIS nodes and deployments (e.g., 1229 [RFC5981]). 1231 * Operational Aspects: 1233 NSIS not only required trust between customers and their provider, 1234 but also among different providers. Especially, QoS signaling 1235 techniques would require some kind of dynamic service level agreement 1236 support that would imply (potentially quite complex) bilateral 1237 negotiations between different Internet service providers. This 1238 complexity was not considered to be justified and increasing the 1239 bandwidth (and thus avoiding bottlenecks) was cheaper than actively 1240 managing network resource bottlenecks by using path-coupled QoS 1241 signaling techniques. Furthermore, an end-to-end path typically 1242 involves several provider domains and these providers need to closely 1243 cooperate in cases of failures. 1245 6.7.2. Lessons Learned 1247 One goal of NSIS was to decrease the complexity of the signaling 1248 protocol, but a path-coupled signaling protocol comes with the 1249 intrinsic complexity of IP-based networks, beyond the complexity of 1250 the signaling protocol itself. Sources of intrinsic complexity 1251 include: 1253 * the presence of asymmetric routes between endpoints and routers 1255 * the lack of security and trust at large in the Internet 1256 infrastructure 1258 * the presence of different trust boundaries 1260 * the effects of best-effort networks (e.g., robustness to packet 1261 loss) 1263 * divergence from the fate sharing principle (e.g., state within the 1264 network). 1266 Any path-coupled signaling protocol has to deal with these realities. 1268 Operators view the use of IPv4 and IPv6 Router Alert Option (RAO) to 1269 signal routers along the path from end systems with suspicion, 1270 because these end systems are usually not authenticated and heavy use 1271 of RAOs can easily increase the CPU load on routers that are designed 1272 to process most packets using a hardware "fast path" and diverting 1273 packets containing RAO to a slower, more capable processor. 1275 6.8. IPv6 Flow Label 1277 The suggested references for IPv6 Flow Label are: 1279 * IPv6 Flow Label Specification [RFC6437] 1280 IPv6 specifies a 20-bit field Flow Label field [RFC6437], included in 1281 the fixed part of the IPv6 header and hence present in every IPv6 1282 packet. An endpoint sets the value in this field to one of a set of 1283 pseudo-randomly assigned values. If a packet is not part of any 1284 flow, the flow label value is set to zero [RFC3697]. A number of 1285 Standards Track and Best Current Practice RFCs (e.g., [RFC8085], 1286 [RFC6437], [RFC6438]) encourage IPv6 endpoints to set a non-zero 1287 value in this field. A multiplexing transport could choose to use 1288 multiple flow labels to allow the network to independently forward 1289 its subflows, or to use one common value for the traffic aggregate. 1290 The flow label is present in all fragments. IPsec was originally put 1291 forward as one important use-case for this mechanism and does encrypt 1292 the field [RFC6438]. 1294 Once set, the flow label can provide information that can help inform 1295 network nodes about subflows present at the transport layer, without 1296 needing to interpret the setting of upper layer protocol fields 1297 [RFC6294]. This information can also be used to coordinate how 1298 aggregates of transport subflows are grouped when queued in the 1299 network and to select appropriate per-flow forwarding when choosing 1300 between alternate paths [RFC6438] (e.g. for Equal Cost Multipath 1301 Routing (ECMP) and Link Aggregation (LAG)). 1303 6.8.1. Reasons for Non-deployment 1305 Despite the field being present in every IPv6 packet, the mechanism 1306 did not receive as much use as originally envisioned. One reason is 1307 that to be useful it requires engagement by two different 1308 stakeholders: 1310 * Endpoint Implementation: 1312 For network nodes along a path to utilize the flow label there needs 1313 to be a non-zero value inserted in the field [RFC6437] at the sending 1314 endpoint. There needs to be an incentive for an endpoint to set an 1315 appropriate non-zero value. The value should appropriately reflect 1316 the level of aggregation the traffic expects to be provided by the 1317 network. However, this requires the stack to know granularity at 1318 which flows should be identified (or conversely which flows should 1319 receive aggregated treatment), i.e., which packets carry the same 1320 flow label. Therefore, setting a non-zero value may result in 1321 additional choices that need to be made by an application developer. 1323 Although the standard [RFC3697] forbids any encoding of meaning into 1324 the flow label value, the opportunity to use the flow label as a 1325 covert channel or to signal other meta-information may have raised 1326 concerns about setting a non-zero value [RFC6437]. 1328 Before methods are widely deployed to use this method, there could be 1329 no incentive for an endpoint to set the field. 1331 * Operational support in network nodes: 1333 A benefit can only be realized when a network node along the path 1334 also uses this information to inform its decisions. Network 1335 equipment (routers and/or middleboxes) need to include appropriate 1336 support so they can utilize the field when making decisions about how 1337 to classify flows, or to inform forwarding choices. Use of any 1338 optional feature in a network node also requires corresponding 1339 updates to operational procedures, and therefore is normally only 1340 introduced when the cost can be justified. 1342 A benefit from utilizing the flow label is expected to be increased 1343 quality of experience for applications - but this comes at some 1344 operational cost to an operator, and requires endpoints to set the 1345 field. 1347 6.8.2. Lessons Learned 1349 The flow label is a general purpose header field for use by the path. 1350 Multiple uses have been proposed. One candidate use was to reduce 1351 the complexity of forwarding decisions. However, modern routers can 1352 use a "fast path", often taking advantage of hardware to accelerate 1353 processing. The method can assist in more complex forwarding, such 1354 as ECMP and load balancing. 1356 Although [RFC6437] recommended that endpoints should by default 1357 choose uniformly-distributed labels for their traffic, the 1358 specification permitted an endpoint to choose to set a zero value. 1359 This ability of endpoints to choose to set a flow label of zero has 1360 had consequences on deployability: 1362 * Before wide-scale support by endpoints, it would be impossible to 1363 rely on a non-zero flow label being set. Network nodes therefore 1364 would need to also employ other techniques to realize equivalent 1365 functions. An example of a method is one assuming semantics of 1366 the source port field to provide entropy input to a network-layer 1367 hash. This use of a 5-tuple to classify a packet represents a 1368 layering violation [RFC6294]. When other methods have been 1369 deployed, they increase the cost of deploying standards-based 1370 methods, even though they may offer less control to endpoints and 1371 result in potential interaction with other uses/interpretation of 1372 the field. 1374 * Even though the flow label is specified as an end-to-end field, 1375 some network paths have been observed to not transparently forward 1376 the flow label. This could result from non-conformant equipment, 1377 or could indicate that some operational networks have chosen to 1378 re-use the protocol field for other (e.g. internal purposes). 1379 This results in lack of transparency, and a deployment hurdle to 1380 endpoints expecting that they can set a flow label that is 1381 utilized by the network. The more recent practice of "greasing" 1382 [GREASE] would suggest that a different outcome could have been 1383 achieved if endpoints were always required to set a non-zero 1384 value. 1386 * [RFC1809] noted that setting the choice of the flow label value 1387 can depend on the expectations of the traffic generated by an 1388 application, which suggests an API should be presented to control 1389 the setting or policy that is used. However, many currently 1390 available APIs do not have this support. 1392 A growth in the use of encrypted transports, (e.g. QUIC [QUIC-WG]) 1393 seems likely to raise similar issues to those discussed above and 1394 could motivate renewed interest in utilizing the flow label. 1396 7. Security Considerations 1398 This document describes Path Aware techniques that were not adopted 1399 and widely deployed on the Internet, so it doesn't affect the 1400 security of the Internet. 1402 If this document meets its goals, we may develop new techniques for 1403 Path Aware Networking that would affect the security of the Internet, 1404 but security considerations for those techniques will be described in 1405 the corresponding RFCs that specify them. 1407 8. IANA Considerations 1409 This document makes no requests of IANA. 1411 9. Acknowledgments 1413 Initial material for Section 6.1 on ST2 was provided by Gorry 1414 Fairhurst. 1416 Initial material for Section 6.2 on IntServ was provided by Ron 1417 Bonica. 1419 Initial material for Section 6.3 on Quick-Start TCP was provided by 1420 Michael Scharf, who also provided suggestions to improve this section 1421 after it was edited. 1423 Initial material for Section 6.4 on ICMP Source Quench was provided 1424 by Gorry Fairhurst. 1426 Initial material for Section 6.5 on Triggers for Transport (TRIGTRAN) 1427 was provided by Spencer Dawkins. 1429 Section 6.6 on Shim6 builds on initial material describing obstacles 1430 provided by Erik Nordmark, with background added by Spencer Dawkins. 1432 Initial material for Section 6.7 on Next Steps In Signaling (NSIS) 1433 was provided by Roland Bless and Martin Stiemerling. 1435 Initial material for Section 6.8 on IPv6 Flow Labels was provided by 1436 Gorry Fairhurst. 1438 Our thanks to C.M. Heard, David Black, Erik Auerswald, Gorry 1439 Fairhurst, Joe Touch, Joeri de Ruiter, Mohamed Boucadair, Roland 1440 Bless, Ruediger Geib, Theresa Enghardt, and Wes Eddy, who provided 1441 review comments on previous versions. 1443 Mallory Knodel reviewed this document for the Internet Engineering 1444 Steering Group, and provided many helpful suggestions. 1446 David Oran also provided helpful comments and text suggestions on 1447 this document during Internet Engineering Steering Group balloting. 1448 In particular, Section 5 reflects his review. 1450 Special thanks to Adrian Farrel for helping Spencer navigate the 1451 twisty little passages of Flow Specs and Filter Specs in IntServ, 1452 RSVP, MPLS, and BGP. They are all alike, except when they are 1453 different [Colossal-Cave]. 1455 10. Informative References 1457 [Colossal-Cave] 1458 "Wikipedia Page for Colossal Cave Adventure", January 1459 2019, 1460 . 1462 [Conviva] "Conviva Precision : Data Sheet", December 2020, 1463 . 1466 [GREASE] Thomson, M., "Long-term Viability of Protocol Extension 1467 Mechanisms", July 2019, . 1470 [I-D.arkko-arch-internet-threat-model] 1471 Arkko, J., "Changes in the Internet Threat Model", Work in 1472 Progress, Internet-Draft, draft-arkko-arch-internet- 1473 threat-model-01, 8 July 2019, . 1477 [I-D.farrell-etm] 1478 Farrell, S., "We're gonna need a bigger threat model", 1479 Work in Progress, Internet-Draft, draft-farrell-etm-03, 6 1480 July 2019, . 1483 [I-D.ietf-tsvwg-intserv-multiple-tspec] 1484 Polk, J. and S. Dhesikan, "Integrated Services (IntServ) 1485 Extension to Allow Signaling of Multiple Traffic 1486 Specifications and Multiple Flow Specifications in 1487 RSVPv1", Work in Progress, Internet-Draft, draft-ietf- 1488 tsvwg-intserv-multiple-tspec-02, 25 February 2013, 1489 . 1492 [I-D.irtf-panrg-questions] 1493 Trammell, B., "Current Open Questions in Path Aware 1494 Networking", Work in Progress, Internet-Draft, draft-irtf- 1495 panrg-questions-08, 23 December 2020, 1496 . 1499 [IEN-119] Forgie, J., "ST - A Proposed Internet Stream Protocol", 1500 September 1979, 1501 . 1503 [ITAT] "IAB Workshop on Internet Technology Adoption and 1504 Transition (ITAT)", December 2013, 1505 . 1507 [model-t] "Model-t -- Discussions of changes in Internet deployment 1508 patterns and their impact on the Internet threat model", 1509 n.d., . 1511 [MOPS-109-Min] 1512 "Media Operations Working Group - IETF-109 Minutes", 1513 November 2020, 1514 . 1517 [MP-TCP] "Multipath TCP Working Group Home Page", n.d., 1518 . 1520 [NANOG-35] "North American Network Operators Group NANOG-35 Agenda", 1521 October 2005, 1522 . 1524 [NSIS-CHARTER-2001] 1525 "Next Steps In Signaling Working Group Charter", March 1526 2011, 1527 . 1529 [PANRG] "Path Aware Networking Research Group (Home Page)", n.d., 1530 . 1532 [PANRG-103-Min] 1533 "Path Aware Networking Research Group - IETF-103 Minutes", 1534 November 2018, 1535 . 1537 [PANRG-105-Min] 1538 "Path Aware Networking Research Group - IETF-105 Minutes", 1539 July 2019, 1540 . 1542 [PANRG-106-Min] 1543 "Path Aware Networking Research Group - IETF-106 Minutes", 1544 November 2019, 1545 . 1547 [PANRG-99] "Path Aware Networking Research Group - IETF-99", July 1548 2017, 1549 . 1551 [PATH-Decade] 1552 Bonaventure, O., "A Decade of Path Awareness", July 2017, 1553 . 1556 [PathProp] Enghardt, T. and C. Kraehenbuehl, "A Vocabulary of Path 1557 Properties", November 2019, . 1560 [QS-SAT] Secchi, R., Sathiaseelan, A., Potorti, F., Gotta, A., and 1561 G. Fairhurst, "Using Quick-Start to enhance TCP-friendly 1562 rate control performance in bidirectional satellite 1563 networks", 2009, 1564 . 1566 [QUIC-WG] "QUIC Working Group Home Page", n.d., 1567 . 1569 [RFC0792] Postel, J., "Internet Control Message Protocol", STD 5, 1570 RFC 792, DOI 10.17487/RFC0792, September 1981, 1571 . 1573 [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, 1574 RFC 793, DOI 10.17487/RFC0793, September 1981, 1575 . 1577 [RFC1016] Prue, W. and J. Postel, "Something a Host Could Do with 1578 Source Quench: The Source Quench Introduced Delay 1579 (SQuID)", RFC 1016, DOI 10.17487/RFC1016, July 1987, 1580 . 1582 [RFC1190] Topolcic, C., "Experimental Internet Stream Protocol: 1583 Version 2 (ST-II)", RFC 1190, DOI 10.17487/RFC1190, 1584 October 1990, . 1586 [RFC1633] Braden, R., Clark, D., and S. Shenker, "Integrated 1587 Services in the Internet Architecture: an Overview", 1588 RFC 1633, DOI 10.17487/RFC1633, June 1994, 1589 . 1591 [RFC1809] Partridge, C., "Using the Flow Label Field in IPv6", 1592 RFC 1809, DOI 10.17487/RFC1809, June 1995, 1593 . 1595 [RFC1819] Delgrossi, L., Ed. and L. Berger, Ed., "Internet Stream 1596 Protocol Version 2 (ST2) Protocol Specification - Version 1597 ST2+", RFC 1819, DOI 10.17487/RFC1819, August 1995, 1598 . 1600 [RFC1887] Rekhter, Y., Ed. and T. Li, Ed., "An Architecture for IPv6 1601 Unicast Address Allocation", RFC 1887, 1602 DOI 10.17487/RFC1887, December 1995, 1603 . 1605 [RFC2001] Stevens, W., "TCP Slow Start, Congestion Avoidance, Fast 1606 Retransmit, and Fast Recovery Algorithms", RFC 2001, 1607 DOI 10.17487/RFC2001, January 1997, 1608 . 1610 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1611 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1612 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1613 September 1997, . 1615 [RFC2210] Wroclawski, J., "The Use of RSVP with IETF Integrated 1616 Services", RFC 2210, DOI 10.17487/RFC2210, September 1997, 1617 . 1619 [RFC2211] Wroclawski, J., "Specification of the Controlled-Load 1620 Network Element Service", RFC 2211, DOI 10.17487/RFC2211, 1621 September 1997, . 1623 [RFC2212] Shenker, S., Partridge, C., and R. Guerin, "Specification 1624 of Guaranteed Quality of Service", RFC 2212, 1625 DOI 10.17487/RFC2212, September 1997, 1626 . 1628 [RFC2215] Shenker, S. and J. Wroclawski, "General Characterization 1629 Parameters for Integrated Service Network Elements", 1630 RFC 2215, DOI 10.17487/RFC2215, September 1997, 1631 . 1633 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1634 and W. Weiss, "An Architecture for Differentiated 1635 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1636 . 1638 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1639 of Explicit Congestion Notification (ECN) to IP", 1640 RFC 3168, DOI 10.17487/RFC3168, September 2001, 1641 . 1643 [RFC3697] Rajahalme, J., Conta, A., Carpenter, B., and S. Deering, 1644 "IPv6 Flow Label Specification", RFC 3697, 1645 DOI 10.17487/RFC3697, March 2004, 1646 . 1648 [RFC4094] Manner, J. and X. Fu, "Analysis of Existing Quality-of- 1649 Service Signaling Protocols", RFC 4094, 1650 DOI 10.17487/RFC4094, May 2005, 1651 . 1653 [RFC4443] Conta, A., Deering, S., and M. Gupta, Ed., "Internet 1654 Control Message Protocol (ICMPv6) for the Internet 1655 Protocol Version 6 (IPv6) Specification", STD 89, 1656 RFC 4443, DOI 10.17487/RFC4443, March 2006, 1657 . 1659 [RFC4782] Floyd, S., Allman, M., Jain, A., and P. Sarolahti, "Quick- 1660 Start for TCP and IP", RFC 4782, DOI 10.17487/RFC4782, 1661 January 2007, . 1663 [RFC5082] Gill, V., Heasley, J., Meyer, D., Savola, P., Ed., and C. 1664 Pignataro, "The Generalized TTL Security Mechanism 1665 (GTSM)", RFC 5082, DOI 10.17487/RFC5082, October 2007, 1666 . 1668 [RFC5218] Thaler, D. and B. Aboba, "What Makes for a Successful 1669 Protocol?", RFC 5218, DOI 10.17487/RFC5218, July 2008, 1670 . 1672 [RFC5533] Nordmark, E. and M. Bagnulo, "Shim6: Level 3 Multihoming 1673 Shim Protocol for IPv6", RFC 5533, DOI 10.17487/RFC5533, 1674 June 2009, . 1676 [RFC5575] Marques, P., Sheth, N., Raszuk, R., Greene, B., Mauch, J., 1677 and D. McPherson, "Dissemination of Flow Specification 1678 Rules", RFC 5575, DOI 10.17487/RFC5575, August 2009, 1679 . 1681 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 1682 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 1683 . 1685 [RFC5971] Schulzrinne, H. and R. Hancock, "GIST: General Internet 1686 Signalling Transport", RFC 5971, DOI 10.17487/RFC5971, 1687 October 2010, . 1689 [RFC5973] Stiemerling, M., Tschofenig, H., Aoun, C., and E. Davies, 1690 "NAT/Firewall NSIS Signaling Layer Protocol (NSLP)", 1691 RFC 5973, DOI 10.17487/RFC5973, October 2010, 1692 . 1694 [RFC5974] Manner, J., Karagiannis, G., and A. McDonald, "NSIS 1695 Signaling Layer Protocol (NSLP) for Quality-of-Service 1696 Signaling", RFC 5974, DOI 10.17487/RFC5974, October 2010, 1697 . 1699 [RFC5981] Manner, J., Stiemerling, M., Tschofenig, H., and R. Bless, 1700 Ed., "Authorization for NSIS Signaling Layer Protocols", 1701 RFC 5981, DOI 10.17487/RFC5981, February 2011, 1702 . 1704 [RFC6294] Hu, Q. and B. Carpenter, "Survey of Proposed Use Cases for 1705 the IPv6 Flow Label", RFC 6294, DOI 10.17487/RFC6294, June 1706 2011, . 1708 [RFC6398] Le Faucheur, F., Ed., "IP Router Alert Considerations and 1709 Usage", BCP 168, RFC 6398, DOI 10.17487/RFC6398, October 1710 2011, . 1712 [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, 1713 "IPv6 Flow Label Specification", RFC 6437, 1714 DOI 10.17487/RFC6437, November 2011, 1715 . 1717 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 1718 for Equal Cost Multipath Routing and Link Aggregation in 1719 Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011, 1720 . 1722 [RFC6633] Gont, F., "Deprecation of ICMP Source Quench Messages", 1723 RFC 6633, DOI 10.17487/RFC6633, May 2012, 1724 . 1726 [RFC6928] Chu, J., Dukkipati, N., Cheng, Y., and M. Mathis, 1727 "Increasing TCP's Initial Window", RFC 6928, 1728 DOI 10.17487/RFC6928, April 2013, 1729 . 1731 [RFC7305] Lear, E., Ed., "Report from the IAB Workshop on Internet 1732 Technology Adoption and Transition (ITAT)", RFC 7305, 1733 DOI 10.17487/RFC7305, July 2014, 1734 . 1736 [RFC7418] Dawkins, S., Ed., "An IRTF Primer for IETF Participants", 1737 RFC 7418, DOI 10.17487/RFC7418, December 2014, 1738 . 1740 [RFC8085] Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage 1741 Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085, 1742 March 2017, . 1744 [RFC8170] Thaler, D., Ed., "Planning for Protocol Adoption and 1745 Subsequent Transitions", RFC 8170, DOI 10.17487/RFC8170, 1746 May 2017, . 1748 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 1749 "Deterministic Networking Architecture", RFC 8655, 1750 DOI 10.17487/RFC8655, October 2019, 1751 . 1753 [RFC8793] Wissingh, B., Wood, C., Afanasyev, A., Zhang, L., Oran, 1754 D., and C. Tschudin, "Information-Centric Networking 1755 (ICN): Content-Centric Networking (CCNx) and Named Data 1756 Networking (NDN) Terminology", RFC 8793, 1757 DOI 10.17487/RFC8793, June 2020, 1758 . 1760 [SAAG-105-Min] 1761 "Security Area Open Meeting - IETF-105 Minutes", July 1762 2019, . 1765 [SAF07] Sarolahti, P., Allman, M., and S. Floyd, "Determining an 1766 appropriate sending rate over an underutilized network 1767 path", Computer Networking Volume 51, Number 7, May 2007. 1769 [Sch11] Scharf, M., "Fast Startup Internet Congestion Control for 1770 Broadband Interactive Applications", Ph.D. Thesis, 1771 University of Stuttgart, April 2011. 1773 [Shim6-35] Meyer, D., Huston, G., Schiller, J., and V. Gill, "IAB 1774 IPv6 Multihoming Panel at NANOG 35", NANOG North American 1775 Network Operator Group, October 2005, 1776 . 1778 [TRIGTRAN-55] 1779 "Triggers for Transport BOF at IETF 55", July 2003, 1780 . 1782 [TRIGTRAN-56] 1783 "Triggers for Transport BOF at IETF 56", November 2003, 1784 . 1786 Author's Address 1788 Spencer Dawkins (editor) 1789 Tencent America 1791 Email: spencerdawkins.ietf@gmail.com