idnits 2.17.1 draft-irtf-panrg-what-not-to-do-14.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (16 November 2020) is 1257 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-04) exists of draft-iab-use-it-or-lose-it-00 == Outdated reference: A later version (-12) exists of draft-irtf-panrg-questions-07 -- Obsolete informational reference (is this intentional?): RFC 793 (Obsoleted by RFC 9293) -- Obsolete informational reference (is this intentional?): RFC 1190 (Obsoleted by RFC 1819) -- Obsolete informational reference (is this intentional?): RFC 2001 (Obsoleted by RFC 2581) -- Obsolete informational reference (is this intentional?): RFC 3697 (Obsoleted by RFC 6437) -- Obsolete informational reference (is this intentional?): RFC 5575 (Obsoleted by RFC 8955) Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 6 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 PANRG S. Dawkins, Ed. 3 Internet-Draft Tencent America 4 Intended status: Informational 16 November 2020 5 Expires: 20 May 2021 7 Path Aware Networking: Obstacles to Deployment (A Bestiary of Roads Not 8 Taken) 9 draft-irtf-panrg-what-not-to-do-14 11 Abstract 13 At the first meeting of the Path Aware Networking Research Group, the 14 research group agreed to catalog and analyze past efforts to develop 15 and deploy Path Aware techniques, most of which were unsuccessful or 16 at most partially successful, in order to extract insights and 17 lessons for path-aware networking researchers. 19 This document contains that catalog and analysis. 21 Status of This Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at https://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on 20 May 2021. 38 Copyright Notice 40 Copyright (c) 2020 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 45 license-info) in effect on the date of publication of this document. 46 Please review these documents carefully, as they describe your rights 47 and restrictions with respect to this document. 49 Table of Contents 51 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 52 1.1. What Does "Path Awareness" Mean in this Document? . . . . 3 53 1.2. Note to RFC Editor . . . . . . . . . . . . . . . . . . . 4 54 2. A Perspective On This Document . . . . . . . . . . . . . . . 4 55 2.1. Notes for the Reader . . . . . . . . . . . . . . . . . . 5 56 2.2. A Note About Path-Aware Techniques Included In This 57 Document . . . . . . . . . . . . . . . . . . . . . . . . 5 58 2.3. Venue for Discussion of this Document . . . . . . . . . . 5 59 2.4. Architectural Guidance . . . . . . . . . . . . . . . . . 5 60 2.5. Terminology Used in this Document . . . . . . . . . . . . 6 61 2.6. Methodology for Contributions . . . . . . . . . . . . . . 6 62 3. Applying the Lessons We've Learned . . . . . . . . . . . . . 7 63 4. Summary of Lessons Learned . . . . . . . . . . . . . . . . . 10 64 4.1. Justifying Deployment . . . . . . . . . . . . . . . . . . 10 65 4.2. Providing Benefits for Early Adopters . . . . . . . . . . 10 66 4.3. Providing Benefits During Partial Deployment . . . . . . 11 67 4.4. Outperforming End-to-end Protocol Mechanisms . . . . . . 11 68 4.5. Paying for Path Aware Techniques . . . . . . . . . . . . 11 69 4.6. Impact on Operational Practices . . . . . . . . . . . . . 11 70 4.7. Per-connection State . . . . . . . . . . . . . . . . . . 11 71 4.8. Keeping Traffic on Fast-paths . . . . . . . . . . . . . . 12 72 4.9. Endpoints Trusting Intermediate Nodes . . . . . . . . . . 12 73 4.10. Intermediate Nodes Trusting Endpoints . . . . . . . . . . 12 74 4.11. Reacting to Distant Signals . . . . . . . . . . . . . . . 12 75 4.12. Support in Endpoint Protocol Stacks . . . . . . . . . . . 13 76 5. Contributions . . . . . . . . . . . . . . . . . . . . . . . . 13 77 5.1. Stream Transport (ST, ST2, ST2+) . . . . . . . . . . . . 13 78 5.1.1. Reasons for Non-deployment . . . . . . . . . . . . . 14 79 5.1.2. Lessons Learned. . . . . . . . . . . . . . . . . . . 14 80 5.2. Integrated Services (IntServ) . . . . . . . . . . . . . . 14 81 5.2.1. Reasons for Non-deployment . . . . . . . . . . . . . 15 82 5.2.2. Lessons Learned. . . . . . . . . . . . . . . . . . . 15 83 5.3. Quick-Start TCP . . . . . . . . . . . . . . . . . . . . . 16 84 5.3.1. Reasons for Non-deployment . . . . . . . . . . . . . 17 85 5.3.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 18 86 5.4. ICMP Source Quench . . . . . . . . . . . . . . . . . . . 18 87 5.4.1. Reasons for Non-deployment . . . . . . . . . . . . . 19 88 5.4.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 19 89 5.5. Triggers for Transport (TRIGTRAN) . . . . . . . . . . . . 20 90 5.5.1. Reasons for Non-deployment . . . . . . . . . . . . . 21 91 5.5.2. Lessons Learned. . . . . . . . . . . . . . . . . . . 22 92 5.6. Shim6 . . . . . . . . . . . . . . . . . . . . . . . . . . 22 93 5.6.1. Reasons for Non-deployment . . . . . . . . . . . . . 23 94 5.6.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 24 95 5.6.3. Addendum on MultiPath TCP . . . . . . . . . . . . . . 24 96 5.7. Next Steps in Signaling (NSIS) . . . . . . . . . . . . . 24 97 5.7.1. Reasons for Non-deployment . . . . . . . . . . . . . 25 98 5.7.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 26 99 5.8. IPv6 Flow Label . . . . . . . . . . . . . . . . . . . . . 27 100 5.8.1. Reasons for Non-deployment . . . . . . . . . . . . . 28 101 5.8.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 29 102 6. Security Considerations . . . . . . . . . . . . . . . . . . . 30 103 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30 104 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 30 105 9. Informative References . . . . . . . . . . . . . . . . . . . 31 106 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 37 108 1. Introduction 110 This document describes the lessons that IETF participants have 111 learned (and learned the hard way) about Path Aware Networking over a 112 period of several decades, and provides an analysis of reasons why 113 various Path Aware Networking techniques have seen limited or no 114 deployment. 116 1.1. What Does "Path Awareness" Mean in this Document? 118 The current definition of "Path Awareness", used by the Path Aware 119 Networking Research Group, appears in Section 1.1 ("Definition") in 120 [I-D.irtf-panrg-questions]. That definition is included here as a 121 convenience to the reader. 123 | For purposes of this document, "path aware networking" describes 124 | endpoint discovery of the properties of paths they use for 125 | communication, and endpoint reaction to these properties that 126 | affects routing and/or transmission; note that this can and 127 | already does happen to some extent in the current Internet 128 | architecture. Expanding on this definition, a "path aware 129 | internetwork" is one in which endpoint discovery of path 130 | properties and endpoint selection of paths used by traffic 131 | exchanged by the endpoint are explicitly supported, regardless of 132 | the specific design of the protocol features which enable this 133 | discovery and selection. 135 Because this document reflects work performed over several decades, 136 some technologies described in Section 5 may not reflect the current 137 definition, but these technologies were considered "path aware" by 138 their contributors, so these contributions are included in this 139 retrospective document. 141 1.2. Note to RFC Editor 143 If the "Definition" in Section 1.1 ("Definition") of 144 [I-D.irtf-panrg-questions] changes, the text in Section 1.1 of this 145 document should be should be changed as well. 147 Whether that happens or not, the RFC Editor is requested to remove 148 this section. 150 2. A Perspective On This Document 152 At the first meeting of the Path Aware Networking Research Group 153 [PANRG], at IETF 99 [PANRG-99], Olivier Bonaventure led a discussion 154 of "A Decade of Path Awareness" [PATH-Decade], on attempts, which 155 were mostly unsuccessful for a variety of reasons, to exploit Path 156 Aware techniques and achieve a variety of goals over the past decade. 157 At the end of that discussion, two things were abundantly clear. 159 * The Internet community has accumulated considerable experience 160 with many Path Aware techniques over a long period of time, and 162 * Although some path aware techniques have been deployed (for 163 example, Differentiated Services, or DiffServ [RFC2475]), most of 164 these techniques haven't seen widespread adoption and deplyment. 165 Even "successful" techniques like DiffServ can face obstacles that 166 prevents wider usage. The reasons for non-adoption and limited 167 adoption and deployment are many, and are worthy of study. 169 The meta-lessons from that experience were 171 * Path aware networking has been more Research than Engineering, so 172 establishing an IRTF Research Group for Path Aware Networking is 173 the right thing to do [RFC7418]. 175 * Analyzing a catalog of past experience to learn the reasons for 176 non-adoption would be a great first step for the Research Group. 178 Allison Mankin, as IRTF Chair, officially chartered the Path Aware 179 Networking Research Group in July, 2018. 181 This document contains the analysis performed by that research group 182 (Section 4), based on that catalog (Section 5). 184 This document represents the consensus of the Path Aware Networking 185 Research Group. 187 2.1. Notes for the Reader 189 This Informational document discusses Path Aware protocol mechanisms 190 considered, and in some cases standardized, by the Internet 191 Engineering Task Force (IETF), and considers Lessons Learned from 192 those mechanisms. The intention is to inform the work of protocol 193 designers, whether in the IRTF, the IETF, or elsewhere in the 194 Internet ecosystem. 196 As an Informational document published in the IRTF stream, this 197 document has no authority beyond the quality of the analysis it 198 contains. 200 2.2. A Note About Path-Aware Techniques Included In This Document 202 This document does not catalog every proposed path aware networking 203 technique that was not adopted and deployed. Instead, we limited our 204 focus to technologies that passed through the IETF community, and 205 still identified enough techniques to provide background for the 206 lessons included in Section 4 to inform researchers and protocol 207 engineers in their work. 209 No shame is intended for the techniques included in this document. 210 As shown in Section 4, the quality of specific techniques had little 211 to do with whether they were deployed or not. Based on the 212 techniques cataloged in this document, it is likely that when these 213 techniques were put forward, the proponents were trying to engineer 214 something that could not be engineered without first carrying out 215 research. Actual shame would be failing to learn from experience, 216 and failing to share that experience with other networking 217 researchers and engineers. 219 2.3. Venue for Discussion of this Document 221 (RFC Editor: please remove this section before publication) 223 Discussion of specific contributed experiences and this document in 224 general should take place on the PANRG mailing list. 226 2.4. Architectural Guidance 228 As background for understanding the Lessons Learned contained in this 229 document, the reader is encouraged to become familiar with the 230 Internet Architecture Board's documents on "What Makes for a 231 Successful Protocol?" [RFC5218] and "Planning for Protocol Adoption 232 and Subsequent Transitions" [RFC8170]. 234 Although these two documents do not specifically target path-aware 235 networking protocols, they are helpful resources for readers seeking 236 to improve their understanding of considerations for successful 237 adoption and deployment of any protocol. For example, the Basic 238 Success Factors described in Setion 2.1 of [RFC5218] are helpful for 239 readers of this document. 241 Because there is an economic aspect to decisions about deployment, 242 the IAB Workshop on Internet Technology Adoption and Transition 243 [ITAT] report [RFC7305] also provides food for thought. 245 Several of the Lessons Learned in Section 4 reflect considerations 246 described in [RFC5218], [RFC7305], and [RFC8170]. 248 2.5. Terminology Used in this Document 250 The terms Node and Element in this document have the meaning defined 251 in [PathProp]. 253 2.6. Methodology for Contributions 255 This document grew out of contributions by various IETF participants 256 with experience with one or more Path Aware Networking techniques. 258 There are many things that could be said about the Path Aware 259 networking techniques that have been developed. For the purposes of 260 this document, contributors were requested to provide 262 * the name of a technique, including an abbreviation if one was used 264 * if available, a long-term pointer to the best reference describing 265 the technique 267 * a short description of the problem the technique was intended to 268 solve 270 * a short description of the reasons why the technique wasn't 271 adopted 273 * a short statement of the lessons that researchers can learn from 274 our experience with this technique. 276 3. Applying the Lessons We've Learned 278 The initial scope for this document was roughly "what mistakes have 279 we made in the decade prior to [PANRG-99], that we shouldn't make 280 again". Some of the contributions in Section 5 predate the initial 281 scope. The earliest Path-Aware Networking technique referred to in 282 Section 5 is Section 5.1, published in the late 1970s. Given that 283 the networking ecosystem has evolved continuously, it seems 284 reasonable to consider how to apply these lessons. 286 The PANRG Research Group reviewed the Lessons Learned (Section 4) 287 contained in the May 23, 2019 version of this document at IETF 105 288 [PANRG-105-Min], and carried out additional discussion at IETF 106 289 [PANRG-106-Min]. Table 1 provides the "sense of the room" about each 290 lesson after those discussions. The intention is to capture whether 291 a specific lesson seems to be 293 * "Invariant" - well-understood and is likely to be applicable for 294 any proposed Path Aware Networking solution. 296 * "Variable" - has impeded deployment in the past, but might not be 297 applicable in a specific technique. Engineering analysis to 298 understand whether the lesson is applicable is prudent. 300 * "Not Now" - this characteristic tends to turn up a minefield full 301 of dragons, and prudent network engineers will wish to avoid 302 gambling on a technique that relies on this, until something 303 significant changes 305 +-----------------------------------------------------+-----------+ 306 | Lesson | Category | 307 +=====================================================+===========+ 308 | Justifying Deployment (Section 4.1) | Invariant | 309 +-----------------------------------------------------+-----------+ 310 | Providing Benefits for Early Adopters (Section 4.2) | Invariant | 311 +-----------------------------------------------------+-----------+ 312 | Providing Benefits during Partial Deployment | Invariant | 313 | (Section 4.3) | | 314 +-----------------------------------------------------+-----------+ 315 | Outperforming End-to-end Protocol Mechanisms | Variable | 316 | (Section 4.4) | | 317 +-----------------------------------------------------+-----------+ 318 | Paying for Path Aware Techniques (Section 4.5) | Invariant | 319 +-----------------------------------------------------+-----------+ 320 | Impact on Operational Practices (Section 4.6) | Invariant | 321 +-----------------------------------------------------+-----------+ 322 | Per-connection State (Section 4.7) | Variable | 323 +-----------------------------------------------------+-----------+ 324 | Keeping Traffic on Fast-paths (Section 4.8) | Variable | 325 +-----------------------------------------------------+-----------+ 326 | Endpoints Trusting Intermediate Nodes (Section 4.9) | Not Now | 327 +-----------------------------------------------------+-----------+ 328 | Intermediate Nodes Trusting Endpoints | Not Now | 329 | (Section 4.10) | | 330 +-----------------------------------------------------+-----------+ 331 | Reacting to Distant Signals (Section 4.11) | Variable | 332 +-----------------------------------------------------+-----------+ 333 | Support in Endpoint Protocol Stacks (Section 4.12) | Variable | 334 +-----------------------------------------------------+-----------+ 336 Table 1 338 "Justifying Deployment", "Providing Benefits for Early Adopters", 339 "Paying for Path Aware Techniques", and "Impact on Operational 340 Practice" were considered to be invariant - the sense of the room was 341 that these would always be considerations for any proposed Path Aware 342 Technique. 344 "Providing Benefits During Partial Deployment" was added after IETF 345 105, during research group last call, and is also considered to be 346 invariant. 348 For "Outperforming End-to-end Protocol Mechanisms", there is a trade- 349 off between improved performance from Path Aware Techniques and 350 additional complexity required by some Path Aware Techniques. 352 * For example, if you can obtain the same understanding of path 353 characteristics from measurements obtained over a few more round 354 trips, endpoint implementers are unlikely to be eager to add 355 complexity, and many attributes can be measured from an endpoint, 356 without assistance from intermediate nodes. 358 For "Per-connection State", the key questions discussed in the 359 research group were "how much state" and "where state is maintained". 361 * IntServ (Section 5.2) required state at every intermediate node 362 for every connection between two endpoints. As the Internet 363 ecosystem has evolved, carrying many connections in a tunnel that 364 appears to intermediate nodes as a single connection has become 365 more common, so that additional end-to-end connections don't add 366 additional state to intermediate nodes between tunnel endpoints. 367 If these tunnels are encrypted, intermediate nodes between tunnel 368 endpoints can't distinguish between connections, even if that were 369 desirable. 371 For "Keeping Traffic on Fast-paths", we noted that this was true for 372 many platforms, but not for all. 374 * For backbone routers, this is likely an invariant, but for 375 platforms that rely more on general-purpose computers to make 376 forwarding decisions, this may not be a fatal flaw for Path Aware 377 Networking techniques. 379 For "Endpoints Trusting Intermediate Nodes" and "Intermediate Nodes 380 Trusting Endpoints", these lessons point to the broader need to 381 revisit the Internet Threat Model. 383 * We noted with relief that discussions about this were already 384 underway in the IETF community at IETF 105 (see the Security Area 385 Open Meeting minutes [SAAG-105-Min] for discussion of 386 [draft-arkko-arch-internet-threat-model] and [draft-farrell-etm]), 387 and the Internet Architecture Board has created a mailing list for 388 continued discussions ([model-t]), but we recognize that there are 389 Path Aware Networking aspects of this effort, requiring research. 391 For "Reacting to Distant Signals", we noted that not all attributes 392 are equal. 394 * If an attribute is stable over an extended period of time, is 395 difficult to observe via end-to-end mechanisms, and is valuable, 396 Path Aware Techniques that rely on that attribute to provide a 397 significant benefit become more attractive. 399 * Analysis to help identify attributes that are useful enough to 400 justify deployment of Path Aware techniques that make use of those 401 attributes would be helpful. 403 For "Support in Endpoint Protocol Stacks", we noted that Path Aware 404 applications must be able to identify and communicate requirements 405 about path characteristics. 407 * The de-facto sockets API has no way of signaling application 408 expectations for the network path to the protocol stack. 410 4. Summary of Lessons Learned 412 This section summarizes the Lessons Learned from the contributed 413 subsections in Section 5. 415 Each Lesson Learned is tagged with one or more contributions that 416 encountered this obstacle as a significant impediment to deployment. 417 Other contributed techniques may have also encountered this obstacle, 418 but this obstacle may not have been the biggest impediment to 419 deployment for those techniques. 421 It is useful to notice that sometimes an obstacle might impede 422 deployment, while at other times, the same obstacle might prevent 423 adoption and deployment entirely. The research group discussed 424 distinguishing between obstacles that impede and obstacles that 425 prevent, but it appears that the boundary between "impede" and 426 "prevent" can shift over time - some of the Lessons Learned are based 427 on both Path Aware techniques that were not deployed, and Path Aware 428 techniques that were deployed, but were not deployed widely or 429 quickly. See Section 5.6 and Section 5.6.3 as one example of this 430 shifting boundary. 432 4.1. Justifying Deployment 434 The benefit of Path Awareness must be great enough to justify making 435 changes in an operational network. The colloquial U.S. American 436 English expression, "If it ain't broke, don't fix it" is a "best 437 current practice" on today's Internet. (See Section 5.3, 438 Section 5.5, and Section 5.4, in addition to [RFC5218]). 440 4.2. Providing Benefits for Early Adopters 442 Providing benefits for early adopters can be key - if everyone must 443 deploy a technique in order for the technique to provide benefits, or 444 even to work at all, the technique is unlikely to be adopted widely 445 or quickly. (See Section 5.2 and Section 5.3, in addition to 446 [RFC5218]). 448 4.3. Providing Benefits During Partial Deployment 450 Some proposals require that all path elements along the full length 451 of the path must be upgraded to support a new technique, before any 452 benefits can be seen. This is likely to require coordination between 453 operators who control a subset of path elements, and between 454 operators and end users if endpoint upgrades are required. If a 455 technique provides benefits when only a part of the path has been 456 upgraded, this is likely to encourage adoption and deployment. (See 457 Section 5.2 and Section 5.3, in addition to [RFC5218]). 459 4.4. Outperforming End-to-end Protocol Mechanisms 461 Adaptive end-to-end protocol mechanisms may respond to feedback 462 quickly enough that the additional realizable benefit from a new Path 463 Aware mechanism that tries to manipulate nodes along a path, or 464 observe the attributes of nodes along a path, may be much smaller 465 than anticipated (Section 5.3 and Section 5.5). 467 4.5. Paying for Path Aware Techniques 469 "Follow the money." If operators can't charge for a Path Aware 470 technique to recover the costs of deploying it, the benefits to the 471 operator must be really significant. Corollary: If operators charge 472 for a Path Aware technique, the benefits to users of that Path Aware 473 technique must be significant enough to justify the cost. (See 474 Section 5.1, Section 5.2, and Section 5.5). 476 4.6. Impact on Operational Practices 478 Impact of a Path Aware technique requiring changes to operational 479 practices can affect how quickly or widely a promising technique is 480 deployed. The impacts of these changes may make deployment more 481 likely, but often discourage deployment. (See Section 5.6, including 482 Section 5.6.3). 484 4.7. Per-connection State 486 Per-connection state in intermediate nodes has been an impediment to 487 adoption and deployment in the past, because of added cost and 488 complexity. Often, similar benefits can be achieved with much less 489 finely-grained state. This is especially true as we move from the 490 edge of the network, further into the routing core (See Section 5.1 491 and Section 5.2). 493 4.8. Keeping Traffic on Fast-paths 495 Many modern platforms, especially high-end routers, have been 496 designed with hardware that can make simple per-packet forwarding 497 decisions ("fast-paths"), but have not been designed to make heavy 498 use of in-band mechanisms such as IPv4 and IPv6 Router Alert Options 499 (RAO) that require more processing to make forwarding decisions. 500 Packets carrying in-band mechanisms are diverted to other processors 501 in the router with much lower packet processing rates. Operators can 502 be reluctant to deploy techniques that rely heavily on in-band 503 mechanisms because they may significantly reduce packet throughput. 504 (See Section 5.7). 506 4.9. Endpoints Trusting Intermediate Nodes 508 If intermediate nodes along the path can't be trusted, it's unlikely 509 that endpoints will rely on signals from intermediate nodes to drive 510 changes to endpoint behaviors. (See Section 5.5, Section 5.4). We 511 note that "trust" is not binary - one, low, level of trust applies 512 when a node issuing a message can confirm that it has visibility of 513 the packets on the path it is seeking to control [RFC8085] (e.g., an 514 ICMP message included a quoted packet from the source). A higher 515 level of trust can arise when an endpoint has established a short 516 term, or even long term, trust relationship with network nodes. 518 4.10. Intermediate Nodes Trusting Endpoints 520 If the endpoints do not have any trust relationship with the 521 intermediate nodes along a path, operators have been reluctant to 522 deploy techniques that rely on endpoints sending unauthenticated 523 control signals to routers. (See Section 5.2 and Section 5.7. We 524 also note this still remains a factor hindering deployment of 525 DiffServ). 527 4.11. Reacting to Distant Signals 529 Because the Internet is a distributed system, if the distance that 530 information from distant path elements travels to a Path Aware host 531 is sufficiently large, the information may no longer accurately 532 represent the state and situation at the distant host or elements 533 along the path when it is received locally. In this case, the 534 benefit that a Path Aware technique provides will be inconsistent, 535 and may not always be beneficial. (See Section 5.3). 537 4.12. Support in Endpoint Protocol Stacks 539 Just because a protocol stack provides a new feature/signal does not 540 mean that applications will use the feature/signal. Protocol stacks 541 may not know how to effectively utilize Path-Aware techniques, 542 because the protocol stack may require information from applications 543 to permit the technique to work effectively, but applications may not 544 a-priori know that information. Even if the application does know 545 that information, the de-facto sockets API has no way of signaling 546 application expectations for the network path to the protocol stack. 547 In order for applications to provide these expectations to protocol 548 stacks, we need an API that signals more than the packets to be sent. 549 (See Section 5.1 and Section 5.2). 551 5. Contributions 553 Contributions on these Path Aware networking techniques were analyzed 554 to arrive at the Lessons Learned captured in Section 4. 556 Our expectation is that most readers will not need to read through 557 this section carefully, but we wanted to record these hard-fought 558 lessons as a service to others who may revisit this document, so 559 they'll have the details close at hand. 561 5.1. Stream Transport (ST, ST2, ST2+) 563 The suggested references for Stream Transport are: 565 * ST - A Proposed Internet Stream Protocol [IEN-119] 567 * Experimental Internet Stream Protocol, Version 2 (ST-II) [RFC1190] 569 * Internet Stream Protocol Version 2 (ST2) Protocol Specification - 570 Version ST2+ [RFC1819] 572 The first version of Stream Transport, ST [IEN-119], was published in 573 the late 1970's and was implemented and deployed on the ARPANET at 574 small scale. It was used throughout the 1980's for experimental 575 transmission of voice, video, and distributed simulation. 577 The second version of the ST specification (ST2) [RFC1190] [RFC1819] 578 was an experimental connection-oriented internetworking protocol that 579 operated at the same layer as connectionless IP. ST2 packets could 580 be distinguished by their IP header protocol numbers (IP, at that 581 time, used protocol number 4, while ST2 used protocol number 5). 583 ST2 used a control plane layered over IP to select routes and reserve 584 capacity for real-time streams across a network path, based on a flow 585 specification communicated by a separate protocol. The flow 586 specification could be associated with QoS state in routers, 587 producing an experimental resource reservation protocol. This 588 allowed ST2 routers along a path to offer end-to-end guarantees, 589 primarily to satisfy the QoS requirements for realtime services over 590 the Internet. 592 5.1.1. Reasons for Non-deployment 594 Although implemented in a range of equipment, ST2 was not widely used 595 after completion of the experiments. It did not offer the 596 scalability and fate-sharing properties that have come to be desired 597 by the Internet community. 599 The ST2 protocol is no longer in use. 601 5.1.2. Lessons Learned. 603 As time passed, the trade-off between router processing and link 604 capacity changed. Links became faster and the cost of router 605 processing became comparatively more expensive. 607 The ST2 control protocol used "hard state" - once a route was 608 established, and resources were reserved, routes and resources 609 existing until they were explicitly released via signaling. A soft- 610 state approach was thought superior to this hard-state approach, and 611 led to development of the IntServ model described in Section 5.2. 613 5.2. Integrated Services (IntServ) 615 The suggested references for IntServ are: 617 * RFC 1633 Integrated Services in the Internet Architecture: an 618 Overview [RFC1633] 620 * RFC 2211 Specification of the Controlled-Load Network Element 621 Service [RFC2211] 623 * RFC 2212 Specification of Guaranteed Quality of Service [RFC2212] 625 * RFC 2215 General Characterization Parameters for Integrated 626 Service Network Elements [RFC2215] 628 * RFC 2205 Resource ReSerVation Protocol (RSVP) [RFC2205] 629 In 1994, when the IntServ architecture document [RFC1633] was 630 published, real-time traffic was first appearing on the Internet. At 631 that time, bandwidth was still a scarce commodity. Internet Service 632 Providers built networks over DS3 (45 Mbps) infrastructure, and sub- 633 rate (< 1 Mpbs) access was common. Therefore, the IETF anticipated a 634 need for a fine-grained QoS mechanism. 636 In the IntServ architecture, some applications can require service 637 guarantees. Therefore, those applications use the Resource 638 Reservation Protocol (RSVP) [RFC2205] to signal QoS reservations 639 across network paths. Every router in the network maintains per-flow 640 soft-state to a) perform call admission control and b) deliver 641 guaranteed service. 643 Applications use Flow Specification (Flow Specs) [RFC2210] to 644 describe the traffic that they emit. RSVP reserves capacity for 645 traffic on a per Flow Spec basis. 647 5.2.1. Reasons for Non-deployment 649 Although IntServ has been used in enterprise and government networks, 650 IntServ was never widely deployed on the Internet because of its 651 cost. The following factors contributed to operational cost: 653 * IntServ must be deployed on every router that is on a path where 654 IntServ is to be used 656 * IntServ maintained per flow state 658 As IntServ was being discussed, the following occurred: 660 * For many expected uses, it became more cost effective to solve the 661 QoS problem by adding bandwidth. Between 1994 and 2000, Internet 662 Service Providers upgraded their infrastructures from DS3 (45 663 Mbps) to OC-48 (2.4 Gbps). This meant that even if an endpoint 664 was using IntServ in an IntServ-enabled network, its requests 665 would never be denied, so endpoints and Internet Service Providers 666 had little reason to enable IntServ. 668 * DiffServ [RFC2475] offered a more cost-effective, albeit less 669 fine-grained, solution to the QoS problem. 671 5.2.2. Lessons Learned. 673 The following lessons were learned: 675 * Any mechanism that requires every onpath router to maintain per- 676 flow state is not likely to succeed, unless the additional cost 677 for offering the feature can be recovered from the user. 679 * Any mechanism that requires an operator to upgrade all of its 680 routers is not likely to succeed, unless the additional cost for 681 offering the feature can be recovered from the user. 683 In environments where IntServ has been deployed, trust relationships 684 with endpoints are very different from trust relationships on the 685 Internet itself, and there are often clearly-defined hierarchies in 686 Service Level Agreements (SLAs), and well-defined transport flows 687 operating with pre-determined capacity and latency requirements over 688 paths where capacity or other attributes are constrained. 690 IntServ was never widely deployed to manage capacity across the 691 Internet. However, the technique that it produced was deployed for 692 reasons other than bandwidth management. RSVP is widely deployed as 693 an MPLS signaling mechanism. BGP reuses the RSVP concept of Filter 694 Specs to distribute firewall filters, although they are called Flow 695 Spec Component Types in BGP [RFC5575]. 697 5.3. Quick-Start TCP 699 The suggested references for Quick-Start TCP are: 701 * Quick-Start for TCP and IP [RFC4782] 703 * Determining an appropriate initial sending rate over an 704 underutilized network path [SAF07] 706 * Fast Startup Internet Congestion Control for Broadband Interactive 707 Applications [Sch11] 709 * Using Quick-Start to enhance TCP-friendly rate control performance 710 in bidirectional satellite networks [QS-SAT] 712 Quick-Start [RFC4782] is an Experimental TCP extension that leverages 713 support from the routers on the path to determine an allowed initial 714 sending rate for a path through the Internet, either at the start of 715 data transfers or after idle periods. Without information about the 716 path, a sender cannot easily determine an appropriate initial sending 717 rate. The default TCP congestion control therefore uses the safe but 718 time-consuming slow-start algorithm [RFC5681]. With Quick-Start, 719 connections are allowed to use higher initial sending rates if there 720 is significant unused bandwidth along the path, and if the sender and 721 all of the routers along the path approve the request. 723 By examining the Time To Live (TTL) field in Quick-Start packets, a 724 sender can determine if routers on the path have approved the Quick- 725 Start request. However, this method is unable to take into account 726 the routers hidden by tunnels or other network nodes invisible at the 727 IP layer. 729 The protocol also includes a nonce that provides protection against 730 cheating routers and receivers. If the Quick-Start request is 731 explicitly approved by all routers along the path, the TCP host can 732 send at up to the approved rate; otherwise TCP would use the default 733 congestion control. Quick-Start requires modifications in the 734 involved end-systems as well in routers. Due to the resulting 735 deployment challenges, Quick-Start was only proposed in [RFC4782] for 736 controlled environments. 738 The Quick-Start mechanism is a lightweight, coarse-grained, in-band, 739 network-assisted fast startup mechanism. The benefits are studied by 740 simulation in a research paper [SAF07] that complements the protocol 741 specification. The study confirms that Quick-Start can significantly 742 speed up mid-sized data transfers. That paper also presents router 743 algorithms that do not require keeping per-flow state. Later studies 744 [Sch11] comprehensively analyzes Quick-Start with a full Linux 745 implementation and with a router fast path prototype using a network 746 processor. In both cases, Quick-Start could be implemented with 747 limited additional complexity. 749 5.3.1. Reasons for Non-deployment 751 However, experiments with Quick-Start in [Sch11] revealed several 752 challenges: 754 * Having information from the routers along the path can reduce the 755 risk of congestion, but cannot avoid it entirely. Determining 756 whether there is unused capacity is not trivial in actual router 757 and host implementations. Data about available capacity visible 758 at the IP layer may be imprecise, and due to the propagation 759 delay, information can already be outdated when it reaches a 760 sender. There is a trade-off between the speedup of data 761 transfers and the risk of congestion even with Quick-Start. This 762 could be mitigated by only allowing Quick-Start to access a 763 proportion of the unused capacity along a path. 765 * For scalable router fast path implementation, it is important to 766 enable parallel processing of packets, as this is a widely used 767 method e.g. in network processors. One challenge is 768 synchronization of information between packets that are processed 769 in parallel, which should be avoided as much as possible. 771 * Only some types of application traffic can benefit from Quick- 772 Start. Capacity needs to be requested and discovered. The 773 discovered capacity needs to be utilized by the flow, or it 774 implicitly becomes available for other flows. Failing to use the 775 requested capacity may have already reduced the pool of Quick- 776 Start capacity that was made available to other competing Quick- 777 Start requests. The benefit is greatest when senders use this 778 only for bulk flows and avoid sending unnecessary Quick-Start 779 requests, e.g. for flows that only send a small amount of data. 780 Choosing an appropriate request size requires application-internal 781 knowledge that is not commonly expressed by the transport API. 782 How a sender can determine the rate for an initial Quick-Start 783 request is still a largely unsolved problem. 785 There is no known deployment of Quick-Start for TCP or other IETF 786 transports. 788 5.3.2. Lessons Learned 790 Some lessons can be learned from Quick-Start. Despite being a very 791 light-weight protocol, Quick-Start suffers from poor incremental 792 deployment properties, both regarding the required modifications in 793 network infrastructure as well as its interactions with applications. 794 Except for corner cases, congestion control can be quite efficiently 795 performed end-to-end in the Internet, and in modern stacks there is 796 not much room for significant improvement by additional network 797 support. 799 After publication of the Quick-Start specification, there have been 800 large-scale experiments with an initial window of up to 10 MSS 801 [RFC6928]. This alternative "IW10" approach can also ramp-up data 802 transfers faster than the standard congestion control, but it only 803 requires sender-side modifications. As a result, this approach can 804 be easier and incrementally deployed in the Internet. While 805 theoretically Quick-Start can outperform "IW10", the improvement in 806 completion time for data transfer times can, in many cases, be small. 807 After publication of [RFC6928], most modern TCP stacks have increased 808 their default initial window. 810 5.4. ICMP Source Quench 812 The suggested references for ICMP Source Quench are: 814 * INTERNET CONTROL MESSAGE PROTOCOL [RFC0792] 815 The ICMP Source Quench message [RFC0792] allowed an on-path router to 816 request the source of a flow to reduce its sending rate. This method 817 allowed a router to provide an early indication of impending 818 congestion on a path to the sources that contribute to that 819 congestion. 821 5.4.1. Reasons for Non-deployment 823 This method was deployed in Internet routers over a period of time, 824 the reaction of endpoints to receiving this signal has varied. For 825 low speed links, with low multiplexing of flows the method could be 826 used to regulate (momentarily reduce) the transmission rate. 827 However, the simple signal does not scale with link speed, or the 828 number of flows sharing a link. 830 The approach was overtaken by the evolution of congestion control 831 methods in TCP [RFC2001], and later also by other IETF transports. 832 Because these methods were based upon measurement of the end-to-end 833 path and an algorithm in the endpoint, they were able to evolve and 834 mature more rapidly than methods relying on interactions between 835 operational routers and endpoint stacks. 837 After ICMP Source Quench was specified, the IETF began to recommend 838 that transports provide end-to-end congestion control [RFC2001]. The 839 Source Quench method has been obsoleted by the IETF [RFC6633], and 840 both hosts and routers must now silently discard this message. 842 5.4.2. Lessons Learned 844 This method had several problems: 846 First, [RFC0792] did not sufficiently specify how the sender would 847 react to the ICMP Source Quench signal from the path (e.g., 848 [RFC1016]). There was ambiguity in how the sender should utilize 849 this additional information. This could lead to unfairness in the 850 way that receivers (or routers) responded to this message. 852 Second, while the message did provide additional information, the 853 Explicit Congestion Notification (ECN) mechanism [RFC3168] provided a 854 more robust and informative signal for network nodes to provide early 855 indication that a path has become congested. 857 The mechanism originated at a time when the Internet trust model was 858 very different. Most endpoint implementations did not attempt to 859 verify that the message originated from an on-path node before they 860 utilized the message. This made it vulnerable to denial of service 861 attacks. In theory, routers might have chosen to use the quoted 862 packet contained in the ICMP payload to validate that the message 863 originated from an on-path node, but this would have increased per- 864 packet processing overhead for each router along the path, would have 865 required transport functionality in the router to verify whether the 866 quoted packet header corresponded to a packet the router had sent. 867 In addition, section 5.2 of [RFC4443] noted ICMPv6-based attacks on 868 hosts that would also have threatened routers processing ICMPv6 869 Source Quench payloads. As time passed, it became increasingly 870 obvious that the lack of validation of the messages exposed receivers 871 to a security vulnerability where the messages could be forged to 872 create a tangible denial of service opportunity. 874 5.5. Triggers for Transport (TRIGTRAN) 876 The suggested references for TRIGTRAN are: 878 * TRIGTRAN BOF at IETF 55 [TRIGTRAN-55] 880 * TRIGTRAN BOF at IETF 56 [TRIGTRAN-56] 882 TCP [RFC0793] has a well-known weakness - the end-to-end flow control 883 mechanism has only a single signal, the loss of a segment, and TCP 884 implementations since the late 1980s have interpreted the loss of a 885 segment as evidence that the path between two endpoints may have 886 become congested enough to exhaust buffers on intermediate hops, so 887 that the TCP sender should "back off" - reduce its sending rate until 888 it knows that its segments are now being delivered without loss 889 [RFC5681]. More modern TCP stacks have added a growing array of 890 strategies about how to establish the sending rate [RFC5681], but 891 when a path is no longer operational, TCP would continue to retry 892 transmissions, which would fail, again, and double their 893 Retransmission Time Out (RTO) timers with each failed transmission, 894 with the result that TCP would wait many seconds before retrying a 895 segment, even if the path becomes operational while the sender is 896 waiting for its next retry. 898 The thinking behind TRIGTRAN was that if a path completely stopped 899 working because a link along the path was "down", somehow something 900 along the path could signal TCP when that link returned to service, 901 and the sending TCP could retry immediately, without waiting for a 902 full retransmission timeout (RTO) period. 904 5.5.1. Reasons for Non-deployment 906 The early dreams for TRIGTRAN were dashed because of an assumption 907 that TRIGTRAN triggers would be unauthenticated. This meant that any 908 "safe" TRIGTRAN mechanism would have relied on a mechanism such as 909 setting the IPv4 TTL or IPv6 Hop Count to 255 at a sender and testing 910 that it was 254 upon receipt, so that a receiver could verify that a 911 signal was generated by an adjacent sender known to be on the path 912 being used, and not some unknown sender which might not even be on 913 the path (e.g., "The Generalized TTL Security Mechanism (GTSM)" 914 [RFC5082]). This situation is very similar to the case for ICMP 915 Source Quench messages as described in Section 5.4, which were also 916 unauthenticated, and could be sent by an off-path attacker, resulting 917 in deprecation of ICMP Source Quench message processing [RFC6633]. 919 TRIGTRAN's scope shrunk from "the path is down" to "the first-hop 920 link is down". 922 But things got worse. 924 Because TRIGTRAN triggers would only be provided when the first-hop 925 link was "down", TRIGTRAN triggers couldn't replace normal TCP 926 retransmission behavior if the path failed because some link further 927 along the network path was "down". So TRIGTRAN triggers added 928 complexity to an already complex TCP state machine, and did not allow 929 any existing complexity to be removed. 931 There was also an issue that the TRIGTRAN signal was not sent in 932 response to a specific host that had been sending packets, and was 933 instead a signal that stimulated a response by any sender on the 934 link. This needs to scale when there are multiple flows trying to 935 use the same resource, yet the sender of a trigger has no 936 understanding how many of the potential traffic sources will respond 937 by sending packets - if recipients of the signal back-off their 938 responses to a trigger to improve scaling, then that immediately 939 mitigates the benefit of the signal. 941 Finally, intermediate forwarding nodes required modification to 942 provide TRIGTRAN triggers, but operators couldn't charge for TRIGTRAN 943 triggers, so there was no way to recover the cost of modifying, 944 testing, and deploying updated intermediate nodes. 946 Two TRIGTRAN BOFs were held, at IETF 55 [TRIGTRAN-55] and IETF 56 947 [TRIGTRAN-56], but this work was not chartered, and there was no 948 interest in deploying TRIGTRAN unless it was chartered and 949 standardized in the IETF. 951 5.5.2. Lessons Learned. 953 The reasons why this work was not chartered, much less deployed, 954 provide several useful lessons for researchers. 956 * TRIGTRAN started with a plausible value proposition, but 957 networking realities in the early 2000s forced reductions in scope 958 that led directly to reductions in potential benefits, but no 959 corresponding reductions in costs and complexity. 961 * These reductions in scope were the direct result of an inability 962 for hosts to trust or authenticate TRIGTRAN signals they received 963 from the network. 965 * Operators did not believe they could charge for TRIGTRAN 966 signaling, because first-hop links didn't fail frequently, and 967 TRIGTRAN provided no reduction in operating expenses, so there was 968 little incentive to purchase and deploy TRIGTRAN-capable network 969 equipment. 971 It is also worth noting that the targeted environment for TRIGTRAN in 972 the late 1990s contained links with a relatively small number of 973 directly-connected hosts - for instance, cellular or satellite links. 974 The transport community was well aware of the dangers of sender 975 synchronization based on multiple senders receiving the same stimulus 976 at the same time, but the working assumption for TRIGTRAN was that 977 there wouldn't be enough senders for this to be a meaningful problem. 978 In the 2010s, it is common for a single "link" to support many 979 senders and receivers on a single link, likely requiring TRIGTRAN 980 senders to wait some random amount of time before sending after 981 receiving a TRIGTRAN signal, which would have reduced the benefits of 982 TRIGTRAN even more. 984 5.6. Shim6 986 The suggested references for Shim6 are: 988 * Shim6: Level 3 Multihoming Shim Protocol for IPv6 [RFC5533] 990 The IPv6 routing architecture [RFC1887] assumed that most sites on 991 the Internet would be identified by Provider Assigned IPv6 prefixes, 992 so that Default-Free Zone routers only contained routes to other 993 providers, resulting in a very small IPv6 global routing table. 995 For a single-homed site, this could work well. A multihomed site 996 with only one upstream provider could also work well, although BGP 997 multihoming from a single upstream provider was often a premium 998 service (costing more than twice as much as two single-homed sites), 999 and if the single upstream provider went out of service, all of the 1000 multihomed paths could fail simultaneously. 1002 IPv4 sites often multihomed by obtaining Provider Independent 1003 prefixes, and advertising these prefixes through multiple upstream 1004 providers. With the assumption that any multihomed IPv4 site would 1005 also multihome in IPv6, it seemed likely that IPv6 routing would be 1006 subject to the same pressures to announce Provider Independent 1007 prefixes, resulting in a global IPv6 routing table that exhibited the 1008 same explosive growth as the global IPv4 routing table. During the 1009 early 2000s, work began on a protocol that would provide multihoming 1010 for IPv6 sites without requiring sites to advertise Provider 1011 Independent prefixes into the IPv6 global routing table. 1013 This protocol, called Shim6, allowed two endpoints to exchange 1014 multiple addresses ("Locators") that all mapped to the same endpoint 1015 ("Identity"). After an endpoint learned multiple Locators for the 1016 other endpoint, it could send to any of those Locators with the 1017 expectation that those packets would all be delivered to the endpoint 1018 with the same Identity. Shim6 was an example of an "Identity/Locator 1019 Split" protocol. 1021 Shim6, as defined in [RFC5533] and related RFCs, provided a workable 1022 solution for IPv6 multihoming using Provider Assigned prefixes, 1023 including capability discovery and negotiation, and allowing end-to- 1024 end application communication to continue even in the face of path 1025 failure, because applications don't see Locator failures, and 1026 continue to communicate with the same Identity using a different 1027 Locator. 1029 5.6.1. Reasons for Non-deployment 1031 Note that the problem being addressed was "site multihoming", but 1032 Shim6 was providing "host multihoming". That meant that the decision 1033 about what path would be used was under host control, not under 1034 router control. 1036 Although more work could have been done to provide a better technical 1037 solution, the biggest impediments to Shim6 deployment were 1038 operational and business considerations. These impediments were 1039 discussed at multiple network operator group meetings, including 1040 [Shim6-35] at [NANOG-35]. 1042 The technique issues centered around concerns that Shim6 relied on 1043 the host to track all the connections, while also tracking Identity/ 1044 Locator mappings in the kernel, and tracking failures to recognize 1045 that a backup path has failed. 1047 The operator issues centered around concerns that operators were 1048 performing traffic engineering, but would have no visibility or 1049 control over hosts when they chose to begin using another path, and 1050 relying on hosts to engineer traffic exposed their networks to 1051 oscillation based on feedback loops, as hosts move from path to path. 1052 At a minimum, traffic engineering policies must be pushed down to 1053 individual hosts. In addition, firewalls that expected to find a 1054 transport-level protocol header in the IP payload, would see a Shim6 1055 Identity header, and be unable to perform transport-protocol-based 1056 firewalling functions because its normal processing logic would not 1057 look past the Identity header. 1059 The business issues centered removing or reducing the ability to sell 1060 BGP multihoming service, which is often more expensive than single- 1061 homed connectivity. 1063 5.6.2. Lessons Learned 1065 It is extremely important to take operational concerns into account 1066 when a path-aware protocol is making decisions about path selection 1067 that may conflict with existing operational practices and business 1068 considerations. 1070 5.6.3. Addendum on MultiPath TCP 1072 During discussions in the PANRG session at IETF 103 [PANRG-103-Min], 1073 Lars Eggert, past Transport Area Director, pointed out that during 1074 charter discussions for the Multipath TCP working group [MP-TCP], 1075 operators expressed concerns that customers could use Multipath TCP 1076 to loadshare TCP connections across operators simultaneously and 1077 compare passive performance measurements across network paths in real 1078 time, changing the balance of power in those business relationships. 1079 Although the Multipath TCP working group was chartered, this concern 1080 could have acted as an obstacle to deployment. 1082 Operator objections to Shim6 were focused on technical concerns, but 1083 this concern could have also been an obstacle to Shim6 deployment if 1084 the technical concerns had been overcome. 1086 5.7. Next Steps in Signaling (NSIS) 1088 The suggested references for Next Steps in Signaling (NSIS) are: 1090 * the concluded working group charter [NSIS-CHARTER-2001] 1092 * GIST: General Internet Signalling Transport [RFC5971] 1094 * NAT/Firewall NSIS Signaling Layer Protocol (NSLP) [RFC5973] 1096 * NSIS Signaling Layer Protocol (NSLP) for Quality-of-Service 1097 Signaling [RFC5974] 1099 * Authorization for NSIS Signaling Layer Protocols [RFC5981] 1101 The NSIS Working Group worked on signaling techniques for network 1102 layer resources (e.g., QoS resource reservations, Firewall and NAT 1103 traversal). 1105 When RSVP [RFC2205] was used in deployments, a number of questions 1106 came up about its perceived limitations and potential missing 1107 features. The issues noted in the NSIS Working Group charter 1108 [NSIS-CHARTER-2001] include interworking between domains with 1109 different QoS architectures, mobility and roaming for IP interfaces, 1110 and complexity. Later, the lack of security in RSVP was also 1111 recognized ([RFC4094]). 1113 The NSIS Working Group was chartered to tackle those issues and 1114 initially focused on QoS signaling as its primary use case. However, 1115 over time a new approach evolved that introduced a modular 1116 architecture using application-specific signaling protocols (the NSIS 1117 Signaling Layer Protocol (NSLP)) on top of a generic signaling 1118 transport protocol (the NSIS Transport Layer Protocol (NTLP)). 1120 The NTLP is defined in [RFC5971]. Two NSLPs are defined: the NSIS 1121 Signaling Layer Protocol (NSLP) for Quality-of-Service Signaling 1122 [RFC5974] as well as the NAT/Firewall NSIS Signaling Layer Protocol 1123 (NSLP) [RFC5973]. 1125 5.7.1. Reasons for Non-deployment 1127 The obstacles for deployment can be grouped into implementation- 1128 related aspects and operational aspects. 1130 * Implementation-related aspects: 1132 Although NSIS provides benefits with respect to flexibility, 1133 mobility, and security compared to other network signaling 1134 techniques, hardware vendors were reluctant to deploy this solution, 1135 because it would require additional implementation effort and would 1136 result in additional complexity for router implementations. 1138 The NTLP mainly operates as path-coupled signaling protocol, i.e, its 1139 messages are processed at the intermediate node's control plane that 1140 are also forwarding the data flows. This requires a mechanism to 1141 intercept signaling packets while they are forwarded in the same 1142 manner (especially along the same path) as data packets. NSIS uses 1143 the IPv4 and IPv6 Router Alert Option (RAO) to allow for interception 1144 of those path-coupled signaling messages, and this technique requires 1145 router implementations to correctly understand and implement the 1146 handling of RAOs, e.g., to only process packet with RAOs of interest 1147 and to leave packets with irrelevant RAOs in the fast forwarding 1148 processing path (a comprehensive discussion of these issues can be 1149 found in [RFC6398]). The latter was an issue with some router 1150 implementations at the time of standardization. 1152 Another reason is that path-coupled signaling protocols that interact 1153 with routers and request manipulation of state at these routers (or 1154 any other network element in general) are under scrutiny: a packet 1155 (or sequence of packets) out of the mainly untrusted data path is 1156 requesting creation and manipulation of network state. This is seen 1157 as potentially dangerous (e.g., opens up a Denial of Service (DoS) 1158 threat to a router's control plane) and difficult for an operator to 1159 control. Path-coupled signaling approaches were considered 1160 problematic (see also section 3 of [RFC6398]). There are 1161 recommendations on how to secure NSIS nodes and deployments (e.g., 1162 [RFC5981]). 1164 * Operational Aspects: 1166 NSIS not only required trust between customers and their provider, 1167 but also among different providers. Especially, QoS signaling 1168 techniques would require some kind of dynamic service level agreement 1169 support that would imply (potentially quite complex) bilateral 1170 negotiations between different Internet service providers. This 1171 complexity was not considered to be justified and increasing the 1172 bandwidth (and thus avoiding bottlenecks) was cheaper than actively 1173 managing network resource bottlenecks by using path-coupled QoS 1174 signaling techniques. Furthermore, an end-to-end path typically 1175 involves several provider domains and these providers need to closely 1176 cooperate in cases of failures. 1178 5.7.2. Lessons Learned 1180 One goal of NSIS was to decrease the complexity of the signaling 1181 protocol, but a path-coupled signaling protocol comes with the 1182 intrinsic complexity of IP-based networks, beyond the complexity of 1183 the signaling protocol itself. Sources of intrinsic complexity 1184 include: 1186 * the presence of asymmetric routes between endpoints and routers 1188 * the lack of security and trust at large in the Internet 1189 infrastructure 1191 * the presence of different trust boundaries 1193 * the effects of best-effort networks (e.g., robustness to packet 1194 loss) 1196 * divergence from the fate sharing principle (e.g., state within the 1197 network). 1199 Any path-coupled signaling protocol has to deal with these realities. 1201 Operators view the use of IPv4 and IPv6 Router Alert Option (RAO) to 1202 signal routers along the path from end systems with suspicion, 1203 because these end systems are usually not authenticated and heavy use 1204 of RAOs can easily increase the CPU load on routers that are designed 1205 to process most packets using a hardware "fast path" and diverting 1206 packets containing RAO to a slower, more capable processor. 1208 5.8. IPv6 Flow Label 1210 The suggested references for IPv6 Flow Label are: 1212 * IPv6 Flow Label Specification [RFC6437] 1214 IPv6 specifies a 20-bit field Flow Label field [RFC6437], included in 1215 the fixed part of the IPv6 header and hence present in every IPv6 1216 packet. An endpoint sets the value in this field to one of a set of 1217 pseudo-randomly assigned values. If a packet is not part of any 1218 flow, the flow label value is set to zero [RFC3697]. A number of 1219 Standards Track and Best Current Practice RFCs (e.g., [RFC8085], 1220 [RFC6437], [RFC6438]) encourage IPv6 endpoints to set a non-zero 1221 value in this field. A multiplexing transport could choose to use 1222 multiple flow labels to allow the network to independently forward 1223 its subflows, or to use one common value for the traffic aggregate. 1224 The flow label is present in all fragments. IPsec was originally put 1225 forward as one important use-case for this mechanism and does encrypt 1226 the field [RFC6438]. 1228 Once set, the flow label can provide information that can help inform 1229 network nodes about subflows present at the transport layer, without 1230 needing to interpret the setting of upper layer protocol fields 1231 [RFC6294]. This information can also be used to coordinate how 1232 aggregates of transport subflows are grouped when queued in the 1233 network and to select appropriate per-flow forwarding when choosing 1234 between alternate paths [RFC6438] (e.g. for Equal Cost Multipath 1235 Routing (ECMP) and Link Aggregation (LAG)). 1237 5.8.1. Reasons for Non-deployment 1239 Despite the field being present in every IPv6 packet, the mechanism 1240 did not receive as much use as originally envisioned. One reason is 1241 that to be useful it requires engagement by two different 1242 stakeholders: 1244 * Endpoint Implementation: 1246 For network nodes along a path to utilize the flow label there needs 1247 to be a non-zero value inserted in the field [RFC6437] at the sending 1248 endpoint. There needs to be an incentive for an endpoint to set an 1249 appropriate non-zero value. The value should appropriately reflect 1250 the level of aggregation the traffic expects to be provided by the 1251 network. However, this requires the stack to know granularity at 1252 which flows should be identified (or conversely which flows should 1253 receive aggregated treatment), i.e., which packets carry the same 1254 flow label. Therefore, setting a non-zero value may result in 1255 additional choices that need to be made by an application developer. 1257 Although the standard [RFC3697] forbids any encoding of meaning into 1258 the flow label value, the opportunity to use the flow label as a 1259 covert channel or to signal other meta-information may have raised 1260 concerns about setting a non-zero value [RFC6437]. 1262 Before methods are widely deployed to use this method, there could be 1263 no incentive for an endpoint to set the field. 1265 * Operational support in network nodes: 1267 A benefit can only be realized when a network node along the path 1268 also uses this information to inform its decisions. Network 1269 equipment (routers and/or middleboxes) need to include appropriate 1270 support so they can utilize the field when making decisions about how 1271 to classify flows, or to inform forwarding choices. Use of any 1272 optional feature in a network node also requires corresponding 1273 updates to operational procedures, and therefore is normally only 1274 introduced when the cost can be justified. 1276 A benefit from utilizing the flow label is expected to be increased 1277 quality of experience for applications - but this comes at some 1278 operational cost to an operator, and requires endpoints to set the 1279 field. 1281 5.8.2. Lessons Learned 1283 The flow label is a general purpose header field for use by the path. 1284 Multiple uses have been proposed. One candidate use was to reduce 1285 the complexity of forwarding decisions. However, modern routers can 1286 use a "fast path", often taking advantage of hardware to accelerate 1287 processing. The method can assist in more complex forwarding, such 1288 as ECMP and load balancing. 1290 Although [RFC6437] recommended that endpoints should by default 1291 choose uniformly-distributed labels for their traffic, the 1292 specification permitted an endpoint to choose to set a zero value. 1293 This ability of endpoints to choose to set a flow label of zero has 1294 had consequences on deployability: 1296 * Before wide-scale support by endpoints, it would be impossible to 1297 rely on a non-zero flow label being set. Network nodes therefore 1298 would need to also employ other techniques to realize equivalent 1299 functions. An example of a method is one assuming semantics of 1300 the source port field to provide entropy input to a network-layer 1301 hash. This use of a 5-tuple to classify a packet represents a 1302 layering violation [RFC6294]. When other methods have been 1303 deployed, they increase the cost of deploying standards-based 1304 methods, even though they may offer less control to endpoints and 1305 result in potential interaction with other uses/interpretation of 1306 the field. 1308 * Even though the flow label is specified as an end-to-end field, 1309 some network paths have been observed to not transparently forward 1310 the flow label. This could result from non-conformant equipment, 1311 or could indicate that some operational networks have chosen to 1312 re-use the protocol field for other (e.g. internal purposes). 1313 This results in lack of transparency, and a deployment hurdle to 1314 endpoints expecting that they can set a flow label that is 1315 utilized by the network. The more recent practice of "greasing" 1316 [GREASE] would suggest that a different outcome could have been 1317 achieved if endpoints were always required to set a non-zero 1318 value. 1320 * [RFC1809] noted that setting the choice of the flow label value 1321 can depend on the expectations of the traffic generated by an 1322 application, which suggests an API should be presented to control 1323 the setting or policy that is used. However, many currently 1324 available APIs do not have this support. 1326 A growth in the use of encrypted transports, (e.g. QUIC [QUIC-WG]) 1327 seems likely to raise similar issues to those discussed above and 1328 could motivate renewed interest in utilizing the flow label. 1330 6. Security Considerations 1332 This document describes Path Aware techniques that were not adopted 1333 and widely deployed on the Internet, so it doesn't affect the 1334 security of the Internet. 1336 If this document meets its goals, we may develop new techniques for 1337 Path Aware Networking that would affect the security of the Internet, 1338 but security considerations for those techniques will be described in 1339 the corresponding RFCs that specify them. 1341 7. IANA Considerations 1343 This document makes no requests of IANA. 1345 8. Acknowledgments 1347 Initial material for Section 5.1 on ST2 was provided by Gorry 1348 Fairhurst. 1350 Initial material for Section 5.2 on IntServ was provided by Ron 1351 Bonica. 1353 Initial material for Section 5.3 on Quick-Start TCP was provided by 1354 Michael Scharf, who also provided suggestions to improve this section 1355 after it was edited. 1357 Initial material for Section 5.4 on ICMP Source Quench was provided 1358 by Gorry Fairhurst. 1360 Initial material for Section 5.5 on Triggers for Transport (TRIGTRAN) 1361 was provided by Spencer Dawkins. 1363 Section 5.6 on Shim6 builds on initial material describing obstacles 1364 provided by Erik Nordmark, with background added by Spencer Dawkins. 1366 Initial material for Section 5.7 on Next Steps In Signaling (NSIS) 1367 was provided by Roland Bless and Martin Stiemerling. 1369 Initial material for Section 5.8 on IPv6 Flow Labels was provided by 1370 Gorry Fairhurst. 1372 Our thanks to C.M. Heard, David Black, Gorry Fairhurst, Joe Touch, 1373 Joeri de Ruiter, Mohamed Boucadair, Roland Bless, Ruediger Geib, 1374 Theresa Enghardt, and Wes Eddy, who provided review comments on 1375 previous versions. 1377 Mallory Knodel reviewed this document for the Internet Engineering 1378 Steering Group, and provided many helpful suggestions. 1380 Special thanks to Adrian Farrel for helping Spencer navigate the 1381 twisty little passages of Flow Specs and Filter Specs in IntServ, 1382 RSVP, MPLS, and BGP. They are all alike, except when they are 1383 different [Colossal-Cave]. 1385 9. Informative References 1387 [Colossal-Cave] 1388 "Wikipedia Page for Colossal Cave Adventure", January 1389 2019, 1390 . 1392 [draft-arkko-arch-internet-threat-model] 1393 Arkko, J., "Changes in the Internet Threat Model", n.d., 1394 . 1397 [draft-farrell-etm] 1398 Farrell, S., "We're gonna need a bigger threat model", 1399 n.d., 1400 . 1402 [GREASE] Thomson, M., "Long-term Viability of Protocol Extension 1403 Mechanisms", July 2019, . 1406 [I-D.irtf-panrg-questions] 1407 Trammell, B., "Current Open Questions in Path Aware 1408 Networking", Work in Progress, Internet-Draft, draft-irtf- 1409 panrg-questions-07, 29 August 2020, . 1412 [IEN-119] Forgie, J., "ST - A Proposed Internet Stream Protocol", 1413 September 1979, 1414 . 1416 [ITAT] "IAB Workshop on Internet Technology Adoption and 1417 Transition (ITAT)", December 2013, 1418 . 1420 [model-t] "Model-t -- Discussions of changes in Internet deployment 1421 patterns and their impact on the Internet threat model", 1422 n.d., . 1424 [MP-TCP] "Multipath TCP Working Group Home Page", n.d., 1425 . 1427 [NANOG-35] "North American Network Operators Group NANOG-35 Agenda", 1428 October 2005, 1429 . 1431 [NSIS-CHARTER-2001] 1432 "Next Steps In Signaling Working Group Charter", March 1433 2011, 1434 . 1436 [PANRG] "Path Aware Networking Research Group (Home Page)", n.d., 1437 . 1439 [PANRG-103-Min] 1440 "Path Aware Networking Research Group - IETF-103 Minutes", 1441 November 2018, 1442 . 1444 [PANRG-105-Min] 1445 "Path Aware Networking Research Group - IETF-105 Minutes", 1446 July 2019, 1447 . 1449 [PANRG-106-Min] 1450 "Path Aware Networking Research Group - IETF-106 Minutes", 1451 November 2019, 1452 . 1454 [PANRG-99] "Path Aware Networking Research Group - IETF-99", July 1455 2017, 1456 . 1458 [PATH-Decade] 1459 Bonaventure, O., "A Decade of Path Awareness", July 2017, 1460 . 1463 [PathProp] Enghardt, T. and C. Kraehenbuehl, "A Vocabulary of Path 1464 Properties", November 2019, . 1467 [QS-SAT] Secchi, R., Sathiaseelan, A., Potorti, F., Gotta, A., and 1468 G. Fairhurst, "Using Quick-Start to enhance TCP-friendly 1469 rate control performance in bidirectional satellite 1470 networks", 2009, 1471 . 1473 [QUIC-WG] "QUIC Working Group Home Page", n.d., 1474 . 1476 [RFC0792] Postel, J., "Internet Control Message Protocol", STD 5, 1477 RFC 792, DOI 10.17487/RFC0792, September 1981, 1478 . 1480 [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, 1481 RFC 793, DOI 10.17487/RFC0793, September 1981, 1482 . 1484 [RFC1016] Prue, W. and J. Postel, "Something a Host Could Do with 1485 Source Quench: The Source Quench Introduced Delay 1486 (SQuID)", RFC 1016, DOI 10.17487/RFC1016, July 1987, 1487 . 1489 [RFC1190] Topolcic, C., "Experimental Internet Stream Protocol: 1490 Version 2 (ST-II)", RFC 1190, DOI 10.17487/RFC1190, 1491 October 1990, . 1493 [RFC1633] Braden, R., Clark, D., and S. Shenker, "Integrated 1494 Services in the Internet Architecture: an Overview", 1495 RFC 1633, DOI 10.17487/RFC1633, June 1994, 1496 . 1498 [RFC1809] Partridge, C., "Using the Flow Label Field in IPv6", 1499 RFC 1809, DOI 10.17487/RFC1809, June 1995, 1500 . 1502 [RFC1819] Delgrossi, L., Ed. and L. Berger, Ed., "Internet Stream 1503 Protocol Version 2 (ST2) Protocol Specification - Version 1504 ST2+", RFC 1819, DOI 10.17487/RFC1819, August 1995, 1505 . 1507 [RFC1887] Rekhter, Y., Ed. and T. Li, Ed., "An Architecture for IPv6 1508 Unicast Address Allocation", RFC 1887, 1509 DOI 10.17487/RFC1887, December 1995, 1510 . 1512 [RFC2001] Stevens, W., "TCP Slow Start, Congestion Avoidance, Fast 1513 Retransmit, and Fast Recovery Algorithms", RFC 2001, 1514 DOI 10.17487/RFC2001, January 1997, 1515 . 1517 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1518 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1519 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1520 September 1997, . 1522 [RFC2210] Wroclawski, J., "The Use of RSVP with IETF Integrated 1523 Services", RFC 2210, DOI 10.17487/RFC2210, September 1997, 1524 . 1526 [RFC2211] Wroclawski, J., "Specification of the Controlled-Load 1527 Network Element Service", RFC 2211, DOI 10.17487/RFC2211, 1528 September 1997, . 1530 [RFC2212] Shenker, S., Partridge, C., and R. Guerin, "Specification 1531 of Guaranteed Quality of Service", RFC 2212, 1532 DOI 10.17487/RFC2212, September 1997, 1533 . 1535 [RFC2215] Shenker, S. and J. Wroclawski, "General Characterization 1536 Parameters for Integrated Service Network Elements", 1537 RFC 2215, DOI 10.17487/RFC2215, September 1997, 1538 . 1540 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1541 and W. Weiss, "An Architecture for Differentiated 1542 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1543 . 1545 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1546 of Explicit Congestion Notification (ECN) to IP", 1547 RFC 3168, DOI 10.17487/RFC3168, September 2001, 1548 . 1550 [RFC3697] Rajahalme, J., Conta, A., Carpenter, B., and S. Deering, 1551 "IPv6 Flow Label Specification", RFC 3697, 1552 DOI 10.17487/RFC3697, March 2004, 1553 . 1555 [RFC4094] Manner, J. and X. Fu, "Analysis of Existing Quality-of- 1556 Service Signaling Protocols", RFC 4094, 1557 DOI 10.17487/RFC4094, May 2005, 1558 . 1560 [RFC4443] Conta, A., Deering, S., and M. Gupta, Ed., "Internet 1561 Control Message Protocol (ICMPv6) for the Internet 1562 Protocol Version 6 (IPv6) Specification", STD 89, 1563 RFC 4443, DOI 10.17487/RFC4443, March 2006, 1564 . 1566 [RFC4782] Floyd, S., Allman, M., Jain, A., and P. Sarolahti, "Quick- 1567 Start for TCP and IP", RFC 4782, DOI 10.17487/RFC4782, 1568 January 2007, . 1570 [RFC5082] Gill, V., Heasley, J., Meyer, D., Savola, P., Ed., and C. 1571 Pignataro, "The Generalized TTL Security Mechanism 1572 (GTSM)", RFC 5082, DOI 10.17487/RFC5082, October 2007, 1573 . 1575 [RFC5218] Thaler, D. and B. Aboba, "What Makes for a Successful 1576 Protocol?", RFC 5218, DOI 10.17487/RFC5218, July 2008, 1577 . 1579 [RFC5533] Nordmark, E. and M. Bagnulo, "Shim6: Level 3 Multihoming 1580 Shim Protocol for IPv6", RFC 5533, DOI 10.17487/RFC5533, 1581 June 2009, . 1583 [RFC5575] Marques, P., Sheth, N., Raszuk, R., Greene, B., Mauch, J., 1584 and D. McPherson, "Dissemination of Flow Specification 1585 Rules", RFC 5575, DOI 10.17487/RFC5575, August 2009, 1586 . 1588 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 1589 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 1590 . 1592 [RFC5971] Schulzrinne, H. and R. Hancock, "GIST: General Internet 1593 Signalling Transport", RFC 5971, DOI 10.17487/RFC5971, 1594 October 2010, . 1596 [RFC5973] Stiemerling, M., Tschofenig, H., Aoun, C., and E. Davies, 1597 "NAT/Firewall NSIS Signaling Layer Protocol (NSLP)", 1598 RFC 5973, DOI 10.17487/RFC5973, October 2010, 1599 . 1601 [RFC5974] Manner, J., Karagiannis, G., and A. McDonald, "NSIS 1602 Signaling Layer Protocol (NSLP) for Quality-of-Service 1603 Signaling", RFC 5974, DOI 10.17487/RFC5974, October 2010, 1604 . 1606 [RFC5981] Manner, J., Stiemerling, M., Tschofenig, H., and R. Bless, 1607 Ed., "Authorization for NSIS Signaling Layer Protocols", 1608 RFC 5981, DOI 10.17487/RFC5981, February 2011, 1609 . 1611 [RFC6294] Hu, Q. and B. Carpenter, "Survey of Proposed Use Cases for 1612 the IPv6 Flow Label", RFC 6294, DOI 10.17487/RFC6294, June 1613 2011, . 1615 [RFC6398] Le Faucheur, F., Ed., "IP Router Alert Considerations and 1616 Usage", BCP 168, RFC 6398, DOI 10.17487/RFC6398, October 1617 2011, . 1619 [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, 1620 "IPv6 Flow Label Specification", RFC 6437, 1621 DOI 10.17487/RFC6437, November 2011, 1622 . 1624 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 1625 for Equal Cost Multipath Routing and Link Aggregation in 1626 Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011, 1627 . 1629 [RFC6633] Gont, F., "Deprecation of ICMP Source Quench Messages", 1630 RFC 6633, DOI 10.17487/RFC6633, May 2012, 1631 . 1633 [RFC6928] Chu, J., Dukkipati, N., Cheng, Y., and M. Mathis, 1634 "Increasing TCP's Initial Window", RFC 6928, 1635 DOI 10.17487/RFC6928, April 2013, 1636 . 1638 [RFC7305] Lear, E., Ed., "Report from the IAB Workshop on Internet 1639 Technology Adoption and Transition (ITAT)", RFC 7305, 1640 DOI 10.17487/RFC7305, July 2014, 1641 . 1643 [RFC7418] Dawkins, S., Ed., "An IRTF Primer for IETF Participants", 1644 RFC 7418, DOI 10.17487/RFC7418, December 2014, 1645 . 1647 [RFC8085] Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage 1648 Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085, 1649 March 2017, . 1651 [RFC8170] Thaler, D., Ed., "Planning for Protocol Adoption and 1652 Subsequent Transitions", RFC 8170, DOI 10.17487/RFC8170, 1653 May 2017, . 1655 [SAAG-105-Min] 1656 "Security Area Open Meeting - IETF-105 Minutes", July 1657 2019, . 1660 [SAF07] Sarolahti, P., Allman, M., and S. Floyd, "Determining an 1661 appropriate sending rate over an underutilized network 1662 path", Computer Networking Volume 51, Number 7, May 2007. 1664 [Sch11] Scharf, M., "Fast Startup Internet Congestion Control for 1665 Broadband Interactive Applications", Ph.D. Thesis, 1666 University of Stuttgart, April 2011. 1668 [Shim6-35] Meyer, D., Huston, G., Schiller, J., and V. Gill, "IAB 1669 IPv6 Multihoming Panel at NANOG 35", NANOG North American 1670 Network Operator Group, October 2005, 1671 . 1673 [TRIGTRAN-55] 1674 "Triggers for Transport BOF at IETF 55", July 2003, 1675 . 1677 [TRIGTRAN-56] 1678 "Triggers for Transport BOF at IETF 56", November 2003, 1679 . 1681 Author's Address 1683 Spencer Dawkins (editor) 1684 Tencent America 1686 Email: spencerdawkins.ietf@gmail.com