idnits 2.17.1 draft-irtf-panrg-what-not-to-do-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (May 13, 2020) is 1443 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-04) exists of draft-iab-use-it-or-lose-it-00 -- Obsolete informational reference (is this intentional?): RFC 793 (Obsoleted by RFC 9293) -- Obsolete informational reference (is this intentional?): RFC 1190 (Obsoleted by RFC 1819) -- Obsolete informational reference (is this intentional?): RFC 2001 (Obsoleted by RFC 2581) -- Obsolete informational reference (is this intentional?): RFC 3697 (Obsoleted by RFC 6437) -- Obsolete informational reference (is this intentional?): RFC 5575 (Obsoleted by RFC 8955) Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 6 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 PANRG S. Dawkins, Ed. 3 Internet-Draft Tencent America 4 Intended status: Informational May 13, 2020 5 Expires: November 14, 2020 7 Path Aware Networking: Obstacles to Deployment (A Bestiary of Roads Not 8 Taken) 9 draft-irtf-panrg-what-not-to-do-08 11 Abstract 13 At the first meeting of the Path Aware Networking Research Group, the 14 research group agreed to catalog and analyze past efforts to develop 15 and deploy Path Aware techniques, most of which were unsuccessful or 16 at most partially successful, in order to extract insights and 17 lessons for path-aware networking researchers. 19 This document contains that catalog and analysis. 21 Status of This Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at https://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on November 14, 2020. 38 Copyright Notice 40 Copyright (c) 2020 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents 45 (https://trustee.ietf.org/license-info) in effect on the date of 46 publication of this document. Please review these documents 47 carefully, as they describe your rights and restrictions with respect 48 to this document. 50 Table of Contents 52 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 53 1.1. Purpose of This Document . . . . . . . . . . . . . . . . 3 54 1.2. What Does "Path Awareness" Mean in this Document? . . . . 4 55 1.3. A Note About Path-Aware Techniques Included In This 56 Document . . . . . . . . . . . . . . . . . . . . . . . . 4 57 1.4. Venue for Discussion of this Document . . . . . . . . . . 4 58 1.5. Architectural Guidance . . . . . . . . . . . . . . . . . 5 59 1.6. Terminology Used in this Document . . . . . . . . . . . . 5 60 2. Summary of Lessons Learned . . . . . . . . . . . . . . . . . 5 61 2.1. Justifying Deployment . . . . . . . . . . . . . . . . . . 6 62 2.2. Providing Benefits for Early Adopters . . . . . . . . . . 6 63 2.3. Providing Benefits During Partial Deployment . . . . . . 6 64 2.4. Outperforming End-to-end Protocol Mechanisms . . . . . . 6 65 2.5. Paying for Path Aware Techniques . . . . . . . . . . . . 6 66 2.6. Impact on Operational Practices . . . . . . . . . . . . . 6 67 2.7. Per-connection State . . . . . . . . . . . . . . . . . . 7 68 2.8. Keeping Traffic on Fast-paths . . . . . . . . . . . . . . 7 69 2.9. Endpoints Trusting Intermediate Nodes . . . . . . . . . . 7 70 2.10. Intermediate Nodes Trusting Endpoints . . . . . . . . . . 7 71 2.11. Reacting to Distant Signals . . . . . . . . . . . . . . . 8 72 2.12. Support in Endpoint Protocol Stacks . . . . . . . . . . . 8 73 3. Do We Understand the Lessons We've Learned? . . . . . . . . . 8 74 4. Template for Contributions . . . . . . . . . . . . . . . . . 11 75 5. Contributions . . . . . . . . . . . . . . . . . . . . . . . . 11 76 5.1. Stream Transport (ST, ST2, ST2+) . . . . . . . . . . . . 11 77 5.1.1. Reasons for Non-deployment . . . . . . . . . . . . . 12 78 5.1.2. Lessons Learned. . . . . . . . . . . . . . . . . . . 12 79 5.2. Integrated Services (IntServ) . . . . . . . . . . . . . . 12 80 5.2.1. Reasons for Non-deployment . . . . . . . . . . . . . 13 81 5.2.2. Lessons Learned. . . . . . . . . . . . . . . . . . . 14 82 5.3. Quick-Start TCP . . . . . . . . . . . . . . . . . . . . . 14 83 5.3.1. Reasons for Non-deployment . . . . . . . . . . . . . 15 84 5.3.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 16 85 5.4. ICMP Source Quench . . . . . . . . . . . . . . . . . . . 16 86 5.4.1. Reasons for Non-deployment . . . . . . . . . . . . . 17 87 5.4.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 17 88 5.5. Triggers for Transport (TRIGTRAN) . . . . . . . . . . . . 18 89 5.5.1. Reasons for Non-deployment . . . . . . . . . . . . . 18 90 5.5.2. Lessons Learned. . . . . . . . . . . . . . . . . . . 19 91 5.6. Shim6 . . . . . . . . . . . . . . . . . . . . . . . . . . 20 92 5.6.1. Reasons for Non-deployment . . . . . . . . . . . . . 21 93 5.6.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 22 94 5.6.3. Addendum on MultiPath TCP . . . . . . . . . . . . . . 22 95 5.7. Next Steps in Signaling (NSIS) . . . . . . . . . . . . . 22 96 5.7.1. Reasons for Non-deployment . . . . . . . . . . . . . 23 97 5.7.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 24 99 5.8. IPv6 Flow Label . . . . . . . . . . . . . . . . . . . . . 25 100 5.8.1. Reasons for Non-deployment . . . . . . . . . . . . . 25 101 5.8.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 26 102 6. Security Considerations . . . . . . . . . . . . . . . . . . . 27 103 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 28 104 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 28 105 9. Informative References . . . . . . . . . . . . . . . . . . . 28 106 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 35 108 1. Introduction 110 At the first meeting of the Path Aware Networking Research Group 111 [PANRG], at IETF 99 [PANRG-99], Olivier Bonaventure led a discussion 112 of "A Decade of Path Awareness" [PATH-Decade], on attempts, which 113 were mostly unsuccessful for a variety of reasons, to exploit Path 114 Aware techniques and achieve a variety of goals over the past decade. 115 At the end of that discussion, two things were abundantly clear. 117 o The Internet community has accumulated considerable experience 118 with many Path Aware techniques over a long period of time, and 120 o Although some path aware techniques have been deployed (for 121 example, Differentiated Services, or DiffServ [RFC2475]), most of 122 these techniques haven't seen widespread adoption and deplyment. 123 Even "successful" techniques like DiffServ can face obstacles that 124 prevents wider usage. The reasons for non-adoption and limited 125 adoption and deployment are many, and are worthy of study. 127 The meta-lessons from that experience were 129 o Path aware networking has been more Research than Engineering, so 130 establishing an IRTF Research Group for Path Aware Networking is 131 the right thing to do [RFC7418]. 133 o Analyzing a catalog of past experience to learn the reasons for 134 non-adoption would be a great first step for the Research Group. 136 Allison Mankin, as IRTF Chair, officially chartered the Path Aware 137 Networking Research Group in July, 2018. 139 This document contains the analysis performed by that research group 140 (Section 2), based on that catalog (Section 5). 142 1.1. Purpose of This Document 144 This Informational document discusses protocol mechanisms considered, 145 and in some cases standardized, by the Internet Engineering Task 146 Force (IETF), and considers Lessons Learned from those mechanisms. 148 The intention is to inform the work of protocol designers, whether in 149 the IRTF, the IETF, or elsewhere in the Internet ecosystem. 151 As an Informational document published in the IRTF stream, this 152 document has no authority beyond the quality of the analysis it 153 contains. 155 1.2. What Does "Path Awareness" Mean in this Document? 157 A word about "Path Awareness" is in order. 159 The Internet architecture assumes a division between the end-to-end 160 functionality of the transport layer and the properties of the path 161 elements between the endpoints. For several decades, various 162 communities have proposed techniques for endpoint transport layers 163 that rely on changing the behavior of path elements along paths, or 164 rely on determining the properties of path elements along paths and 165 adjust their own behavior based on that understanding. 167 Examples of both of these strategies are included in this document. 168 More recent proposals have tended toward determining path element 169 behavior and adjusting endpoint behavior. 171 1.3. A Note About Path-Aware Techniques Included In This Document 173 This document does not catalog every proposed path aware networking 174 technique that was not adopted and deployed. Instead, we limited our 175 focus to technologies that passed through the IETF community, and 176 still identified enough techniques to provide background for the 177 lessons included in Section 2 to inform researchers and protocol 178 engineers in their work. 180 No shame is intended for the techniques included in this document. 181 As shown in Section 2, the quality of specific techniques had little 182 to do with whether they were deployed or not. Based on the 183 techniques cataloged in this document, it is likely that when these 184 techniques were put forward, the proponents were trying to engineer 185 something that could not be engineered without first carrying out 186 research. Actual shame would be failing to learn from experience, 187 and failing to share that experience with other networking 188 researchers and engineers. 190 1.4. Venue for Discussion of this Document 192 (RFC Editor: please remove this section before publication) 194 Discussion of specific contributed experiences and this document in 195 general should take place on the PANRG mailing list. 197 1.5. Architectural Guidance 199 As background for understanding the Lessons Learned contained in this 200 document, the reader is encouraged to become familiar with the 201 Internet Architecture Board's documents on "What Makes for a 202 Successful Protocol?" [RFC5218] and "Planning for Protocol Adoption 203 and Subsequent Transitions" [RFC8170]. 205 Although these two documents do not specifically target path-aware 206 networking protocols, they are helpful resources for readers seeking 207 to improve their understanding of considerations for successful 208 adoption and deployment of any protocol. For example, the Basic 209 Success Factors described in Setion 2.1 of [RFC5218] are helpful for 210 readers of this document. 212 Because there is an economic aspect to decisions about deployment, 213 the IAB Workshop on Internet Technology Adoption and Transition 214 [ITAT] report [RFC7305] also provides food for thought. 216 Several of the Lessons Learned in Section 2 reflect considerations 217 described in [RFC5218], [RFC7305], and [RFC8170]. 219 1.6. Terminology Used in this Document 221 The terms Node and Element in this document have the meaning defined 222 in [PathProp]. 224 2. Summary of Lessons Learned 226 This section summarizes the Lessons Learned from the contributed 227 subsections in Section 5. 229 Each Lesson Learned is tagged with one or more contributions that 230 encountered this obstacle as a significant impediment to deployment. 231 Other contributed techniques may have also encountered this obstacle, 232 but this obstacle may not have been the biggest impediment to 233 deployment for those techniques. 235 It is useful to notice that sometimes an obstacle might impede 236 deployment, while at other times, the same obstacle might prevent 237 adoption and deployment entirely. The research group discussed 238 distinguishing between obstacles that impede and obstacles that 239 prevent, but it appears that the boundary between "impede" and 240 "prevent" can shift over time - some of the Lessons Learned are based 241 on both Path Aware techniques that were not deployed, and Path Aware 242 techniques that were deployed, but were not deployed widely or 243 quickly. See Section 5.6 and Section 5.6.3 as one example of this 244 shifting boundary. 246 2.1. Justifying Deployment 248 The benefit of Path Awareness must be great enough to justify making 249 changes in an operational network. The colloquial American English 250 expression, "If it ain't broke, don't fix it" is a "best current 251 practice" on today's Internet. (See Section 5.3, Section 5.5, and 252 Section 5.4). 254 2.2. Providing Benefits for Early Adopters 256 Providing benefits for early adopters can be key - if everyone must 257 deploy a technique in order for the technique to provide benefits, or 258 even to work at all, the technique is unlikely to be adopted widely 259 or quickly. (See Section 5.2 and Section 5.3). 261 2.3. Providing Benefits During Partial Deployment 263 Some proposals require that all path elements along the full length 264 of the path must be upgraded to support a new technique, before any 265 benefits can be seen. This is likely to require coordination between 266 operators who control a subset of path elements, and between 267 operators and end users if endpoint upgrades are required. If a 268 technique provides benefits when only a part of the path has been 269 upgraded, this is likely to encourage adoption and deployment. (See 270 Section 5.2 and Section 5.3). 272 2.4. Outperforming End-to-end Protocol Mechanisms 274 Adaptive end-to-end protocol mechanisms may respond to feedback 275 quickly enough that the additional realizable benefit from a new Path 276 Aware mechanism that tries to manipulate nodes along a path, or 277 observe the attributes of nodes along a path, may be much smaller 278 than anticipated (Section 5.3 and Section 5.5). 280 2.5. Paying for Path Aware Techniques 282 "Follow the money." If operators can't charge for a Path Aware 283 technique to recover the costs of deploying it, the benefits to the 284 operator must be really significant. Corollary: If operators charge 285 for a Path Aware technique, the benefits to users of that Path Aware 286 technique must be significant enough to justify the cost. (See 287 Section 5.1, Section 5.2, and Section 5.5). 289 2.6. Impact on Operational Practices 291 Impact of a Path Aware technique requiring changes to operational 292 practices can affect how quickly or widely a promising technique is 293 deployed. These impacts may make deployment more likely, but often 294 discourage deployment. (See Section 5.6, including Section 5.6.3). 296 2.7. Per-connection State 298 Per-connection state in intermediate nodes has been an impediment to 299 adoption and deployment in the past. This is especially true as we 300 move from the edge of the network, further into the routing core (See 301 Section 5.1 and Section 5.2). 303 2.8. Keeping Traffic on Fast-paths 305 Many modern platforms, especially high-end routers, have been 306 designed with hardware that can make simple per-packet forwarding 307 decisions ("fast-paths"), but have not been designed to make heavy 308 use of in-band mechanisms such as IPv4 and IPv6 Router Alert Options 309 (RAO) that require more processing to make forwarding decisions. 310 Packets carrying in-band mechanisms are diverted to other processors 311 in the router with much lower packet processing rates. Operators can 312 be reluctant to deploy techniques that rely heavily on in-band 313 mechanisms because they may significantly reduce packet throughput. 314 (See Section 5.7). 316 2.9. Endpoints Trusting Intermediate Nodes 318 If intermediate nodes along the path can't be trusted, it's unlikely 319 that endpoints will rely on signals from intermediate nodes to drive 320 changes to endpoint behaviors. (See Section 5.5, Section 5.4). We 321 note that "trust" is not binary - one, low, level of trust applies 322 when a node issuing a message can confirm that it has visibility of 323 the packets on the path it is seeking to control [RFC8085] (e.g., an 324 ICMP message included a quoted packet from the source). A higher 325 level of trust can arise when an endpoint has established a short 326 term, or even long term, trust relationship with network nodes. 328 2.10. Intermediate Nodes Trusting Endpoints 330 If the endpoints do not have any trust relationship with the 331 intermediate nodes along a path, operators have been reluctant to 332 deploy techniques that rely on endpoints sending unauthenticated 333 control signals to routers. (See Section 5.2 and Section 5.7. We 334 also note this still remains a factor hindering deployment of 335 DiffServ). 337 2.11. Reacting to Distant Signals 339 Because the Internet is a distributed system, if the distance that 340 information from distant path elements travels to a Path Aware host 341 is sufficiently large, the information may no longer accurately 342 represent the state and situation at the distant host or elements 343 along the path when it is received locally. In this case, the 344 benefit that a Path Aware technique provides will be inconsistent, 345 and may not always be beneficial. (See Section 5.3). 347 2.12. Support in Endpoint Protocol Stacks 349 Just because a protocol stack provides a new feature/signal does not 350 mean that applications will use the feature/signal. Protocol stacks 351 may not know how to effectively utilize Path-Aware techniques, 352 because the protocol stack may require information from applications 353 to permit the technique to work effectively, but applications may not 354 a-priori know that information. Even if the application does know 355 that information, the de-facto sockets API has no way of signaling 356 application expectations for the network path to the protocol stack. 357 In order for applications to provide these expectations to protocol 358 stacks, we need an API that signals more than the packets to be sent. 359 TAPS is exploring such an API [TAPS-WG], yet even with such an API, 360 policy is needed to bind the application expectations to the network 361 characteristics. (See Section 5.1 and Section 5.2). 363 3. Do We Understand the Lessons We've Learned? 365 The initial scope for this document was roughly "what mistakes have 366 we made in the decade prior to [PANRG-99], that we shouldn't make 367 again". Some of the contributions in Section 5 predate the initial 368 scope. The earliest Path-Aware Networking technique referred to in 369 Section 5 is Section 5.1, published in the late 1970s. Given that 370 the networking ecosystem has evolved continuously, it seemed 371 reasonable to ask whether the Lessons we've Learned are still true. 373 The PANRG Research Group reviewed the Lessons Learned contained in 374 the May 23, 2019 version of this document at IETF 105 375 [PANRG-105-Min], and carried out additional discussion at IETF 106 376 [PANRG-106-Min]. Table 1 provides the "sense of the room" after 377 those discussions. The intention is to capture whether a specific 378 lesson seems to be 380 o "Invariant" - well-understood and is likely to be applicable for 381 any proposed Path Aware Networking solution. 383 o "Variable" - has impeded deployment in the past, but might not be 384 applicable in a specific technique. Engineering analysis to 385 understand whether the lesson is applicable is prudent. 387 o "Not Now" - this characteristic tends to turn up a minefield full 388 of dragons, and prudent network engineers will wish to avoid 389 gambling on a technique that relies on this, until something 390 significant changes 392 +-------------------------------------------------------+-----------+ 393 | Lesson | Category | 394 +-------------------------------------------------------+-----------+ 395 | Justifying Deployment (Section 2.1) | Invariant | 396 | Providing Benefits for Early Adopters (Section 2.2) | Invariant | 397 | Providing Benefits during Partial Deployment | Invariant | 398 | (Section 2.3) | | 399 | Outperforming End-to-end Protocol Mechanisms | Variable | 400 | (Section 2.4) | | 401 | Paying for Path Aware Techniques (Section 2.5) | Invariant | 402 | Impact on Operational Practices (Section 2.6) | Invariant | 403 | Per-connection State (Section 2.7) | Variable | 404 | Keeping Traffic on Fast-paths (Section 2.8) | Variable | 405 | Endpoints Trusting Intermediate Nodes (Section 2.9) | Not Now | 406 | Intermediate Nodes Trusting Endpoints (Section 2.10) | Not Now | 407 | Reacting to Distant Signals (Section 2.11) | Variable | 408 | Support in Endpoint Protocol Stacks (Section 2.12) | Variable | 409 +-------------------------------------------------------+-----------+ 411 Table 1 413 "Justifying Deployment", "Providing Benefits for Early Adopters", 414 "Paying for Path Aware Techniques", and "Impact on Operational 415 Practice" were considered to be invariant - the sense of the room was 416 that these would always be considerations for any proposed Path Aware 417 Technique. 419 "Providing Benefits During Partial Deployment" was added after IETF 420 105, during research group last call, and is also considered to be 421 invariant. 423 For "Outperforming End-to-end Protocol Mechanisms", there is a trade- 424 off between improved performance from Path Aware Techniques and 425 additional complexity required by some Path Aware Techniques. 427 o For example, if you can obtain the same understanding of path 428 characteristics from measurements obtained over a few more round 429 trips, endpoint implementers are unlikely to be eager to add 430 complexity, and many attributes can be measured from an endpoint, 431 without assistance from intermediate nodes. 433 For "Per-connection State", the key questions discussed in the 434 research group were "how much state" and "where state is maintained". 436 o IntServ (Section 5.2) required state at every intermediate node 437 for every connection between two endpoints. As the Internet 438 ecosystem has evolved, carrying many connections in a tunnel that 439 appears to intermediate nodes as a single connection has become 440 more common, so that additional end-to-end connections don't add 441 additional state to intermediate nodes between tunnel endpoints. 442 If these tunnels are encrypted, intermediate nodes between tunnel 443 endpoints can't distinguish between connections, even if that were 444 desirable. 446 For "Keeping Traffic on Fast-paths", we noted that this was true for 447 many platforms, but not for all. 449 o For backbone routers, this is likely an invariant, but for 450 platforms that rely more on general-purpose computers to make 451 forwarding decisions, this may not be a fatal flaw for Path Aware 452 Networking techniques. 454 For "Endpoints Trusting Intermediate Nodes" and "Intermediate Nodes 455 Trusting Endpoints", these lessons point to the broader need to 456 revisit the Internet Threat Model. 458 o We noted with relief that discussions about this were already 459 underway in the IETF community at IETF 105 (see the Security Area 460 Open Meeting minutes [SAAG-105-Min] for discussion of 461 [draft-arkko-arch-internet-threat-model] and [draft-farrell-etm]), 462 and the Internet Architecture Board has created a mailing list for 463 continued discussions ([model-t]), but we recognize that there are 464 Path Aware Networking aspects of this effort, requiring research. 466 For "Reacting to Distant Signals", we noted that not all attributes 467 are equal. 469 o If an attribute is stable over an extended period of time, is 470 difficult to observe via end-to-end mechanisms, and is valuable, 471 Path Aware Techniques that rely on that attribute to provide a 472 significant benefit become more attractive. 474 o Analysis to help identify attributes that are useful enough to 475 justify deployment of Path Aware techniques that make use of those 476 attributes would be helpful. 478 For "Support in Endpoint Protocol Stacks", we noted that current work 479 in the TAPS working group ([TAPS-WG]) could provide an API that 480 allows applications to more easily make use of Path Aware techniques, 481 but this isn't guaranteed. 483 4. Template for Contributions 485 There are many things that could be said about the Path Aware 486 networking techniques that have been developed. For the purposes of 487 this document, contributors were requested to provide 489 o the name of a technique, including an abbreviation if one was used 491 o if available, a long-term pointer to the best reference describing 492 the technique 494 o a short description of the problem the technique was intended to 495 solve 497 o a short description of the reasons why the technique wasn't 498 adopted 500 o a short statement of the lessons that researchers can learn from 501 our experience with this technique. 503 5. Contributions 505 Contributions on these Path Aware networking techniques were analyzed 506 to arrive at the Lessons Learned captured in Section 2. 508 5.1. Stream Transport (ST, ST2, ST2+) 510 The suggested references for Stream Transport are: 512 o ST - A Proposed Internet Stream Protocol [IEN-119] 514 o Experimental Internet Stream Protocol, Version 2 (ST-II) [RFC1190] 516 o Internet Stream Protocol Version 2 (ST2) Protocol Specification - 517 Version ST2+ [RFC1819] 519 The first version of Stream Transport, ST [IEN-119], was published in 520 the late 1970's and was implemented and deployed on the ARPANET at 521 small scale. It was used throughout the 1980's for experimental 522 transmission of voice, video, and distributed simulation. 524 The second version of the ST specification (ST2) [RFC1190] [RFC1819] 525 was an experimental connection-oriented internetworking protocol that 526 operated at the same layer as connectionless IP. ST2 packets could 527 be distinguished by their IP header protocol numbers (IP, at that 528 time, used protocol number 4, while ST2 used protocol number 5). 530 ST2 used a control plane layered over IP to select routes and reserve 531 capacity for real-time streams across a network path, based on a flow 532 specification communicated by a separate protocol. The flow 533 specification could be associated with QoS state in routers, 534 producing an experimental resource reservation protocol. This 535 allowed ST2 routers along a path to offer end-to-end guarantees, 536 primarily to satisfy the QoS requirements for realtime services over 537 the Internet. 539 5.1.1. Reasons for Non-deployment 541 Although implemented in a range of equipment, ST2 was not widely used 542 after completion of the experiments. It did not offer the 543 scalability and fate-sharing properties that have come to be desired 544 by the Internet community. 546 The ST2 protocol is no longer in use. 548 5.1.2. Lessons Learned. 550 As time passed, the trade-off between router processing and link 551 capacity changed. Links became faster and the cost of router 552 processing became comparatively more expensive. 554 The ST2 control protocol used "hard state" - once a route was 555 established, and resources were reserved, routes and resources 556 existing until they were explicitly released via signaling. A soft- 557 state approach was thought superior to this hard-state approach, and 558 led to development of the IntServ model described in Section 5.2. 560 5.2. Integrated Services (IntServ) 562 The suggested references for IntServ are: 564 o RFC 1633 Integrated Services in the Internet Architecture: an 565 Overview [RFC1633] 567 o RFC 2211 Specification of the Controlled-Load Network Element 568 Service [RFC2211] 570 o RFC 2212 Specification of Guaranteed Quality of Service [RFC2212] 572 o RFC 2215 General Characterization Parameters for Integrated 573 Service Network Elements [RFC2215] 575 o RFC 2205 Resource ReSerVation Protocol (RSVP) [RFC2205] 577 In 1994, when the IntServ architecture document [RFC1633] was 578 published, real-time traffic was first appearing on the Internet. At 579 that time, bandwidth was still a scarce commodity. Internet Service 580 Providers built networks over DS3 (45 Mbps) infrastructure, and sub- 581 rate (< 1 Mpbs) access was common. Therefore, the IETF anticipated a 582 need for a fine-grained QoS mechanism. 584 In the IntServ architecture, some applications can require service 585 guarantees. Therefore, those applications use the Resource 586 Reservation Protocol (RSVP) [RFC2205] to signal QoS reservations 587 across network paths. Every router in the network maintains per-flow 588 soft-state to a) perform call admission control and b) deliver 589 guaranteed service. 591 Applications use Flow Specification (Flow Specs) [RFC2210] to 592 describe the traffic that they emit. RSVP reserves capacity for 593 traffic on a per Flow Spec basis. 595 5.2.1. Reasons for Non-deployment 597 Although IntServ has been used in enterprise and government networks, 598 IntServ was never widely deployed on the Internet because of its 599 cost. The following factors contributed to operational cost: 601 o IntServ must be deployed on every router that is on a path where 602 IntServ is to be used 604 o IntServ maintained per flow state 606 As IntServ was being discussed, the following occurred: 608 o For many expected uses, it became more cost effective to solve the 609 QoS problem by adding bandwidth. Between 1994 and 2000, Internet 610 Service Providers upgraded their infrastructures from DS3 (45 611 Mbps) to OC-48 (2.4 Gbps). This meant that even if an endpoint 612 was using IntServ in an IntServ-enabled network, its requests 613 would never be denied, so endpoints and Internet Service Providers 614 had little reason to enable IntServ. 616 o DiffServ [RFC2475] offered a more cost-effective, albeit less 617 fine-grained, solution to the QoS problem. 619 5.2.2. Lessons Learned. 621 The following lessons were learned: 623 o Any mechanism that requires every onpath router to maintain per- 624 flow state is not likely to succeed, unless the additional cost 625 for offering the feature can be recovered from the user. 627 o Any mechanism that requires an operator to upgrade all of its 628 routers is not likely to succeed, unless the additional cost for 629 offering the feature can be recovered from the user. 631 In environments where IntServ has been deployed, trust relationships 632 with endpoints are very different from trust relationships on the 633 Internet itself, and there are often clearly-defined hierarchies in 634 Service Level Agreements (SLAs), and well-defined transport flows 635 operating with pre-determined capacity and latency requirements over 636 paths where capacity or other attributes are constrained. 638 IntServ was never widely deployed to manage capacity across the 639 Internet. However, the technique that it produced was deployed for 640 reasons other than bandwidth management. RSVP is widely deployed as 641 an MPLS signaling mechanism. BGP reuses the RSVP concept of Filter 642 Specs to distribute firewall filters, although they are called Flow 643 Spec Component Types in BGP [RFC5575]. 645 5.3. Quick-Start TCP 647 The suggested references for Quick-Start TCP are: 649 o Quick-Start for TCP and IP [RFC4782] 651 o Determining an appropriate initial sending rate over an 652 underutilized network path [SAF07] 654 o Fast Startup Internet Congestion Control for Broadband Interactive 655 Applications [Sch11] 657 o Using Quick-Start to enhance TCP-friendly rate control performance 658 in bidirectional satellite networks [QS-SAT] 660 Quick-Start [RFC4782] is an Experimental TCP extension that leverages 661 support from the routers on the path to determine an allowed initial 662 sending rate for a path through the Internet, either at the start of 663 data transfers or after idle periods. Without information about the 664 path, a sender cannot easily determine an appropriate initial sending 665 rate. The default TCP congestion control therefore uses the safe but 666 time-consuming slow-start algorithm [RFC5681]. With Quick-Start, 667 connections are allowed to use higher initial sending rates if there 668 is significant unused bandwidth along the path, and if the sender and 669 all of the routers along the path approve the request. 671 By examining the Time To Live (TTL) field in Quick-Start packets, a 672 sender can determine if routers on the path have approved the Quick- 673 Start request. However, this method is unable to take into account 674 the routers hidden by tunnels or other network nodes invisible at the 675 IP layer. 677 The protocol also includes a nonce that provides protection against 678 cheating routers and receivers. If the Quick-Start request is 679 explicitly approved by all routers along the path, the TCP host can 680 send at up to the approved rate; otherwise TCP would use the default 681 congestion control. Quick-Start requires modifications in the 682 involved end-systems as well in routers. Due to the resulting 683 deployment challenges, Quick-Start was only proposed in [RFC4782] for 684 controlled environments. 686 The Quick-Start mechanism is a lightweight, coarse-grained, in-band, 687 network-assisted fast startup mechanism. The benefits are studied by 688 simulation in a research paper [SAF07] that complements the protocol 689 specification. The study confirms that Quick-Start can significantly 690 speed up mid-sized data transfers. That paper also presents router 691 algorithms that do not require keeping per-flow state. Later studies 692 [Sch11] comprehensively analyzes Quick-Start with a full Linux 693 implementation and with a router fast path prototype using a network 694 processor. In both cases, Quick-Start could be implemented with 695 limited additional complexity. 697 5.3.1. Reasons for Non-deployment 699 However, experiments with Quick-Start in [Sch11] revealed several 700 challenges: 702 o Having information from the routers along the path can reduce the 703 risk of congestion, but cannot avoid it entirely. Determining 704 whether there is unused capacity is not trivial in actual router 705 and host implementations. Data about available capacity visible 706 at the IP layer may be imprecise, and due to the propagation 707 delay, information can already be outdated when it reaches a 708 sender. There is a trade-off between the speedup of data 709 transfers and the risk of congestion even with Quick-Start. This 710 could be mitigated by only allowing Quick-Start to access a 711 proportion of the unused capacity along a path. 713 o For scalable router fast path implementation, it is important to 714 enable parallel processing of packets, as this is a widely used 715 method e.g. in network processors. One challenge is 716 synchronization of information between packets that are processed 717 in parallel, which should be avoided as much as possible. 719 o Only some types of application traffic can benefit from Quick- 720 Start. Capacity needs to be requested and discovered. The 721 discovered capacity needs to be utilized by the flow, or it 722 implicitly becomes available for other flows. Failing to use the 723 requested capacity may have already reduced the pool of Quick- 724 Start capacity that was made available to other competing Quick- 725 Start requests. The benefit is greatest when senders use this 726 only for bulk flows and avoid sending unnecessary Quick-Start 727 requests, e.g. for flows that only send a small amount of data. 728 Choosing an appropriate request size requires application-internal 729 knowledge that is not commonly expressed by the transport API. 730 How a sender can determine the rate for an initial Quick-Start 731 request is still a largely unsolved problem. 733 There is no known deployment of Quick-Start for TCP or other IETF 734 transports. 736 5.3.2. Lessons Learned 738 Some lessons can be learned from Quick-Start. Despite being a very 739 light-weight protocol, Quick-Start suffers from poor incremental 740 deployment properties, both regarding the required modifications in 741 network infrastructure as well as its interactions with applications. 742 Except for corner cases, congestion control can be quite efficiently 743 performed end-to-end in the Internet, and in modern stacks there is 744 not much room for significant improvement by additional network 745 support. 747 After publication of the Quick-Start specification, there have been 748 large-scale experiments with an initial window of up to 10 MSS 749 [RFC6928]. This alternative "IW10" approach can also ramp-up data 750 transfers faster than the standard congestion control, but it only 751 requires sender-side modifications. As a result, this approach can 752 be easier and incrementally deployed in the Internet. While 753 theoretically Quick-Start can outperform "IW10", the improvement in 754 completion time for data transfer times can, in many cases, be small. 755 After publication of [RFC6928], most modern TCP stacks have increased 756 their default initial window. 758 5.4. ICMP Source Quench 760 The suggested references for ICMP Source Quench are: 762 o INTERNET CONTROL MESSAGE PROTOCOL [RFC0792] 763 The ICMP Source Quench message [RFC0792] allowed an on-path router to 764 request the source of a flow to reduce its sending rate. This method 765 allowed a router to provide an early indication of impending 766 congestion on a path to the sources that contribute to that 767 congestion. 769 5.4.1. Reasons for Non-deployment 771 This method was deployed in Internet routers over a period of time, 772 the reaction of endpoints to receiving this signal has varied. For 773 low speed links, with low multiplexing of flows the method could be 774 used to regulate (momentarily reduce) the transmission rate. 775 However, the simple signal does not scale with link speed, or the 776 number of flows sharing a link. 778 The approach was overtaken by the evolution of congestion control 779 methods in TCP [RFC2001], and later also by other IETF transports. 780 Because these methods were based upon measurement of the end-to-end 781 path and an algorithm in the endpoint, they were able to evolve and 782 mature more rapidly than methods relying on interactions between 783 operational routers and endpoint stacks. 785 After ICMP Source Quench was specified, the IETF began to recommend 786 that transports provide end-to-end congestion control [RFC2001]. The 787 Source Quench method has been obsoleted by the IETF [RFC6633], and 788 both hosts and routers must now silently discard this message. 790 5.4.2. Lessons Learned 792 This method had several problems: 794 First, [RFC0792] did not sufficiently specify how the sender would 795 react to the ICMP Source Quench signal from the path (e.g., 796 [RFC1016]). There was ambiguity in how the sender should utilize 797 this additional information. This could lead to unfairness in the 798 way that receivers (or routers) responded to this message. 800 Second, while the message did provide additional information, the 801 Explicit Congestion Notification (ECN) mechanism [RFC3168] provided a 802 more robust and informative signal for network nodes to provide early 803 indication that a path has become congested. 805 The mechanism originated at a time when the Internet trust model was 806 very different. Most endpoint implementations did not attempt to 807 verify that the message originated from an on-path node before they 808 utilized the message. This made it vulnerable to denial of service 809 attacks. In theory, routers might have chosen to use the quoted 810 packet contained in the ICMP payload to validate that the message 811 originated from an on-path node, but this would have increased per- 812 packet processing overhead for each router along the path, would have 813 required transport functionality in the router to verify whether the 814 quoted packet header corresponded to a packet the router had sent. 815 In addition, section 5.2 of [RFC4443] noted ICMPv6-based attacks on 816 hosts that would also have threatened routers processing ICMPv6 817 Source Quench payloads. As time passed, it became increasingly 818 obvious that the lack of validation of the messages exposed receivers 819 to a security vulnerability where the messages could be forged to 820 create a tangible denial of service opportunity. 822 5.5. Triggers for Transport (TRIGTRAN) 824 The suggested references for TRIGTRAN are: 826 o TRIGTRAN BOF at IETF 55 [TRIGTRAN-55] 828 o TRIGTRAN BOF at IETF 56 [TRIGTRAN-56] 830 TCP [RFC0793] has a well-known weakness - the end-to-end flow control 831 mechanism has only a single signal, the loss of a segment, and TCP 832 implementations since the late 1980s have interpreted the loss of a 833 segment as evidence that the path between two endpoints may have 834 become congested enough to exhaust buffers on intermediate hops, so 835 that the TCP sender should "back off" - reduce its sending rate until 836 it knows that its segments are now being delivered without loss 837 [RFC5681]. More modern TCP stacks have added a growing array of 838 strategies about how to establish the sending rate [RFC5681], but 839 when a path is no longer operational, TCP would continue to retry 840 transmissions, which would fail, again, and double their 841 Retransmission Time Out (RTO) timers with each failed transmission, 842 with the result that TCP would wait many seconds before retrying a 843 segment, even if the path becomes operational while the sender is 844 waiting for its next retry. 846 The thinking behind TRIGTRAN was that if a path completely stopped 847 working because a link along the path was "down", somehow something 848 along the path could signal TCP when that link returned to service, 849 and the sending TCP could retry immediately, without waiting for a 850 full retransmission timeout (RTO) period. 852 5.5.1. Reasons for Non-deployment 854 The early dreams for TRIGTRAN were dashed because of an assumption 855 that TRIGTRAN triggers would be unauthenticated. This meant that any 856 "safe" TRIGTRAN mechanism would have relied on a mechanism such as 857 setting the IPv4 TTL or IPv6 Hop Count to 255 at a sender and testing 858 that it was 254 upon receipt, so that a receiver could verify that a 859 signal was generated by an adjacent sender known to be on the path 860 being used, and not some unknown sender which might not even be on 861 the path (e.g., "The Generalized TTL Security Mechanism (GTSM)" 862 [RFC5082]). This situation is very similar to the case for ICMP 863 Source Quench messages as described in Section 5.4, which were also 864 unauthenticated, and could be sent by an off-path attacker, resulting 865 in deprecation of ICMP Source Quench message processing [RFC6633]. 867 TRIGTRAN's scope shrunk from "the path is down" to "the first-hop 868 link is down". 870 But things got worse. 872 Because TRIGTRAN triggers would only be provided when the first-hop 873 link was "down", TRIGTRAN triggers couldn't replace normal TCP 874 retransmission behavior if the path failed because some link further 875 along the network path was "down". So TRIGTRAN triggers added 876 complexity to an already complex TCP state machine, and did not allow 877 any existing complexity to be removed. 879 There was also an issue that the TRIGTRAN signal was not sent in 880 response to a specific host that had been sending packets, and was 881 instead a signal that stimulated a response by any sender on the 882 link. This needs to scale when there are multiple flows trying to 883 use the same resource, yet the sender of a trigger has no 884 understanding how many of the potential traffic sources will respond 885 by sending packets - if recipients of the signal back-off their 886 responses to a trigger to improve scaling, then that immediately 887 mitigates the benefit of the signal. 889 Finally, intermediate forwarding nodes required modification to 890 provide TRIGTRAN triggers, but operators couldn't charge for TRIGTRAN 891 triggers, so there was no way to recover the cost of modifying, 892 testing, and deploying updated intermediate nodes. 894 Two TRIGTRAN BOFs were held, at IETF 55 [TRIGTRAN-55] and IETF 56 895 [TRIGTRAN-56], but this work was not chartered, and there was no 896 interest in deploying TRIGTRAN unless it was chartered and 897 standardized in the IETF. 899 5.5.2. Lessons Learned. 901 The reasons why this work was not chartered, much less deployed, 902 provide several useful lessons for researchers. 904 o TRIGTRAN started with a plausible value proposition, but 905 networking realities in the early 2000s forced reductions in scope 906 that led directly to reductions in potential benefits, but no 907 corresponding reductions in costs and complexity. 909 o These reductions in scope were the direct result of an inability 910 for hosts to trust or authenticate TRIGTRAN signals they received 911 from the network. 913 o Operators did not believe they could charge for TRIGTRAN 914 signaling, because first-hop links didn't fail frequently, and 915 TRIGTRAN provided no reduction in operating expenses, so there was 916 little incentive to purchase and deploy TRIGTRAN-capable network 917 equipment. 919 It is also worth noting that the targeted environment for TRIGTRAN in 920 the late 1990s contained links with a relatively small number of 921 directly-connected hosts - for instance, cellular or satellite links. 922 The transport community was well aware of the dangers of sender 923 synchronization based on multiple senders receiving the same stimulus 924 at the same time, but the working assumption for TRIGTRAN was that 925 there wouldn't be enough senders for this to be a meaningful problem. 926 In the 2010s, it is common for a single "link" to support many 927 senders and receivers on a single link, likely requiring TRIGTRAN 928 senders to wait some random amount of time before sending after 929 receiving a TRIGTRAN signal, which would have reduced the benefits of 930 TRIGTRAN even more. 932 5.6. Shim6 934 The suggested references for Shim6 are: 936 o Shim6: Level 3 Multihoming Shim Protocol for IPv6 [RFC5533] 938 The IPv6 routing architecture [RFC1887] assumed that most sites on 939 the Internet would be identified by Provider Assigned IPv6 prefixes, 940 so that Default-Free Zone routers only contained routes to other 941 providers, resulting in a very small IPv6 global routing table. 943 For a single-homed site, this could work well. A multihomed site 944 with only one upstream provider could also work well, although BGP 945 multihoming from a single upstream provider was often a premium 946 service (costing more than twice as much as two single-homed sites), 947 and if the single upstream provider went out of service, all of the 948 multihomed paths could fail simultaneously. 950 IPv4 sites often multihomed by obtaining Provider Independent 951 prefixes, and advertising these prefixes through multiple upstream 952 providers. With the assumption that any multihomed IPv4 site would 953 also multihome in IPv6, it seemed likely that IPv6 routing would be 954 subject to the same pressures to announce Provider Independent 955 prefixes, resulting in a global IPv6 routing table that exhibited the 956 same explosive growth as the global IPv4 routing table. During the 957 early 2000s, work began on a protocol that would provide multihoming 958 for IPv6 sites without requiring sites to advertise Provider 959 Independent prefixes into the IPv6 global routing table. 961 This protocol, called Shim6, allowed two endpoints to exchange 962 multiple addresses ("Locators") that all mapped to the same endpoint 963 ("Identity"). After an endpoint learned multiple Locators for the 964 other endpoint, it could send to any of those Locators with the 965 expectation that those packets would all be delivered to the endpoint 966 with the same Identity. Shim6 was an example of an "Identity/Locator 967 Split" protocol. 969 Shim6, as defined in [RFC5533] and related RFCs, provided a workable 970 solution for IPv6 multihoming using Provider Assigned prefixes, 971 including capability discovery and negotiation, and allowing end-to- 972 end application communication to continue even in the face of path 973 failure, because applications don't see Locator failures, and 974 continue to communicate with the same Identity using a different 975 Locator. 977 5.6.1. Reasons for Non-deployment 979 Note that the problem being addressed was "site multihoming", but 980 Shim6 was providing "host multihoming". That meant that the decision 981 about what path would be used was under host control, not under 982 router control. 984 Although more work could have been done to provide a better technical 985 solution, the biggest impediments to Shim6 deployment were 986 operational and business considerations. These impediments were 987 discussed at multiple network operator group meetings, including 988 [Shim6-35] at [NANOG-35]. 990 The technique issues centered around concerns that Shim6 relied on 991 the host to track all the connections, while also tracking Identity/ 992 Locator mappings in the kernel, and tracking failures to recognize 993 that a backup path has failed. 995 The operator issues centered around concerns that operators were 996 performing traffic engineering, but would have no visibility or 997 control over hosts when they chose to begin using another path, and 998 relying on hosts to engineer traffic exposed their networks to 999 oscillation based on feedback loops, as hosts move from path to path. 1000 At a minimum, traffic engineering policies must be pushed down to 1001 individual hosts. In addition, firewalls that expected to find a 1002 transport-level protocol header in the IP payload, would see a Shim6 1003 Identity header, and be unable to perform transport-protocol-based 1004 firewalling functions because its normal processing logic would not 1005 look past the Identity header. 1007 The business issues centered removing or reducing the ability to sell 1008 BGP multihoming service, which is often more expensive than single- 1009 homed connectivity. 1011 5.6.2. Lessons Learned 1013 It is extremely important to take operational concerns into account 1014 when a path-aware protocol is making decisions about path selection 1015 that may conflict with existing operational practices and business 1016 considerations. 1018 5.6.3. Addendum on MultiPath TCP 1020 During discussions in the PANRG session at IETF 103 [PANRG-103-Min], 1021 Lars Eggert, past Transport Area Director, pointed out that during 1022 charter discussions for the Multipath TCP working group [MP-TCP], 1023 operators expressed concerns that customers could use Multipath TCP 1024 to loadshare TCP connections across operators simultaneously and 1025 compare passive performance measurements across network paths in real 1026 time, changing the balance of power in those business relationships. 1027 Although the Multipath TCP working group was chartered, this concern 1028 could have acted as an obstacle to deployment. 1030 Operator objections to Shim6 were focused on technical concerns, but 1031 this concern could have also been an obstacle to Shim6 deployment if 1032 the technical concerns had been overcome. 1034 5.7. Next Steps in Signaling (NSIS) 1036 The suggested references for Next Steps in Signaling (NSIS) are: 1038 o the concluded working group charter [NSIS-CHARTER-2001] 1040 o GIST: General Internet Signalling Transport [RFC5971] 1042 o NAT/Firewall NSIS Signaling Layer Protocol (NSLP) [RFC5973] 1044 o NSIS Signaling Layer Protocol (NSLP) for Quality-of-Service 1045 Signaling [RFC5974] 1047 o Authorization for NSIS Signaling Layer Protocols [RFC5981] 1048 The NSIS Working Group worked on signaling techniques for network 1049 layer resources (e.g., QoS resource reservations, Firewall and NAT 1050 traversal). 1052 When RSVP [RFC2205] was used in deployments, a number of questions 1053 came up about its perceived limitations and potential missing 1054 features. The issues noted in the NSIS Working Group charter 1055 [NSIS-CHARTER-2001] include interworking between domains with 1056 different QoS architectures, mobility and roaming for IP interfaces, 1057 and complexity. Later, the lack of security in RSVP was also 1058 recognized ([RFC4094]). 1060 The NSIS Working Group was chartered to tackle those issues and 1061 initially focused on QoS signaling as its primary use case. However, 1062 over time a new approach evolved that introduced a modular 1063 architecture using application-specific signaling protocols (the NSIS 1064 Signaling Layer Protocol (NSLP)) on top of a generic signaling 1065 transport protocol (the NSIS Transport Layer Protocol (NTLP)). 1067 The NTLP is defined in [RFC5971]. Two NSLPs are defined: the NSIS 1068 Signaling Layer Protocol (NSLP) for Quality-of-Service Signaling 1069 [RFC5974] as well as the NAT/Firewall NSIS Signaling Layer Protocol 1070 (NSLP) [RFC5973]. 1072 5.7.1. Reasons for Non-deployment 1074 The obstacles for deployment can be grouped into implementation- 1075 related aspects and operational aspects. 1077 o Implementation-related aspects: 1079 Although NSIS provides benefits with respect to flexibility, 1080 mobility, and security compared to other network signaling 1081 techniques, hardware vendors were reluctant to deploy this solution, 1082 because it would require additional implementation effort and would 1083 result in additional complexity for router implementations. 1085 The NTLP mainly operates as path-coupled signaling protocol, i.e, its 1086 messages are processed at the intermediate node's control plane that 1087 are also forwarding the data flows. This requires a mechanism to 1088 intercept signaling packets while they are forwarded in the same 1089 manner (especially along the same path) as data packets. NSIS uses 1090 the IPv4 and IPv6 Router Alert Option (RAO) to allow for interception 1091 of those path-coupled signaling messages, and this technique requires 1092 router implementations to correctly understand and implement the 1093 handling of RAOs, e.g., to only process packet with RAOs of interest 1094 and to leave packets with irrelevant RAOs in the fast forwarding 1095 processing path (a comprehensive discussion of these issues can be 1096 found in [RFC6398]). The latter was an issue with some router 1097 implementations at the time of standardization. 1099 Another reason is that path-coupled signaling protocols that interact 1100 with routers and request manipulation of state at these routers (or 1101 any other network element in general) are under scrutiny: a packet 1102 (or sequence of packets) out of the mainly untrusted data path is 1103 requesting creation and manipulation of network state. This is seen 1104 as potentially dangerous (e.g., opens up a Denial of Service (DoS) 1105 threat to a router's control plane) and difficult for an operator to 1106 control. Path-coupled signaling approaches were considered 1107 problematic (see also section 3 of [RFC6398]). There are 1108 recommendations on how to secure NSIS nodes and deployments (e.g., 1109 [RFC5981]). 1111 o Operational Aspects: 1113 NSIS not only required trust between customers and their provider, 1114 but also among different providers. Especially, QoS signaling 1115 techniques would require some kind of dynamic service level agreement 1116 support that would imply (potentially quite complex) bilateral 1117 negotiations between different Internet service providers. This 1118 complexity was not considered to be justified and increasing the 1119 bandwidth (and thus avoiding bottlenecks) was cheaper than actively 1120 managing network resource bottlenecks by using path-coupled QoS 1121 signaling techniques. Furthermore, an end-to-end path typically 1122 involves several provider domains and these providers need to closely 1123 cooperate in cases of failures. 1125 5.7.2. Lessons Learned 1127 One goal of NSIS was to decrease the complexity of the signaling 1128 protocol, but a path-coupled signaling protocol comes with the 1129 intrinsic complexity of IP-based networks, beyond the complexity of 1130 the signaling protocol itself. Sources of intrinsic complexity 1131 include: 1133 o the presence of asymmetric routes between endpoints and routers 1135 o the lack of security and trust at large in the Internet 1136 infrastructure 1138 o the presence of different trust boundaries 1140 o the effects of best-effort networks (e.g., robustness to packet 1141 loss) 1143 o divergence from the fate sharing principle (e.g., state within the 1144 network). 1146 Any path-coupled signaling protocol has to deal with these realities. 1148 Operators view the use of IPv4 and IPv6 Router Alert Option (RAO) to 1149 signal routers along the path from end systems with suspicion, 1150 because these end systems are usually not authenticated and heavy use 1151 of RAOs can easily increase the CPU load on routers that are designed 1152 to process most packets using a hardware "fast path" and diverting 1153 packets containing RAO to a slower, more capable processor. 1155 5.8. IPv6 Flow Label 1157 The suggested references for IPv6 Flow Label are: 1159 o IPv6 Flow Label Specification [RFC6437] 1161 IPv6 specifies a 20-bit field Flow Label field [RFC6437], included in 1162 the fixed part of the IPv6 header and hence present in every IPv6 1163 packet. An endpoint sets the value in this field to one of a set of 1164 pseudo-randomly assigned values. If a packet is not part of any 1165 flow, the flow label value is set to zero [RFC3697]. A number of 1166 Standards Track and Best Current Practice RFCs (e.g., [RFC8085], 1167 [RFC6437], [RFC6438]) encourage IPv6 endpoints to set a non-zero 1168 value in this field. A multiplexing transport could choose to use 1169 multiple flow labels to allow the network to independently forward 1170 its subflows, or to use one common value for the traffic aggregate. 1171 The flow label is present in all fragments. IPsec was originally put 1172 forward as one important use-case for this mechanism and does encrypt 1173 the field [RFC6438]. 1175 Once set, the flow label can provide information that can help inform 1176 network nodes about subflows present at the transport layer, without 1177 needing to interpret the setting of upper layer protocol fields 1178 [RFC6294]. This information can also be used to coordinate how 1179 aggregates of transport subflows are grouped when queued in the 1180 network and to select appropriate per-flow forwarding when choosing 1181 between alternate paths [RFC6438] (e.g. for Equal Cost Multipath 1182 Routing (ECMP) and Link Aggregation (LAG)). 1184 5.8.1. Reasons for Non-deployment 1186 Despite the field being present in every IPv6 packet, the mechanism 1187 did not receive as much use as originally envisioned. One reason is 1188 that to be useful it requires engagement by two different 1189 stakeholders: 1191 o Endpoint Implementation: 1193 For network nodes along a path to utilize the flow label there needs 1194 to be a non-zero value inserted in the field [RFC6437] at the sending 1195 endpoint. There needs to be an incentive for an endpoint to set an 1196 appropriate non-zero value. The value should appropriately reflect 1197 the level of aggregation the traffic expects to be provided by the 1198 network. However, this requires the stack to know granularity at 1199 which flows should be identified (or conversely which flows should 1200 receive aggregated treatment), i.e., which packets carry the same 1201 flow label. Therefore, setting a non-zero value may result in 1202 additional choices that need to be made by an application developer. 1204 Although the standard [RFC3697] forbids any encoding of meaning into 1205 the flow label value, the opportunity to use the flow label as a 1206 covert channel or to signal other meta-information may have raised 1207 concerns about setting a non-zero value [RFC6437]. 1209 Before methods are widely deployed to use this method, there could be 1210 no incentive for an endpoint to set the field. 1212 o Operational support in network nodes: 1214 A benefit can only be realized when a network node along the path 1215 also uses this information to inform its decisions. Network 1216 equipment (routers and/or middleboxes) need to include appropriate 1217 support so they can utilize the field when making decisions about how 1218 to classify flows, or to inform forwarding choices. Use of any 1219 optional feature in a network node also requires corresponding 1220 updates to operational procedures, and therefore is normally only 1221 introduced when the cost can be justified. 1223 A benefit from utilizing the flow label is expected to be increased 1224 quality of experience for applications - but this comes at some 1225 operational cost to an operator, and requires endpoints to set the 1226 field. 1228 5.8.2. Lessons Learned 1230 The flow label is a general purpose header field for use by the path. 1231 Multiple uses have been proposed. One candidate use was to reduce 1232 the complexity of forwarding decisions. However, modern routers can 1233 use a "fast path", often taking advantage of hardware to accelerate 1234 processing. The method can assist in more complex forwarding, such 1235 as ECMP and load balancing. 1237 Although [RFC6437] recommended that endpoints should by default 1238 choose uniformly-distributed labels for their traffic, the 1239 specification permitted an endpoint to choose to set a zero value. 1240 This ability of endpoints to choose to set a flow label of zero has 1241 had consequences on deployability: 1243 o Before wide-scale support by endpoints, it would be impossible to 1244 rely on a non-zero flow label being set. Network nodes therefore 1245 would need to also employ other techniques to realize equivalent 1246 functions. An example of a method is one assuming semantics of 1247 the source port field to provide entropy input to a network-layer 1248 hash. This use of a 5-tuple to classify a packet represents a 1249 layering violation [RFC6294]. When other methods have been 1250 deployed, they increase the cost of deploying standards-based 1251 methods, even though they may offer less control to endpoints and 1252 result in potential interaction with other uses/interpretation of 1253 the field. 1255 o Even though the flow label is specified as an end-to-end field, 1256 some network paths have been observed to not transparently forward 1257 the flow label. This could result from non-conformant equipment, 1258 or could indicate that some operational networks have chosen to 1259 re-use the protocol field for other (e.g. internal purposes). 1260 This results in lack of transparency, and a deployment hurdle to 1261 endpoints expecting that they can set a flow label that is 1262 utilized by the network. The more recent practice of "greasing" 1263 [GREASE] would suggest that a different outcome could have been 1264 achieved if endpoints were always required to set a non-zero 1265 value. 1267 o [RFC1809] noted that setting the choice of the flow label value 1268 can depend on the expectations of the traffic generated by an 1269 application, which suggests an API should be presented to control 1270 the setting or policy that is used. However, many currently 1271 available APIs do not have this support. 1273 A growth in the use of encrypted transports, (e.g. QUIC [QUIC-WG]) 1274 seems likely to raise similar issues to those discussed above and 1275 could motivate renewed interest in utilizing the flow label. 1277 6. Security Considerations 1279 This document describes Path Aware techniques that were not adopted 1280 and widely deployed on the Internet, so it doesn't affect the 1281 security of the Internet. 1283 If this document meets its goals, we may develop new techniques for 1284 Path Aware Networking that would affect the security of the Internet, 1285 but security considerations for those techniques will be described in 1286 the corresponding RFCs that specify them. 1288 7. IANA Considerations 1290 This document makes no requests of IANA. 1292 8. Acknowledgments 1294 Initial material for Section 5.1 on ST2 was provided by Gorry 1295 Fairhurst. 1297 Initial material for Section 5.2 on IntServ was provided by Ron 1298 Bonica. 1300 Initial material for Section 5.3 on Quick-Start TCP was provided by 1301 Michael Scharf, who also provided suggestions to improve this section 1302 after it was edited. 1304 Initial material for Section 5.4 on ICMP Source Quench was provided 1305 by Gorry Fairhurst. 1307 Initial material for Section 5.5 on Triggers for Transport (TRIGTRAN) 1308 was provided by Spencer Dawkins. 1310 Section 5.6 on Shim6 builds on initial material describing obstacles 1311 provided by Erik Nordmark, with background added by Spencer Dawkins. 1313 Initial material for Section 5.7 on Next Steps In Signaling (NSIS) 1314 was provided by Roland Bless and Martin Stiemerling. 1316 Initial material for Section 5.8 on IPv6 Flow Labels was provided by 1317 Gorry Fairhurst. 1319 Our thanks to C.M. Heard, David Black, Gorry Fairhurst, Joe Touch, 1320 Joeri de Ruiter, Mohamed Boucadair, Roland Bless, Ruediger Geib, 1321 Theresa Enghardt, and Wes Eddy, who provided review comments on 1322 previous versions. 1324 Special thanks to Adrian Farrel for helping Spencer navigate the 1325 twisty little passages of Flow Specs and Filter Specs in IntServ, 1326 RSVP, MPLS, and BGP. They are all alike, except when they are 1327 different [Colossal-Cave]. 1329 9. Informative References 1331 [Colossal-Cave] 1332 "Wikipedia Page for Colossal Cave Adventure", January 1333 2019, 1334 . 1336 [draft-arkko-arch-internet-threat-model] 1337 Arkko, J., "Changes in the Internet Threat Model", n.d., 1338 . 1341 [draft-farrell-etm] 1342 Farrell, S., "We're gonna need a bigger threat model", 1343 n.d., 1344 . 1346 [GREASE] Thomson, M., "Long-term Viability of Protocol Extension 1347 Mechanisms", July 2019, . 1350 [IEN-119] Forgie, J., "ST - A Proposed Internet Stream Protocol", 1351 September 1979, 1352 . 1354 [ITAT] "IAB Workshop on Internet Technology Adoption and 1355 Transition (ITAT)", December 2013, 1356 . 1358 [model-t] "Model-t -- Discussions of changes in Internet deployment 1359 patterns and their impact on the Internet threat model", 1360 n.d., . 1362 [MP-TCP] "Multipath TCP Working Group Home Page", n.d., 1363 . 1365 [NANOG-35] 1366 "North American Network Operators Group NANOG-35 Agenda", 1367 October 2005, 1368 . 1370 [NSIS-CHARTER-2001] 1371 "Next Steps In Signaling Working Group Charter", March 1372 2011, 1373 . 1375 [PANRG] "Path Aware Networking Research Group (Home Page)", n.d., 1376 . 1378 [PANRG-103-Min] 1379 "Path Aware Networking Research Group - IETF-103 Minutes", 1380 November 2018, 1381 . 1383 [PANRG-105-Min] 1384 "Path Aware Networking Research Group - IETF-105 Minutes", 1385 July 2019, 1386 . 1388 [PANRG-106-Min] 1389 "Path Aware Networking Research Group - IETF-106 Minutes", 1390 November 2019, 1391 . 1393 [PANRG-99] 1394 "Path Aware Networking Research Group - IETF-99", July 1395 2017, 1396 . 1398 [PATH-Decade] 1399 Bonaventure, O., "A Decade of Path Awareness", July 2017, 1400 . 1403 [PathProp] 1404 Enghardt, T. and C. Kraehenbuehl, "A Vocabulary of Path 1405 Properties", November 2019, . 1408 [QS-SAT] Secchi, R., Sathiaseelan, A., Potorti, F., Gotta, A., and 1409 G. Fairhurst, "Using Quick-Start to enhance TCP-friendly 1410 rate control performance in bidirectional satellite 1411 networks", 2009, 1412 . 1414 [QUIC-WG] "QUIC Working Group Home Page", n.d., 1415 . 1417 [RFC0792] Postel, J., "Internet Control Message Protocol", STD 5, 1418 RFC 792, DOI 10.17487/RFC0792, September 1981, 1419 . 1421 [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, 1422 RFC 793, DOI 10.17487/RFC0793, September 1981, 1423 . 1425 [RFC1016] Prue, W. and J. Postel, "Something a Host Could Do with 1426 Source Quench: The Source Quench Introduced Delay 1427 (SQuID)", RFC 1016, DOI 10.17487/RFC1016, July 1987, 1428 . 1430 [RFC1190] Topolcic, C., "Experimental Internet Stream Protocol: 1431 Version 2 (ST-II)", RFC 1190, DOI 10.17487/RFC1190, 1432 October 1990, . 1434 [RFC1633] Braden, R., Clark, D., and S. Shenker, "Integrated 1435 Services in the Internet Architecture: an Overview", 1436 RFC 1633, DOI 10.17487/RFC1633, June 1994, 1437 . 1439 [RFC1809] Partridge, C., "Using the Flow Label Field in IPv6", 1440 RFC 1809, DOI 10.17487/RFC1809, June 1995, 1441 . 1443 [RFC1819] Delgrossi, L., Ed. and L. Berger, Ed., "Internet Stream 1444 Protocol Version 2 (ST2) Protocol Specification - Version 1445 ST2+", RFC 1819, DOI 10.17487/RFC1819, August 1995, 1446 . 1448 [RFC1887] Rekhter, Y., Ed. and T. Li, Ed., "An Architecture for IPv6 1449 Unicast Address Allocation", RFC 1887, 1450 DOI 10.17487/RFC1887, December 1995, 1451 . 1453 [RFC2001] Stevens, W., "TCP Slow Start, Congestion Avoidance, Fast 1454 Retransmit, and Fast Recovery Algorithms", RFC 2001, 1455 DOI 10.17487/RFC2001, January 1997, 1456 . 1458 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1459 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1460 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1461 September 1997, . 1463 [RFC2210] Wroclawski, J., "The Use of RSVP with IETF Integrated 1464 Services", RFC 2210, DOI 10.17487/RFC2210, September 1997, 1465 . 1467 [RFC2211] Wroclawski, J., "Specification of the Controlled-Load 1468 Network Element Service", RFC 2211, DOI 10.17487/RFC2211, 1469 September 1997, . 1471 [RFC2212] Shenker, S., Partridge, C., and R. Guerin, "Specification 1472 of Guaranteed Quality of Service", RFC 2212, 1473 DOI 10.17487/RFC2212, September 1997, 1474 . 1476 [RFC2215] Shenker, S. and J. Wroclawski, "General Characterization 1477 Parameters for Integrated Service Network Elements", 1478 RFC 2215, DOI 10.17487/RFC2215, September 1997, 1479 . 1481 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1482 and W. Weiss, "An Architecture for Differentiated 1483 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1484 . 1486 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1487 of Explicit Congestion Notification (ECN) to IP", 1488 RFC 3168, DOI 10.17487/RFC3168, September 2001, 1489 . 1491 [RFC3697] Rajahalme, J., Conta, A., Carpenter, B., and S. Deering, 1492 "IPv6 Flow Label Specification", RFC 3697, 1493 DOI 10.17487/RFC3697, March 2004, 1494 . 1496 [RFC4094] Manner, J. and X. Fu, "Analysis of Existing Quality-of- 1497 Service Signaling Protocols", RFC 4094, 1498 DOI 10.17487/RFC4094, May 2005, 1499 . 1501 [RFC4443] Conta, A., Deering, S., and M. Gupta, Ed., "Internet 1502 Control Message Protocol (ICMPv6) for the Internet 1503 Protocol Version 6 (IPv6) Specification", STD 89, 1504 RFC 4443, DOI 10.17487/RFC4443, March 2006, 1505 . 1507 [RFC4782] Floyd, S., Allman, M., Jain, A., and P. Sarolahti, "Quick- 1508 Start for TCP and IP", RFC 4782, DOI 10.17487/RFC4782, 1509 January 2007, . 1511 [RFC5082] Gill, V., Heasley, J., Meyer, D., Savola, P., Ed., and C. 1512 Pignataro, "The Generalized TTL Security Mechanism 1513 (GTSM)", RFC 5082, DOI 10.17487/RFC5082, October 2007, 1514 . 1516 [RFC5218] Thaler, D. and B. Aboba, "What Makes for a Successful 1517 Protocol?", RFC 5218, DOI 10.17487/RFC5218, July 2008, 1518 . 1520 [RFC5533] Nordmark, E. and M. Bagnulo, "Shim6: Level 3 Multihoming 1521 Shim Protocol for IPv6", RFC 5533, DOI 10.17487/RFC5533, 1522 June 2009, . 1524 [RFC5575] Marques, P., Sheth, N., Raszuk, R., Greene, B., Mauch, J., 1525 and D. McPherson, "Dissemination of Flow Specification 1526 Rules", RFC 5575, DOI 10.17487/RFC5575, August 2009, 1527 . 1529 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 1530 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 1531 . 1533 [RFC5971] Schulzrinne, H. and R. Hancock, "GIST: General Internet 1534 Signalling Transport", RFC 5971, DOI 10.17487/RFC5971, 1535 October 2010, . 1537 [RFC5973] Stiemerling, M., Tschofenig, H., Aoun, C., and E. Davies, 1538 "NAT/Firewall NSIS Signaling Layer Protocol (NSLP)", 1539 RFC 5973, DOI 10.17487/RFC5973, October 2010, 1540 . 1542 [RFC5974] Manner, J., Karagiannis, G., and A. McDonald, "NSIS 1543 Signaling Layer Protocol (NSLP) for Quality-of-Service 1544 Signaling", RFC 5974, DOI 10.17487/RFC5974, October 2010, 1545 . 1547 [RFC5981] Manner, J., Stiemerling, M., Tschofenig, H., and R. Bless, 1548 Ed., "Authorization for NSIS Signaling Layer Protocols", 1549 RFC 5981, DOI 10.17487/RFC5981, February 2011, 1550 . 1552 [RFC6294] Hu, Q. and B. Carpenter, "Survey of Proposed Use Cases for 1553 the IPv6 Flow Label", RFC 6294, DOI 10.17487/RFC6294, June 1554 2011, . 1556 [RFC6398] Le Faucheur, F., Ed., "IP Router Alert Considerations and 1557 Usage", BCP 168, RFC 6398, DOI 10.17487/RFC6398, October 1558 2011, . 1560 [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, 1561 "IPv6 Flow Label Specification", RFC 6437, 1562 DOI 10.17487/RFC6437, November 2011, 1563 . 1565 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 1566 for Equal Cost Multipath Routing and Link Aggregation in 1567 Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011, 1568 . 1570 [RFC6633] Gont, F., "Deprecation of ICMP Source Quench Messages", 1571 RFC 6633, DOI 10.17487/RFC6633, May 2012, 1572 . 1574 [RFC6928] Chu, J., Dukkipati, N., Cheng, Y., and M. Mathis, 1575 "Increasing TCP's Initial Window", RFC 6928, 1576 DOI 10.17487/RFC6928, April 2013, 1577 . 1579 [RFC7305] Lear, E., Ed., "Report from the IAB Workshop on Internet 1580 Technology Adoption and Transition (ITAT)", RFC 7305, 1581 DOI 10.17487/RFC7305, July 2014, 1582 . 1584 [RFC7418] Dawkins, S., Ed., "An IRTF Primer for IETF Participants", 1585 RFC 7418, DOI 10.17487/RFC7418, December 2014, 1586 . 1588 [RFC8085] Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage 1589 Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085, 1590 March 2017, . 1592 [RFC8170] Thaler, D., Ed., "Planning for Protocol Adoption and 1593 Subsequent Transitions", RFC 8170, DOI 10.17487/RFC8170, 1594 May 2017, . 1596 [SAAG-105-Min] 1597 "Security Area Open Meeting - IETF-105 Minutes", July 1598 2019, . 1601 [SAF07] Sarolahti, P., Allman, M., and S. Floyd, "Determining an 1602 appropriate sending rate over an underutilized network 1603 path", Computer Networking Volume 51, Number 7, May 2007. 1605 [Sch11] Scharf, M., "Fast Startup Internet Congestion Control for 1606 Broadband Interactive Applications", Ph.D. Thesis, 1607 University of Stuttgart, April 2011. 1609 [Shim6-35] 1610 Meyer, D., Huston, G., Schiller, J., and V. Gill, "IAB 1611 IPv6 Multihoming Panel at NANOG 35", NANOG North American 1612 Network Operator Group, October 2005, 1613 . 1615 [TAPS-WG] "Transport Services Working Group Home Page", n.d., 1616 . 1618 [TRIGTRAN-55] 1619 "Triggers for Transport BOF at IETF 55", July 2003, 1620 . 1622 [TRIGTRAN-56] 1623 "Triggers for Transport BOF at IETF 56", November 2003, 1624 . 1626 Author's Address 1628 Spencer Dawkins (editor) 1629 Tencent America 1631 Email: spencerdawkins.ietf@gmail.com