idnits 2.17.1 draft-irtf-panrg-what-not-to-do-13.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (1 September 2020) is 1334 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-04) exists of draft-iab-use-it-or-lose-it-00 == Outdated reference: A later version (-12) exists of draft-irtf-panrg-questions-07 -- Obsolete informational reference (is this intentional?): RFC 793 (Obsoleted by RFC 9293) -- Obsolete informational reference (is this intentional?): RFC 1190 (Obsoleted by RFC 1819) -- Obsolete informational reference (is this intentional?): RFC 2001 (Obsoleted by RFC 2581) -- Obsolete informational reference (is this intentional?): RFC 3697 (Obsoleted by RFC 6437) -- Obsolete informational reference (is this intentional?): RFC 5575 (Obsoleted by RFC 8955) Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 6 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 PANRG S. Dawkins, Ed. 3 Internet-Draft Tencent America 4 Intended status: Informational 1 September 2020 5 Expires: 5 March 2021 7 Path Aware Networking: Obstacles to Deployment (A Bestiary of Roads Not 8 Taken) 9 draft-irtf-panrg-what-not-to-do-13 11 Abstract 13 At the first meeting of the Path Aware Networking Research Group, the 14 research group agreed to catalog and analyze past efforts to develop 15 and deploy Path Aware techniques, most of which were unsuccessful or 16 at most partially successful, in order to extract insights and 17 lessons for path-aware networking researchers. 19 This document contains that catalog and analysis. 21 Status of This Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at https://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on 5 March 2021. 38 Copyright Notice 40 Copyright (c) 2020 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 45 license-info) in effect on the date of publication of this document. 46 Please review these documents carefully, as they describe your rights 47 and restrictions with respect to this document. 49 Table of Contents 51 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 52 1.1. Purpose of This Document . . . . . . . . . . . . . . . . 4 53 1.2. What Does "Path Awareness" Mean in this Document? . . . . 4 54 1.3. Note to RFC Editor . . . . . . . . . . . . . . . . . . . 4 55 1.4. A Note About Path-Aware Techniques Included In This 56 Document . . . . . . . . . . . . . . . . . . . . . . . . 5 57 1.5. Venue for Discussion of this Document . . . . . . . . . . 5 58 1.6. Architectural Guidance . . . . . . . . . . . . . . . . . 5 59 1.7. Terminology Used in this Document . . . . . . . . . . . . 6 60 2. Summary of Lessons Learned . . . . . . . . . . . . . . . . . 6 61 2.1. Justifying Deployment . . . . . . . . . . . . . . . . . . 6 62 2.2. Providing Benefits for Early Adopters . . . . . . . . . . 6 63 2.3. Providing Benefits During Partial Deployment . . . . . . 7 64 2.4. Outperforming End-to-end Protocol Mechanisms . . . . . . 7 65 2.5. Paying for Path Aware Techniques . . . . . . . . . . . . 7 66 2.6. Impact on Operational Practices . . . . . . . . . . . . . 7 67 2.7. Per-connection State . . . . . . . . . . . . . . . . . . 7 68 2.8. Keeping Traffic on Fast-paths . . . . . . . . . . . . . . 8 69 2.9. Endpoints Trusting Intermediate Nodes . . . . . . . . . . 8 70 2.10. Intermediate Nodes Trusting Endpoints . . . . . . . . . . 8 71 2.11. Reacting to Distant Signals . . . . . . . . . . . . . . . 8 72 2.12. Support in Endpoint Protocol Stacks . . . . . . . . . . . 9 73 3. Do We Understand the Lessons We've Learned? . . . . . . . . . 9 74 4. Template for Contributions . . . . . . . . . . . . . . . . . 12 75 5. Contributions . . . . . . . . . . . . . . . . . . . . . . . . 12 76 5.1. Stream Transport (ST, ST2, ST2+) . . . . . . . . . . . . 12 77 5.1.1. Reasons for Non-deployment . . . . . . . . . . . . . 13 78 5.1.2. Lessons Learned. . . . . . . . . . . . . . . . . . . 13 79 5.2. Integrated Services (IntServ) . . . . . . . . . . . . . . 13 80 5.2.1. Reasons for Non-deployment . . . . . . . . . . . . . 14 81 5.2.2. Lessons Learned. . . . . . . . . . . . . . . . . . . 15 82 5.3. Quick-Start TCP . . . . . . . . . . . . . . . . . . . . . 15 83 5.3.1. Reasons for Non-deployment . . . . . . . . . . . . . 16 84 5.3.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 17 85 5.4. ICMP Source Quench . . . . . . . . . . . . . . . . . . . 18 86 5.4.1. Reasons for Non-deployment . . . . . . . . . . . . . 18 87 5.4.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 18 88 5.5. Triggers for Transport (TRIGTRAN) . . . . . . . . . . . . 19 89 5.5.1. Reasons for Non-deployment . . . . . . . . . . . . . 20 90 5.5.2. Lessons Learned. . . . . . . . . . . . . . . . . . . 21 91 5.6. Shim6 . . . . . . . . . . . . . . . . . . . . . . . . . . 21 92 5.6.1. Reasons for Non-deployment . . . . . . . . . . . . . 22 93 5.6.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 23 94 5.6.3. Addendum on MultiPath TCP . . . . . . . . . . . . . . 23 95 5.7. Next Steps in Signaling (NSIS) . . . . . . . . . . . . . 23 96 5.7.1. Reasons for Non-deployment . . . . . . . . . . . . . 24 97 5.7.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 25 98 5.8. IPv6 Flow Label . . . . . . . . . . . . . . . . . . . . . 26 99 5.8.1. Reasons for Non-deployment . . . . . . . . . . . . . 27 100 5.8.2. Lessons Learned . . . . . . . . . . . . . . . . . . . 28 101 6. Security Considerations . . . . . . . . . . . . . . . . . . . 29 102 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29 103 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 29 104 9. Informative References . . . . . . . . . . . . . . . . . . . 30 105 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 36 107 1. Introduction 109 At the first meeting of the Path Aware Networking Research Group 110 [PANRG], at IETF 99 [PANRG-99], Olivier Bonaventure led a discussion 111 of "A Decade of Path Awareness" [PATH-Decade], on attempts, which 112 were mostly unsuccessful for a variety of reasons, to exploit Path 113 Aware techniques and achieve a variety of goals over the past decade. 114 At the end of that discussion, two things were abundantly clear. 116 * The Internet community has accumulated considerable experience 117 with many Path Aware techniques over a long period of time, and 119 * Although some path aware techniques have been deployed (for 120 example, Differentiated Services, or DiffServ [RFC2475]), most of 121 these techniques haven't seen widespread adoption and deplyment. 122 Even "successful" techniques like DiffServ can face obstacles that 123 prevents wider usage. The reasons for non-adoption and limited 124 adoption and deployment are many, and are worthy of study. 126 The meta-lessons from that experience were 128 * Path aware networking has been more Research than Engineering, so 129 establishing an IRTF Research Group for Path Aware Networking is 130 the right thing to do [RFC7418]. 132 * Analyzing a catalog of past experience to learn the reasons for 133 non-adoption would be a great first step for the Research Group. 135 Allison Mankin, as IRTF Chair, officially chartered the Path Aware 136 Networking Research Group in July, 2018. 138 This document contains the analysis performed by that research group 139 (Section 2), based on that catalog (Section 5). 141 This document represents the consensus of the Path Aware Networking 142 Research Group. 144 1.1. Purpose of This Document 146 This Informational document discusses protocol mechanisms considered, 147 and in some cases standardized, by the Internet Engineering Task 148 Force (IETF), and considers Lessons Learned from those mechanisms. 149 The intention is to inform the work of protocol designers, whether in 150 the IRTF, the IETF, or elsewhere in the Internet ecosystem. 152 As an Informational document published in the IRTF stream, this 153 document has no authority beyond the quality of the analysis it 154 contains. 156 1.2. What Does "Path Awareness" Mean in this Document? 158 The current definition of "Path Awareness", used by the Path Aware 159 Networking Research Group, appears in Section 1.1 ("Definition") in 160 [I-D.irtf-panrg-questions]. That definition is included here as a 161 convenience to the reader. 163 | For purposes of this document, "path aware networking" describes 164 | endpoint discovery of the properties of paths they use for 165 | communication, and endpoint reaction to these properties that 166 | affects routing and/or transmission; note that this can and 167 | already does happen to some extent in the current Internet 168 | architecture. Expanding on this definition, a "path aware 169 | internetwork" is one in which endpoint discovery of path 170 | properties and endpoint selection of paths used by traffic 171 | exchanged by the endpoint are explicitly supported, regardless of 172 | the specific design of the protocol features which enable this 173 | discovery and selection. 175 Because this document reflects work performed over several decades, 176 some technologies described in Section 5 may not reflect the current 177 definition, but these technologies were considered "path aware" by 178 their contributors, so these contributions are included in this 179 retrospective document. 181 1.3. Note to RFC Editor 183 If the "Definition" in Section 1.1 ("Definition") of 184 [I-D.irtf-panrg-questions] changes, the text in Section 1.2 of this 185 document should be should be changed as well. 187 Whether that happens or not, the RFC Editor is requested to remove 188 this section. 190 1.4. A Note About Path-Aware Techniques Included In This Document 192 This document does not catalog every proposed path aware networking 193 technique that was not adopted and deployed. Instead, we limited our 194 focus to technologies that passed through the IETF community, and 195 still identified enough techniques to provide background for the 196 lessons included in Section 2 to inform researchers and protocol 197 engineers in their work. 199 No shame is intended for the techniques included in this document. 200 As shown in Section 2, the quality of specific techniques had little 201 to do with whether they were deployed or not. Based on the 202 techniques cataloged in this document, it is likely that when these 203 techniques were put forward, the proponents were trying to engineer 204 something that could not be engineered without first carrying out 205 research. Actual shame would be failing to learn from experience, 206 and failing to share that experience with other networking 207 researchers and engineers. 209 1.5. Venue for Discussion of this Document 211 (RFC Editor: please remove this section before publication) 213 Discussion of specific contributed experiences and this document in 214 general should take place on the PANRG mailing list. 216 1.6. Architectural Guidance 218 As background for understanding the Lessons Learned contained in this 219 document, the reader is encouraged to become familiar with the 220 Internet Architecture Board's documents on "What Makes for a 221 Successful Protocol?" [RFC5218] and "Planning for Protocol Adoption 222 and Subsequent Transitions" [RFC8170]. 224 Although these two documents do not specifically target path-aware 225 networking protocols, they are helpful resources for readers seeking 226 to improve their understanding of considerations for successful 227 adoption and deployment of any protocol. For example, the Basic 228 Success Factors described in Setion 2.1 of [RFC5218] are helpful for 229 readers of this document. 231 Because there is an economic aspect to decisions about deployment, 232 the IAB Workshop on Internet Technology Adoption and Transition 233 [ITAT] report [RFC7305] also provides food for thought. 235 Several of the Lessons Learned in Section 2 reflect considerations 236 described in [RFC5218], [RFC7305], and [RFC8170]. 238 1.7. Terminology Used in this Document 240 The terms Node and Element in this document have the meaning defined 241 in [PathProp]. 243 2. Summary of Lessons Learned 245 This section summarizes the Lessons Learned from the contributed 246 subsections in Section 5. 248 Each Lesson Learned is tagged with one or more contributions that 249 encountered this obstacle as a significant impediment to deployment. 250 Other contributed techniques may have also encountered this obstacle, 251 but this obstacle may not have been the biggest impediment to 252 deployment for those techniques. 254 It is useful to notice that sometimes an obstacle might impede 255 deployment, while at other times, the same obstacle might prevent 256 adoption and deployment entirely. The research group discussed 257 distinguishing between obstacles that impede and obstacles that 258 prevent, but it appears that the boundary between "impede" and 259 "prevent" can shift over time - some of the Lessons Learned are based 260 on both Path Aware techniques that were not deployed, and Path Aware 261 techniques that were deployed, but were not deployed widely or 262 quickly. See Section 5.6 and Section 5.6.3 as one example of this 263 shifting boundary. 265 2.1. Justifying Deployment 267 The benefit of Path Awareness must be great enough to justify making 268 changes in an operational network. The colloquial American English 269 expression, "If it ain't broke, don't fix it" is a "best current 270 practice" on today's Internet. (See Section 5.3, Section 5.5, and 271 Section 5.4). 273 2.2. Providing Benefits for Early Adopters 275 Providing benefits for early adopters can be key - if everyone must 276 deploy a technique in order for the technique to provide benefits, or 277 even to work at all, the technique is unlikely to be adopted widely 278 or quickly. (See Section 5.2 and Section 5.3). 280 2.3. Providing Benefits During Partial Deployment 282 Some proposals require that all path elements along the full length 283 of the path must be upgraded to support a new technique, before any 284 benefits can be seen. This is likely to require coordination between 285 operators who control a subset of path elements, and between 286 operators and end users if endpoint upgrades are required. If a 287 technique provides benefits when only a part of the path has been 288 upgraded, this is likely to encourage adoption and deployment. (See 289 Section 5.2 and Section 5.3). 291 2.4. Outperforming End-to-end Protocol Mechanisms 293 Adaptive end-to-end protocol mechanisms may respond to feedback 294 quickly enough that the additional realizable benefit from a new Path 295 Aware mechanism that tries to manipulate nodes along a path, or 296 observe the attributes of nodes along a path, may be much smaller 297 than anticipated (Section 5.3 and Section 5.5). 299 2.5. Paying for Path Aware Techniques 301 "Follow the money." If operators can't charge for a Path Aware 302 technique to recover the costs of deploying it, the benefits to the 303 operator must be really significant. Corollary: If operators charge 304 for a Path Aware technique, the benefits to users of that Path Aware 305 technique must be significant enough to justify the cost. (See 306 Section 5.1, Section 5.2, and Section 5.5). 308 2.6. Impact on Operational Practices 310 Impact of a Path Aware technique requiring changes to operational 311 practices can affect how quickly or widely a promising technique is 312 deployed. These impacts may make deployment more likely, but often 313 discourage deployment. (See Section 5.6, including Section 5.6.3). 315 2.7. Per-connection State 317 Per-connection state in intermediate nodes has been an impediment to 318 adoption and deployment in the past. This is especially true as we 319 move from the edge of the network, further into the routing core (See 320 Section 5.1 and Section 5.2). 322 2.8. Keeping Traffic on Fast-paths 324 Many modern platforms, especially high-end routers, have been 325 designed with hardware that can make simple per-packet forwarding 326 decisions ("fast-paths"), but have not been designed to make heavy 327 use of in-band mechanisms such as IPv4 and IPv6 Router Alert Options 328 (RAO) that require more processing to make forwarding decisions. 329 Packets carrying in-band mechanisms are diverted to other processors 330 in the router with much lower packet processing rates. Operators can 331 be reluctant to deploy techniques that rely heavily on in-band 332 mechanisms because they may significantly reduce packet throughput. 333 (See Section 5.7). 335 2.9. Endpoints Trusting Intermediate Nodes 337 If intermediate nodes along the path can't be trusted, it's unlikely 338 that endpoints will rely on signals from intermediate nodes to drive 339 changes to endpoint behaviors. (See Section 5.5, Section 5.4). We 340 note that "trust" is not binary - one, low, level of trust applies 341 when a node issuing a message can confirm that it has visibility of 342 the packets on the path it is seeking to control [RFC8085] (e.g., an 343 ICMP message included a quoted packet from the source). A higher 344 level of trust can arise when an endpoint has established a short 345 term, or even long term, trust relationship with network nodes. 347 2.10. Intermediate Nodes Trusting Endpoints 349 If the endpoints do not have any trust relationship with the 350 intermediate nodes along a path, operators have been reluctant to 351 deploy techniques that rely on endpoints sending unauthenticated 352 control signals to routers. (See Section 5.2 and Section 5.7. We 353 also note this still remains a factor hindering deployment of 354 DiffServ). 356 2.11. Reacting to Distant Signals 358 Because the Internet is a distributed system, if the distance that 359 information from distant path elements travels to a Path Aware host 360 is sufficiently large, the information may no longer accurately 361 represent the state and situation at the distant host or elements 362 along the path when it is received locally. In this case, the 363 benefit that a Path Aware technique provides will be inconsistent, 364 and may not always be beneficial. (See Section 5.3). 366 2.12. Support in Endpoint Protocol Stacks 368 Just because a protocol stack provides a new feature/signal does not 369 mean that applications will use the feature/signal. Protocol stacks 370 may not know how to effectively utilize Path-Aware techniques, 371 because the protocol stack may require information from applications 372 to permit the technique to work effectively, but applications may not 373 a-priori know that information. Even if the application does know 374 that information, the de-facto sockets API has no way of signaling 375 application expectations for the network path to the protocol stack. 376 In order for applications to provide these expectations to protocol 377 stacks, we need an API that signals more than the packets to be sent. 378 TAPS is exploring such an API [TAPS-WG], yet even with such an API, 379 policy is needed to bind the application expectations to the network 380 characteristics. (See Section 5.1 and Section 5.2). 382 3. Do We Understand the Lessons We've Learned? 384 The initial scope for this document was roughly "what mistakes have 385 we made in the decade prior to [PANRG-99], that we shouldn't make 386 again". Some of the contributions in Section 5 predate the initial 387 scope. The earliest Path-Aware Networking technique referred to in 388 Section 5 is Section 5.1, published in the late 1970s. Given that 389 the networking ecosystem has evolved continuously, it seemed 390 reasonable to ask whether the Lessons we've Learned are still true. 392 The PANRG Research Group reviewed the Lessons Learned contained in 393 the May 23, 2019 version of this document at IETF 105 394 [PANRG-105-Min], and carried out additional discussion at IETF 106 395 [PANRG-106-Min]. Table 1 provides the "sense of the room" after 396 those discussions. The intention is to capture whether a specific 397 lesson seems to be 399 * "Invariant" - well-understood and is likely to be applicable for 400 any proposed Path Aware Networking solution. 402 * "Variable" - has impeded deployment in the past, but might not be 403 applicable in a specific technique. Engineering analysis to 404 understand whether the lesson is applicable is prudent. 406 * "Not Now" - this characteristic tends to turn up a minefield full 407 of dragons, and prudent network engineers will wish to avoid 408 gambling on a technique that relies on this, until something 409 significant changes 411 +-----------------------------------------------------+-----------+ 412 | Lesson | Category | 413 +=====================================================+===========+ 414 | Justifying Deployment (Section 2.1) | Invariant | 415 +-----------------------------------------------------+-----------+ 416 | Providing Benefits for Early Adopters (Section 2.2) | Invariant | 417 +-----------------------------------------------------+-----------+ 418 | Providing Benefits during Partial Deployment | Invariant | 419 | (Section 2.3) | | 420 +-----------------------------------------------------+-----------+ 421 | Outperforming End-to-end Protocol Mechanisms | Variable | 422 | (Section 2.4) | | 423 +-----------------------------------------------------+-----------+ 424 | Paying for Path Aware Techniques (Section 2.5) | Invariant | 425 +-----------------------------------------------------+-----------+ 426 | Impact on Operational Practices (Section 2.6) | Invariant | 427 +-----------------------------------------------------+-----------+ 428 | Per-connection State (Section 2.7) | Variable | 429 +-----------------------------------------------------+-----------+ 430 | Keeping Traffic on Fast-paths (Section 2.8) | Variable | 431 +-----------------------------------------------------+-----------+ 432 | Endpoints Trusting Intermediate Nodes (Section 2.9) | Not Now | 433 +-----------------------------------------------------+-----------+ 434 | Intermediate Nodes Trusting Endpoints | Not Now | 435 | (Section 2.10) | | 436 +-----------------------------------------------------+-----------+ 437 | Reacting to Distant Signals (Section 2.11) | Variable | 438 +-----------------------------------------------------+-----------+ 439 | Support in Endpoint Protocol Stacks (Section 2.12) | Variable | 440 +-----------------------------------------------------+-----------+ 442 Table 1 444 "Justifying Deployment", "Providing Benefits for Early Adopters", 445 "Paying for Path Aware Techniques", and "Impact on Operational 446 Practice" were considered to be invariant - the sense of the room was 447 that these would always be considerations for any proposed Path Aware 448 Technique. 450 "Providing Benefits During Partial Deployment" was added after IETF 451 105, during research group last call, and is also considered to be 452 invariant. 454 For "Outperforming End-to-end Protocol Mechanisms", there is a trade- 455 off between improved performance from Path Aware Techniques and 456 additional complexity required by some Path Aware Techniques. 458 * For example, if you can obtain the same understanding of path 459 characteristics from measurements obtained over a few more round 460 trips, endpoint implementers are unlikely to be eager to add 461 complexity, and many attributes can be measured from an endpoint, 462 without assistance from intermediate nodes. 464 For "Per-connection State", the key questions discussed in the 465 research group were "how much state" and "where state is maintained". 467 * IntServ (Section 5.2) required state at every intermediate node 468 for every connection between two endpoints. As the Internet 469 ecosystem has evolved, carrying many connections in a tunnel that 470 appears to intermediate nodes as a single connection has become 471 more common, so that additional end-to-end connections don't add 472 additional state to intermediate nodes between tunnel endpoints. 473 If these tunnels are encrypted, intermediate nodes between tunnel 474 endpoints can't distinguish between connections, even if that were 475 desirable. 477 For "Keeping Traffic on Fast-paths", we noted that this was true for 478 many platforms, but not for all. 480 * For backbone routers, this is likely an invariant, but for 481 platforms that rely more on general-purpose computers to make 482 forwarding decisions, this may not be a fatal flaw for Path Aware 483 Networking techniques. 485 For "Endpoints Trusting Intermediate Nodes" and "Intermediate Nodes 486 Trusting Endpoints", these lessons point to the broader need to 487 revisit the Internet Threat Model. 489 * We noted with relief that discussions about this were already 490 underway in the IETF community at IETF 105 (see the Security Area 491 Open Meeting minutes [SAAG-105-Min] for discussion of 492 [draft-arkko-arch-internet-threat-model] and [draft-farrell-etm]), 493 and the Internet Architecture Board has created a mailing list for 494 continued discussions ([model-t]), but we recognize that there are 495 Path Aware Networking aspects of this effort, requiring research. 497 For "Reacting to Distant Signals", we noted that not all attributes 498 are equal. 500 * If an attribute is stable over an extended period of time, is 501 difficult to observe via end-to-end mechanisms, and is valuable, 502 Path Aware Techniques that rely on that attribute to provide a 503 significant benefit become more attractive. 505 * Analysis to help identify attributes that are useful enough to 506 justify deployment of Path Aware techniques that make use of those 507 attributes would be helpful. 509 For "Support in Endpoint Protocol Stacks", we noted that current work 510 in the TAPS working group ([TAPS-WG]) could provide an API that 511 allows applications to more easily make use of Path Aware techniques, 512 but this isn't guaranteed. 514 4. Template for Contributions 516 There are many things that could be said about the Path Aware 517 networking techniques that have been developed. For the purposes of 518 this document, contributors were requested to provide 520 * the name of a technique, including an abbreviation if one was used 522 * if available, a long-term pointer to the best reference describing 523 the technique 525 * a short description of the problem the technique was intended to 526 solve 528 * a short description of the reasons why the technique wasn't 529 adopted 531 * a short statement of the lessons that researchers can learn from 532 our experience with this technique. 534 5. Contributions 536 Contributions on these Path Aware networking techniques were analyzed 537 to arrive at the Lessons Learned captured in Section 2. 539 5.1. Stream Transport (ST, ST2, ST2+) 541 The suggested references for Stream Transport are: 543 * ST - A Proposed Internet Stream Protocol [IEN-119] 545 * Experimental Internet Stream Protocol, Version 2 (ST-II) [RFC1190] 547 * Internet Stream Protocol Version 2 (ST2) Protocol Specification - 548 Version ST2+ [RFC1819] 550 The first version of Stream Transport, ST [IEN-119], was published in 551 the late 1970's and was implemented and deployed on the ARPANET at 552 small scale. It was used throughout the 1980's for experimental 553 transmission of voice, video, and distributed simulation. 555 The second version of the ST specification (ST2) [RFC1190] [RFC1819] 556 was an experimental connection-oriented internetworking protocol that 557 operated at the same layer as connectionless IP. ST2 packets could 558 be distinguished by their IP header protocol numbers (IP, at that 559 time, used protocol number 4, while ST2 used protocol number 5). 561 ST2 used a control plane layered over IP to select routes and reserve 562 capacity for real-time streams across a network path, based on a flow 563 specification communicated by a separate protocol. The flow 564 specification could be associated with QoS state in routers, 565 producing an experimental resource reservation protocol. This 566 allowed ST2 routers along a path to offer end-to-end guarantees, 567 primarily to satisfy the QoS requirements for realtime services over 568 the Internet. 570 5.1.1. Reasons for Non-deployment 572 Although implemented in a range of equipment, ST2 was not widely used 573 after completion of the experiments. It did not offer the 574 scalability and fate-sharing properties that have come to be desired 575 by the Internet community. 577 The ST2 protocol is no longer in use. 579 5.1.2. Lessons Learned. 581 As time passed, the trade-off between router processing and link 582 capacity changed. Links became faster and the cost of router 583 processing became comparatively more expensive. 585 The ST2 control protocol used "hard state" - once a route was 586 established, and resources were reserved, routes and resources 587 existing until they were explicitly released via signaling. A soft- 588 state approach was thought superior to this hard-state approach, and 589 led to development of the IntServ model described in Section 5.2. 591 5.2. Integrated Services (IntServ) 593 The suggested references for IntServ are: 595 * RFC 1633 Integrated Services in the Internet Architecture: an 596 Overview [RFC1633] 598 * RFC 2211 Specification of the Controlled-Load Network Element 599 Service [RFC2211] 601 * RFC 2212 Specification of Guaranteed Quality of Service [RFC2212] 603 * RFC 2215 General Characterization Parameters for Integrated 604 Service Network Elements [RFC2215] 606 * RFC 2205 Resource ReSerVation Protocol (RSVP) [RFC2205] 608 In 1994, when the IntServ architecture document [RFC1633] was 609 published, real-time traffic was first appearing on the Internet. At 610 that time, bandwidth was still a scarce commodity. Internet Service 611 Providers built networks over DS3 (45 Mbps) infrastructure, and sub- 612 rate (< 1 Mpbs) access was common. Therefore, the IETF anticipated a 613 need for a fine-grained QoS mechanism. 615 In the IntServ architecture, some applications can require service 616 guarantees. Therefore, those applications use the Resource 617 Reservation Protocol (RSVP) [RFC2205] to signal QoS reservations 618 across network paths. Every router in the network maintains per-flow 619 soft-state to a) perform call admission control and b) deliver 620 guaranteed service. 622 Applications use Flow Specification (Flow Specs) [RFC2210] to 623 describe the traffic that they emit. RSVP reserves capacity for 624 traffic on a per Flow Spec basis. 626 5.2.1. Reasons for Non-deployment 628 Although IntServ has been used in enterprise and government networks, 629 IntServ was never widely deployed on the Internet because of its 630 cost. The following factors contributed to operational cost: 632 * IntServ must be deployed on every router that is on a path where 633 IntServ is to be used 635 * IntServ maintained per flow state 637 As IntServ was being discussed, the following occurred: 639 * For many expected uses, it became more cost effective to solve the 640 QoS problem by adding bandwidth. Between 1994 and 2000, Internet 641 Service Providers upgraded their infrastructures from DS3 (45 642 Mbps) to OC-48 (2.4 Gbps). This meant that even if an endpoint 643 was using IntServ in an IntServ-enabled network, its requests 644 would never be denied, so endpoints and Internet Service Providers 645 had little reason to enable IntServ. 647 * DiffServ [RFC2475] offered a more cost-effective, albeit less 648 fine-grained, solution to the QoS problem. 650 5.2.2. Lessons Learned. 652 The following lessons were learned: 654 * Any mechanism that requires every onpath router to maintain per- 655 flow state is not likely to succeed, unless the additional cost 656 for offering the feature can be recovered from the user. 658 * Any mechanism that requires an operator to upgrade all of its 659 routers is not likely to succeed, unless the additional cost for 660 offering the feature can be recovered from the user. 662 In environments where IntServ has been deployed, trust relationships 663 with endpoints are very different from trust relationships on the 664 Internet itself, and there are often clearly-defined hierarchies in 665 Service Level Agreements (SLAs), and well-defined transport flows 666 operating with pre-determined capacity and latency requirements over 667 paths where capacity or other attributes are constrained. 669 IntServ was never widely deployed to manage capacity across the 670 Internet. However, the technique that it produced was deployed for 671 reasons other than bandwidth management. RSVP is widely deployed as 672 an MPLS signaling mechanism. BGP reuses the RSVP concept of Filter 673 Specs to distribute firewall filters, although they are called Flow 674 Spec Component Types in BGP [RFC5575]. 676 5.3. Quick-Start TCP 678 The suggested references for Quick-Start TCP are: 680 * Quick-Start for TCP and IP [RFC4782] 682 * Determining an appropriate initial sending rate over an 683 underutilized network path [SAF07] 685 * Fast Startup Internet Congestion Control for Broadband Interactive 686 Applications [Sch11] 688 * Using Quick-Start to enhance TCP-friendly rate control performance 689 in bidirectional satellite networks [QS-SAT] 691 Quick-Start [RFC4782] is an Experimental TCP extension that leverages 692 support from the routers on the path to determine an allowed initial 693 sending rate for a path through the Internet, either at the start of 694 data transfers or after idle periods. Without information about the 695 path, a sender cannot easily determine an appropriate initial sending 696 rate. The default TCP congestion control therefore uses the safe but 697 time-consuming slow-start algorithm [RFC5681]. With Quick-Start, 698 connections are allowed to use higher initial sending rates if there 699 is significant unused bandwidth along the path, and if the sender and 700 all of the routers along the path approve the request. 702 By examining the Time To Live (TTL) field in Quick-Start packets, a 703 sender can determine if routers on the path have approved the Quick- 704 Start request. However, this method is unable to take into account 705 the routers hidden by tunnels or other network nodes invisible at the 706 IP layer. 708 The protocol also includes a nonce that provides protection against 709 cheating routers and receivers. If the Quick-Start request is 710 explicitly approved by all routers along the path, the TCP host can 711 send at up to the approved rate; otherwise TCP would use the default 712 congestion control. Quick-Start requires modifications in the 713 involved end-systems as well in routers. Due to the resulting 714 deployment challenges, Quick-Start was only proposed in [RFC4782] for 715 controlled environments. 717 The Quick-Start mechanism is a lightweight, coarse-grained, in-band, 718 network-assisted fast startup mechanism. The benefits are studied by 719 simulation in a research paper [SAF07] that complements the protocol 720 specification. The study confirms that Quick-Start can significantly 721 speed up mid-sized data transfers. That paper also presents router 722 algorithms that do not require keeping per-flow state. Later studies 723 [Sch11] comprehensively analyzes Quick-Start with a full Linux 724 implementation and with a router fast path prototype using a network 725 processor. In both cases, Quick-Start could be implemented with 726 limited additional complexity. 728 5.3.1. Reasons for Non-deployment 730 However, experiments with Quick-Start in [Sch11] revealed several 731 challenges: 733 * Having information from the routers along the path can reduce the 734 risk of congestion, but cannot avoid it entirely. Determining 735 whether there is unused capacity is not trivial in actual router 736 and host implementations. Data about available capacity visible 737 at the IP layer may be imprecise, and due to the propagation 738 delay, information can already be outdated when it reaches a 739 sender. There is a trade-off between the speedup of data 740 transfers and the risk of congestion even with Quick-Start. This 741 could be mitigated by only allowing Quick-Start to access a 742 proportion of the unused capacity along a path. 744 * For scalable router fast path implementation, it is important to 745 enable parallel processing of packets, as this is a widely used 746 method e.g. in network processors. One challenge is 747 synchronization of information between packets that are processed 748 in parallel, which should be avoided as much as possible. 750 * Only some types of application traffic can benefit from Quick- 751 Start. Capacity needs to be requested and discovered. The 752 discovered capacity needs to be utilized by the flow, or it 753 implicitly becomes available for other flows. Failing to use the 754 requested capacity may have already reduced the pool of Quick- 755 Start capacity that was made available to other competing Quick- 756 Start requests. The benefit is greatest when senders use this 757 only for bulk flows and avoid sending unnecessary Quick-Start 758 requests, e.g. for flows that only send a small amount of data. 759 Choosing an appropriate request size requires application-internal 760 knowledge that is not commonly expressed by the transport API. 761 How a sender can determine the rate for an initial Quick-Start 762 request is still a largely unsolved problem. 764 There is no known deployment of Quick-Start for TCP or other IETF 765 transports. 767 5.3.2. Lessons Learned 769 Some lessons can be learned from Quick-Start. Despite being a very 770 light-weight protocol, Quick-Start suffers from poor incremental 771 deployment properties, both regarding the required modifications in 772 network infrastructure as well as its interactions with applications. 773 Except for corner cases, congestion control can be quite efficiently 774 performed end-to-end in the Internet, and in modern stacks there is 775 not much room for significant improvement by additional network 776 support. 778 After publication of the Quick-Start specification, there have been 779 large-scale experiments with an initial window of up to 10 MSS 780 [RFC6928]. This alternative "IW10" approach can also ramp-up data 781 transfers faster than the standard congestion control, but it only 782 requires sender-side modifications. As a result, this approach can 783 be easier and incrementally deployed in the Internet. While 784 theoretically Quick-Start can outperform "IW10", the improvement in 785 completion time for data transfer times can, in many cases, be small. 786 After publication of [RFC6928], most modern TCP stacks have increased 787 their default initial window. 789 5.4. ICMP Source Quench 791 The suggested references for ICMP Source Quench are: 793 * INTERNET CONTROL MESSAGE PROTOCOL [RFC0792] 795 The ICMP Source Quench message [RFC0792] allowed an on-path router to 796 request the source of a flow to reduce its sending rate. This method 797 allowed a router to provide an early indication of impending 798 congestion on a path to the sources that contribute to that 799 congestion. 801 5.4.1. Reasons for Non-deployment 803 This method was deployed in Internet routers over a period of time, 804 the reaction of endpoints to receiving this signal has varied. For 805 low speed links, with low multiplexing of flows the method could be 806 used to regulate (momentarily reduce) the transmission rate. 807 However, the simple signal does not scale with link speed, or the 808 number of flows sharing a link. 810 The approach was overtaken by the evolution of congestion control 811 methods in TCP [RFC2001], and later also by other IETF transports. 812 Because these methods were based upon measurement of the end-to-end 813 path and an algorithm in the endpoint, they were able to evolve and 814 mature more rapidly than methods relying on interactions between 815 operational routers and endpoint stacks. 817 After ICMP Source Quench was specified, the IETF began to recommend 818 that transports provide end-to-end congestion control [RFC2001]. The 819 Source Quench method has been obsoleted by the IETF [RFC6633], and 820 both hosts and routers must now silently discard this message. 822 5.4.2. Lessons Learned 824 This method had several problems: 826 First, [RFC0792] did not sufficiently specify how the sender would 827 react to the ICMP Source Quench signal from the path (e.g., 828 [RFC1016]). There was ambiguity in how the sender should utilize 829 this additional information. This could lead to unfairness in the 830 way that receivers (or routers) responded to this message. 832 Second, while the message did provide additional information, the 833 Explicit Congestion Notification (ECN) mechanism [RFC3168] provided a 834 more robust and informative signal for network nodes to provide early 835 indication that a path has become congested. 837 The mechanism originated at a time when the Internet trust model was 838 very different. Most endpoint implementations did not attempt to 839 verify that the message originated from an on-path node before they 840 utilized the message. This made it vulnerable to denial of service 841 attacks. In theory, routers might have chosen to use the quoted 842 packet contained in the ICMP payload to validate that the message 843 originated from an on-path node, but this would have increased per- 844 packet processing overhead for each router along the path, would have 845 required transport functionality in the router to verify whether the 846 quoted packet header corresponded to a packet the router had sent. 847 In addition, section 5.2 of [RFC4443] noted ICMPv6-based attacks on 848 hosts that would also have threatened routers processing ICMPv6 849 Source Quench payloads. As time passed, it became increasingly 850 obvious that the lack of validation of the messages exposed receivers 851 to a security vulnerability where the messages could be forged to 852 create a tangible denial of service opportunity. 854 5.5. Triggers for Transport (TRIGTRAN) 856 The suggested references for TRIGTRAN are: 858 * TRIGTRAN BOF at IETF 55 [TRIGTRAN-55] 860 * TRIGTRAN BOF at IETF 56 [TRIGTRAN-56] 862 TCP [RFC0793] has a well-known weakness - the end-to-end flow control 863 mechanism has only a single signal, the loss of a segment, and TCP 864 implementations since the late 1980s have interpreted the loss of a 865 segment as evidence that the path between two endpoints may have 866 become congested enough to exhaust buffers on intermediate hops, so 867 that the TCP sender should "back off" - reduce its sending rate until 868 it knows that its segments are now being delivered without loss 869 [RFC5681]. More modern TCP stacks have added a growing array of 870 strategies about how to establish the sending rate [RFC5681], but 871 when a path is no longer operational, TCP would continue to retry 872 transmissions, which would fail, again, and double their 873 Retransmission Time Out (RTO) timers with each failed transmission, 874 with the result that TCP would wait many seconds before retrying a 875 segment, even if the path becomes operational while the sender is 876 waiting for its next retry. 878 The thinking behind TRIGTRAN was that if a path completely stopped 879 working because a link along the path was "down", somehow something 880 along the path could signal TCP when that link returned to service, 881 and the sending TCP could retry immediately, without waiting for a 882 full retransmission timeout (RTO) period. 884 5.5.1. Reasons for Non-deployment 886 The early dreams for TRIGTRAN were dashed because of an assumption 887 that TRIGTRAN triggers would be unauthenticated. This meant that any 888 "safe" TRIGTRAN mechanism would have relied on a mechanism such as 889 setting the IPv4 TTL or IPv6 Hop Count to 255 at a sender and testing 890 that it was 254 upon receipt, so that a receiver could verify that a 891 signal was generated by an adjacent sender known to be on the path 892 being used, and not some unknown sender which might not even be on 893 the path (e.g., "The Generalized TTL Security Mechanism (GTSM)" 894 [RFC5082]). This situation is very similar to the case for ICMP 895 Source Quench messages as described in Section 5.4, which were also 896 unauthenticated, and could be sent by an off-path attacker, resulting 897 in deprecation of ICMP Source Quench message processing [RFC6633]. 899 TRIGTRAN's scope shrunk from "the path is down" to "the first-hop 900 link is down". 902 But things got worse. 904 Because TRIGTRAN triggers would only be provided when the first-hop 905 link was "down", TRIGTRAN triggers couldn't replace normal TCP 906 retransmission behavior if the path failed because some link further 907 along the network path was "down". So TRIGTRAN triggers added 908 complexity to an already complex TCP state machine, and did not allow 909 any existing complexity to be removed. 911 There was also an issue that the TRIGTRAN signal was not sent in 912 response to a specific host that had been sending packets, and was 913 instead a signal that stimulated a response by any sender on the 914 link. This needs to scale when there are multiple flows trying to 915 use the same resource, yet the sender of a trigger has no 916 understanding how many of the potential traffic sources will respond 917 by sending packets - if recipients of the signal back-off their 918 responses to a trigger to improve scaling, then that immediately 919 mitigates the benefit of the signal. 921 Finally, intermediate forwarding nodes required modification to 922 provide TRIGTRAN triggers, but operators couldn't charge for TRIGTRAN 923 triggers, so there was no way to recover the cost of modifying, 924 testing, and deploying updated intermediate nodes. 926 Two TRIGTRAN BOFs were held, at IETF 55 [TRIGTRAN-55] and IETF 56 927 [TRIGTRAN-56], but this work was not chartered, and there was no 928 interest in deploying TRIGTRAN unless it was chartered and 929 standardized in the IETF. 931 5.5.2. Lessons Learned. 933 The reasons why this work was not chartered, much less deployed, 934 provide several useful lessons for researchers. 936 * TRIGTRAN started with a plausible value proposition, but 937 networking realities in the early 2000s forced reductions in scope 938 that led directly to reductions in potential benefits, but no 939 corresponding reductions in costs and complexity. 941 * These reductions in scope were the direct result of an inability 942 for hosts to trust or authenticate TRIGTRAN signals they received 943 from the network. 945 * Operators did not believe they could charge for TRIGTRAN 946 signaling, because first-hop links didn't fail frequently, and 947 TRIGTRAN provided no reduction in operating expenses, so there was 948 little incentive to purchase and deploy TRIGTRAN-capable network 949 equipment. 951 It is also worth noting that the targeted environment for TRIGTRAN in 952 the late 1990s contained links with a relatively small number of 953 directly-connected hosts - for instance, cellular or satellite links. 954 The transport community was well aware of the dangers of sender 955 synchronization based on multiple senders receiving the same stimulus 956 at the same time, but the working assumption for TRIGTRAN was that 957 there wouldn't be enough senders for this to be a meaningful problem. 958 In the 2010s, it is common for a single "link" to support many 959 senders and receivers on a single link, likely requiring TRIGTRAN 960 senders to wait some random amount of time before sending after 961 receiving a TRIGTRAN signal, which would have reduced the benefits of 962 TRIGTRAN even more. 964 5.6. Shim6 966 The suggested references for Shim6 are: 968 * Shim6: Level 3 Multihoming Shim Protocol for IPv6 [RFC5533] 970 The IPv6 routing architecture [RFC1887] assumed that most sites on 971 the Internet would be identified by Provider Assigned IPv6 prefixes, 972 so that Default-Free Zone routers only contained routes to other 973 providers, resulting in a very small IPv6 global routing table. 975 For a single-homed site, this could work well. A multihomed site 976 with only one upstream provider could also work well, although BGP 977 multihoming from a single upstream provider was often a premium 978 service (costing more than twice as much as two single-homed sites), 979 and if the single upstream provider went out of service, all of the 980 multihomed paths could fail simultaneously. 982 IPv4 sites often multihomed by obtaining Provider Independent 983 prefixes, and advertising these prefixes through multiple upstream 984 providers. With the assumption that any multihomed IPv4 site would 985 also multihome in IPv6, it seemed likely that IPv6 routing would be 986 subject to the same pressures to announce Provider Independent 987 prefixes, resulting in a global IPv6 routing table that exhibited the 988 same explosive growth as the global IPv4 routing table. During the 989 early 2000s, work began on a protocol that would provide multihoming 990 for IPv6 sites without requiring sites to advertise Provider 991 Independent prefixes into the IPv6 global routing table. 993 This protocol, called Shim6, allowed two endpoints to exchange 994 multiple addresses ("Locators") that all mapped to the same endpoint 995 ("Identity"). After an endpoint learned multiple Locators for the 996 other endpoint, it could send to any of those Locators with the 997 expectation that those packets would all be delivered to the endpoint 998 with the same Identity. Shim6 was an example of an "Identity/Locator 999 Split" protocol. 1001 Shim6, as defined in [RFC5533] and related RFCs, provided a workable 1002 solution for IPv6 multihoming using Provider Assigned prefixes, 1003 including capability discovery and negotiation, and allowing end-to- 1004 end application communication to continue even in the face of path 1005 failure, because applications don't see Locator failures, and 1006 continue to communicate with the same Identity using a different 1007 Locator. 1009 5.6.1. Reasons for Non-deployment 1011 Note that the problem being addressed was "site multihoming", but 1012 Shim6 was providing "host multihoming". That meant that the decision 1013 about what path would be used was under host control, not under 1014 router control. 1016 Although more work could have been done to provide a better technical 1017 solution, the biggest impediments to Shim6 deployment were 1018 operational and business considerations. These impediments were 1019 discussed at multiple network operator group meetings, including 1020 [Shim6-35] at [NANOG-35]. 1022 The technique issues centered around concerns that Shim6 relied on 1023 the host to track all the connections, while also tracking Identity/ 1024 Locator mappings in the kernel, and tracking failures to recognize 1025 that a backup path has failed. 1027 The operator issues centered around concerns that operators were 1028 performing traffic engineering, but would have no visibility or 1029 control over hosts when they chose to begin using another path, and 1030 relying on hosts to engineer traffic exposed their networks to 1031 oscillation based on feedback loops, as hosts move from path to path. 1032 At a minimum, traffic engineering policies must be pushed down to 1033 individual hosts. In addition, firewalls that expected to find a 1034 transport-level protocol header in the IP payload, would see a Shim6 1035 Identity header, and be unable to perform transport-protocol-based 1036 firewalling functions because its normal processing logic would not 1037 look past the Identity header. 1039 The business issues centered removing or reducing the ability to sell 1040 BGP multihoming service, which is often more expensive than single- 1041 homed connectivity. 1043 5.6.2. Lessons Learned 1045 It is extremely important to take operational concerns into account 1046 when a path-aware protocol is making decisions about path selection 1047 that may conflict with existing operational practices and business 1048 considerations. 1050 5.6.3. Addendum on MultiPath TCP 1052 During discussions in the PANRG session at IETF 103 [PANRG-103-Min], 1053 Lars Eggert, past Transport Area Director, pointed out that during 1054 charter discussions for the Multipath TCP working group [MP-TCP], 1055 operators expressed concerns that customers could use Multipath TCP 1056 to loadshare TCP connections across operators simultaneously and 1057 compare passive performance measurements across network paths in real 1058 time, changing the balance of power in those business relationships. 1059 Although the Multipath TCP working group was chartered, this concern 1060 could have acted as an obstacle to deployment. 1062 Operator objections to Shim6 were focused on technical concerns, but 1063 this concern could have also been an obstacle to Shim6 deployment if 1064 the technical concerns had been overcome. 1066 5.7. Next Steps in Signaling (NSIS) 1068 The suggested references for Next Steps in Signaling (NSIS) are: 1070 * the concluded working group charter [NSIS-CHARTER-2001] 1072 * GIST: General Internet Signalling Transport [RFC5971] 1074 * NAT/Firewall NSIS Signaling Layer Protocol (NSLP) [RFC5973] 1076 * NSIS Signaling Layer Protocol (NSLP) for Quality-of-Service 1077 Signaling [RFC5974] 1079 * Authorization for NSIS Signaling Layer Protocols [RFC5981] 1081 The NSIS Working Group worked on signaling techniques for network 1082 layer resources (e.g., QoS resource reservations, Firewall and NAT 1083 traversal). 1085 When RSVP [RFC2205] was used in deployments, a number of questions 1086 came up about its perceived limitations and potential missing 1087 features. The issues noted in the NSIS Working Group charter 1088 [NSIS-CHARTER-2001] include interworking between domains with 1089 different QoS architectures, mobility and roaming for IP interfaces, 1090 and complexity. Later, the lack of security in RSVP was also 1091 recognized ([RFC4094]). 1093 The NSIS Working Group was chartered to tackle those issues and 1094 initially focused on QoS signaling as its primary use case. However, 1095 over time a new approach evolved that introduced a modular 1096 architecture using application-specific signaling protocols (the NSIS 1097 Signaling Layer Protocol (NSLP)) on top of a generic signaling 1098 transport protocol (the NSIS Transport Layer Protocol (NTLP)). 1100 The NTLP is defined in [RFC5971]. Two NSLPs are defined: the NSIS 1101 Signaling Layer Protocol (NSLP) for Quality-of-Service Signaling 1102 [RFC5974] as well as the NAT/Firewall NSIS Signaling Layer Protocol 1103 (NSLP) [RFC5973]. 1105 5.7.1. Reasons for Non-deployment 1107 The obstacles for deployment can be grouped into implementation- 1108 related aspects and operational aspects. 1110 * Implementation-related aspects: 1112 Although NSIS provides benefits with respect to flexibility, 1113 mobility, and security compared to other network signaling 1114 techniques, hardware vendors were reluctant to deploy this solution, 1115 because it would require additional implementation effort and would 1116 result in additional complexity for router implementations. 1118 The NTLP mainly operates as path-coupled signaling protocol, i.e, its 1119 messages are processed at the intermediate node's control plane that 1120 are also forwarding the data flows. This requires a mechanism to 1121 intercept signaling packets while they are forwarded in the same 1122 manner (especially along the same path) as data packets. NSIS uses 1123 the IPv4 and IPv6 Router Alert Option (RAO) to allow for interception 1124 of those path-coupled signaling messages, and this technique requires 1125 router implementations to correctly understand and implement the 1126 handling of RAOs, e.g., to only process packet with RAOs of interest 1127 and to leave packets with irrelevant RAOs in the fast forwarding 1128 processing path (a comprehensive discussion of these issues can be 1129 found in [RFC6398]). The latter was an issue with some router 1130 implementations at the time of standardization. 1132 Another reason is that path-coupled signaling protocols that interact 1133 with routers and request manipulation of state at these routers (or 1134 any other network element in general) are under scrutiny: a packet 1135 (or sequence of packets) out of the mainly untrusted data path is 1136 requesting creation and manipulation of network state. This is seen 1137 as potentially dangerous (e.g., opens up a Denial of Service (DoS) 1138 threat to a router's control plane) and difficult for an operator to 1139 control. Path-coupled signaling approaches were considered 1140 problematic (see also section 3 of [RFC6398]). There are 1141 recommendations on how to secure NSIS nodes and deployments (e.g., 1142 [RFC5981]). 1144 * Operational Aspects: 1146 NSIS not only required trust between customers and their provider, 1147 but also among different providers. Especially, QoS signaling 1148 techniques would require some kind of dynamic service level agreement 1149 support that would imply (potentially quite complex) bilateral 1150 negotiations between different Internet service providers. This 1151 complexity was not considered to be justified and increasing the 1152 bandwidth (and thus avoiding bottlenecks) was cheaper than actively 1153 managing network resource bottlenecks by using path-coupled QoS 1154 signaling techniques. Furthermore, an end-to-end path typically 1155 involves several provider domains and these providers need to closely 1156 cooperate in cases of failures. 1158 5.7.2. Lessons Learned 1160 One goal of NSIS was to decrease the complexity of the signaling 1161 protocol, but a path-coupled signaling protocol comes with the 1162 intrinsic complexity of IP-based networks, beyond the complexity of 1163 the signaling protocol itself. Sources of intrinsic complexity 1164 include: 1166 * the presence of asymmetric routes between endpoints and routers 1168 * the lack of security and trust at large in the Internet 1169 infrastructure 1171 * the presence of different trust boundaries 1173 * the effects of best-effort networks (e.g., robustness to packet 1174 loss) 1176 * divergence from the fate sharing principle (e.g., state within the 1177 network). 1179 Any path-coupled signaling protocol has to deal with these realities. 1181 Operators view the use of IPv4 and IPv6 Router Alert Option (RAO) to 1182 signal routers along the path from end systems with suspicion, 1183 because these end systems are usually not authenticated and heavy use 1184 of RAOs can easily increase the CPU load on routers that are designed 1185 to process most packets using a hardware "fast path" and diverting 1186 packets containing RAO to a slower, more capable processor. 1188 5.8. IPv6 Flow Label 1190 The suggested references for IPv6 Flow Label are: 1192 * IPv6 Flow Label Specification [RFC6437] 1194 IPv6 specifies a 20-bit field Flow Label field [RFC6437], included in 1195 the fixed part of the IPv6 header and hence present in every IPv6 1196 packet. An endpoint sets the value in this field to one of a set of 1197 pseudo-randomly assigned values. If a packet is not part of any 1198 flow, the flow label value is set to zero [RFC3697]. A number of 1199 Standards Track and Best Current Practice RFCs (e.g., [RFC8085], 1200 [RFC6437], [RFC6438]) encourage IPv6 endpoints to set a non-zero 1201 value in this field. A multiplexing transport could choose to use 1202 multiple flow labels to allow the network to independently forward 1203 its subflows, or to use one common value for the traffic aggregate. 1204 The flow label is present in all fragments. IPsec was originally put 1205 forward as one important use-case for this mechanism and does encrypt 1206 the field [RFC6438]. 1208 Once set, the flow label can provide information that can help inform 1209 network nodes about subflows present at the transport layer, without 1210 needing to interpret the setting of upper layer protocol fields 1211 [RFC6294]. This information can also be used to coordinate how 1212 aggregates of transport subflows are grouped when queued in the 1213 network and to select appropriate per-flow forwarding when choosing 1214 between alternate paths [RFC6438] (e.g. for Equal Cost Multipath 1215 Routing (ECMP) and Link Aggregation (LAG)). 1217 5.8.1. Reasons for Non-deployment 1219 Despite the field being present in every IPv6 packet, the mechanism 1220 did not receive as much use as originally envisioned. One reason is 1221 that to be useful it requires engagement by two different 1222 stakeholders: 1224 * Endpoint Implementation: 1226 For network nodes along a path to utilize the flow label there needs 1227 to be a non-zero value inserted in the field [RFC6437] at the sending 1228 endpoint. There needs to be an incentive for an endpoint to set an 1229 appropriate non-zero value. The value should appropriately reflect 1230 the level of aggregation the traffic expects to be provided by the 1231 network. However, this requires the stack to know granularity at 1232 which flows should be identified (or conversely which flows should 1233 receive aggregated treatment), i.e., which packets carry the same 1234 flow label. Therefore, setting a non-zero value may result in 1235 additional choices that need to be made by an application developer. 1237 Although the standard [RFC3697] forbids any encoding of meaning into 1238 the flow label value, the opportunity to use the flow label as a 1239 covert channel or to signal other meta-information may have raised 1240 concerns about setting a non-zero value [RFC6437]. 1242 Before methods are widely deployed to use this method, there could be 1243 no incentive for an endpoint to set the field. 1245 * Operational support in network nodes: 1247 A benefit can only be realized when a network node along the path 1248 also uses this information to inform its decisions. Network 1249 equipment (routers and/or middleboxes) need to include appropriate 1250 support so they can utilize the field when making decisions about how 1251 to classify flows, or to inform forwarding choices. Use of any 1252 optional feature in a network node also requires corresponding 1253 updates to operational procedures, and therefore is normally only 1254 introduced when the cost can be justified. 1256 A benefit from utilizing the flow label is expected to be increased 1257 quality of experience for applications - but this comes at some 1258 operational cost to an operator, and requires endpoints to set the 1259 field. 1261 5.8.2. Lessons Learned 1263 The flow label is a general purpose header field for use by the path. 1264 Multiple uses have been proposed. One candidate use was to reduce 1265 the complexity of forwarding decisions. However, modern routers can 1266 use a "fast path", often taking advantage of hardware to accelerate 1267 processing. The method can assist in more complex forwarding, such 1268 as ECMP and load balancing. 1270 Although [RFC6437] recommended that endpoints should by default 1271 choose uniformly-distributed labels for their traffic, the 1272 specification permitted an endpoint to choose to set a zero value. 1273 This ability of endpoints to choose to set a flow label of zero has 1274 had consequences on deployability: 1276 * Before wide-scale support by endpoints, it would be impossible to 1277 rely on a non-zero flow label being set. Network nodes therefore 1278 would need to also employ other techniques to realize equivalent 1279 functions. An example of a method is one assuming semantics of 1280 the source port field to provide entropy input to a network-layer 1281 hash. This use of a 5-tuple to classify a packet represents a 1282 layering violation [RFC6294]. When other methods have been 1283 deployed, they increase the cost of deploying standards-based 1284 methods, even though they may offer less control to endpoints and 1285 result in potential interaction with other uses/interpretation of 1286 the field. 1288 * Even though the flow label is specified as an end-to-end field, 1289 some network paths have been observed to not transparently forward 1290 the flow label. This could result from non-conformant equipment, 1291 or could indicate that some operational networks have chosen to 1292 re-use the protocol field for other (e.g. internal purposes). 1293 This results in lack of transparency, and a deployment hurdle to 1294 endpoints expecting that they can set a flow label that is 1295 utilized by the network. The more recent practice of "greasing" 1296 [GREASE] would suggest that a different outcome could have been 1297 achieved if endpoints were always required to set a non-zero 1298 value. 1300 * [RFC1809] noted that setting the choice of the flow label value 1301 can depend on the expectations of the traffic generated by an 1302 application, which suggests an API should be presented to control 1303 the setting or policy that is used. However, many currently 1304 available APIs do not have this support. 1306 A growth in the use of encrypted transports, (e.g. QUIC [QUIC-WG]) 1307 seems likely to raise similar issues to those discussed above and 1308 could motivate renewed interest in utilizing the flow label. 1310 6. Security Considerations 1312 This document describes Path Aware techniques that were not adopted 1313 and widely deployed on the Internet, so it doesn't affect the 1314 security of the Internet. 1316 If this document meets its goals, we may develop new techniques for 1317 Path Aware Networking that would affect the security of the Internet, 1318 but security considerations for those techniques will be described in 1319 the corresponding RFCs that specify them. 1321 7. IANA Considerations 1323 This document makes no requests of IANA. 1325 8. Acknowledgments 1327 Initial material for Section 5.1 on ST2 was provided by Gorry 1328 Fairhurst. 1330 Initial material for Section 5.2 on IntServ was provided by Ron 1331 Bonica. 1333 Initial material for Section 5.3 on Quick-Start TCP was provided by 1334 Michael Scharf, who also provided suggestions to improve this section 1335 after it was edited. 1337 Initial material for Section 5.4 on ICMP Source Quench was provided 1338 by Gorry Fairhurst. 1340 Initial material for Section 5.5 on Triggers for Transport (TRIGTRAN) 1341 was provided by Spencer Dawkins. 1343 Section 5.6 on Shim6 builds on initial material describing obstacles 1344 provided by Erik Nordmark, with background added by Spencer Dawkins. 1346 Initial material for Section 5.7 on Next Steps In Signaling (NSIS) 1347 was provided by Roland Bless and Martin Stiemerling. 1349 Initial material for Section 5.8 on IPv6 Flow Labels was provided by 1350 Gorry Fairhurst. 1352 Our thanks to C.M. Heard, David Black, Gorry Fairhurst, Joe Touch, 1353 Joeri de Ruiter, Mohamed Boucadair, Roland Bless, Ruediger Geib, 1354 Theresa Enghardt, and Wes Eddy, who provided review comments on 1355 previous versions. 1357 Special thanks to Adrian Farrel for helping Spencer navigate the 1358 twisty little passages of Flow Specs and Filter Specs in IntServ, 1359 RSVP, MPLS, and BGP. They are all alike, except when they are 1360 different [Colossal-Cave]. 1362 9. Informative References 1364 [Colossal-Cave] 1365 "Wikipedia Page for Colossal Cave Adventure", January 1366 2019, 1367 . 1369 [draft-arkko-arch-internet-threat-model] 1370 Arkko, J., "Changes in the Internet Threat Model", n.d., 1371 . 1374 [draft-farrell-etm] 1375 Farrell, S., "We're gonna need a bigger threat model", 1376 n.d., 1377 . 1379 [GREASE] Thomson, M., "Long-term Viability of Protocol Extension 1380 Mechanisms", July 2019, . 1383 [I-D.irtf-panrg-questions] 1384 Trammell, B., "Current Open Questions in Path Aware 1385 Networking", Work in Progress, Internet-Draft, draft-irtf- 1386 panrg-questions-07, 29 August 2020, . 1389 [IEN-119] Forgie, J., "ST - A Proposed Internet Stream Protocol", 1390 September 1979, 1391 . 1393 [ITAT] "IAB Workshop on Internet Technology Adoption and 1394 Transition (ITAT)", December 2013, 1395 . 1397 [model-t] "Model-t -- Discussions of changes in Internet deployment 1398 patterns and their impact on the Internet threat model", 1399 n.d., . 1401 [MP-TCP] "Multipath TCP Working Group Home Page", n.d., 1402 . 1404 [NANOG-35] "North American Network Operators Group NANOG-35 Agenda", 1405 October 2005, 1406 . 1408 [NSIS-CHARTER-2001] 1409 "Next Steps In Signaling Working Group Charter", March 1410 2011, 1411 . 1413 [PANRG] "Path Aware Networking Research Group (Home Page)", n.d., 1414 . 1416 [PANRG-103-Min] 1417 "Path Aware Networking Research Group - IETF-103 Minutes", 1418 November 2018, 1419 . 1421 [PANRG-105-Min] 1422 "Path Aware Networking Research Group - IETF-105 Minutes", 1423 July 2019, 1424 . 1426 [PANRG-106-Min] 1427 "Path Aware Networking Research Group - IETF-106 Minutes", 1428 November 2019, 1429 . 1431 [PANRG-99] "Path Aware Networking Research Group - IETF-99", July 1432 2017, 1433 . 1435 [PATH-Decade] 1436 Bonaventure, O., "A Decade of Path Awareness", July 2017, 1437 . 1440 [PathProp] Enghardt, T. and C. Kraehenbuehl, "A Vocabulary of Path 1441 Properties", November 2019, . 1444 [QS-SAT] Secchi, R., Sathiaseelan, A., Potorti, F., Gotta, A., and 1445 G. Fairhurst, "Using Quick-Start to enhance TCP-friendly 1446 rate control performance in bidirectional satellite 1447 networks", 2009, 1448 . 1450 [QUIC-WG] "QUIC Working Group Home Page", n.d., 1451 . 1453 [RFC0792] Postel, J., "Internet Control Message Protocol", STD 5, 1454 RFC 792, DOI 10.17487/RFC0792, September 1981, 1455 . 1457 [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, 1458 RFC 793, DOI 10.17487/RFC0793, September 1981, 1459 . 1461 [RFC1016] Prue, W. and J. Postel, "Something a Host Could Do with 1462 Source Quench: The Source Quench Introduced Delay 1463 (SQuID)", RFC 1016, DOI 10.17487/RFC1016, July 1987, 1464 . 1466 [RFC1190] Topolcic, C., "Experimental Internet Stream Protocol: 1467 Version 2 (ST-II)", RFC 1190, DOI 10.17487/RFC1190, 1468 October 1990, . 1470 [RFC1633] Braden, R., Clark, D., and S. Shenker, "Integrated 1471 Services in the Internet Architecture: an Overview", 1472 RFC 1633, DOI 10.17487/RFC1633, June 1994, 1473 . 1475 [RFC1809] Partridge, C., "Using the Flow Label Field in IPv6", 1476 RFC 1809, DOI 10.17487/RFC1809, June 1995, 1477 . 1479 [RFC1819] Delgrossi, L., Ed. and L. Berger, Ed., "Internet Stream 1480 Protocol Version 2 (ST2) Protocol Specification - Version 1481 ST2+", RFC 1819, DOI 10.17487/RFC1819, August 1995, 1482 . 1484 [RFC1887] Rekhter, Y., Ed. and T. Li, Ed., "An Architecture for IPv6 1485 Unicast Address Allocation", RFC 1887, 1486 DOI 10.17487/RFC1887, December 1995, 1487 . 1489 [RFC2001] Stevens, W., "TCP Slow Start, Congestion Avoidance, Fast 1490 Retransmit, and Fast Recovery Algorithms", RFC 2001, 1491 DOI 10.17487/RFC2001, January 1997, 1492 . 1494 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1495 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1496 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1497 September 1997, . 1499 [RFC2210] Wroclawski, J., "The Use of RSVP with IETF Integrated 1500 Services", RFC 2210, DOI 10.17487/RFC2210, September 1997, 1501 . 1503 [RFC2211] Wroclawski, J., "Specification of the Controlled-Load 1504 Network Element Service", RFC 2211, DOI 10.17487/RFC2211, 1505 September 1997, . 1507 [RFC2212] Shenker, S., Partridge, C., and R. Guerin, "Specification 1508 of Guaranteed Quality of Service", RFC 2212, 1509 DOI 10.17487/RFC2212, September 1997, 1510 . 1512 [RFC2215] Shenker, S. and J. Wroclawski, "General Characterization 1513 Parameters for Integrated Service Network Elements", 1514 RFC 2215, DOI 10.17487/RFC2215, September 1997, 1515 . 1517 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1518 and W. Weiss, "An Architecture for Differentiated 1519 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1520 . 1522 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1523 of Explicit Congestion Notification (ECN) to IP", 1524 RFC 3168, DOI 10.17487/RFC3168, September 2001, 1525 . 1527 [RFC3697] Rajahalme, J., Conta, A., Carpenter, B., and S. Deering, 1528 "IPv6 Flow Label Specification", RFC 3697, 1529 DOI 10.17487/RFC3697, March 2004, 1530 . 1532 [RFC4094] Manner, J. and X. Fu, "Analysis of Existing Quality-of- 1533 Service Signaling Protocols", RFC 4094, 1534 DOI 10.17487/RFC4094, May 2005, 1535 . 1537 [RFC4443] Conta, A., Deering, S., and M. Gupta, Ed., "Internet 1538 Control Message Protocol (ICMPv6) for the Internet 1539 Protocol Version 6 (IPv6) Specification", STD 89, 1540 RFC 4443, DOI 10.17487/RFC4443, March 2006, 1541 . 1543 [RFC4782] Floyd, S., Allman, M., Jain, A., and P. Sarolahti, "Quick- 1544 Start for TCP and IP", RFC 4782, DOI 10.17487/RFC4782, 1545 January 2007, . 1547 [RFC5082] Gill, V., Heasley, J., Meyer, D., Savola, P., Ed., and C. 1548 Pignataro, "The Generalized TTL Security Mechanism 1549 (GTSM)", RFC 5082, DOI 10.17487/RFC5082, October 2007, 1550 . 1552 [RFC5218] Thaler, D. and B. Aboba, "What Makes for a Successful 1553 Protocol?", RFC 5218, DOI 10.17487/RFC5218, July 2008, 1554 . 1556 [RFC5533] Nordmark, E. and M. Bagnulo, "Shim6: Level 3 Multihoming 1557 Shim Protocol for IPv6", RFC 5533, DOI 10.17487/RFC5533, 1558 June 2009, . 1560 [RFC5575] Marques, P., Sheth, N., Raszuk, R., Greene, B., Mauch, J., 1561 and D. McPherson, "Dissemination of Flow Specification 1562 Rules", RFC 5575, DOI 10.17487/RFC5575, August 2009, 1563 . 1565 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 1566 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 1567 . 1569 [RFC5971] Schulzrinne, H. and R. Hancock, "GIST: General Internet 1570 Signalling Transport", RFC 5971, DOI 10.17487/RFC5971, 1571 October 2010, . 1573 [RFC5973] Stiemerling, M., Tschofenig, H., Aoun, C., and E. Davies, 1574 "NAT/Firewall NSIS Signaling Layer Protocol (NSLP)", 1575 RFC 5973, DOI 10.17487/RFC5973, October 2010, 1576 . 1578 [RFC5974] Manner, J., Karagiannis, G., and A. McDonald, "NSIS 1579 Signaling Layer Protocol (NSLP) for Quality-of-Service 1580 Signaling", RFC 5974, DOI 10.17487/RFC5974, October 2010, 1581 . 1583 [RFC5981] Manner, J., Stiemerling, M., Tschofenig, H., and R. Bless, 1584 Ed., "Authorization for NSIS Signaling Layer Protocols", 1585 RFC 5981, DOI 10.17487/RFC5981, February 2011, 1586 . 1588 [RFC6294] Hu, Q. and B. Carpenter, "Survey of Proposed Use Cases for 1589 the IPv6 Flow Label", RFC 6294, DOI 10.17487/RFC6294, June 1590 2011, . 1592 [RFC6398] Le Faucheur, F., Ed., "IP Router Alert Considerations and 1593 Usage", BCP 168, RFC 6398, DOI 10.17487/RFC6398, October 1594 2011, . 1596 [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, 1597 "IPv6 Flow Label Specification", RFC 6437, 1598 DOI 10.17487/RFC6437, November 2011, 1599 . 1601 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 1602 for Equal Cost Multipath Routing and Link Aggregation in 1603 Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011, 1604 . 1606 [RFC6633] Gont, F., "Deprecation of ICMP Source Quench Messages", 1607 RFC 6633, DOI 10.17487/RFC6633, May 2012, 1608 . 1610 [RFC6928] Chu, J., Dukkipati, N., Cheng, Y., and M. Mathis, 1611 "Increasing TCP's Initial Window", RFC 6928, 1612 DOI 10.17487/RFC6928, April 2013, 1613 . 1615 [RFC7305] Lear, E., Ed., "Report from the IAB Workshop on Internet 1616 Technology Adoption and Transition (ITAT)", RFC 7305, 1617 DOI 10.17487/RFC7305, July 2014, 1618 . 1620 [RFC7418] Dawkins, S., Ed., "An IRTF Primer for IETF Participants", 1621 RFC 7418, DOI 10.17487/RFC7418, December 2014, 1622 . 1624 [RFC8085] Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage 1625 Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085, 1626 March 2017, . 1628 [RFC8170] Thaler, D., Ed., "Planning for Protocol Adoption and 1629 Subsequent Transitions", RFC 8170, DOI 10.17487/RFC8170, 1630 May 2017, . 1632 [SAAG-105-Min] 1633 "Security Area Open Meeting - IETF-105 Minutes", July 1634 2019, . 1637 [SAF07] Sarolahti, P., Allman, M., and S. Floyd, "Determining an 1638 appropriate sending rate over an underutilized network 1639 path", Computer Networking Volume 51, Number 7, May 2007. 1641 [Sch11] Scharf, M., "Fast Startup Internet Congestion Control for 1642 Broadband Interactive Applications", Ph.D. Thesis, 1643 University of Stuttgart, April 2011. 1645 [Shim6-35] Meyer, D., Huston, G., Schiller, J., and V. Gill, "IAB 1646 IPv6 Multihoming Panel at NANOG 35", NANOG North American 1647 Network Operator Group, October 2005, 1648 . 1650 [TAPS-WG] "Transport Services Working Group Home Page", n.d., 1651 . 1653 [TRIGTRAN-55] 1654 "Triggers for Transport BOF at IETF 55", July 2003, 1655 . 1657 [TRIGTRAN-56] 1658 "Triggers for Transport BOF at IETF 56", November 2003, 1659 . 1661 Author's Address 1663 Spencer Dawkins (editor) 1664 Tencent America 1666 Email: spencerdawkins.ietf@gmail.com