idnits 2.17.1 draft-ietf-teas-rfc3272bis-12.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (May 15, 2021) is 1077 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-28) exists of draft-ietf-alto-performance-metrics-15 == Outdated reference: A later version (-21) exists of draft-ietf-bess-evpn-unequal-lb-08 == Outdated reference: A later version (-26) exists of draft-ietf-idr-segment-routing-te-policy-11 == Outdated reference: A later version (-26) exists of draft-ietf-lsr-flex-algo-15 == Outdated reference: A later version (-17) exists of draft-ietf-lsr-ip-flexalgo-02 == Outdated reference: A later version (-22) exists of draft-ietf-spring-segment-routing-policy-11 == Outdated reference: A later version (-17) exists of draft-ietf-teas-enhanced-vpn-07 == Outdated reference: A later version (-25) exists of draft-ietf-teas-ietf-network-slices-00 == Outdated reference: A later version (-09) exists of draft-irtf-nmrg-ibn-concepts-definitions-03 -- Obsolete informational reference (is this intentional?): RFC 3272 (Obsoleted by RFC 9522) -- Obsolete informational reference (is this intentional?): RFC 7752 (Obsoleted by RFC 9552) Summary: 0 errors (**), 0 flaws (~~), 10 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TEAS Working Group A. Farrel, Ed. 3 Internet-Draft Old Dog Consulting 4 Obsoletes: 3272 (if approved) May 15, 2021 5 Intended status: Informational 6 Expires: November 16, 2021 8 Overview and Principles of Internet Traffic Engineering 9 draft-ietf-teas-rfc3272bis-12 11 Abstract 13 This document describes the principles of traffic engineering (TE) in 14 the Internet. The document is intended to promote better 15 understanding of the issues surrounding traffic engineering in IP 16 networks and the networks that support IP networking, and to provide 17 a common basis for the development of traffic engineering 18 capabilities for the Internet. The principles, architectures, and 19 methodologies for performance evaluation and performance optimization 20 of operational networks are also discussed. 22 This work was first published as RFC 3272 in May 2002. This document 23 obsoletes RFC 3272 by making a complete update to bring the text in 24 line with best current practices for Internet traffic engineering and 25 to include references to the latest relevant work in the IETF. 27 Status of This Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at https://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on November 16, 2021. 44 Copyright Notice 46 Copyright (c) 2021 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents 51 (https://trustee.ietf.org/license-info) in effect on the date of 52 publication of this document. Please review these documents 53 carefully, as they describe your rights and restrictions with respect 54 to this document. Code Components extracted from this document must 55 include Simplified BSD License text as described in Section 4.e of 56 the Trust Legal Provisions and are provided without warranty as 57 described in the Simplified BSD License. 59 Table of Contents 61 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 62 1.1. What is Internet Traffic Engineering? . . . . . . . . . . 4 63 1.2. Components of Traffic Engineering . . . . . . . . . . . . 6 64 1.3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 8 65 1.4. Terminology . . . . . . . . . . . . . . . . . . . . . . . 8 66 2. Background . . . . . . . . . . . . . . . . . . . . . . . . . 11 67 2.1. Context of Internet Traffic Engineering . . . . . . . . . 11 68 2.2. Network Domain Context . . . . . . . . . . . . . . . . . 12 69 2.3. Problem Context . . . . . . . . . . . . . . . . . . . . . 14 70 2.3.1. Congestion and its Ramifications . . . . . . . . . . 15 71 2.4. Solution Context . . . . . . . . . . . . . . . . . . . . 16 72 2.4.1. Combating the Congestion Problem . . . . . . . . . . 18 73 2.5. Implementation and Operational Context . . . . . . . . . 21 74 3. Traffic Engineering Process Models . . . . . . . . . . . . . 21 75 3.1. Components of the Traffic Engineering Process Model . . . 22 76 4. Taxonomy of Traffic Engineering Systems . . . . . . . . . . . 22 77 4.1. Time-Dependent Versus State-Dependent Versus Event- 78 Dependent . . . . . . . . . . . . . . . . . . . . . . . . 23 79 4.2. Offline Versus Online . . . . . . . . . . . . . . . . . . 24 80 4.3. Centralized Versus Distributed . . . . . . . . . . . . . 25 81 4.3.1. Hybrid Systems . . . . . . . . . . . . . . . . . . . 25 82 4.3.2. Considerations for Software Defined Networking . . . 26 83 4.4. Local Versus Global . . . . . . . . . . . . . . . . . . . 26 84 4.5. Prescriptive Versus Descriptive . . . . . . . . . . . . . 27 85 4.5.1. Intent-Based Networking . . . . . . . . . . . . . . . 27 86 4.6. Open-Loop Versus Closed-Loop . . . . . . . . . . . . . . 28 87 4.7. Tactical versus Strategic . . . . . . . . . . . . . . . . 28 88 5. Review of TE Techniques . . . . . . . . . . . . . . . . . . . 28 89 5.1. Overview of IETF Projects Related to Traffic Engineering 28 90 5.1.1. Constraint-Based Routing . . . . . . . . . . . . . . 29 91 5.1.2. Integrated Services . . . . . . . . . . . . . . . . . 31 92 5.1.3. RSVP . . . . . . . . . . . . . . . . . . . . . . . . 31 93 5.1.4. Differentiated Services . . . . . . . . . . . . . . . 32 94 5.1.5. QUIC . . . . . . . . . . . . . . . . . . . . . . . . 33 95 5.1.6. Multiprotocol Label Switching (MPLS) . . . . . . . . 34 96 5.1.7. Generalized MPLS (GMPLS) . . . . . . . . . . . . . . 34 97 5.1.8. IP Performance Metrics . . . . . . . . . . . . . . . 35 98 5.1.9. Flow Measurement . . . . . . . . . . . . . . . . . . 35 99 5.1.10. Endpoint Congestion Management . . . . . . . . . . . 36 100 5.1.11. TE Extensions to the IGPs . . . . . . . . . . . . . . 36 101 5.1.12. Link-State BGP . . . . . . . . . . . . . . . . . . . 37 102 5.1.13. Path Computation Element . . . . . . . . . . . . . . 37 103 5.1.14. Application-Layer Traffic Optimization . . . . . . . 38 104 5.1.15. Segment Routing with MPLS Encapsulation (SR-MPLS) . . 39 105 5.1.16. Segment Routing Policy . . . . . . . . . . . . . . . 40 106 5.1.17. Network Virtualization and Abstraction . . . . . . . 41 107 5.1.18. Network Slicing . . . . . . . . . . . . . . . . . . . 41 108 5.1.19. Deterministic Networking . . . . . . . . . . . . . . 42 109 5.1.20. Network TE State Definition and Presentation . . . . 43 110 5.1.21. System Management and Control Interfaces . . . . . . 43 111 5.2. Content Distribution . . . . . . . . . . . . . . . . . . 43 112 6. Recommendations for Internet Traffic Engineering . . . . . . 44 113 6.1. Generic Non-functional Recommendations . . . . . . . . . 44 114 6.2. Routing Recommendations . . . . . . . . . . . . . . . . . 46 115 6.3. Traffic Mapping Recommendations . . . . . . . . . . . . . 49 116 6.4. Measurement Recommendations . . . . . . . . . . . . . . . 50 117 6.5. Policing, Planning, and Access Control . . . . . . . . . 50 118 6.6. Network Survivability . . . . . . . . . . . . . . . . . . 51 119 6.6.1. Survivability in MPLS Based Networks . . . . . . . . 53 120 6.6.2. Protection Options . . . . . . . . . . . . . . . . . 54 121 6.7. Multi-Layer Traffic Engineering . . . . . . . . . . . . . 55 122 6.8. Traffic Engineering in Diffserv Environments . . . . . . 56 123 6.9. Network Controllability . . . . . . . . . . . . . . . . . 57 124 7. Inter-Domain Considerations . . . . . . . . . . . . . . . . . 58 125 8. Overview of Contemporary TE Practices in Operational IP 126 Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 60 127 9. Security Considerations . . . . . . . . . . . . . . . . . . . 62 128 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 63 129 11. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 63 130 12. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 64 131 13. Informative References . . . . . . . . . . . . . . . . . . . 65 132 Appendix A. Historic Overview . . . . . . . . . . . . . . . . . 78 133 A.1. Traffic Engineering in Classical Telephone Networks . . . 78 134 A.2. Evolution of Traffic Engineering in Packet Networks . . . 79 135 A.2.1. Adaptive Routing in the ARPANET . . . . . . . . . . . 80 136 A.2.2. Dynamic Routing in the Internet . . . . . . . . . . . 80 137 A.2.3. ToS Routing . . . . . . . . . . . . . . . . . . . . . 81 138 A.2.4. Equal Cost Multi-Path . . . . . . . . . . . . . . . . 81 139 A.2.5. Nimrod . . . . . . . . . . . . . . . . . . . . . . . 82 140 A.3. Development of Internet Traffic Engineering . . . . . . . 82 141 A.3.1. Overlay Model . . . . . . . . . . . . . . . . . . . . 82 142 Appendix B. Overview of Traffic Engineering Related Work in 143 Other SDOs . . . . . . . . . . . . . . . . . . . . . 83 144 B.1. Overview of ITU Activities Related to Traffic Engineering 83 146 Appendix C. Summary of Changes Since RFC 3272 . . . . . . . . . 84 147 C.1. RFC 3272 . . . . . . . . . . . . . . . . . . . . . . . . 84 148 C.2. This Document . . . . . . . . . . . . . . . . . . . . . . 87 149 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 90 151 1. Introduction 153 This document describes the principles of Internet traffic 154 engineering (TE). The objective of the document is to articulate the 155 general issues and principles for Internet traffic engineering, and 156 where appropriate to provide recommendations, guidelines, and options 157 for the development of preplanned (offline) and dynamic (online) 158 Internet traffic engineering capabilities and support systems. 160 This document provides a terminology and taxonomy for describing and 161 understanding common Internet traffic engineering concepts. 163 Even though Internet traffic engineering is most effective when 164 applied end-to-end, the focus of this document is traffic engineering 165 within a given domain (such as an autonomous system). However, 166 because a preponderance of Internet traffic tends to originate in one 167 autonomous system and terminate in another, this document also 168 provides an overview of aspects pertaining to inter-domain traffic 169 engineering. 171 This work was first published as [RFC3272] in May 2002. This 172 document obsoletes [RFC3272] by making a complete update to bring the 173 text in line with best current practices for Internet traffic 174 engineering and to include references to the latest relevant work in 175 the IETF. It is worth noting around three fifths of the RFCs 176 referenced in this document post-date the publication of RFC 3272. 177 Appendix C provides a summary of changes between RFC 3272 and this 178 document. 180 1.1. What is Internet Traffic Engineering? 182 One of the most significant functions performed in the Internet is 183 the routing and forwarding of traffic from ingress nodes to egress 184 nodes. Therefore, one of the most distinctive functions performed by 185 Internet traffic engineering is the control and optimization of these 186 routing and forwarding functions, to steer traffic through the 187 network. 189 Internet traffic engineering is defined as that aspect of Internet 190 network engineering dealing with the issues of performance evaluation 191 and performance optimization of operational IP networks. Traffic 192 engineering encompasses the application of technology and scientific 193 principles to the measurement, characterization, modeling, and 194 control of Internet traffic [RFC2702], [AWD2]. 196 It is the performance of the network as seen by end users of network 197 services that is paramount. The characteristics visible to end users 198 are the emergent properties of the network, which are the 199 characteristics of the network when viewed as a whole. A central 200 goal of the service provider, therefore, is to enhance the emergent 201 properties of the network while taking economic considerations into 202 account. This is accomplished by addressing traffic oriented 203 performance requirements while utilizing network resources 204 economically and reliably. Traffic oriented performance measures 205 include delay, delay variation, packet loss, and throughput. 207 Internet traffic engineering responds to network events. Aspects of 208 capacity management respond at intervals ranging from days to years. 209 Routing control functions operate at intervals ranging from 210 milliseconds to days. Packet level processing functions operate at 211 very fine levels of temporal resolution, ranging from picoseconds to 212 milliseconds while reacting to the real-time statistical behavior of 213 traffic. 215 Thus, the optimization aspects of traffic engineering can be viewed 216 from a control perspective, and can be both pro-active and reactive. 217 In the pro-active case, the traffic engineering control system takes 218 preventive action to protect against predicted unfavorable future 219 network states, for example, by engineering backup paths. It may 220 also take action that will lead to a more desirable future network 221 state. In the reactive case, the control system responds to correct 222 issues and adapt to network events, such as routing after failure. 224 Another important objective of Internet traffic engineering is to 225 facilitate reliable network operations [RFC2702]. Reliable network 226 operations can be facilitated by providing mechanisms that enhance 227 network integrity and by embracing policies emphasizing network 228 survivability. This reduces the vulnerability of services to outages 229 arising from errors, faults, and failures occurring within the 230 network infrastructure. 232 The optimization aspects of traffic engineering can be achieved 233 through capacity management and traffic management. In this 234 document, capacity management includes capacity planning, routing 235 control, and resource management. Network resources of particular 236 interest include link bandwidth, buffer space, and computational 237 resources. In this document, traffic management includes: 239 1. Nodal traffic control functions such as traffic conditioning, 240 queue management, and scheduling 242 2. Other functions that regulate the flow of traffic through the 243 network or that arbitrate access to network resources between 244 different packets or between different traffic flows. 246 One major challenge of Internet traffic engineering is the 247 realization of automated control capabilities that adapt quickly and 248 cost effectively to significant changes in network state, while still 249 maintaining stability of the network. Performance evaluation can 250 assess the effectiveness of traffic engineering methods, and the 251 results of this evaluation can be used to identify existing problems, 252 guide network re-optimization, and aid in the prediction of potential 253 future problems. However, this process can also be time consuming 254 and may not be suitable to act on short-lived changes in the network. 256 Performance evaluation can be achieved in many different ways. The 257 most notable techniques include analytical methods, simulation, and 258 empirical methods based on measurements. 260 Traffic engineering comes in two flavors: either a background process 261 that constantly monitors traffic and optimizes the use of resources 262 to improve performance; or a form of a pre-planned optimized traffic 263 distribution that is considered optimal. In the later case, any 264 deviation from the optimum distribution (e.g., caused by a fiber cut) 265 is reverted upon repair without further optimization. However, this 266 form of traffic engineering relies upon the notion that the planned 267 state of the network is optimal. Hence, in such a mode there are two 268 levels of traffic engineering: the TE-planning task to enable optimum 269 traffic distribution, and the routing and forwarding tasks that keep 270 traffic flows attached to the pre-planned distribution. 272 As a general rule, traffic engineering concepts and mechanisms must 273 be sufficiently specific and well-defined to address known 274 requirements, but simultaneously flexible and extensible to 275 accommodate unforeseen future demands (see Section 6.1). 277 1.2. Components of Traffic Engineering 279 As mentioned in Section 1.1, Internet traffic engineering provides 280 performance optimization of IP networks while utilizing network 281 resources economically and reliably. Such optimization is supported 282 at the control/controller level and within the data/forwarding plane. 284 The key elements required in any TE solution are as follows: 286 1. Policy 288 2. Path steering 289 3. Resource management 291 Some TE solutions rely on these elements to a lesser or greater 292 extent. Debate remains about whether a solution can truly be called 293 traffic engineering if it does not include all of these elements. 294 For the sake of this document, we assert that all TE solutions must 295 include some aspects of all of these elements. Other solutions can 296 be classed as "partial TE" and also fall in scope of this document. 298 Policy allows for the selection of paths (including next hops) based 299 on information beyond basic reachability. Early definitions of 300 routing policy, e.g., [RFC1102] and [RFC1104], discuss routing policy 301 being applied to restrict access to network resources at an aggregate 302 level. BGP is an example of a commonly used mechanism for applying 303 such policies, see [RFC4271] and [RFC8955]. In the traffic 304 engineering context, policy decisions are made within the control 305 plane or by controllers, and govern the selection of paths. Examples 306 can be found in [RFC4655] and [RFC5394]. Standard TE solutions may 307 cover the mechanisms to distribute and/or enforce polices, but 308 specific policy definition is generally unspecified. 310 Path steering is the ability to forward packets using more 311 information than just knowledge of the next hop. Examples of path 312 steering include IPv4 source routes [RFC0791], RSVP-TE explicit 313 routes [RFC3209], Segment Routing [RFC8402], and Service Function 314 Chaining [RFC7665]. Path steering for TE can be supported via 315 control plane protocols, by encoding in the data plane headers, or by 316 a combination of the two. This includes when control is provided by 317 a controller using a network-facing control protocol. 319 Resource management provides resource-aware control and forwarding. 320 Examples of resources are bandwidth, buffers, and queues, all of 321 which can be managed to control loss and latency. 323 Resource reservation is the control aspect of resource management. 324 It provides for domain-wide consensus about which network 325 resources are used by a particular flow. This determination may 326 be made at a very course or very fine level. Note that this 327 consensus exists at the network control or controller level, not 328 within the data plane. It may be composed purely of accounting/ 329 bookkeeping, but it typically includes an ability to admit, 330 reject, or reclassify a flow based on policy. Such accounting can 331 be done based on any combination of a static understanding of 332 resource requirements, and the use of dynamic mechanisms to 333 collect requirements (e.g., via [RFC3209]) and resource 334 availability (e.g., via [RFC4203]). 336 Resource allocation is the data plane aspect of resource 337 management. It provides for the allocation of specific node and 338 link resources to specific flows. Example resources include 339 buffers, policing, and rate-shaping mechanisms that are typically 340 supported via queuing. It also includes the matching of a flow 341 (i.e., flow classification) to a particular set of allocated 342 resources. The method of flow classification and granularity of 343 resource management is technology specific. Examples include 344 Diffserv with dropping and remarking [RFC4594], MPLS-TE [RFC3209], 345 and GMPLS based label switched paths [RFC3945], as well as 346 controller-based solutions [RFC8453]. This level of resource 347 control, while optional, is important in networks that wish to 348 support congestion management policies to control or regulate the 349 offered traffic to deliver different levels of service and 350 alleviate congestion problems, or those networks that wish to 351 control latencies experienced by specific traffic flows. 353 1.3. Scope 355 The scope of this document is intra-domain traffic engineering. That 356 is, traffic engineering within a given autonomous system in the 357 Internet. This document discusses concepts pertaining to intra- 358 domain traffic control, including such issues as routing control, 359 micro and macro resource allocation, and the control coordination 360 problems that arise consequently. 362 This document describes and characterizes techniques already in use 363 or in advanced development for Internet traffic engineering. The way 364 these techniques fit together is discussed and scenarios in which 365 they are useful will be identified. 367 Although the emphasis in this document is on intra-domain traffic 368 engineering, an overview of the high level considerations pertaining 369 to inter-domain traffic engineering is provided in Section 7. Inter- 370 domain Internet traffic engineering is crucial to the performance 371 enhancement of the global Internet infrastructure. 373 Whenever possible, relevant requirements from existing IETF documents 374 and other sources are incorporated by reference. 376 1.4. Terminology 378 This section provides terminology which is useful for Internet 379 traffic engineering. The definitions presented apply to this 380 document. These terms may have other meanings elsewhere. 382 Busy hour: A one hour period within a specified interval of time 383 (typically 24 hours) in which the traffic load in a network or 384 sub-network is greatest. 386 Congestion: A state of a network resource in which the traffic 387 incident on the resource exceeds its output capacity over an 388 interval of time. 390 Congestion avoidance: An approach to congestion management that 391 attempts to obviate the occurrence of congestion. 393 Congestion control: An approach to congestion management that 394 attempts to remedy congestion problems that have already occurred. 396 Constraint-based routing: A class of routing protocols that take 397 specified traffic attributes, network constraints, and policy 398 constraints into account when making routing decisions. 399 Constraint-based routing is applicable to traffic aggregates as 400 well as flows. It is a generalization of QoS-based routing. 402 Demand side congestion management: A congestion management scheme 403 that addresses congestion problems by regulating or conditioning 404 offered load. 406 Effective bandwidth: The minimum amount of bandwidth that can be 407 assigned to a flow or traffic aggregate in order to deliver 408 'acceptable service quality' to the flow or traffic aggregate. 410 Hot-spot: A network element or subsystem which is in a state of 411 congestion. 413 Inter-domain traffic: Traffic that originates in one Autonomous 414 system and terminates in another. 416 Metric: A parameter defined in terms of standard units of 417 measurement. 419 Measurement methodology: A repeatable measurement technique used to 420 derive one or more metrics of interest. 422 Network survivability: The capability to provide a prescribed level 423 of QoS for existing services after a given number of failures 424 occur within the network. 426 Offline traffic engineering: A traffic engineering system that 427 exists outside of the network. 429 Online traffic engineering: A traffic engineering system that exists 430 within the network, typically implemented on or as adjuncts to 431 operational network elements. 433 Performance measures: Metrics that provide quantitative or 434 qualitative measures of the performance of systems or subsystems 435 of interest. 437 Performance metric: A performance parameter defined in terms of 438 standard units of measurement. 440 Provisioning: The process of assigning or configuring network 441 resources to meet certain requests. 443 QoS routing: Class of routing systems that selects paths to be used 444 by a flow based on the QoS requirements of the flow. 446 Service Level Agreement (SLA): A contract between a provider and a 447 customer that guarantees specific levels of performance and 448 reliability at a certain cost. 450 Service Level Objective (SLO): A key element of an SLA between a 451 provider and a customer. SLOs are agreed upon as a means of 452 measuring the performance of the Service Provider and are outlined 453 as a way of avoiding disputes between the two parties based on 454 misunderstanding. 456 Stability: An operational state in which a network does not 457 oscillate in a disruptive manner from one mode to another mode. 459 Supply-side congestion management: A congestion management scheme 460 that provisions additional network resources to address existing 461 and/or anticipated congestion problems. 463 Traffic characteristic: A description of the temporal behavior or a 464 description of the attributes of a given traffic flow or traffic 465 aggregate. 467 Traffic engineering system: A collection of objects, mechanisms, and 468 protocols that are used together to accomplish traffic engineering 469 objectives. 471 Traffic flow: A stream of packets between two end-points that can be 472 characterized in a certain way. A common classification for a 473 traffic flow selects packets with the "five-tuple" of source and 474 destination addresses, source and destination ports, and protocol 475 ID. 477 Traffic matrix: A representation of the traffic demand between a set 478 of origin and destination abstract nodes. An abstract node can 479 consist of one or more network elements. 481 Traffic monitoring: The process of observing traffic characteristics 482 at a given point in a network and collecting the traffic 483 information for analysis and further action. 485 Traffic trunk: An aggregation of traffic flows belonging to the same 486 class which are forwarded through a common path. A traffic trunk 487 may be characterized by an ingress and egress node, and a set of 488 attributes which determine its behavioral characteristics and 489 requirements from the network. 491 2. Background 493 The Internet aims to convey IP packets from ingress nodes to egress 494 nodes efficiently, expeditiously, and economically. Furthermore, in 495 a multiclass service environment (e.g., Diffserv capable networks - 496 see Section 5.1.4), the resource sharing parameters of the network 497 must be appropriately determined and configured according to 498 prevailing policies and service models to resolve resource contention 499 issues arising from mutual interference between packets traversing 500 through the network. Thus, consideration must be given to resolving 501 competition for network resources between traffic flows belonging to 502 the same service class (intra-class contention resolution) and 503 traffic flows belonging to different classes (inter-class contention 504 resolution). 506 2.1. Context of Internet Traffic Engineering 508 The context of Internet traffic engineering includes: 510 1. A network domain context that defines the scope under 511 consideration, and in particular the situations in which the 512 traffic engineering problems occur. The network domain context 513 includes network structure, network policies, network 514 characteristics, network constraints, network quality attributes, 515 and network optimization criteria. 517 2. A problem context defining the general and concrete issues that 518 traffic engineering addresses. The problem context includes 519 identification, abstraction of relevant features, representation, 520 formulation, specification of the requirements on the solution 521 space, and specification of the desirable features of acceptable 522 solutions. 524 3. A solution context suggesting how to address the issues 525 identified by the problem context. The solution context includes 526 analysis, evaluation of alternatives, prescription, and 527 resolution. 529 4. An implementation and operational context in which the solutions 530 are instantiated. The implementation and operational context 531 includes planning, organization, and execution. 533 The context of Internet traffic engineering and the different problem 534 scenarios are discussed in the following subsections. 536 2.2. Network Domain Context 538 IP networks range in size from small clusters of routers situated 539 within a given location, to thousands of interconnected routers, 540 switches, and other components distributed all over the world. 542 At the most basic level of abstraction, an IP network can be 543 represented as a distributed dynamic system consisting of: 545 o a set of interconnected resources which provide transport services 546 for IP traffic subject to certain constraints 548 o a demand system representing the offered load to be transported 549 through the network 551 o a response system consisting of network processes, protocols, and 552 related mechanisms which facilitate the movement of traffic 553 through the network (see also [AWD2]). 555 The network elements and resources may have specific characteristics 556 restricting the manner in which the traffic demand is handled. 557 Additionally, network resources may be equipped with traffic control 558 mechanisms managing the way in which the demand is serviced. Traffic 559 control mechanisms may be used to: 561 o control packet processing activities within a given resource 563 o arbitrate contention for access to the resource by different 564 packets 566 o regulate traffic behavior through the resource. 568 A configuration management and provisioning system may allow the 569 settings of the traffic control mechanisms to be manipulated by 570 external or internal entities in order to exercise control over the 571 way in which the network elements respond to internal and external 572 stimuli. 574 The details of how the network carries packets are specified in the 575 policies of the network administrators and are installed through 576 network configuration management and policy based provisioning 577 systems. Generally, the types of service provided by the network 578 also depend upon the technology and characteristics of the network 579 elements and protocols, the prevailing service and utility models, 580 and the ability of the network administrators to translate policies 581 into network configurations. 583 Internet networks have two significant characteristics: 585 o they provide real-time services 587 o their operating environments are very dynamic. 589 The dynamic characteristics of IP and IP/MPLS networks can be 590 attributed in part to fluctuations in demand, to the interaction 591 between various network protocols and processes, to the rapid 592 evolution of the infrastructure which demands the constant inclusion 593 of new technologies and new network elements, and to transient and 594 persistent faults which occur within the system. 596 Packets contend for the use of network resources as they are conveyed 597 through the network. A network resource is considered to be 598 congested if, for an interval of time, the arrival rate of packets 599 exceed the output capacity of the resource. Congestion may result in 600 some of the arriving packets being delayed or even dropped. 602 Congestion increases transit delay, delay variation, may lead to 603 packet loss, and reduces the predictability of network services. 604 Clearly, congestion is highly undesirable. Combating congestion at a 605 reasonable cost is a major objective of Internet traffic engineering. 607 Efficient sharing of network resources by multiple traffic flows is a 608 basic operational premise for the Internet. A fundamental challenge 609 in network operation is to increase resource utilization while 610 minimizing the possibility of congestion. 612 The Internet has to function in the presence of different classes of 613 traffic with different service requirements. This requirement is 614 clarified in [RFC2475] which also provides an architecture for 615 Differentiated Services (Diffserv). That document describes how 616 packets can be grouped into behavior aggregates such that each 617 aggregate has a common set of behavioral characteristics or a common 618 set of delivery requirements. Delivery requirements of a specific 619 set of packets may be specified explicitly or implicitly. Two of the 620 most important traffic delivery requirements are: 622 o Capacity constraints can be expressed statistically as peak rates, 623 mean rates, burst sizes, or as some deterministic notion of 624 effective bandwidth. 626 o QoS requirements can be expressed in terms of: 628 * integrity constraints such as packet loss 630 * temporal constraints such as timing restrictions for the 631 delivery of each packet (delay) and timing restrictions for the 632 delivery of consecutive packets belonging to the same traffic 633 stream (delay variation). 635 2.3. Problem Context 637 There are several problems associated with operating a network 638 described in the previous section. This section analyzes the problem 639 context in relation to traffic engineering. The identification, 640 abstraction, representation, and measurement of network features 641 relevant to traffic engineering are significant issues. 643 A particular challenge is to formulate the problems that traffic 644 engineering attempts to solve. For example: 646 o how to identify the requirements on the solution space 648 o how to specify the desirable features of solutions 650 o how to actually solve the problems 652 o how to measure and characterize the effectiveness of solutions. 654 Another class of problems is how to measure and estimate relevant 655 network state parameters. Effective traffic engineering relies on a 656 good estimate of the offered traffic load as well as a view of the 657 underlying topology and associated resource constraints. A network- 658 wide view of the topology is also a must for offline planning. 660 Still another class of problem is how to characterize the state of 661 the network and how to evaluate its performance. The performance 662 evaluation problem is two-fold: one aspect relates to the evaluation 663 of the system-level performance of the network; the other aspect 664 relates to the evaluation of resource-level performance, which 665 restricts attention to the performance analysis of individual network 666 resources. 668 In this document, we refer to the system-level characteristics of the 669 network as the "macro-states" and the resource-level characteristics 670 as the "micro-states." The system-level characteristics are also 671 known as the emergent properties of the network. Correspondingly, we 672 refer to the traffic engineering schemes dealing with network 673 performance optimization at the systems level as "macro-TE" and the 674 schemes that optimize at the individual resource level as "micro-TE." 675 Under certain circumstances, the system-level performance can be 676 derived from the resource-level performance using appropriate rules 677 of composition, depending upon the particular performance measures of 678 interest. 680 Another fundamental class of problem concerns how to effectively 681 optimize network performance. Performance optimization may entail 682 translating solutions for specific traffic engineering problems into 683 network configurations. Optimization may also entail some degree of 684 resource management control, routing control, and capacity 685 augmentation. 687 2.3.1. Congestion and its Ramifications 689 Congestion is one of the most significant problems in an operational 690 IP context. A network element is said to be congested if it 691 experiences sustained overload over an interval of time. Congestion 692 almost always results in degradation of service quality to end users. 693 Congestion control schemes can include demand-side policies and 694 supply-side policies. Demand-side policies may restrict access to 695 congested resources or dynamically regulate the demand to alleviate 696 the overload situation. Supply-side policies may expand or augment 697 network capacity to better accommodate offered traffic. Supply-side 698 policies may also re-allocate network resources by redistributing 699 traffic over the infrastructure. Traffic redistribution and resource 700 re-allocation serve to increase the 'effective capacity' of the 701 network. 703 The emphasis of this document is primarily on congestion management 704 schemes falling within the scope of the network, rather than on 705 congestion management systems dependent upon sensitivity and 706 adaptivity from end-systems. That is, the aspects that are 707 considered in this document with respect to congestion management are 708 those solutions that can be provided by control entities operating on 709 the network and by the actions of network administrators and network 710 operations systems. 712 2.4. Solution Context 714 The solution context for Internet traffic engineering involves 715 analysis, evaluation of alternatives, and choice between alternative 716 courses of action. Generally the solution context is based on making 717 inferences about the current or future state of the network, and 718 making decisions that may involve a preference between alternative 719 sets of action. More specifically, the solution context demands 720 reasonable estimates of traffic workload, characterization of network 721 state, derivation of solutions which may be implicitly or explicitly 722 formulated, and possibly instantiating a set of control actions. 723 Control actions may involve the manipulation of parameters associated 724 with routing, control over tactical capacity acquisition, and control 725 over the traffic management functions. 727 The following list of instruments may be applicable to the solution 728 context of Internet traffic engineering. 730 o A set of policies, objectives, and requirements (which may be 731 context dependent) for network performance evaluation and 732 performance optimization. 734 o A collection of online and possibly offline tools and mechanisms 735 for measurement, characterization, modeling, and control traffic, 736 and control over the placement and allocation of network 737 resources, as well as control over the mapping or distribution of 738 traffic onto the infrastructure. 740 o A set of constraints on the operating environment, the network 741 protocols, and the traffic engineering system itself. 743 o A set of quantitative and qualitative techniques and methodologies 744 for abstracting, formulating, and solving traffic engineering 745 problems. 747 o A set of administrative control parameters which may be 748 manipulated through a configuration management system. Such 749 system itself may include a configuration control subsystem, a 750 configuration repository, a configuration accounting subsystem, 751 and a configuration auditing subsystem. 753 o A set of guidelines for network performance evaluation, 754 performance optimization, and performance improvement. 756 Determining traffic characteristics through measurement or estimation 757 is very useful within the realm the traffic engineering solution 758 space. Traffic estimates can be derived from customer subscription 759 information, traffic projections, traffic models, and from actual 760 measurements. The measurements may be performed at different levels, 761 e.g., at the traffic-aggregate level or at the flow level. 762 Measurements at the flow level or on small traffic aggregates may be 763 performed at edge nodes, when traffic enters and leaves the network. 764 Measurements for large traffic-aggregates may be performed within the 765 core of the network. 767 To conduct performance studies and to support planning of existing 768 and future networks, a routing analysis may be performed to determine 769 the paths the routing protocols will choose for various traffic 770 demands, and to ascertain the utilization of network resources as 771 traffic is routed through the network. Routing analysis captures the 772 selection of paths through the network, the assignment of traffic 773 across multiple feasible routes, and the multiplexing of IP traffic 774 over traffic trunks (if such constructs exist) and over the 775 underlying network infrastructure. A model of network topology is 776 necessary to perform routing analysis. A network topology model may 777 be extracted from: 779 o network architecture documents 781 o network designs 783 o information contained in router configuration files 785 o routing databases 787 o routing tables 789 o automated tools that discover and collate network topology 790 information. 792 Topology information may also be derived from servers that monitor 793 network state, and from servers that perform provisioning functions. 795 Routing in operational IP networks can be administratively controlled 796 at various levels of abstraction including the manipulation of BGP 797 attributes and interior gateway protocol (IGP) metrics. For path 798 oriented technologies such as MPLS, routing can be further controlled 799 by the manipulation of relevant traffic engineering parameters, 800 resource parameters, and administrative policy constraints. Within 801 the context of MPLS, the path of an explicitly routed label switched 802 path (LSP) can be computed and established in various ways including: 804 o manually 806 o automatically, online using constraint-based routing processes 807 implemented on label switching routers 809 o automatically, offline using constraint-based routing entities 810 implemented on external traffic engineering support systems. 812 2.4.1. Combating the Congestion Problem 814 Minimizing congestion is a significant aspect of Internet traffic 815 engineering. This subsection gives an overview of the general 816 approaches that have been used or proposed to combat congestion. 818 Congestion management policies can be categorized based upon the 819 following criteria (see [YARE95] for a more detailed taxonomy of 820 congestion control schemes): 822 1. Congestion Management Based on Response Time Scales 824 * Long (weeks to months): Expanding network capacity by adding 825 new equipment, routers, and links takes time and is 826 comparatively costly. Capacity planning needs to take this 827 into consideration. Network capacity is expanded based on 828 estimates or forecasts of future traffic development and 829 traffic distribution. These upgrades are typically carried 830 out over weeks or months, or maybe even years. 832 * Medium (minutes to days): Several control policies fall within 833 the medium timescale category. Examples include: 835 a. Adjusting routing protocol parameters to route traffic 836 away or towards certain segments of the network. 838 b. Setting up or adjusting explicitly routed LSPs in MPLS 839 networks to route traffic trunks away from possibly 840 congested resources or toward possibly more favorable 841 routes. 843 c. Re-configuring the logical topology of the network to make 844 it correlate more closely with the spatial traffic 845 distribution using, for example, an underlying path- 846 oriented technology such as MPLS LSPs or optical channel 847 trails. 849 Many of these adaptive schemes rely on measurement systems. A 850 measurement system monitors changes in traffic distribution, 851 traffic loads, and network resource utilization and then 852 provides feedback to the online or offline traffic engineering 853 mechanisms and tools so that they can trigger control actions 854 within the network. The traffic engineering mechanisms and 855 tools can be implemented in a distributed or centralized 856 fashion. A centralized scheme may have global visibility into 857 the network state and may produce more optimal solutions. 858 However, centralized schemes are prone to single points of 859 failure and may not scale as well as distributed schemes. 860 Moreover, the information utilized by a centralized scheme may 861 be stale and might not reflect the actual state of the 862 network. It is not an objective of this document to make a 863 recommendation between distributed and centralized schemes: 864 that is a choice that network administrators must make based 865 on their specific needs. 867 * Short (picoseconds to minutes): This category includes packet 868 level processing functions and events that are recorded on the 869 order of several round trip times. It also includes router 870 mechanisms such as passive and active buffer management. All 871 of these mechanisms are used to control congestion or signal 872 congestion to end systems so that they can adaptively regulate 873 the rate at which traffic is injected into the network. One 874 of the most popular active queue management schemes, 875 especially for TCP traffic, is Random Early Detection (RED) 876 [FLJA93]. During congestion (but before the queue is filled), 877 the RED scheme chooses arriving packets to "mark" according to 878 a probabilistic algorithm which takes into account the average 879 queue size. A router that does not utilize explicit 880 congestion notification (ECN) [FLOY94] can simply drop marked 881 packets to alleviate congestion and implicitly notify the 882 receiver about the congestion. On the other hand, if the 883 router supports ECN, it can set the ECN field in the packet 884 header. Several variations of RED have been proposed to 885 support different drop precedence levels in multi-class 886 environments [RFC2597]. RED provides congestion avoidance 887 which is not worse than traditional Tail-Drop (TD) queue 888 management (drop arriving packets only when the queue is 889 full). Importantly, RED reduces the possibility of global 890 synchronization where retransmission burst become synchronized 891 across the whole network, and improves fairness among 892 different TCP sessions. However, RED by itself cannot prevent 893 congestion and unfairness caused by sources unresponsive to 894 RED, e.g., UDP traffic and some misbehaved greedy connections. 895 Other schemes have been proposed to improve the performance 896 and fairness in the presence of unresponsive traffic. Some of 897 those schemes (such as Longest Queue Drop (LQD) and Dynamic 898 Soft Partitioning with Random Drop (RND) [SLDC98]) were 899 proposed as theoretical frameworks and are typically not 900 available in existing commercial products. 902 2. Reactive Versus Preventive Congestion Management Schemes 903 * Reactive (recovery) congestion management policies react to 904 existing congestion problems. All the policies described 905 above for the long and medium time scales can be categorized 906 as being reactive. They are based on monitoring and 907 identifying congestion problems that exist in the network, and 908 on the initiation of relevant actions to ease a situation. 910 * Preventive (predictive/avoidance) policies take proactive 911 action to prevent congestion based on estimates and 912 predictions of future congestion problems (e.g., traffic 913 matrix forecasts). Some of the policies described for the 914 long and medium time scales fall into this category. 915 Preventive policies do not necessarily respond immediately to 916 existing congestion problems. Instead, forecasts of traffic 917 demand and workload distribution are considered, and action 918 may be taken to prevent potential future congestion problems. 919 The schemes described for the short time scale can also be 920 used for congestion avoidance because dropping or marking 921 packets before queues actually overflow would trigger 922 corresponding TCP sources to slow down. 924 3. Supply-Side Versus Demand-Side Congestion Management Schemes 926 * Supply-side congestion management policies increase the 927 effective capacity available to traffic in order to control or 928 reduce congestion. This can be accomplished by increasing 929 capacity or by balancing distribution of traffic over the 930 network. Capacity planning aims to provide a physical 931 topology and associated link bandwidths that match or exceed 932 estimated traffic workload and traffic distribution subject to 933 traffic forecasts and budgetary or other constraints. If the 934 actual traffic distribution does not fit the topology derived 935 from capacity panning, then the traffic can be mapped onto the 936 topology by using routing control mechanisms, by applying path 937 oriented technologies (e.g., MPLS LSPs and optical channel 938 trails) to modify the logical topology, or by employing some 939 other load redistribution mechanisms. 941 * Demand-side congestion management policies control or regulate 942 the offered traffic to alleviate congestion problems. For 943 example, some of the short time scale mechanisms described 944 earlier as well as policing and rate-shaping mechanisms 945 attempt to regulate the offered load in various ways. 947 2.5. Implementation and Operational Context 949 The operational context of Internet traffic engineering is 950 characterized by constant changes that occur at multiple levels of 951 abstraction. The implementation context demands effective planning, 952 organization, and execution. The planning aspects may involve 953 determining prior sets of actions to achieve desired objectives. 954 Organizing involves arranging and assigning responsibility to the 955 various components of the traffic engineering system and coordinating 956 the activities to accomplish the desired TE objectives. Execution 957 involves measuring and applying corrective or perfective actions to 958 attain and maintain desired TE goals. 960 3. Traffic Engineering Process Models 962 This section describes a generic process model that captures the 963 high-level practical aspects of Internet traffic engineering in an 964 operational context. The process model is described as a sequence of 965 actions that must be carried out to optimize the performance of an 966 operational network (see also [RFC2702], [AWD2]). This process model 967 may be enacted explicitly or implicitly, by a software process or by 968 a human. 970 The traffic engineering process model is iterative [AWD2]. The four 971 phases of the process model described below are repeated as a 972 continual sequence. 974 o Define the relevant control policies that govern the operation of 975 the network. 977 o Acquire measurement data from the operational network. 979 o Analyze the network state and characterize the traffic workload. 980 Proactive analysis identifies potential problems that could 981 manifest in the future. Reactive analysis identifies existing 982 problems and determines their causes. 984 o Optimize the performance of the network. This involves a decision 985 process which selects and implements a set of actions from a set 986 of alternatives given the results of the three previous steps. 987 Optimization actions may include the use of techniques to control 988 the offered traffic and to control the distribution of traffic 989 across the network. 991 3.1. Components of the Traffic Engineering Process Model 993 The key components of the traffic engineering process model are as 994 follows. 996 1. Measurement is crucial to the traffic engineering function. The 997 operational state of a network can only be conclusively 998 determined through measurement. Measurement is also critical to 999 the optimization function because it provides feedback data which 1000 is used by traffic engineering control subsystems. This data is 1001 used to adaptively optimize network performance in response to 1002 events and stimuli originating within and outside the network. 1003 Measurement in support of the TE function can occur at different 1004 levels of abstraction. For example, measurement can be used to 1005 derive packet level characteristics, flow level characteristics, 1006 user or customer level characteristics, traffic aggregate 1007 characteristics, component level characteristics, and network 1008 wide characteristics. 1010 2. Modeling, analysis, and simulation are important aspects of 1011 Internet traffic engineering. Modeling involves constructing an 1012 abstract or physical representation which depicts relevant 1013 traffic characteristics and network attributes. A network model 1014 is an abstract representation of the network which captures 1015 relevant network features, attributes, and characteristic. 1016 Network simulation tools are extremely useful for traffic 1017 engineering. Because of the complexity of realistic quantitative 1018 analysis of network behavior, certain aspects of network 1019 performance studies can only be conducted effectively using 1020 simulation. 1022 3. Network performance optimization involves resolving network 1023 issues by transforming such issues into concepts that enable a 1024 solution, identification of a solution, and implementation of the 1025 solution. Network performance optimization can be corrective or 1026 perfective. In corrective optimization, the goal is to remedy a 1027 problem that has occurred or that is incipient. In perfective 1028 optimization, the goal is to improve network performance even 1029 when explicit problems do not exist and are not anticipated. 1031 4. Taxonomy of Traffic Engineering Systems 1033 This section presents a short taxonomy of traffic engineering systems 1034 constructed based on traffic engineering styles and views as listed 1035 below and described in greater detail in the following subsections of 1036 this document. 1038 o Time-dependent versus State-dependent versus Event-dependent 1039 o Offline versus Online 1041 o Centralized versus Distributed 1043 o Local versus Global Information 1045 o Prescriptive versus Descriptive 1047 o Open Loop versus Closed Loop 1049 o Tactical versus Strategic 1051 4.1. Time-Dependent Versus State-Dependent Versus Event-Dependent 1053 Traffic engineering methodologies can be classified as time- 1054 dependent, state-dependent, or event-dependent. All TE schemes are 1055 considered to be dynamic in this document. Static TE implies that no 1056 traffic engineering methodology or algorithm is being applied - it is 1057 a feature of network planning, but lacks the reactive and flexible 1058 nature of traffic engineering. 1060 In time-dependent TE, historical information based on periodic 1061 variations in traffic (such as time of day) is used to pre-program 1062 routing and other TE control mechanisms. Additionally, customer 1063 subscription or traffic projection may be used. Pre-programmed 1064 routing plans typically change on a relatively long time scale (e.g., 1065 daily). Time-dependent algorithms do not attempt to adapt to short- 1066 term variations in traffic or changing network conditions. An 1067 example of a time-dependent algorithm is a global centralized 1068 optimizer where the input to the system is a traffic matrix and 1069 multi-class QoS requirements as described [MR99]. Another example of 1070 such a methodology is the application of data mining to Internet 1071 traffic [AJ19] which enables the use of various machine learning 1072 algorithms to identify patterns within historically collected 1073 datasets about Internet traffic, and to extract information in order 1074 to guide decision-making, and to improve efficiency and productivity 1075 of operational processes. 1077 State-dependent TE adapts the routing plans based on the current 1078 state of the network which provides additional information on 1079 variations in actual traffic (i.e., perturbations from regular 1080 variations) that could not be predicted using historical information. 1081 Constraint-based routing is an example of state-dependent TE 1082 operating in a relatively long time scale. An example operating in a 1083 relatively short timescale is a load-balancing algorithm described in 1084 [MATE]. The state of the network can be based on parameters flooded 1085 by the routers. Another approach is for a particular router 1086 performing adaptive TE to send probe packets along a path to gather 1087 the state of that path. [RFC6374] defines protocol extensions to 1088 collect performance measurements from MPLS networks. Another 1089 approach is for a management system to gather the relevant 1090 information directly from network elements using telemetry data 1091 collection "publication/subscription" techniques [RFC7923]. Timely 1092 gathering and distribution of state information is critical for 1093 adaptive TE. While time-dependent algorithms are suitable for 1094 predictable traffic variations, state-dependent algorithms may be 1095 applied to increase network efficiency and resilience to adapt to the 1096 prevailing network state. 1098 Event-dependent TE methods can also be used for TE path selection. 1099 Event-dependent TE methods are distinct from time-dependent and 1100 state-dependent TE methods in the manner in which paths are selected. 1101 These algorithms are adaptive and distributed in nature and typically 1102 use learning models to find good paths for TE in a network. While 1103 state-dependent TE models typically use available-link-bandwidth 1104 (ALB) flooding for TE path selection, event-dependent TE methods do 1105 not require ALB flooding. Rather, event-dependent TE methods 1106 typically search out capacity by learning models, as in the success- 1107 to-the-top (STT) method. ALB flooding can be resource intensive, 1108 since it requires link bandwidth to carry LSAs, processor capacity to 1109 process LSAs, and the overhead can limit area/Autonomous System (AS) 1110 size. Modeling results suggest that event-dependent TE methods could 1111 lead to a reduction in ALB flooding overhead without loss of network 1112 throughput performance [I-D.ietf-tewg-qos-routing]. 1114 4.2. Offline Versus Online 1116 Traffic engineering requires the computation of routing plans. The 1117 computation may be performed offline or online. The computation can 1118 be done offline for scenarios where routing plans need not be 1119 executed in real-time. For example, routing plans computed from 1120 forecast information may be computed offline. Typically, offline 1121 computation is also used to perform extensive searches on multi- 1122 dimensional solution spaces. 1124 Online computation is required when the routing plans must adapt to 1125 changing network conditions as in state-dependent algorithms. Unlike 1126 offline computation (which can be computationally demanding), online 1127 computation is geared toward relative simple and fast calculations to 1128 select routes, fine-tune the allocations of resources, and perform 1129 load balancing. 1131 4.3. Centralized Versus Distributed 1133 Under centralized control there is a central authority which 1134 determines routing plans and perhaps other TE control parameters on 1135 behalf of each router. The central authority periodically collects 1136 network-state information from all routers, and sends routing 1137 information to the routers. The update cycle for information 1138 exchange in both directions is a critical parameter directly 1139 impacting the performance of the network being controlled. 1140 Centralized control may need high processing power and high bandwidth 1141 control channels. 1143 Distributed control determines route selection by each router 1144 autonomously based on the router's view of the state of the network. 1145 The network state information may be obtained by the router using a 1146 probing method or distributed by other routers on a periodic basis 1147 using link state advertisements. Network state information may also 1148 be disseminated under exception conditions. Examples of protocol 1149 extensions used to advertise network link state information are 1150 defined in [RFC5305], [RFC6119], [RFC7471], [RFC8570], and [RFC8571]. 1151 See also Section 5.1.11. 1153 4.3.1. Hybrid Systems 1155 In practice, most TE systems will be a hybrid of central and 1156 distributed control. For example, a popular MPLS approach to TE is 1157 to use a central controller based on an active, stateful PCE, but to 1158 use routing and signaling protocols to make local decisions at 1159 routers within the network. Local decisions may be able to respond 1160 more quickly to network events, but may result in conflicts with 1161 decisions made by other routers. 1163 Network operations for TE systems may also use a hybrid of offline 1164 and online computation. TE paths may be precomputed based on stable- 1165 state network information and planned traffic demands, but may then 1166 be modified in the active network depending on variations in network 1167 state and traffic load. Furthermore, responses to network events may 1168 be precomputed offline to allow rapid reactions without further 1169 computation, or may be derived online depending on the nature of the 1170 events. 1172 Lastly, note that a fully functional TE system is likely to use all 1173 aspects of time-dependent, state-dependent, and event-dependent 1174 methodologies as described in Section 4.1. 1176 4.3.2. Considerations for Software Defined Networking 1178 As discussed in Section 5.1.17, one of the main drivers for SDN is a 1179 decoupling of the network control plane from the data plane 1180 [RFC7149]. However, SDN may also combine centralized control of 1181 resources, and facilitate application-to-network interaction via an 1182 application programming interface (API) such as [RFC8040]. Combining 1183 these features provides a flexible network architecture that can 1184 adapt to network requirements of a variety of higher-layer 1185 applications, a concept often referred to as the "programmable 1186 network" [RFC7426]. 1188 The centralized control aspect of SDN helps improve global network 1189 resource utilization compared with distributed network control, where 1190 local policy may often override global optimization goals. In an SDN 1191 environment, the data plane forwards traffic to its desired 1192 destination. However, before traffic reaches the data plane, the 1193 logically centralized SDN control plane often determines the end-to- 1194 end path the application traffic will take in the network. 1195 Therefore, the SDN control plane needs to be aware of the underlying 1196 network topology, capabilities and current node and link resource 1197 state. 1199 Using a PCE-based SDN control framework [RFC7491], the available 1200 network topology may be discovered by running a passive instance of 1201 OSPF or IS-IS, or via BGP-LS [RFC7752], to generate a TED (see 1202 Section 5.1.20). The PCE is used to compute a path (see 1203 Section 5.1.13) based on the TED and available bandwidth, and further 1204 path optimization may be based on requested objective functions 1205 [RFC5541]. When a suitable path has been computed the programming of 1206 the explicit network path may be performed using either end-to-end 1207 signaling protocol [RFC3209] or per-hop with each node being directly 1208 programmed [RFC8283] by the SDN controller. 1210 By utilizing a centralized approach to network control, additional 1211 network benefits are also available, including Global Concurrent 1212 Optimization (GCO) [RFC5557]. A GCO path computation request will 1213 simultaneously use the network topology and set of new end-to-end 1214 path requests, along with their respective constraints, for optimal 1215 placement in the network. Correspondingly, a GCO-based computation 1216 may be applied to recompute existing network paths to groom traffic 1217 and to mitigate congestion. 1219 4.4. Local Versus Global 1221 Traffic engineering algorithms may require local and global network- 1222 state information. 1224 Local information is the state of a portion of the domain. Examples 1225 include the bandwidth and packet loss rate of a particular path, or 1226 the state and capabilities of a network link. Local state 1227 information may be sufficient for certain instances of distributed 1228 control TE. 1230 Global information is the state of the entire TE domain. Examples 1231 include a global traffic matrix, and loading information on each link 1232 throughout the domain of interest. Global state information is 1233 typically required with centralized control. Distributed TE systems 1234 may also need global information in some cases. 1236 4.5. Prescriptive Versus Descriptive 1238 TE systems may also be classified as prescriptive or descriptive. 1240 Prescriptive traffic engineering evaluates alternatives and 1241 recommends a course of action. Prescriptive traffic engineering can 1242 be further categorized as either corrective or perfective. 1243 Corrective TE prescribes a course of action to address an existing or 1244 predicted anomaly. Perfective TE prescribes a course of action to 1245 evolve and improve network performance even when no anomalies are 1246 evident. 1248 Descriptive traffic engineering, on the other hand, characterizes the 1249 state of the network and assesses the impact of various policies 1250 without recommending any particular course of action. 1252 4.5.1. Intent-Based Networking 1254 One way to express a service request is through "intent". Intent- 1255 Based Networking aims to produce networks that are simpler to manage 1256 and operate, requiring only minimal intervention. Intent is defined 1257 in [I-D.irtf-nmrg-ibn-concepts-definitions] as a set of operational 1258 goals (that a network should meet) and outcomes (that a network is 1259 supposed to deliver), defined in a declarative manner without 1260 specifying how to achieve or implement them. 1262 Intent provides data and functional abstraction so that users and 1263 operators do not need to be concerned with low-level device 1264 configuration or the mechanisms used to achieve a given intent. This 1265 approach can be conceptually easier for a user, but may be less 1266 expressive in terms of constraints and guidelines. 1268 Intent-Based Networking is applicable to traffic engineering because 1269 many of the high-level objectives may be expressed as "intent." For 1270 example, load balancing, delivery of services, and robustness against 1271 failures. The intent is converted by the management system into 1272 traffic engineering actions within the network. 1274 4.6. Open-Loop Versus Closed-Loop 1276 Open-loop traffic engineering control is where control action does 1277 not use feedback information from the current network state. The 1278 control action may use its own local information for accounting 1279 purposes, however. 1281 Closed-loop traffic engineering control is where control action 1282 utilizes feedback information from the network state. The feedback 1283 information may be in the form of historical information or current 1284 measurement. 1286 4.7. Tactical versus Strategic 1288 Tactical traffic engineering aims to address specific performance 1289 problems (such as hot-spots) that occur in the network from a 1290 tactical perspective, without consideration of overall strategic 1291 imperatives. Without proper planning and insights, tactical TE tends 1292 to be ad hoc in nature. 1294 Strategic traffic engineering approaches the TE problem from a more 1295 organized and systematic perspective, taking into consideration the 1296 immediate and longer term consequences of specific policies and 1297 actions. 1299 5. Review of TE Techniques 1301 This section briefly reviews different traffic engineering approaches 1302 proposed and implemented in telecommunications and computer networks 1303 using IETF protocols and architectures. The discussion is not 1304 intended to be comprehensive. It is primarily intended to illuminate 1305 existing approaches to traffic engineering in the Internet. A 1306 historic overview of traffic engineering in telecommunications 1307 networks is provided in Appendix A, while Appendix B describes 1308 approaches in other standards bodies. 1310 5.1. Overview of IETF Projects Related to Traffic Engineering 1312 This subsection reviews a number of IETF activities pertinent to 1313 Internet traffic engineering. 1315 5.1.1. Constraint-Based Routing 1317 Constraint-based routing refers to a class of routing systems that 1318 compute routes through a network subject to the satisfaction of a set 1319 of constraints and requirements. In the most general case, 1320 constraint-based routing may also seek to optimize overall network 1321 performance while minimizing costs. 1323 The constraints and requirements may be imposed by the network itself 1324 or by administrative policies. Constraints may include bandwidth, 1325 hop count, delay, and policy instruments such as resource class 1326 attributes. Constraints may also include domain specific attributes 1327 of certain network technologies and contexts which impose 1328 restrictions on the solution space of the routing function. Path 1329 oriented technologies such as MPLS have made constraint-based routing 1330 feasible and attractive in public IP networks. 1332 The concept of constraint-based routing within the context of MPLS 1333 traffic engineering requirements in IP networks was first described 1334 in [RFC2702] and led to developments such as MPLS-TE [RFC3209] as 1335 described in Section 5.1.6. 1337 Unlike QoS-based routing (for example, see [RFC2386], [MA], and 1338 [I-D.ietf-idr-performance-routing]) which generally addresses the 1339 issue of routing individual traffic flows to satisfy prescribed flow- 1340 based QoS requirements subject to network resource availability, 1341 constraint-based routing is applicable to traffic aggregates as well 1342 as flows and may be subject to a wide variety of constraints which 1343 may include policy restrictions. 1345 5.1.1.1. IGP Flexible Algorithms (Flex-Algos) 1347 The traditional approach to routing in an IGP network relies on the 1348 IGPs deriving "shortest paths" over the network based solely on the 1349 IGP metric assigned to the links. Such an approach is often limited: 1350 traffic may tend to converge toward the destination, possibly causing 1351 congestion; and it is not possible to steer traffic onto paths 1352 depending on the end-to-end qualities demanded by the applications. 1354 To overcome this limitation, various sorts of traffic engineering 1355 have been widely deployed (as described in this document), where the 1356 TE component is responsible for computing the path based on 1357 additional metrics and/or constraints. Such paths (or tunnels) need 1358 to be installed in the routers' forwarding tables in addition to, or 1359 as a replacement for the original paths computed by IGPs. The main 1360 drawback of these TE approaches is the additional complexity of 1361 protocols and management, and the state that may need to be 1362 maintained within the network. 1364 IGP flexible algorithms (flex-algos) [I-D.ietf-lsr-flex-algo] allow 1365 IGPs to construct constraint-based paths over the network by 1366 computing constraint- based next hops. The intent of flex-algos is 1367 to reduce TE complexity by letting an IGP perform some basic TE 1368 computation capabilities. Flex-algo includes a set of extensions to 1369 the IGPs that enable a router to send TLVs that: 1371 o describe a set of constraints on the topology 1373 o identify calculation-type 1375 o describe a metric-type that is to be used to compute the best 1376 paths through the constrained topology. 1378 A given combination of calculation-type, metric-type, and constraints 1379 is known as a "Flexible Algorithm Definition" (or FAD). A router 1380 that sends such a set of TLVs also assigns a specific identifier (the 1381 Flexible Algorithm) to the specified combination of calculation-type, 1382 metric-type, and constraints. 1384 There are two use cases for flex-algo: in IP networks 1385 [I-D.ietf-lsr-ip-flexalgo] and in segment routing networks 1386 [I-D.ietf-lsr-flex-algo]. In the first case, flex-algo computes 1387 paths to an IPv4 or IPv6 address, in the second case, flex-algo 1388 computes paths to a prefix SID (see Section 5.1.15). 1390 There are many use cases where flex-algo can bring big value, such 1391 as: 1393 o Expansion of functionality of IP Performance metrics [RFC5664] 1394 when points of interest could instantiate specific constraint- 1395 based routing (flex-algo) based on the measurement results. 1397 o Nested usage of flex-algo and TE extensions for IGP (see 1398 Section 5.1.11) when we can form 'underlay' by means of flex-algo 1399 and 'overlay' by TE. 1401 o Flex-algo in SR-MPLS (Section 5.1.15) is a base use case when we 1402 can easily benefit from TE-like topology that will be built 1403 without external TE component on routers or PCE (see 1404 Section 5.1.13). 1406 o Building of network slices [I-D.ietf-teas-ietf-network-slices] 1407 where particular IETF network slice SLO can be guaranteed by flex- 1408 algo. 1410 5.1.2. Integrated Services 1412 The IETF developed the Integrated Services (Intserv) model that 1413 requires resources, such as bandwidth and buffers, to be reserved a 1414 priori for a given traffic flow to ensure that the quality of service 1415 requested by the traffic flow is satisfied. The Integrated Services 1416 model includes additional components beyond those used in the best- 1417 effort model such as packet classifiers, packet schedulers, and 1418 admission control. A packet classifier is used to identify flows 1419 that are to receive a certain level of service. A packet scheduler 1420 handles the scheduling of service to different packet flows to ensure 1421 that QoS commitments are met. Admission control is used to determine 1422 whether a router has the necessary resources to accept a new flow. 1424 The main issue with the Integrated Services model has been 1425 scalability [RFC2998], especially in large public IP networks which 1426 may potentially have millions of active traffic flows in transit 1427 concurrently. 1429 A notable feature of the Integrated Services model is that it 1430 requires explicit signaling of QoS requirements from end systems to 1431 routers [RFC2753]. The Resource Reservation Protocol (RSVP) performs 1432 this signaling function and is a critical component of the Integrated 1433 Services model. RSVP is described in Section 5.1.3. 1435 5.1.3. RSVP 1437 RSVP is a soft state signaling protocol [RFC2205]. It supports 1438 receiver initiated establishment of resource reservations for both 1439 multicast and unicast flows. RSVP was originally developed as a 1440 signaling protocol within the Integrated Services framework (see 1441 Section 5.1.2) for applications to communicate QoS requirements to 1442 the network and for the network to reserve relevant resources to 1443 satisfy the QoS requirements [RFC2205]. 1445 In RSVP, the traffic sender or source node sends a PATH message to 1446 the traffic receiver with the same source and destination addresses 1447 as the traffic which the sender will generate. The PATH message 1448 contains: (1) a sender traffic specification describing the 1449 characteristics of the traffic, (2) a sender template specifying the 1450 format of the traffic, and (3) an optional advertisement 1451 specification which is used to support the concept of One Pass With 1452 Advertising (OPWA) [RFC2205]. Every intermediate router along the 1453 path forwards the PATH message to the next hop determined by the 1454 routing protocol. Upon receiving a PATH message, the receiver 1455 responds with a RESV message which includes a flow descriptor used to 1456 request resource reservations. The RESV message travels to the 1457 sender or source node in the opposite direction along the path that 1458 the PATH message traversed. Every intermediate router along the path 1459 can reject or accept the reservation request of the RESV message. If 1460 the request is rejected, the rejecting router will send an error 1461 message to the receiver and the signaling process will terminate. If 1462 the request is accepted, link bandwidth and buffer space are 1463 allocated for the flow and the related flow state information is 1464 installed in the router. 1466 One of the issues with the original RSVP specification was 1467 Scalability. This is because reservations were required for micro- 1468 flows, so that the amount of state maintained by network elements 1469 tends to increase linearly with the number of traffic flows. These 1470 issues are described in [RFC2961] which also modifies and extends 1471 RSVP to mitigate the scaling problems to make RSVP a versatile 1472 signaling protocol for the Internet. For example, RSVP has been 1473 extended to reserve resources for aggregation of flows, to set up 1474 MPLS explicit label switched paths (see Section 5.1.6), and to 1475 perform other signaling functions within the Internet. [RFC2961] 1476 also describes a mechanism to reduce the amount of Refresh messages 1477 required to maintain established RSVP sessions. 1479 5.1.4. Differentiated Services 1481 The goal of Differentiated Services (Diffserv) within the IETF was to 1482 devise scalable mechanisms for categorization of traffic into 1483 behavior aggregates, which ultimately allows each behavior aggregate 1484 to be treated differently, especially when there is a shortage of 1485 resources such as link bandwidth and buffer space [RFC2475]. One of 1486 the primary motivations for Diffserv was to devise alternative 1487 mechanisms for service differentiation in the Internet that mitigate 1488 the scalability issues encountered with the Intserv model. 1490 Diffserv uses the Differentiated Services field in the IP header (the 1491 DS field) consisting of six bits in what was formerly known as the 1492 Type of Service (TOS) octet. The DS field is used to indicate the 1493 forwarding treatment that a packet should receive at a transit node 1494 [RFC2474]. Diffserv includes the concept of Per-Hop Behavior (PHB) 1495 groups. Using the PHBs, several classes of services can be defined 1496 using different classification, policing, shaping, and scheduling 1497 rules. 1499 For an end-user of network services to utilize Differentiated 1500 Services provided by its Internet Service Provider (ISP), it may be 1501 necessary for the user to have an SLA with the ISP. An SLA may 1502 explicitly or implicitly specify a Traffic Conditioning Agreement 1503 (TCA) which defines classifier rules as well as metering, marking, 1504 discarding, and shaping rules. 1506 Packets are classified, and possibly policed and shaped at the 1507 ingress to a Diffserv network. When a packet traverses the boundary 1508 between different Diffserv domains, the DS field of the packet may be 1509 re-marked according to existing agreements between the domains. 1511 Differentiated Services allows only a finite number of service 1512 classes to be specified by the DS field. The main advantage of the 1513 Diffserv approach relative to the Intserv model is scalability. 1514 Resources are allocated on a per-class basis and the amount of state 1515 information is proportional to the number of classes rather than to 1516 the number of application flows. 1518 The Diffserv model deals with traffic management issues on a per hop 1519 basis. The Diffserv control model consists of a collection of micro- 1520 TE control mechanisms. Other traffic engineering capabilities, such 1521 as capacity management (including routing control), are also required 1522 in order to deliver acceptable service quality in Diffserv networks. 1523 The concept of Per Domain Behaviors has been introduced to better 1524 capture the notion of Differentiated Services across a complete 1525 domain [RFC3086]. 1527 Diffserv procedures can also be applied in an MPLS context. See 1528 Section 6.8 for more information. 1530 5.1.5. QUIC 1532 QUIC [I-D.ietf-quic-transport] is a UDP-based multiplexed and secure 1533 transport protocol. QUIC provides applications with flow-controlled 1534 streams for structured communication, low-latency connection 1535 establishment, and network path migration. 1537 QUIC is a connection-oriented protocol that creates a stateful 1538 interaction between a client and server. QUIC uses a handshake 1539 procedure that combines negotiation of cryptographic and transport 1540 parameters. This is a key differentiation from other transport 1541 protocols. 1543 Endpoints communicate in QUIC by exchanging QUIC packets that use a 1544 customized framing for protection. Most QUIC packets contain frames, 1545 which carry control information and application data between 1546 endpoints. QUIC authenticates all packets and encrypts as much as is 1547 practical. QUIC packets are carried in UDP datagrams to better 1548 facilitate deployment within existing systems and networks. 1550 Application protocols exchange information over a QUIC connection via 1551 streams, which are ordered sequences of bytes. Two types of stream 1552 can be created: bidirectional streams, which allow both endpoints to 1553 send data; and unidirectional streams, which allow a single endpoint 1554 to send data. A credit-based scheme is used to limit stream creation 1555 and to bound the amount of data that can be sent. 1557 QUIC provides the necessary feedback to implement reliable delivery 1558 and congestion control to avoid network congestion. 1560 5.1.6. Multiprotocol Label Switching (MPLS) 1562 MPLS is an advanced forwarding scheme which also includes extensions 1563 to conventional IP control plane protocols. MPLS extends the 1564 Internet routing model and enhances packet forwarding and path 1565 control [RFC3031]. 1567 At the ingress to an MPLS domain, Label Switching Routers (LSRs) 1568 classify IP packets into Forwarding Equivalence Classes (FECs) based 1569 on a variety of factors, including, e.g., a combination of the 1570 information carried in the IP header of the packets and the local 1571 routing information maintained by the LSRs. An MPLS label stack 1572 entry is then prepended to each packet according to their forwarding 1573 equivalence classes. The MPLS label stack entry is 32 bits long and 1574 contains a 20-bit label field. 1576 An LSR makes forwarding decisions by using the label prepended to 1577 packets as the index into a local next hop label forwarding entry 1578 (NHLFE). The packet is then processed as specified in the NHLFE. 1579 The incoming label may be replaced by an outgoing label (label swap), 1580 and the packet may be forwarded to the next LSR. Before a packet 1581 leaves an MPLS domain, its MPLS label may be removed (label pop). A 1582 Label Switched Path (LSP) is the path between an ingress LSRs and an 1583 egress LSRs through which a labeled packet traverses. The path of an 1584 explicit LSP is defined at the originating (ingress) node of the LSP. 1585 MPLS can use a signaling protocol such as RSVP or LDP to set up LSPs. 1587 MPLS is a very powerful technology for Internet traffic engineering 1588 because it supports explicit LSPs which allow constraint-based 1589 routing to be implemented efficiently in IP networks [AWD2]. The 1590 requirements for traffic engineering over MPLS are described in 1591 [RFC2702]. Extensions to RSVP to support instantiation of explicit 1592 LSP are discussed in [RFC3209]. 1594 5.1.7. Generalized MPLS (GMPLS) 1596 GMPLS extends MPLS control protocols to encompass time-division 1597 (e.g., Synchronous Optical Network / Synchronous Digital Hierarchy 1598 (SONET/SDH), Plesiochronous Digital Hierarchy (PDH), Optical 1599 Transport Network (OTN)), wavelength (lambdas), and spatial switching 1600 (e.g., incoming port or fiber to outgoing port or fiber) as well as 1601 continuing to support packet switching. GMPLS provides a common set 1602 of control protocols for all of these layers (including some 1603 technology-specific extensions) each of which has a diverse data or 1604 forwarding plane. GMPLS covers both the signaling and the routing 1605 part of that control plane and is based on the Traffic Engineering 1606 extensions to MPLS (see Section 5.1.6). 1608 In GMPLS, the original MPLS architecture is extended to include LSRs 1609 whose forwarding planes rely on circuit switching, and therefore 1610 cannot forward data based on the information carried in either packet 1611 or cell headers. Specifically, such LSRs include devices where the 1612 switching is based on time slots, wavelengths, or physical ports. 1613 These additions impact basic LSP properties: how labels are requested 1614 and communicated, the unidirectional nature of MPLS LSPs, how errors 1615 are propagated, and information provided for synchronizing the 1616 ingress and egress LSRs. 1618 5.1.8. IP Performance Metrics 1620 The IETF IP Performance Metrics (IPPM) working group has developed a 1621 set of standard metrics that can be used to monitor the quality, 1622 performance, and reliability of Internet services. These metrics can 1623 be applied by network operators, end-users, and independent testing 1624 groups to provide users and service providers with a common 1625 understanding of the performance and reliability of the Internet 1626 component 'clouds' they use/provide [RFC2330]. The criteria for 1627 performance metrics developed by the IPPM working group are described 1628 in [RFC2330]. Examples of performance metrics include one-way packet 1629 loss [RFC7680], one-way delay [RFC7679], and connectivity measures 1630 between two nodes [RFC2678]. Other metrics include second-order 1631 measures of packet loss and delay. 1633 Some of the performance metrics specified by the IPPM working group 1634 are useful for specifying SLAs. SLAs are sets of service level 1635 objectives negotiated between users and service providers, wherein 1636 each objective is a combination of one or more performance metrics, 1637 possibly subject to certain constraints. 1639 5.1.9. Flow Measurement 1641 The IETF Real Time Flow Measurement (RTFM) working group produced an 1642 architecture that defines a method to specify traffic flows as well 1643 as a number of components for flow measurement (meters, meter 1644 readers, manager) [RFC2722]. A flow measurement system enables 1645 network traffic flows to be measured and analyzed at the flow level 1646 for a variety of purposes. As noted in RFC 2722, a flow measurement 1647 system can be very useful in the following contexts: 1649 o understanding the behavior of existing networks 1650 o planning for network development and expansion 1652 o quantification of network performance 1654 o verifying the quality of network service 1656 o attribution of network usage to users. 1658 A flow measurement system consists of meters, meter readers, and 1659 managers. A meter observes packets passing through a measurement 1660 point, classifies them into groups, accumulates usage data (such as 1661 the number of packets and bytes for each group), and stores the usage 1662 data in a flow table. A group may represent any collection of user 1663 applications, hosts, networks, etc. A meter reader gathers usage 1664 data from various meters so it can be made available for analysis. A 1665 manager is responsible for configuring and controlling meters and 1666 meter readers. The instructions received by a meter from a manager 1667 include flow specifications, meter control parameters, and sampling 1668 techniques. The instructions received by a meter reader from a 1669 manager include the address of the meter whose date is to be 1670 collected, the frequency of data collection, and the types of flows 1671 to be collected. 1673 5.1.10. Endpoint Congestion Management 1675 [RFC3124] provides a set of congestion control mechanisms for the use 1676 of transport protocols. It is also allows the development of 1677 mechanisms for unifying congestion control across a subset of an 1678 endpoint's active unicast connections (called a congestion group). A 1679 congestion manager continuously monitors the state of the path for 1680 each congestion group under its control. The manager uses that 1681 information to instruct a scheduler on how to partition bandwidth 1682 among the connections of that congestion group. 1684 5.1.11. TE Extensions to the IGPs 1686 [RFC5305] describes the extensions to the Intermediate System to 1687 Intermediate System (IS-IS) protocol to support TE, similarly 1688 [RFC3630] specifies TE extensions for OSPFv2 ([RFC5329] has the same 1689 description for OSPFv3). 1691 The idea of redistribution TE extensions such as link type and ID, 1692 local and remote IP addresses, TE metric, maximum bandwidth, maximum 1693 reservable bandwidth and unreserved bandwidth, admin group in IGP is 1694 a common for both IS-IS and OSPF. 1696 The difference is in the details of their transmission: IS-IS uses 1697 the Extended IS Reachability TLV (type 22) and Sub-TLVs for those TE 1698 parameters, OSPFv2 uses Opaque LSA [RFC5250] type 10 (OSPFv3 uses 1699 Intra-Area-TE-LSA) with two top-level TLV (Router Address and Link) 1700 also with Sub-TLVs for that purpose. 1702 IS-IS also uses the Extended IP Reachability TLV (type 135, which 1703 have the new 32 bit metric) and the TE Router ID TLV (type 134). 1704 Those Sub-TLV details are described in [RFC8570] for IS-IS and in 1705 [RFC7471] for OSPFv2 ([RFC5329] for OSPFv3). 1707 5.1.12. Link-State BGP 1709 In a number of environments, a component external to a network is 1710 called upon to perform computations based on the network topology and 1711 current state of the connections within the network, including 1712 traffic engineering information. This is information typically 1713 distributed by IGP routing protocols within the network (see 1714 Section 5.1.11. 1716 The Border Gateway Protocol (BGP) (see also Section 7) is one of the 1717 essential routing protocols that glue the Internet together. BGP 1718 Link State (BGP-LS) [RFC7752] is a mechanism by which link-state and 1719 traffic engineering information can be collected from networks and 1720 shared with external components using the BGP routing protocol. The 1721 mechanism is applicable to physical and virtual IGP links, and is 1722 subject to policy control. 1724 Information collected by BGP-LS can be used to construct the Traffic 1725 Engineering Database (TED, see Section 5.1.20) for use by the Path 1726 Computation Element (PCE, see Section 5.1.13), or may be used by 1727 Application-Layer Traffic Optimization (ALTO) servers (see 1728 Section 5.1.14). 1730 5.1.13. Path Computation Element 1732 Constraint-based path computation is a fundamental building block for 1733 traffic engineering in MPLS and GMPLS networks. Path computation in 1734 large, multi-domain networks is complex and may require special 1735 computational components and cooperation between the elements in 1736 different domains. The Path Computation Element (PCE) [RFC4655] is 1737 an entity (component, application, or network node) that is capable 1738 of computing a network path or route based on a network graph and 1739 applying computational constraints. 1741 Thus, a PCE can provide a central component in a traffic engineering 1742 system operating on the Traffic Engineering Database (TED, see 1743 Section 5.1.20) with delegated responsibility for determining paths 1744 in MPLS, GMPLS, or Segment Routing networks. The PCE uses the Path 1745 Computation Element Communication Protocol (PCEP) [RFC5440] to 1746 communicate with Path Computation Clients (PCCs), such as MPLS LSRs, 1747 to answer their requests for computed paths or to instruct them to 1748 initiate new paths [RFC8281] and maintain state about paths already 1749 installed in the network [RFC8231]. 1751 PCEs form key components of a number of traffic engineering systems. 1752 More information about the applicability of PCE can be found in 1753 [RFC8051], while [RFC6805] describes the application of PCE to 1754 determining paths across multiple domains. PCE also has potential 1755 use in Abstraction and Control of TE Networks (ACTN) (see 1756 Section 5.1.17), Centralized Network Control [RFC8283], and Software 1757 Defined Networking (SDN) (see Section 4.3.2). 1759 5.1.14. Application-Layer Traffic Optimization 1761 This document describes various TE mechanisms available in the 1762 network. However, distributed applications in general and, in 1763 particular, bandwidth-greedy P2P applications that are used, for 1764 example, for file sharing, cannot directly use those techniques. As 1765 per [RFC5693], applications could greatly improve traffic 1766 distribution and quality by cooperating with external services that 1767 are aware of the network topology. Addressing the Application-Layer 1768 Traffic Optimization (ALTO) problem means, on the one hand, deploying 1769 an ALTO service to provide applications with information regarding 1770 the underlying network (e.g., basic network location structure and 1771 preferences of network paths) and, on the other hand, enhancing 1772 applications in order to use such information to perform better-than- 1773 random selection of the endpoints with which they establish 1774 connections. 1776 The basic function of ALTO is based on abstract maps of a network. 1777 These maps provide a simplified view, yet enough information about a 1778 network for applications to effectively utilize them. Additional 1779 services are built on top of the maps. [RFC7285] describes a 1780 protocol implementing the ALTO services as an information-publishing 1781 interface that allows a network to publish its network information 1782 such as network locations, costs between them at configurable 1783 granularities, and end-host properties to network applications. The 1784 information published by the ALTO Protocol should benefit both the 1785 network and the applications. The ALTO Protocol uses a REST-ful 1786 design and encodes its requests and responses using JSON [RFC8259] 1787 with a modular design by dividing ALTO information publication into 1788 multiple ALTO services (e.g., the Map service, the Map-Filtering 1789 Service, the Endpoint Property Service, and the Endpoint Cost 1790 Service). 1792 [RFC8189] defines a new service that allows an ALTO Client to 1793 retrieve several cost metrics in a single request for an ALTO 1794 filtered cost map and endpoint cost map. [RFC8896] extends the ALTO 1795 cost information service so that applications decide not only 'where' 1796 to connect, but also 'when'. This is useful for applications that 1797 need to perform bulk data transfer and would like to schedule these 1798 transfers during an off-peak hour, for example. 1799 [I-D.ietf-alto-performance-metrics] introducing network performance 1800 metrics, including network delay, jitter, packet loss rate, hop 1801 count, and bandwidth. The ALTO server may derive and aggregate such 1802 performance metrics from BGP-LS (see Section 5.1.12) or IGP-TE (see 1803 Section 5.1.11), or management tools, and then expose the information 1804 to allow applications to determine 'where' to connect based on 1805 network performance criteria. ALTO WG is evaluating the use of 1806 network TE properties while making application decisions for new use- 1807 cases such as Edge computing and Datacenter interconnect. 1809 5.1.15. Segment Routing with MPLS Encapsulation (SR-MPLS) 1811 Segment Routing (SR) [RFC8402] leverages the source routing and 1812 tunneling paradigms. The path a packet takes is defined at the 1813 ingress and the packet is tunneled to the egress. A node steers a 1814 packet through a controlled set of instructions, called segments, by 1815 prepending the packet with an SR header: a label stack in MPLS case. 1817 A segment can represent any instruction, topological or service- 1818 based, thanks to the MPLS architecture [RFC3031]. Labels can be 1819 looked up in a global context (platform wide) as well as in some 1820 other context (see "context labels" in Section 3 of [RFC5331]). 1822 Segments are identified by Segment Identifiers (SIDs). There are 1823 four types of SID that are relevant for traffic engineering. 1825 Prefix SID: Unique within the routing domain used to identify a 1826 prefix. 1828 Node SID: A Prefix SID with the 'N' bit set to identify a node. 1830 Adjacency SID: Identifies a unidirectional adjacency. 1832 Binding SID: A Binding SID has two purposes: 1834 1. Used to advertise the mappings of prefixes to SIDs/Labels. 1836 2. Used to advertise a path available for a Forwarding 1837 Equivalence Class. 1839 5.1.16. Segment Routing Policy 1841 SR Policy [I-D.ietf-spring-segment-routing-policy] is an evolution of 1842 Segment Routing to enhance the TE capabilities. It is a framework 1843 that enables instantiation of an ordered list of segments on a node 1844 for implementing a source routing policy with a specific intent for 1845 traffic steering from that node. 1847 An SR Policy is identified through the tuple . The headend is the IP address of the node where the 1849 policy is instantiated. The endpoint is the IP address of the 1850 destination of the policy. The color is an index that associates the 1851 SR Policy with an intent (e.g., low-latency). 1853 The headend node is notified of SR Policies and associated SR paths 1854 via configuration or by a extensions to protocols such as PCEP 1855 [RFC8664] or BGP [I-D.ietf-idr-segment-routing-te-policy]. Each SR 1856 path consists of a Segment-List (an SR source-routed path), and the 1857 headend uses the endpoint and color parameters to classify packets to 1858 match the SR policy and so determine along which path to forward 1859 them. If an SR Policy is associated with a set of SR paths, each is 1860 associated with a weight for weighted load balancing. Furthermore, 1861 multiple SR Policies may be associated with a set of SR paths to 1862 allow multiple traffic flows to be placed on the same paths. 1864 An SR Binding SID (BSID) are also be associated with each candidate 1865 path associated with an SR Policy, or with the SR Policy itself. The 1866 headend node installs a BSID-keyed entry in the forwarding plane and 1867 assigns it the action of steering packets that match the entry to the 1868 selected path of the SR Policy. This steering can be done in various 1869 ways: 1871 o SID Steering: Incoming packets have an active SID matching a local 1872 BSID at the headend. 1874 o Per-destination Steering: Incoming packets match a BGP/Service 1875 route which indicates an SR Policy. 1877 o Per-flow Steering: Incoming packets match a forwarding array (for 1878 example, the classic 5-tuple) which indicates an SR Policies. 1880 o Policy-based Steering: Incoming packets match a routing policy 1881 which directs them to an SR Policy. 1883 5.1.17. Network Virtualization and Abstraction 1885 One of the main drivers for Software Defined Networking (SDN) 1886 [RFC7149] is a decoupling of the network control plane from the data 1887 plane. This separation has been achieved for TE networks with the 1888 development of MPLS/GMPLS (see Section 5.1.6 and Section 5.1.7) and 1889 the Path Computation Element (PCE) (Section 5.1.13). One of the 1890 advantages of SDN is its logically centralized control regime that 1891 allows a global view of the underlying networks. Centralized control 1892 in SDN helps improve network resource utilization compared with 1893 distributed network control. 1895 Abstraction and Control of TE Networks (ACTN) [RFC8453] defines a 1896 hierarchical SDN architecture which describes the functional entities 1897 and methods for the coordination of resources across multiple 1898 domains, to provide end-to-end traffic engineered services. ACTN 1899 facilitates end-to-end connections and provides them to the user. 1900 ACTN is focused on: 1902 o Abstraction of the underlying network resources and how they are 1903 provided to higher-layer applications and customers. 1905 o Virtualization of underlying resources for use by the customer, 1906 application, or service. The creation of a virtualized 1907 environment allows operators to view and control multi-domain 1908 networks as a single virtualized network. 1910 o Presentation to customers of networks as a virtual network via 1911 open and programmable interfaces. 1913 The ACTN managed infrastructure is built from traffic engineered 1914 network resources, which may include statistical packet bandwidth, 1915 physical forwarding plane sources (such as wavelengths and time 1916 slots), forwarding and cross-connect capabilities. The type of 1917 network virtualization seen in ACTN allows customers and applications 1918 (tenants) to utilize and independently control allocated virtual 1919 network resources as if resources as if they were physically their 1920 own resource. The ACTN network is "sliced", with tenants being given 1921 a different partial and abstracted topology view of the physical 1922 underlying network. 1924 5.1.18. Network Slicing 1926 An IETF Network Slice is a logical network topology connecting a 1927 number of endpoints using a set of shared or dedicated network 1928 resources [I-D.ietf-teas-ietf-network-slices]. The resources are 1929 used to satisfy specific Service Level Objectives (SLOs) specified by 1930 the consumer. 1932 IETF network slices are not, of themselves, TE constructs. However, 1933 a network operator that offers IETF network slices is likely to use 1934 many TE tools in order to manage their network and provide the 1935 services. 1937 IETF Network Slices are defined such that they are independent of the 1938 underlying infrastructure connectivity and technologies used. From a 1939 customer's perspective an IETF Network Slice looks like a VPN 1940 connectivity matrix with additional information about the level of 1941 service required between endpoints. From an operator's perspective 1942 the IETF Network Slice looks like a set of routing or tunneling 1943 instructions with the network resource reservations necessary to 1944 provide the required service levels as specified by the SLOs. The 1945 concept of an IETF network slice is consistent with an enhanced VPN 1946 (VPN+) [I-D.ietf-teas-enhanced-vpn]. 1948 5.1.19. Deterministic Networking 1950 Deterministic Networking (DetNet) [RFC8655] is an architecture for 1951 applications with critical timing and reliability requirements. The 1952 layered architecture particularly focuses on developing DetNet 1953 service capabilities in the data plane [RFC8938]. The DetNet service 1954 sub-layer provides a set of Packet Replication, Elimination, and 1955 Ordering Functions (PREOF) functions to provide end-to-end service 1956 assurance. The DetNet forwarding sub-layer provides corresponding 1957 forwarding assurance (low packet loss, bounded latency, and in-order 1958 delivery) functions using resource allocations and explicit route 1959 mechanisms. 1961 The separation into two sub-layers allows a greater flexibility to 1962 adapt Detnet capability over a number of TE data plane mechanisms 1963 such as IP, MPLS, and Segment Routing. More importantly it 1964 interconnects IEEE 802.1 Time Sensitive Networking (TSN) 1965 [I-D.ietf-detnet-ip-over-tsn] deployed in Industry Control and 1966 Automation Systems (ICAS). 1968 DetNet can be seen as a specialized branch of TE, since it sets up 1969 explicit optimized paths with allocation of resources as requested. 1970 A DetNet application can express its QoS attributes or traffic 1971 behavior using any combination of DetNet functions described in sub- 1972 layers. They are then distributed and provisioned using well- 1973 established control and provisioning mechanisms adopted for traffic- 1974 engineering. 1976 In DetNet, a considerable state information is required to maintain 1977 per flow queuing disciplines and resource reservation for a large 1978 number of individual flows. This can be quite challenging for 1979 network operations during network events such as faults, change in 1980 traffic volume or re-provisioning. Therefore, DetNet recommends 1981 support for aggregated flows, however, it still requires large amount 1982 of control signaling to establish and maintain DetNet flows. 1984 5.1.20. Network TE State Definition and Presentation 1986 The network states that are relevant to the traffic engineering need 1987 to be stored in the system and presented to the user. The Traffic 1988 Engineering Database (TED) is a collection of all TE information 1989 about all TE nodes and TE links in the network, which is an essential 1990 component of a TE system, such as MPLS-TE [RFC2702] and GMPLS 1991 [RFC3945]. In order to formally define the data in the TED and to 1992 present the data to the user with high usability, the data modeling 1993 language YANG [RFC7950] can be used as described in [RFC8795]. 1995 5.1.21. System Management and Control Interfaces 1997 The traffic engineering control system needs to have a management 1998 interface that is human-friendly and a control interfaces that is 1999 programmable for automation. The Network Configuration Protocol 2000 (NETCONF) [RFC6241] or the RESTCONF Protocol [RFC8040] provide 2001 programmable interfaces that are also human-friendly. These 2002 protocols use XML or JSON encoded messages. When message compactness 2003 or protocol bandwidth consumption needs to be optimized for the 2004 control interface, other protocols, such as Group Communication for 2005 the Constrained Application Protocol (CoAP) [RFC7390] or gRPC, are 2006 available, especially when the protocol messages are encoded in a 2007 binary format. Along with any of these protocols, the data modeling 2008 language YANG [RFC7950] can be used to formally and precisely define 2009 the interface data. 2011 The Path Computation Element Communication Protocol (PCEP) [RFC5440] 2012 is another protocol that has evolved to be an option for the TE 2013 system control interface. The messages of PCEP are TLV-based, not 2014 defined by a data modeling language such as YANG. 2016 5.2. Content Distribution 2018 The Internet is dominated by client-server interactions, principally 2019 Web traffic although in the future, more sophisticated media servers 2020 may become dominant. The location and performance of major 2021 information servers has a significant impact on the traffic patterns 2022 within the Internet as well as on the perception of service quality 2023 by end users. 2025 A number of dynamic load balancing techniques have been devised to 2026 improve the performance of replicated information servers. These 2027 techniques can cause spatial traffic characteristics to become more 2028 dynamic in the Internet because information servers can be 2029 dynamically picked based upon the location of the clients, the 2030 location of the servers, the relative utilization of the servers, the 2031 relative performance of different networks, and the relative 2032 performance of different parts of a network. This process of 2033 assignment of distributed servers to clients is called traffic 2034 directing. It is an application layer function. 2036 Traffic directing schemes that allocate servers in multiple 2037 geographically dispersed locations to clients may require empirical 2038 network performance statistics to make more effective decisions. In 2039 the future, network measurement systems may need to provide this type 2040 of information. 2042 When congestion exists in the network, traffic directing and traffic 2043 engineering systems should act in a coordinated manner. This topic 2044 is for further study. 2046 The issues related to location and replication of information 2047 servers, particularly web servers, are important for Internet traffic 2048 engineering because these servers contribute a substantial proportion 2049 of Internet traffic. 2051 6. Recommendations for Internet Traffic Engineering 2053 This section describes high-level recommendations for traffic 2054 engineering in the Internet in general terms. 2056 The recommendations describe the capabilities needed to solve a 2057 traffic engineering problem or to achieve a traffic engineering 2058 objective. Broadly speaking, these recommendations can be 2059 categorized as either functional or non-functional recommendations. 2061 o Functional recommendations describe the functions that a traffic 2062 engineering system should perform. These functions are needed to 2063 realize traffic engineering objectives by addressing traffic 2064 engineering problems. 2066 o Non-functional recommendations relate to the quality attributes or 2067 state characteristics of a traffic engineering system. These 2068 recommendations may contain conflicting assertions and may 2069 sometimes be difficult to quantify precisely. 2071 6.1. Generic Non-functional Recommendations 2073 The generic non-functional recommendations for Internet traffic 2074 engineering are listed in the paragraphs that follow. In a given 2075 context, some of these recommendations may be critical while others 2076 may be optional. Therefore, prioritization may be required during 2077 the development phase of a traffic engineering system to tailor it to 2078 a specific operational context. 2080 Usability: Usability is a human aspect of traffic engineering 2081 systems. It refers to the ease with which a traffic engineering 2082 system can be deployed and operated. In general, it is desirable 2083 to have a TE system that can be readily deployed in an existing 2084 network. It is also desirable to have a TE system that is easy to 2085 operate and maintain. 2087 Automation: Whenever feasible, a TE system should automate as many 2088 TE functions as possible to minimize the amount of human effort 2089 needed to analyze and control operational networks. Automation is 2090 particularly important in large-scale public networks because of 2091 the high cost of the human aspects of network operations and the 2092 high risk of network problems caused by human errors. Automation 2093 may entail the incorporation of automatic feedback and 2094 intelligence into some components of the TE system. 2096 Scalability: Public networks continue to grow rapidly with respect 2097 to network size and traffic volume. Therefore, to remain 2098 applicable as the network evolves, a TE system should be scalable. 2099 In particular, a TE system should remain functional as the network 2100 expands with regard to the number of routers and links, and with 2101 respect to the traffic volume. A TE system should have a scalable 2102 architecture, should not adversely impair other functions and 2103 processes in a network element, and should not consume too many 2104 network resources when collecting and distributing state 2105 information, or when exerting control. 2107 Stability: Stability is a very important consideration in TE systems 2108 that respond to changes in the state of the network. State- 2109 dependent TE methodologies typically include a trade-off between 2110 responsiveness and stability. It is strongly recommended that 2111 when a trade-off between responsiveness and stability is needed, 2112 it should be made in favor of stability (especially in public IP 2113 backbone networks). 2115 Flexibility: A TE system should allow for changes in optimization 2116 policy. In particular, a TE system should provide sufficient 2117 configuration options so that a network administrator can tailor 2118 the system to a particular environment. It may also be desirable 2119 to have both online and offline TE subsystems which can be 2120 independently enabled and disabled. TE systems that are used in 2121 multi-class networks should also have options to support class 2122 based performance evaluation and optimization. 2124 Visibility: Mechanisms should exist as part of the TE system to 2125 collect statistics from the network and to analyze these 2126 statistics to determine how well the network is functioning. 2127 Derived statistics such as traffic matrices, link utilization, 2128 latency, packet loss, and other performance measures of interest 2129 which are determined from network measurements can be used as 2130 indicators of prevailing network conditions. The capabilities of 2131 the various components of the routing system are other examples of 2132 status information which should be observable. 2134 Simplicity: A TE system should be as simple as possible. Simplicity 2135 in user interface does not necessarily imply that the TE system 2136 will use naive algorithms. When complex algorithms and internal 2137 structures are used, the user interface should hide such 2138 complexities from the network administrator as much as possible. 2140 Interoperability: Whenever feasible, TE systems and their components 2141 should be developed with open standards-based interfaces to allow 2142 interoperation with other systems and components. 2144 Security: Security is a critical consideration in TE systems. Such 2145 systems typically exert control over functional aspects of the 2146 network to achieve the desired performance objectives. Therefore, 2147 adequate measures must be taken to safeguard the integrity of the 2148 TE system. Adequate measures must also be taken to protect the 2149 network from vulnerabilities that originate from security breaches 2150 and other impairments within the TE system. 2152 The remaining subsections of this section focus on some of the high- 2153 level functional recommendations for traffic engineering. 2155 6.2. Routing Recommendations 2157 Routing control is a significant aspect of Internet traffic 2158 engineering. Routing impacts many of the key performance measures 2159 associated with networks, such as throughput, delay, and utilization. 2160 Generally, it is very difficult to provide good service quality in a 2161 wide area network without effective routing control. A desirable TE 2162 routing system is one that takes traffic characteristics and network 2163 constraints into account during route selection while maintaining 2164 stability. 2166 Shortest path first (SPF) IGPs are based on shortest path algorithms 2167 and have limited control capabilities for TE [RFC2702], [AWD2]. 2168 These limitations include: 2170 1. Pure SPF protocols do not take network constraints and traffic 2171 characteristics into account during route selection. For 2172 example, IGPs always select the shortest paths based on link 2173 metrics assigned by administrators) so load sharing cannot be 2174 performed across paths of different costs. Using shortest paths 2175 to forward traffic may cause the following problems: 2177 * If traffic from a source to a destination exceeds the capacity 2178 of a link along the shortest path, the link (and hence the 2179 shortest path) becomes congested while a longer path between 2180 these two nodes may be under-utilized 2182 * The shortest paths from different sources can overlap at some 2183 links. If the total traffic from the sources exceeds the 2184 capacity of any of these links, congestion will occur. 2186 * Problems can also occur because traffic demand changes over 2187 time, but network topology and routing configuration cannot be 2188 changed as rapidly. This causes the network topology and 2189 routing configuration to become sub-optimal over time, which 2190 may result in persistent congestion problems. 2192 2. The Equal-Cost Multi-Path (ECMP) capability of SPF IGPs supports 2193 sharing of traffic among equal cost paths between two nodes. 2194 However, ECMP attempts to divide the traffic as equally as 2195 possible among the equal cost shortest paths. Generally, ECMP 2196 does not support configurable load sharing ratios among equal 2197 cost paths. The result is that one of the paths may carry 2198 significantly more traffic than other paths because it may also 2199 carry traffic from other sources. This situation can result in 2200 congestion along the path that carries more traffic. Weighted 2201 ECMP (WECMP) (see, for example, [I-D.ietf-bess-evpn-unequal-lb]) 2202 provides some mitigation. 2204 3. Modifying IGP metrics to control traffic routing tends to have 2205 network-wide effects. Consequently, undesirable and 2206 unanticipated traffic shifts can be triggered as a result. Work 2207 described in Section 8 may be capable of better control [FT00], 2208 [FT01]. 2210 Because of these limitations, new capabilities are needed to enhance 2211 the routing function in IP networks. Some of these capabilities are 2212 summarized below. 2214 o Constraint-based routing computes routes to fulfill requirements 2215 subject to constraints. This can be useful in public IP backbones 2216 with complex topologies. Constraints may include bandwidth, hop 2217 count, delay, and administrative policy instruments such as 2218 resource class attributes [RFC2702], [RFC2386]. This makes it 2219 possible to select routes that satisfy a given set of 2220 requirements. Routes computed by constraint-based routing are not 2221 necessarily the shortest paths. Constraint-based routing works 2222 best with path-oriented technologies that support explicit 2223 routing, such as MPLS. 2225 Constraint-based routing can also be used as a way to distribute 2226 traffic onto the infrastructure, including for best effort 2227 traffic. For example, congestion problems caused by uneven 2228 traffic distribution may be avoided or reduced by knowing the 2229 reservable bandwidth attributes of the network links and by 2230 specifying the bandwidth requirements for path selection. 2232 o A number of enhancements to the link state IGPs are needed to 2233 allow them to distribute additional state information required for 2234 constraint-based routing. The extensions to OSPF are described in 2235 [RFC3630], and to IS-IS in [RFC5305]. Some of the additional 2236 topology state information includes link attributes such as 2237 reservable bandwidth and link resource class attribute (an 2238 administratively specified property of the link). The resource 2239 class attribute concept is defined in [RFC2702]. The additional 2240 topology state information is carried in new TLVs and sub-TLVs in 2241 IS-IS, or in the Opaque LSA in OSPF [RFC5305], [RFC3630]. 2243 An enhanced link-state IGP may flood information more frequently 2244 than a normal IGP. This is because even without changes in 2245 topology, changes in reservable bandwidth or link affinity can 2246 trigger the enhanced IGP to initiate flooding. A trade-off 2247 between the timeliness of the information flooded and the flooding 2248 frequency is typically implemented using a threshold based on the 2249 percentage change of the advertised resources to avoid excessive 2250 consumption of link bandwidth and computational resources, and to 2251 avoid instability in the TED. 2253 o In a TE system, it is also desirable for the routing subsystem to 2254 make the load splitting ratio among multiple paths (with equal 2255 cost or different cost) configurable. This capability gives 2256 network administrators more flexibility in the control of traffic 2257 distribution across the network. It can be very useful for 2258 avoiding/relieving congestion in certain situations. Examples can 2259 be found in [XIAO] and [I-D.ietf-bess-evpn-unequal-lb]. 2261 o The routing system should also have the capability to control the 2262 routes of subsets of traffic without affecting the routes of other 2263 traffic if sufficient resources exist for this purpose. This 2264 capability allows a more refined control over the distribution of 2265 traffic across the network. For example, the ability to move 2266 traffic away from its original path to another path (without 2267 affecting other traffic paths) allows the traffic to be moved from 2268 resource-poor network segments to resource-rich segments. Path 2269 oriented technologies such as MPLS-TE inherently support this 2270 capability as discussed in [AWD2]. 2272 o Additionally, the routing subsystem should be able to select 2273 different paths for different classes of traffic (or for different 2274 traffic behavior aggregates) if the network supports multiple 2275 classes of service (different behavior aggregates). 2277 6.3. Traffic Mapping Recommendations 2279 Traffic mapping is the assignment of traffic workload onto (pre- 2280 established) paths to meet certain requirements. Thus, while 2281 constraint-based routing deals with path selection, traffic mapping 2282 deals with the assignment of traffic to established paths which may 2283 have been generated by constraint-based routing or by some other 2284 means. Traffic mapping can be performed by time-dependent or state- 2285 dependent mechanisms, as described in Section 4.1. 2287 An important aspect of the traffic mapping function is the ability to 2288 establish multiple paths between an originating node and a 2289 destination node, and the capability to distribute the traffic 2290 between the two nodes across the paths according to some policies. A 2291 pre-condition for this scheme is the existence of flexible mechanisms 2292 to partition traffic and then assign the traffic partitions onto the 2293 parallel paths as noted in [RFC2702]. When traffic is assigned to 2294 multiple parallel paths, it is recommended that special care should 2295 be taken to ensure proper ordering of packets belonging to the same 2296 application (or traffic flow) at the destination node of the parallel 2297 paths. 2299 Mechanisms that perform the traffic mapping functions should aim to 2300 map the traffic onto the network infrastructure to minimize 2301 congestion. If the total traffic load cannot be accommodated, or if 2302 the routing and mapping functions cannot react fast enough to 2303 changing traffic conditions, then a traffic mapping system may use 2304 short time scale congestion control mechanisms (such as queue 2305 management, scheduling, etc.) to mitigate congestion. Thus, 2306 mechanisms that perform the traffic mapping functions complement 2307 existing congestion control mechanisms. In an operational network, 2308 traffic should be mapped onto the infrastructure such that intra- 2309 class and inter-class resource contention are minimized (see 2310 Section 2). 2312 When traffic mapping techniques that depend on dynamic state feedback 2313 (e.g., MATE [MATE] and such like) are used, special care must be 2314 taken to guarantee network stability. 2316 6.4. Measurement Recommendations 2318 The importance of measurement in traffic engineering has been 2319 discussed throughout this document. A TE system should include 2320 mechanisms to measure and collect statistics from the network to 2321 support the TE function. Additional capabilities may be needed to 2322 help in the analysis of the statistics. The actions of these 2323 mechanisms should not adversely affect the accuracy and integrity of 2324 the statistics collected. The mechanisms for statistical data 2325 acquisition should also be able to scale as the network evolves. 2327 Traffic statistics may be classified according to long-term or short- 2328 term timescales. Long-term traffic statistics are very useful for 2329 traffic engineering. Long-term traffic statistics may periodicity 2330 record network workload (such as hourly, daily, and weekly variations 2331 in traffic profiles) as well as traffic trends. Aspects of the 2332 traffic statistics may also describe class of service characteristics 2333 for a network supporting multiple classes of service. Analysis of 2334 the long-term traffic statistics may yield other information such as 2335 busy hour characteristics, traffic growth patterns, persistent 2336 congestion problems, hot-spot, and imbalances in link utilization 2337 caused by routing anomalies. 2339 A mechanism for constructing traffic matrices for both long-term and 2340 short-term traffic statistics should be in place. In multi-service 2341 IP networks, the traffic matrices may be constructed for different 2342 service classes. Each element of a traffic matrix represents a 2343 statistic about the traffic flow between a pair of abstract nodes. 2344 An abstract node may represent a router, a collection of routers, or 2345 a site in a VPN. 2347 Traffic statistics should provide reasonable and reliable indicators 2348 of the current state of the network on the short-term scale. Some 2349 short term traffic statistics may reflect link utilization and link 2350 congestion status. Examples of congestion indicators include 2351 excessive packet delay, packet loss, and high resource utilization. 2352 Examples of mechanisms for distributing this kind of information 2353 include SNMP, probing tools, FTP, IGP link state advertisements, and 2354 NETCONF/RESTCONF, etc. 2356 6.5. Policing, Planning, and Access Control 2358 The recommendations in Section 6.2 and Section 6.3 may be sub-optimal 2359 or even ineffective if the amount of traffic flowing on a route or 2360 path exceeds the capacity of the resource on that route or path. 2361 Several approaches can be used to increase the performance of TE 2362 systems. 2364 o The fundamental approach is some form of planning where traffic is 2365 steered onto paths so that it is distributed accross the available 2366 resources. This planning may be centralized or distributed, and 2367 must be aware of the planned traffic volumes amd available 2368 resources. However, this approach is only of value if the traffic 2369 is presented conformant to the planned traffic volumes. 2371 o Traffic flows may be policed at the edges of a network. This is a 2372 simple way to check that the actual traffic volumes are consistent 2373 with the planned volumes. Some form of measurement (see 2374 Section 6.4) is used to determine the rate of arrival of traffic 2375 and excess traffic could be discarded. Alternatively, excess 2376 traffic could be forwarded as best-effort within the network. 2377 However, this approach is only completley effective if the 2378 planning is stringent and network-wide, and if a harsh approach is 2379 taken to disposing of excess traffic. 2381 o Resource-based admission control is the process whereby network 2382 nodes decide whether to grant access to resources. The basis for 2383 the decision on a packet-by-packet basis is determination of the 2384 flow to which the packet belongs. This information is combined 2385 with policy instructions that have been locally configured, or 2386 installed through the management or control planes. The end 2387 result is that a packet may be allowed to access (or use) specific 2388 resources on the node if and only if the policy is conformed with 2389 for the flow to which the packet belongs. 2391 Combining some element of all three of these measures is advisable to 2392 achieve a better TE system. 2394 6.6. Network Survivability 2396 Network survivability refers to the capability of a network to 2397 maintain service continuity in the presence of faults. This can be 2398 accomplished by promptly recovering from network impairments and 2399 maintaining the required QoS for existing services after recovery. 2400 Survivability is an issue of great concern within the Internet 2401 community due to the demand to carry mission critical traffic, real- 2402 time traffic, and other high priority traffic over the Internet. 2403 Survivability can be addressed at the device level by developing 2404 network elements that are more reliable; and at the network level by 2405 incorporating redundancy into the architecture, design, and operation 2406 of networks. It is recommended that a philosophy of robustness and 2407 survivability should be adopted in the architecture, design, and 2408 operation of traffic engineering that control IP networks (especially 2409 public IP networks). Because different contexts may demand different 2410 levels of survivability, the mechanisms developed to support network 2411 survivability should be flexible so that they can be tailored to 2412 different needs. A number of tools and techniques have been 2413 developed to enable network survivability including MPLS Fast Reroute 2414 [RFC4090], RSVP-TE Extensions in Support of End-to-End GMPLS Recovery 2415 [RFC4872], and GMPLS Segment Recovery [RFC4873]. 2417 The impact of service outages varies significantly for different 2418 service classes depending on the duration of the outage which can 2419 vary from milliseconds (with minor service impact) to seconds (with 2420 possible call drops for IP telephony and session time-outs for 2421 connection oriented transactions) to minutes and hours (with 2422 potentially considerable social and business impact). Different 2423 duration outages have different impacts depending on the throughput 2424 of the traffic flows that are interrupted. 2426 Failure protection and restoration capabilities are available in 2427 multiple layers as network technologies have continued to evolve. 2428 Optical networks are capable of providing dynamic ring and mesh 2429 restoration functionality at the wavelength level. At the SONET/SDH 2430 layer survivability capability is provided with Automatic Protection 2431 Switching (APS) as well as self-healing ring and mesh architectures. 2432 Similar functionality is provided by layer 2 technologies such as 2433 Ethernet. 2435 Rerouting is used at the IP layer to restore service following link 2436 and node outages. Rerouting at the IP layer occurs after a period of 2437 routing convergence which may require seconds to minutes to complete. 2438 Path-oriented technologies such as MPLS ([RFC3469]) can be used to 2439 enhance the survivability of IP networks in a potentially cost 2440 effective manner. 2442 An important of multi-layer survivability is that technologies at 2443 different layers may provide protection and restoration capabilities 2444 at different granularities in terms of time scales and at different 2445 bandwidth granularity (from packet-level to wavelength level). 2446 Protection and restoration capabilities can also be sensitive to 2447 different service classes and different network utility models. 2448 Coordinating different protection and restoration capabilities across 2449 multiple layers in a cohesive manner to ensure network survivability 2450 is maintained at reasonable cost is a challenging task. Protection 2451 and restoration coordination across layers may not always be 2452 feasible, because networks at different layers may belong to 2453 different administrative domains. 2455 The following paragraphs present some of the general recommendations 2456 for protection and restoration coordination. 2458 o Protection and restoration capabilities from different layers 2459 should be coordinated to provide network survivability in a 2460 flexible and cost effective manner. Avoiding duplication of 2461 functions in different layers is one way to achieve the 2462 coordination. Escalation of alarms and other fault indicators 2463 from lower to higher layers may also be performed in a coordinated 2464 manner. The order of timing of restoration triggers from 2465 different layers is another way to coordinate multi-layer 2466 protection/restoration. 2468 o Network capacity reserved in one layer to provide protection and 2469 restoration is not available to carry traffic in a higher layer: 2470 it is not visible as spare capacity in the higher layer. Placing 2471 protection/restoration functions in many layers may increase 2472 redundancy and robustness, but it can result in significant 2473 inefficiencies in network resource utilization. Careful planning 2474 is needed to balance the trade-off between the desire for 2475 survivablity and the optimal use of resources. 2477 o It is generally desirable to have protection and restoration 2478 schemes that are intrinsically bandwidth efficient. 2480 o Failure notifications throughout the network should be timely and 2481 reliable if they are to be acted on as triggers for effective 2482 protection and restoration actions. 2484 o Alarms and other fault monitoring and reporting capabilities 2485 should be provided at the right network layers so that the 2486 protection and restoration actions can be taken in those layers. 2488 6.6.1. Survivability in MPLS Based Networks 2490 Because MPLS is path-oriented, it has the potential to provide faster 2491 and more predictable protection and restoration capabilities than 2492 conventional hop by hop routed IP systems. Protection types for MPLS 2493 networks can be divided into four categories. 2495 o Link Protection: The objective of link protection is to protect an 2496 LSP from the failure of a given link. Under link protection, a 2497 protection or backup LSP (the secondary LSP) follows a path that 2498 is disjoint from the path of the working or operational LSP (the 2499 primary LSP) at the particular link where link protection is 2500 required. When the protected link fails, traffic on the working 2501 LSP is switched to the protection LSP at the head-end of the 2502 failed link. As a local repair method, link protection can be 2503 fast. This form of protection may be most appropriate in 2504 situations where some network elements along a given path are 2505 known to be less reliable than others. 2507 o Node Protection: The objective of node protection is to protect an 2508 LSP from the failure of a given node. Under node protection, the 2509 secondary LSP follows a path that is disjoint from the path of the 2510 primary LSP at the particular node where node protection is 2511 required. The secondary LSP is also disjoint from the primary LSP 2512 at all links attached to the node to be protected. When the 2513 protected node fails, traffic on the working LSP is switched over 2514 to the protection LSP at the upstream LSR directly connected to 2515 the failed node. Node protection covers a slightly larger part of 2516 the network compared to link protection, but is otherwise 2517 fundamentally the same. 2519 o Path Protection: The goal of LSP path protection (or end-to-end 2520 protection) is to protect an LSP from any failure along its routed 2521 path. Under path protection, the path of the protection LSP is 2522 completely disjoint from the path of the working LSP. The 2523 advantage of path protection is that the backup LSP protects the 2524 working LSP from all possible link and node failures along the 2525 path, except for failures of ingress or egress LSR. Additionally, 2526 path protection may be more efficient in terms of resource usage 2527 than link or node protection applied at every jop along the path. 2528 However, path protection may be slower than link and node 2529 protection because the fault notifications have to be propagated 2530 further. 2532 o Segment Protection: An MPLS domain may be partitioned into 2533 multiple subdomains (protection domains). Path protection is 2534 applied to the path of each LSP as it crosses the domain from its 2535 ingress to the domain to where it egresses the domain. In cases 2536 where an LSP traverses multiple protection domains, a protection 2537 mechanism within a domain only needs to protect the segment of the 2538 LSP that lies within the domain. Segment protection will 2539 generally be faster than end-to-end path protection because 2540 recovery generally occurs closer to the fault and the notification 2541 doesn't have to propagate as far. 2543 See [RFC3469] and [RFC6372] for a more comprehensive discussion of 2544 MPLS based recovery. 2546 6.6.2. Protection Options 2548 Another issue to consider is the concept of protection options. We 2549 use notation such as "m:n protection", where m is the number of 2550 protection LSPs used to protect n working LSPs. In all cases except 2551 1+1 protection, the resources associated with the protection LSPs can 2552 be used to carry preemptable best-effort traffic when the working LSP 2553 is functioning correctly. 2555 o 1:1 protection: One working LSP is protected/restored by one 2556 protection LSP. 2558 o 1:n protection: One protection LSP is used to protect/restore n 2559 working LSPs. Only one failed LSP can be restored at any time. 2561 o n:1 protection: One working LSP is protected/restored by n 2562 protection LSPs, possibly with load splitting across the 2563 protection LSPs. This may be especially useful when it is not 2564 feasible to find one path for the backup that can satisfy the 2565 bandwidth requirement of the primary LSP. 2567 o 1+1 protection: Traffic is sent concurrently on both the working 2568 LSP and a protection LSP. The egress LSR selects one of the two 2569 LSPs based on local policy (usually based on traffic integrity). 2570 When a fault disrupts the traffic on one LSP, the egress switches 2571 to receive traffic from the other LSP. This approach is expensive 2572 in how it consumes network but recovers from failures most 2573 rapidly. 2575 6.7. Multi-Layer Traffic Engineering 2577 Networks are often arranged as layers. A layer relationship may 2578 represent the interaction between technologies (for example, an IP 2579 network operated over an optical network), or the relationship 2580 between different network operators (for example, a customer network 2581 operated over a service provider's network). Note that a multi-layer 2582 network does not imply the use of multiple technologies, although 2583 some form of encapsulation is often applied. 2585 Multi-layer traffic engineering presents a number of challenges 2586 associated with scalability and confidentiality. These issues are 2587 addressed in [RFC7926] which discusses the sharing of information 2588 between domains through policy filters, aggregation, abstraction, and 2589 virtualization. That document also discusses how existing protocols 2590 can support this scenario with special reference to BGP-LS (see 2591 Section 5.1.12). 2593 PCE (see Section 5.1.13) is also a useful tool for multi-layer 2594 networks as described in [RFC6805] and [RFC8685]. Signaling 2595 techniques for multi-layer traffic engineering are described in 2596 [RFC6107]. 2598 See also Appendix A.3.1 for a discussion of how the overlay model has 2599 been important in the development of traffic engineering. 2601 6.8. Traffic Engineering in Diffserv Environments 2603 Increasing requirements to support multiple classes of traffic in the 2604 Internet, such as best effort and mission critical data, calls for IP 2605 networks to differentiate traffic according to some criteria and to 2606 give preferential treatment to certain types of traffic. Large 2607 numbers of flows can be aggregated into a few behavior aggregates 2608 based on some criteria based on common performance requirements in 2609 terms of packet loss ratio, delay, and jitter, or in terms of common 2610 fields within the IP packet headers. 2612 Differentiated Services (Diffserv) [RFC2475] can be used to ensure 2613 that SLAs defined to differentiate between traffic flows are met. 2614 Classes of service (CoS) can be supported in a Diffserv environment 2615 by concatenating per-hop behaviors (PHBs) along the routing path. A 2616 PHB is the forwarding behavior that a packet receives at a Diffserv- 2617 compliant node, and it can be configured at each router. PHBs are 2618 delivered using buffer management and packet scheduling mechanisms 2619 and require that the ingress nodes use traffic classification, 2620 marking, policing, and shaping. 2622 Traffic engineering can complement Diffserv to improve utilization of 2623 network resources. Traffic engineering can be operated on an 2624 aggregated basis across all service classes [RFC3270], or on a per 2625 service class basis. The former is used to provide better 2626 distribution of the traffic load over the network resources (see 2627 [RFC3270] for detailed mechanisms to support aggregate traffic 2628 engineering). The latter case is discussed below since it is 2629 specific to the Diffserv environment, with so called Diffserv-aware 2630 traffic engineering [RFC4124]. 2632 For some Diffserv networks, it may be desirable to control the 2633 performance of some service classes by enforcing relationships 2634 between the traffic workload contributed by each service class and 2635 the amount of network resources allocated or provisioned for that 2636 service class. Such relationships between demand and resource 2637 allocation can be enforced using a combination of, for example: 2639 o TE mechanisms on a per service class basis that enforce the 2640 relationship between the amount of traffic contributed by a given 2641 service class and the resources allocated to that class. 2643 o Mechanisms that dynamically adjust the resources allocated to a 2644 given service class to relate to the amount of traffic contributed 2645 by that service class. 2647 It may also be desirable to limit the performance impact of high 2648 priority traffic on relatively low priority traffic. This can be 2649 achieved, for example, by controlling the percentage of high priority 2650 traffic that is routed through a given link. Another way to 2651 accomplish this is to increase link capacities appropriately so that 2652 lower priority traffic can still enjoy adequate service quality. 2653 When the ratio of traffic workload contributed by different service 2654 classes varies significantly from router to router, it may not be 2655 enough to rely on conventional IGP routing protocols or on TE 2656 mechanisms that are not sensitive to different service classes. 2657 Instead, it may be desirable to perform traffic engineering, 2658 especially routing control and mapping functions, on a per service 2659 class basis. One way to accomplish this in a domain that supports 2660 both MPLS and Diffserv is to define class specific LSPs and to map 2661 traffic from each class onto one or more LSPs that correspond to that 2662 service class. An LSP corresponding to a given service class can 2663 then be routed and protected/restored in a class dependent manner, 2664 according to specific policies. 2666 Performing traffic engineering on a per class basis may require per- 2667 class parameters to be distributed. It is common to have some 2668 classes share some aggregate constraints (e.g., maximum bandwidth 2669 requirement) without enforcing the constraint on each individual 2670 class. These classes can be grouped into class-types, and per-class- 2671 type parameters can be distributed to improve scalability. This also 2672 allows better bandwidth sharing between classes in the same class- 2673 type. A class-type is a set of classes that satisfy the following 2674 two conditions: 2676 o Classes in the same class-type have common aggregate requirements 2677 to satisfy required performance levels. 2679 o There is no requirement to be enforced at the level of an 2680 individual class in the class-type. Note that it is, 2681 nevertheless, still possible to implement some priority policies 2682 for classes in the same class-type to permit preferential access 2683 to the class-type bandwidth through the use of preemption 2684 priorities. 2686 See [RFC4124] for detailed requirements on Diffserv-aware traffic 2687 engineering. 2689 6.9. Network Controllability 2691 Offline and online (see Section 4.2) TE considerations are of limited 2692 utility if the network cannot be controlled effectively to implement 2693 the results of TE decisions and to achieve the desired network 2694 performance objectives. 2696 Capacity augmentation is a coarse-grained solution to TE issues. 2697 However, it is simple and may be advantageous if bandwidth is 2698 abundant and cheap. However, bandwidth is not always abundant and 2699 cheap, and additional capacity might not always be the best solution. 2700 Adjustments of administrative weights and other parameters associated 2701 with routing protocols provide finer-grained control, but this 2702 approach is difficult to use and imprecise because of the the way the 2703 routing protocols interact occur across the network. 2705 Control mechanisms can be manual (e.g., static configuration), 2706 partially-automated (e.g., scripts), or fully-automated (e.g., policy 2707 based management systems). Automated mechanisms are particularly 2708 useful in large scale networks. Multi-vendor interoperability can be 2709 facilitated by standardized management systems (e.g., YANG models) to 2710 support the control functions required to address TE objectives. 2712 Network control functions should be secure, reliable, and stable as 2713 these are often needed to operate correctly in times of network 2714 impairments (e.g., during network congestion or security attacks). 2716 7. Inter-Domain Considerations 2718 Inter-domain TE is concerned with performance optimization for 2719 traffic that originates in one administrative domain and terminates 2720 in a different one. 2722 BGP [RFC4271] is the standard exterior gateway protocol used to 2723 exchange routing information between autonomous systems (ASes) in the 2724 Internet. BGP includes a sequential decision process that calculates 2725 the preference for routes to a given destination network. There are 2726 two fundamental aspects to inter-domain TE using BGP: 2728 o Route Redistribution: Controlling the import and export of routes 2729 between ASes, and controlling the redistribution of routes between 2730 BGP and other protocols within an AS. 2732 o Best path selection: Selecting the best path when there are 2733 multiple candidate paths to a given destination network. This is 2734 performed by the BGP decision process, selecting preferred exit 2735 points out of an AS towards specific destination networks taking a 2736 number of different considerations into account. The BGP path 2737 selection process can be influenced by manipulating the attributes 2738 associated with the process, including NEXT-HOP, WEIGHT, LOCAL- 2739 PREFERENCE, AS-PATH, ROUTE-ORIGIN, MULTI-EXIT-DESCRIMINATOR (MED), 2740 IGP METRIC, etc. 2742 Route-maps provide the flexibility to implement complex BGP policies 2743 based on pre-configured logical conditions. They can be used to 2744 control import and export policies for incoming and outgoing routes, 2745 control the redistribution of routes between BGP and other protocols, 2746 and influence the selection of best paths by manipulating the 2747 attributes associated with the BGP decision process. Very complex 2748 logical expressions that implement various types of policies can be 2749 implemented using a combination of Route-maps, BGP-attributes, 2750 Access-lists, and Community attributes. 2752 When considering inter-domain TE with BGP, note that the outbound 2753 traffic exit point is controllable, whereas the interconnection point 2754 where inbound traffic is received typically is not. Therefore, it is 2755 up to each individual network to implement TE strategies that deal 2756 with the efficient delivery of outbound traffic from its customers to 2757 its peering points. The vast majority of TE policy is based on a 2758 "closest exit" strategy, which offloads inter-domain traffic at the 2759 nearest outbound peering point towards the destination AS. Most 2760 methods of manipulating the point at which inbound traffic enters a 2761 are either ineffective, or not accepted in the peering community. 2763 Inter-domain TE with BGP is generally effective, but it is usually 2764 applied in a trial-and-error fashion because a TE system usually only 2765 has a view of the available network resources within one domain (an 2766 AS in this case). A systematic approach for inter-domain TE requires 2767 cooperation between the domains. Further, what may be considered a 2768 good solution in one domain may not necessarily be a good solution in 2769 another. Moreover, it is generally considered inadvisable for one 2770 domain to permit a control process from another domain to influence 2771 the routing and management of traffic in its network. 2773 MPLS TE-tunnels (LSPs) can add a degree of flexibility in the 2774 selection of exit points for inter-domain routing by applying rhe 2775 concept of relative and absolute metrics. If BGP attributes are 2776 defined such that the BGP decision process depends on IGP metrics to 2777 select exit points for inter-domain traffic, then some inter-domain 2778 traffic destined to a given peer network can be made to prefer a 2779 specific exit point by establishing a TE-tunnel between the router 2780 making the selection and the peering point via a TE-tunnel and 2781 assigning the TE-tunnel a metric which is smaller than the IGP cost 2782 to all other peering points. 2784 Similarly to intra-domain TE, inter-domain TE is best accomplished 2785 when a traffic matrix can be derived to depict the volume of traffic 2786 from one AS to another. 2788 8. Overview of Contemporary TE Practices in Operational IP Networks 2790 This section provides an overview of some traffic engineering 2791 practices in IP networks. The focus is on aspects of control of the 2792 routing function in operational contexts. The intent here is to 2793 provide an overview of the commonly used practices: the discussion is 2794 not intended to be exhaustive. 2796 Service providers apply many of the traffic engineering mechanisms 2797 described in this document to optimize the performance of their IP 2798 networks. These techniques include capacity planning for long 2799 timescales; routing control using IGP metrics and MPLS, as well as 2800 path planning and path control using MPLS and Segment Routing for 2801 medium timescales; and traffic management mechanisms for short 2802 timescale. 2804 Capacity planning is an important component of how a service provider 2805 plans an effective IP network. These plans may take the following 2806 aspects into account: location of and new links or nodes, existing 2807 and predicted traffic patterns, costs, link capacity, topology, 2808 routing design, and survivability. 2810 Performance optimization of operational networks is usually an 2811 ongoing process in which traffic statistics, performance parameters, 2812 and fault indicators are continually collected from the network. 2813 This empirical data is analyzed and used to trigger TE mechanisms. 2814 Tools that perform what-if analysis can also be used to assist the TE 2815 process by reviewing scenarios before a new set of configurations are 2816 implemented in the operational network. 2818 Real-time intra-domain TE using the IGP is done by increasing the 2819 OSPF or IS-IS metric of a congested link until enough traffic has 2820 been diverted away from that link. This approach has some 2821 limitations as discussed in Section 6.2. Intra-domain TE approaches 2822 ([RR94] [FT00] [FT01] [WANG]) take traffic matrix, network topology, 2823 and network performance objectives as input, and produce link metrics 2824 and load-sharing ratios. These processes open the possibility for 2825 intra-domain TE with IGP to be done in a more systematic way. 2827 Administrators of MPLS-TE networks specify and configure link 2828 attributes and resource constraints such as maximum reservable 2829 bandwidth and resource class attributes for the links in the domain. 2830 A link state IGP that supports TE extensions (IS-IS-TE or OSPF-TE) is 2831 used to propagate information about network topology and link 2832 attributes to all routers in the domain. Network administrators 2833 specify the LSPs that are to originate at each router. For each LSP, 2834 the network administrator specifies the destination node and the 2835 attributes of the LSP which indicate the requirements that are to be 2836 satisfied during the path selection process. The attributes may 2837 include and explicit path for the LSP to follow, or originating 2838 router uses a local constraint-based routing process to compute the 2839 path of the LSP. RSVP-TE is used as a signaling protocol to 2840 instantiate the LSPs. By assigning proper bandwidth values to links 2841 and LSPs, congestion caused by uneven traffic distribution can be 2842 avoided or mitigated. 2844 The bandwidth attributes of an LSP relates to the bandwidth 2845 requirements of traffic that flows through the LSP. The traffic 2846 attribute of an LSP can be modified to accommodate persistent shifts 2847 in demand (traffic growth or reduction). If network congestion 2848 occurs due to some unexpected events, existing LSPs can be rerouted 2849 to alleviate the situation or network administrator can configure new 2850 LSPs to divert some traffic to alternative paths. The reservable 2851 bandwidth of the congested links can also be reduced to force some 2852 LSPs to be rerouted to other paths. A traffic matrix in an MPLS 2853 domain can also be estimated by monitoring the traffic on LSPs. Such 2854 traffic statistics can be used for a variety of purposes including 2855 network planning and network optimization. 2857 Network management and planning systems have evolved and taken over a 2858 lot of the responsibility for determining traffic paths in TE 2859 networks. This allows a network-wide view of resources, and 2860 facilitates coordination of the use of resources for all traffic 2861 flows in the network. Initial solutions using a PCE to perform path 2862 computation on behalf of network routers have given way to an 2863 approach that follows the SDN architecture. A stateful PCE is able 2864 to track all of the LSPs in the network and can redistribute them to 2865 make better use of the available resources. Such a PCE can forms 2866 part of a network orchestrator that uses PCEP or some other 2867 southbound interface to instruct the signaling protocol or directly 2868 program the routers. 2870 Segment routing leverages a centralized TE controller and either an 2871 MPLS or IPv6 forwarding plane, but does not need to use a signaling 2872 protocol or management plane protocol to reserve resources in the 2873 routers. All resource reservation is logical within the controller, 2874 and not distributed to the routers. Packets are steered through the 2875 network using segment routing. 2877 As mentioned in Section 7, there is usually no direct control over 2878 the distribution of inbound traffic to a domain. Therefore, the main 2879 goal of inter-domain TE is to optimize the distribution of outbound 2880 traffic between multiple inter-domain links. When operating a global 2881 network, maintaining the ability to operate the network in a regional 2882 fashion where desired, while continuing to take advantage of the 2883 benefits of a global network, also becomes an important objective. 2885 Inter-domain TE with BGP begins with the placement of multiple 2886 peering interconnection points that are in close proximity to traffic 2887 sources/destination, and offer lowest cost paths across the network 2888 between the peering points and the sources/destinations. Some 2889 location-decision problems that arise in association with inter- 2890 domain routing are discussed in [AWD5]. 2892 Once the locations of the peering interconnects have been determined 2893 and implemented, the network operator decides how best to handle the 2894 routes advertised by the peer, as well as how to propagate the peer's 2895 routes within their network. One way to engineer outbound traffic 2896 flows in a network with many peering interconnects is to create a 2897 hierarchy of peers. Generally, the shortest AS paths will be chosen 2898 to forward traffic but BGP metrics can be used to prefer some peers 2899 and so favor particular paths. Preferred peers are those peers 2900 attached through peering interconnects with the most available 2901 capacity. Changes may be needed, for example, to deal with a 2902 "problem peer" who is difficult to work with on upgrades or is 2903 charging high prices for connectivity to their network. In that 2904 case, the peer may be given a reduced preference. This type of 2905 change can affect a large amount of traffic, and is only used after 2906 other methods have failed to provide the desired results. 2908 When there are multiple exit points toward a given peer, and only one 2909 of them is congested, it is not necessary to shift traffic away from 2910 the peer entirely, but only from the one congested connections. This 2911 can be achieved by using passive IGP-metrics, AS-path filtering, or 2912 prefix filtering. 2914 9. Security Considerations 2916 This document does not introduce new security issues. 2918 Network security is, of course, an important issue. In general, TE 2919 mechanisms are security neutral: they may use tunnels which can 2920 slightly help protect traffic from inspection and which, in some 2921 cases, can be secured using encryption; they put traffic onto 2922 predictable paths within the network that may make it easier to find 2923 and attack; they increase the complexity or operation and management 2924 of the network; and they enable traffic to be steered onto more 2925 secure links or to more secure parts of the network. 2927 The consequences of attacks on the control and management protocols 2928 used to operate TE networks can be significant: traffic can be 2929 hijacked to pass through specific nodes that perform inspection, or 2930 even to be delivered to the wrong place; traffic can be steered onto 2931 paths that deliver quality that is below the desired quality; and, 2932 networks can be congested or have resources on key links consumed. 2934 Thus, it is important to use adequate protection mechanisms on all 2935 protocols used to deliver TE. 2937 Certain aspects of a network may be deduced from the details of the 2938 TE paths that are used. For example, the link connectivity of the 2939 network, and the quality and load on individual links may be assumed 2940 from knowing the paths of traffic and the requirements they place on 2941 the network (for example, by seeing the control messages or through 2942 path- trace techniques). Such knowledge can be used to launch 2943 targeted attacks (for example, taking down critical links) or can 2944 reveal commercially sensitive information (for example, whether a 2945 network is close to capacity). Network operators may, therefore, 2946 choose techniques that mask or hide information from within the 2947 network. 2949 10. IANA Considerations 2951 This draft makes no requests for IANA action. 2953 11. Acknowledgments 2955 Much of the text in this document is derived from RFC 3272. The 2956 authors of this document would like to express their gratitude to all 2957 involved in that work. Although the source text has been edited in 2958 the production of this document, the original authors should be 2959 considered as Contributors to this work. They were: 2961 Daniel O. Awduche 2962 Movaz Networks 2964 Angela Chiu 2965 Celion Networks 2967 Anwar Elwalid 2968 Lucent Technologies 2970 Indra Widjaja 2971 Bell Labs, Lucent Technologies 2973 XiPeng Xiao 2974 Redback Networks 2976 The acknowledgements in RFC3272 were as below. All people who helped 2977 in the production of that document also need to be thanked for the 2978 carry-over into this new document. 2980 The authors would like to thank Jim Boyle for inputs on the 2981 recommendations section, Francois Le Faucheur for inputs on 2982 Diffserv aspects, Blaine Christian for inputs on measurement, 2983 Gerald Ash for inputs on routing in telephone networks and for 2984 text on event-dependent TE methods, Steven Wright for inputs 2985 on network controllability, and Jonathan Aufderheide for 2986 inputs on inter-domain TE with BGP. Special thanks to 2987 Randy Bush for proposing the TE taxonomy based on "tactical versus 2988 strategic" methods. The subsection describing an "Overview of 2989 ITU Activities Related to Traffic Engineering" was adapted from 2990 a contribution by Waisum Lai. Useful feedback and pointers to 2991 relevant materials were provided by J. Noel Chiappa. 2992 Additional comments were provided by Glenn Grotefeld during 2993 the working last call process. Finally, the authors would like 2994 to thank Ed Kern, the TEWG co-chair, for his comments and 2995 support. 2997 The early versions of this document were produced by the TEAS Working 2998 Group's RFC3272bis Design Team. The full list of members of this 2999 team is: 3001 Acee Lindem 3002 Adrian Farrel 3003 Aijun Wang 3004 Daniele Ceccarelli 3005 Dieter Beller 3006 Jeff Tantsura 3007 Julien Meuric 3008 Liu Hua 3009 Loa Andersson 3010 Luis Miguel Contreras 3011 Martin Horneffer 3012 Tarek Saad 3013 Xufeng Liu 3015 The production of this document includes a fix to the original text 3016 resulting from an Errata Report by Jean-Michel Grimaldi. 3018 The author of this document would also like to thank Dhurv Dhody for 3019 review comments. 3021 12. Contributors 3023 The following people contributed substantive text to this document: 3025 Gert Grammel 3026 EMail: ggrammel@juniper.net 3028 Loa Andersson 3029 EMail: loa@pi.nu 3031 Xufeng Liu 3032 EMail: xufeng.liu.ietf@gmail.com 3034 Lou Berger 3035 EMail: lberger@labn.net 3037 Jeff Tantsura 3038 EMail: jefftant.ietf@gmail.com 3040 Daniel King 3041 EMail: daniel@olddog.co.uk 3043 Boris Hassanov 3044 EMail: bhassanov@yandex-team.ru 3046 Kiran Makhijani 3047 Email: kiranm@futurewei.com 3049 Dhruv Dhody 3050 Email: dhruv.ietf@gmail.com 3052 13. Informative References 3054 [AJ19] Adekitan, A., Abolade, J., and O. Shobayo, "Data mining 3055 approach for predicting the daily Internet data traffic of 3056 a smart university", Article Journal of Big Data, 2019, 3057 Volume 6, Number 1, Page 1, 1998. 3059 [ASH2] Ash, J., "Dynamic Routing in Telecommunications Networks", 3060 Book McGraw Hill, 1998. 3062 [AWD2] Awduche, D., "MPLS and Traffic Engineering in IP 3063 Networks", Article IEEE Communications Magazine, December 3064 1999. 3066 [AWD5] Awduche, D., "An Approach to Optimal Peering Between 3067 Autonomous Systems in the Internet", Paper International 3068 Conference on Computer Communications and Networks 3069 (ICCCN'98), October 1998. 3071 [FLJA93] Floyd, S. and V. Jacobson, "Random Early Detection 3072 Gateways for Congestion Avoidance", Article IEEE/ACM 3073 Transactions on Networking, Vol. 1, p. 387-413, November 3074 1993. 3076 [FLOY94] Floyd, S., "TCP and Explicit Congestion Notification", 3077 Article ACM Computer Communication Review, V. 24, No. 5, 3078 p. 10-23, October 1994. 3080 [FT00] Fortz, B. and M. Thorup, "Internet Traffic Engineering by 3081 Optimizing OSPF Weights", Article IEEE INFOCOM 2000, March 3082 2000. 3084 [FT01] Fortz, B. and M. Thorup, "Optimizing OSPF/IS-IS Weights in 3085 a Changing World", n.d., 3086 . 3088 [HUSS87] Hurley, B., Seidl, C., and W. Sewel, "A Survey of Dynamic 3089 Routing Methods for Circuit-Switched Traffic", 3090 Article IEEE Communication Magazine, September 1987. 3092 [I-D.ietf-alto-performance-metrics] 3093 Wu, Q., Yang, Y. R., Lee, Y., Dhody, D., Randriamasy, S., 3094 and L. M. Contreras, "ALTO Performance Cost Metrics", 3095 draft-ietf-alto-performance-metrics-15 (work in progress), 3096 February 2021. 3098 [I-D.ietf-bess-evpn-unequal-lb] 3099 Malhotra, N., Sajassi, A., Rabadan, J., Drake, J., 3100 Lingala, A., and S. Thoria, "Weighted Multi-Path 3101 Procedures for EVPN All-Active Multi-Homing", draft-ietf- 3102 bess-evpn-unequal-lb-08 (work in progress), February 2021. 3104 [I-D.ietf-detnet-ip-over-tsn] 3105 Varga, B., Farkas, J., Malis, A. G., and S. Bryant, 3106 "DetNet Data Plane: IP over IEEE 802.1 Time Sensitive 3107 Networking (TSN)", draft-ietf-detnet-ip-over-tsn-07 (work 3108 in progress), February 2021. 3110 [I-D.ietf-idr-performance-routing] 3111 Xu, X., Hegde, S., Talaulikar, K., Boucadair, M., and C. 3112 Jacquenet, "Performance-based BGP Routing Mechanism", 3113 draft-ietf-idr-performance-routing-03 (work in progress), 3114 December 2020. 3116 [I-D.ietf-idr-segment-routing-te-policy] 3117 Previdi, S., Filsfils, C., Talaulikar, K., Mattes, P., 3118 Rosen, E., Jain, D., and S. Lin, "Advertising Segment 3119 Routing Policies in BGP", draft-ietf-idr-segment-routing- 3120 te-policy-11 (work in progress), November 2020. 3122 [I-D.ietf-lsr-flex-algo] 3123 Psenak, P., Hegde, S., Filsfils, C., Talaulikar, K., and 3124 A. Gulko, "IGP Flexible Algorithm", draft-ietf-lsr-flex- 3125 algo-15 (work in progress), April 2021. 3127 [I-D.ietf-lsr-ip-flexalgo] 3128 Britto, W., Hegde, S., Kaneriya, P., Shetty, R., Bonica, 3129 R., and P. Psenak, "IGP Flexible Algorithms (Flex- 3130 Algorithm) In IP Networks", draft-ietf-lsr-ip-flexalgo-02 3131 (work in progress), April 2021. 3133 [I-D.ietf-quic-transport] 3134 Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed 3135 and Secure Transport", draft-ietf-quic-transport-34 (work 3136 in progress), January 2021. 3138 [I-D.ietf-spring-segment-routing-policy] 3139 Filsfils, C., Talaulikar, K., Voyer, D., Bogdanov, A., and 3140 P. Mattes, "Segment Routing Policy Architecture", draft- 3141 ietf-spring-segment-routing-policy-11 (work in progress), 3142 April 2021. 3144 [I-D.ietf-teas-enhanced-vpn] 3145 Dong, J., Bryant, S., Li, Z., Miyasaka, T., and Y. Lee, "A 3146 Framework for Enhanced Virtual Private Network (VPN+) 3147 Services", draft-ietf-teas-enhanced-vpn-07 (work in 3148 progress), February 2021. 3150 [I-D.ietf-teas-ietf-network-slices] 3151 Farrel, A., Gray, E., Drake, J., Rokui, R., Homma, S., 3152 Makhijani, K., Contreras, L. M., and J. Tantsura, 3153 "Framework for IETF Network Slices", draft-ietf-teas-ietf- 3154 network-slices-00 (work in progress), April 2021. 3156 [I-D.ietf-tewg-qos-routing] 3157 Ash, G., "Traffic Engineering & QoS Methods for IP-, ATM-, 3158 & Based Multiservice Networks", draft-ietf-tewg-qos- 3159 routing-04 (work in progress), October 2001. 3161 [I-D.irtf-nmrg-ibn-concepts-definitions] 3162 Clemm, A., Ciavaglia, L., Granville, L. Z., and J. 3163 Tantsura, "Intent-Based Networking - Concepts and 3164 Definitions", draft-irtf-nmrg-ibn-concepts-definitions-03 3165 (work in progress), February 2021. 3167 [ITU-E600] 3168 "Terms and Definitions of Traffic Engineering", 3169 Recommendation ITU-T Recommendation E.600, March 1993. 3171 [ITU-E701] 3172 "Reference Connections for Traffic Engineering", 3173 Recommendation ITU-T Recommendation E.701, October 1993. 3175 [ITU-E801] 3176 "Framework for Service Quality Agreement", 3177 Recommendation ITU-T Recommendation E.801, October 1996. 3179 [MA] Ma, Q., "Quality of Service Routing in Integrated Services 3180 Networks", Ph.D. PhD Dissertation, CMU-CS-98-138, CMU, 3181 1998. 3183 [MATE] Elwalid, A., Jin, C., Low, S., and I. Widjaja, "MATE - 3184 MPLS Adaptive Traffic Engineering", 3185 Proceedings INFOCOM'01, April 2001. 3187 [MCQ80] McQuillan, J., Richer, I., and E. Rosen, "The New Routing 3188 Algorithm for the ARPANET", Transaction IEEE Transactions 3189 on Communications, vol. 28, no. 5, p. 711-719, May 1980. 3191 [MR99] Mitra, D. and K. Ramakrishnan, "A Case Study of 3192 Multiservice, Multipriority Traffic Engineering Design for 3193 Data Networks", Proceedings Globecom'99, December 1999. 3195 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 3196 DOI 10.17487/RFC0791, September 1981, 3197 . 3199 [RFC1102] Clark, D., "Policy routing in Internet protocols", 3200 RFC 1102, DOI 10.17487/RFC1102, May 1989, 3201 . 3203 [RFC1104] Braun, H., "Models of policy based routing", RFC 1104, 3204 DOI 10.17487/RFC1104, June 1989, 3205 . 3207 [RFC1992] Castineyra, I., Chiappa, N., and M. Steenstrup, "The 3208 Nimrod Routing Architecture", RFC 1992, 3209 DOI 10.17487/RFC1992, August 1996, 3210 . 3212 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 3213 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 3214 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 3215 September 1997, . 3217 [RFC2328] Moy, J., "OSPF Version 2", STD 54, RFC 2328, 3218 DOI 10.17487/RFC2328, April 1998, 3219 . 3221 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 3222 "Framework for IP Performance Metrics", RFC 2330, 3223 DOI 10.17487/RFC2330, May 1998, 3224 . 3226 [RFC2386] Crawley, E., Nair, R., Rajagopalan, B., and H. Sandick, "A 3227 Framework for QoS-based Routing in the Internet", 3228 RFC 2386, DOI 10.17487/RFC2386, August 1998, 3229 . 3231 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 3232 "Definition of the Differentiated Services Field (DS 3233 Field) in the IPv4 and IPv6 Headers", RFC 2474, 3234 DOI 10.17487/RFC2474, December 1998, 3235 . 3237 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 3238 and W. Weiss, "An Architecture for Differentiated 3239 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 3240 . 3242 [RFC2597] Heinanen, J., Baker, F., Weiss, W., and J. Wroclawski, 3243 "Assured Forwarding PHB Group", RFC 2597, 3244 DOI 10.17487/RFC2597, June 1999, 3245 . 3247 [RFC2678] Mahdavi, J. and V. Paxson, "IPPM Metrics for Measuring 3248 Connectivity", RFC 2678, DOI 10.17487/RFC2678, September 3249 1999, . 3251 [RFC2702] Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M., and J. 3252 McManus, "Requirements for Traffic Engineering Over MPLS", 3253 RFC 2702, DOI 10.17487/RFC2702, September 1999, 3254 . 3256 [RFC2722] Brownlee, N., Mills, C., and G. Ruth, "Traffic Flow 3257 Measurement: Architecture", RFC 2722, 3258 DOI 10.17487/RFC2722, October 1999, 3259 . 3261 [RFC2753] Yavatkar, R., Pendarakis, D., and R. Guerin, "A Framework 3262 for Policy-based Admission Control", RFC 2753, 3263 DOI 10.17487/RFC2753, January 2000, 3264 . 3266 [RFC2961] Berger, L., Gan, D., Swallow, G., Pan, P., Tommasi, F., 3267 and S. Molendini, "RSVP Refresh Overhead Reduction 3268 Extensions", RFC 2961, DOI 10.17487/RFC2961, April 2001, 3269 . 3271 [RFC2998] Bernet, Y., Ford, P., Yavatkar, R., Baker, F., Zhang, L., 3272 Speer, M., Braden, R., Davie, B., Wroclawski, J., and E. 3273 Felstaine, "A Framework for Integrated Services Operation 3274 over Diffserv Networks", RFC 2998, DOI 10.17487/RFC2998, 3275 November 2000, . 3277 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 3278 Label Switching Architecture", RFC 3031, 3279 DOI 10.17487/RFC3031, January 2001, 3280 . 3282 [RFC3086] Nichols, K. and B. Carpenter, "Definition of 3283 Differentiated Services Per Domain Behaviors and Rules for 3284 their Specification", RFC 3086, DOI 10.17487/RFC3086, 3285 April 2001, . 3287 [RFC3124] Balakrishnan, H. and S. Seshan, "The Congestion Manager", 3288 RFC 3124, DOI 10.17487/RFC3124, June 2001, 3289 . 3291 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 3292 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 3293 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 3294 . 3296 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 3297 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 3298 Protocol Label Switching (MPLS) Support of Differentiated 3299 Services", RFC 3270, DOI 10.17487/RFC3270, May 2002, 3300 . 3302 [RFC3272] Awduche, D., Chiu, A., Elwalid, A., Widjaja, I., and X. 3303 Xiao, "Overview and Principles of Internet Traffic 3304 Engineering", RFC 3272, DOI 10.17487/RFC3272, May 2002, 3305 . 3307 [RFC3469] Sharma, V., Ed. and F. Hellstrand, Ed., "Framework for 3308 Multi-Protocol Label Switching (MPLS)-based Recovery", 3309 RFC 3469, DOI 10.17487/RFC3469, February 2003, 3310 . 3312 [RFC3630] Katz, D., Kompella, K., and D. Yeung, "Traffic Engineering 3313 (TE) Extensions to OSPF Version 2", RFC 3630, 3314 DOI 10.17487/RFC3630, September 2003, 3315 . 3317 [RFC3945] Mannie, E., Ed., "Generalized Multi-Protocol Label 3318 Switching (GMPLS) Architecture", RFC 3945, 3319 DOI 10.17487/RFC3945, October 2004, 3320 . 3322 [RFC4090] Pan, P., Ed., Swallow, G., Ed., and A. Atlas, Ed., "Fast 3323 Reroute Extensions to RSVP-TE for LSP Tunnels", RFC 4090, 3324 DOI 10.17487/RFC4090, May 2005, 3325 . 3327 [RFC4124] Le Faucheur, F., Ed., "Protocol Extensions for Support of 3328 Diffserv-aware MPLS Traffic Engineering", RFC 4124, 3329 DOI 10.17487/RFC4124, June 2005, 3330 . 3332 [RFC4203] Kompella, K., Ed. and Y. Rekhter, Ed., "OSPF Extensions in 3333 Support of Generalized Multi-Protocol Label Switching 3334 (GMPLS)", RFC 4203, DOI 10.17487/RFC4203, October 2005, 3335 . 3337 [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A 3338 Border Gateway Protocol 4 (BGP-4)", RFC 4271, 3339 DOI 10.17487/RFC4271, January 2006, 3340 . 3342 [RFC4594] Babiarz, J., Chan, K., and F. Baker, "Configuration 3343 Guidelines for DiffServ Service Classes", RFC 4594, 3344 DOI 10.17487/RFC4594, August 2006, 3345 . 3347 [RFC4655] Farrel, A., Vasseur, J., and J. Ash, "A Path Computation 3348 Element (PCE)-Based Architecture", RFC 4655, 3349 DOI 10.17487/RFC4655, August 2006, 3350 . 3352 [RFC4872] Lang, J., Ed., Rekhter, Y., Ed., and D. Papadimitriou, 3353 Ed., "RSVP-TE Extensions in Support of End-to-End 3354 Generalized Multi-Protocol Label Switching (GMPLS) 3355 Recovery", RFC 4872, DOI 10.17487/RFC4872, May 2007, 3356 . 3358 [RFC4873] Berger, L., Bryskin, I., Papadimitriou, D., and A. Farrel, 3359 "GMPLS Segment Recovery", RFC 4873, DOI 10.17487/RFC4873, 3360 May 2007, . 3362 [RFC5250] Berger, L., Bryskin, I., Zinin, A., and R. Coltun, "The 3363 OSPF Opaque LSA Option", RFC 5250, DOI 10.17487/RFC5250, 3364 July 2008, . 3366 [RFC5305] Li, T. and H. Smit, "IS-IS Extensions for Traffic 3367 Engineering", RFC 5305, DOI 10.17487/RFC5305, October 3368 2008, . 3370 [RFC5329] Ishiguro, K., Manral, V., Davey, A., and A. Lindem, Ed., 3371 "Traffic Engineering Extensions to OSPF Version 3", 3372 RFC 5329, DOI 10.17487/RFC5329, September 2008, 3373 . 3375 [RFC5331] Aggarwal, R., Rekhter, Y., and E. Rosen, "MPLS Upstream 3376 Label Assignment and Context-Specific Label Space", 3377 RFC 5331, DOI 10.17487/RFC5331, August 2008, 3378 . 3380 [RFC5394] Bryskin, I., Papadimitriou, D., Berger, L., and J. Ash, 3381 "Policy-Enabled Path Computation Framework", RFC 5394, 3382 DOI 10.17487/RFC5394, December 2008, 3383 . 3385 [RFC5440] Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation 3386 Element (PCE) Communication Protocol (PCEP)", RFC 5440, 3387 DOI 10.17487/RFC5440, March 2009, 3388 . 3390 [RFC5541] Le Roux, JL., Vasseur, JP., and Y. Lee, "Encoding of 3391 Objective Functions in the Path Computation Element 3392 Communication Protocol (PCEP)", RFC 5541, 3393 DOI 10.17487/RFC5541, June 2009, 3394 . 3396 [RFC5557] Lee, Y., Le Roux, JL., King, D., and E. Oki, "Path 3397 Computation Element Communication Protocol (PCEP) 3398 Requirements and Protocol Extensions in Support of Global 3399 Concurrent Optimization", RFC 5557, DOI 10.17487/RFC5557, 3400 July 2009, . 3402 [RFC5664] Halevy, B., Welch, B., and J. Zelenka, "Object-Based 3403 Parallel NFS (pNFS) Operations", RFC 5664, 3404 DOI 10.17487/RFC5664, January 2010, 3405 . 3407 [RFC5693] Seedorf, J. and E. Burger, "Application-Layer Traffic 3408 Optimization (ALTO) Problem Statement", RFC 5693, 3409 DOI 10.17487/RFC5693, October 2009, 3410 . 3412 [RFC6107] Shiomoto, K., Ed. and A. Farrel, Ed., "Procedures for 3413 Dynamically Signaled Hierarchical Label Switched Paths", 3414 RFC 6107, DOI 10.17487/RFC6107, February 2011, 3415 . 3417 [RFC6119] Harrison, J., Berger, J., and M. Bartlett, "IPv6 Traffic 3418 Engineering in IS-IS", RFC 6119, DOI 10.17487/RFC6119, 3419 February 2011, . 3421 [RFC6241] Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed., 3422 and A. Bierman, Ed., "Network Configuration Protocol 3423 (NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011, 3424 . 3426 [RFC6372] Sprecher, N., Ed. and A. Farrel, Ed., "MPLS Transport 3427 Profile (MPLS-TP) Survivability Framework", RFC 6372, 3428 DOI 10.17487/RFC6372, September 2011, 3429 . 3431 [RFC6374] Frost, D. and S. Bryant, "Packet Loss and Delay 3432 Measurement for MPLS Networks", RFC 6374, 3433 DOI 10.17487/RFC6374, September 2011, 3434 . 3436 [RFC6805] King, D., Ed. and A. Farrel, Ed., "The Application of the 3437 Path Computation Element Architecture to the Determination 3438 of a Sequence of Domains in MPLS and GMPLS", RFC 6805, 3439 DOI 10.17487/RFC6805, November 2012, 3440 . 3442 [RFC7149] Boucadair, M. and C. Jacquenet, "Software-Defined 3443 Networking: A Perspective from within a Service Provider 3444 Environment", RFC 7149, DOI 10.17487/RFC7149, March 2014, 3445 . 3447 [RFC7285] Alimi, R., Ed., Penno, R., Ed., Yang, Y., Ed., Kiesel, S., 3448 Previdi, S., Roome, W., Shalunov, S., and R. Woundy, 3449 "Application-Layer Traffic Optimization (ALTO) Protocol", 3450 RFC 7285, DOI 10.17487/RFC7285, September 2014, 3451 . 3453 [RFC7390] Rahman, A., Ed. and E. Dijk, Ed., "Group Communication for 3454 the Constrained Application Protocol (CoAP)", RFC 7390, 3455 DOI 10.17487/RFC7390, October 2014, 3456 . 3458 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 3459 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 3460 Defined Networking (SDN): Layers and Architecture 3461 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 3462 2015, . 3464 [RFC7471] Giacalone, S., Ward, D., Drake, J., Atlas, A., and S. 3465 Previdi, "OSPF Traffic Engineering (TE) Metric 3466 Extensions", RFC 7471, DOI 10.17487/RFC7471, March 2015, 3467 . 3469 [RFC7491] King, D. and A. Farrel, "A PCE-Based Architecture for 3470 Application-Based Network Operations", RFC 7491, 3471 DOI 10.17487/RFC7491, March 2015, 3472 . 3474 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 3475 Chaining (SFC) Architecture", RFC 7665, 3476 DOI 10.17487/RFC7665, October 2015, 3477 . 3479 [RFC7679] Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, 3480 Ed., "A One-Way Delay Metric for IP Performance Metrics 3481 (IPPM)", STD 81, RFC 7679, DOI 10.17487/RFC7679, January 3482 2016, . 3484 [RFC7680] Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, 3485 Ed., "A One-Way Loss Metric for IP Performance Metrics 3486 (IPPM)", STD 82, RFC 7680, DOI 10.17487/RFC7680, January 3487 2016, . 3489 [RFC7752] Gredler, H., Ed., Medved, J., Previdi, S., Farrel, A., and 3490 S. Ray, "North-Bound Distribution of Link-State and 3491 Traffic Engineering (TE) Information Using BGP", RFC 7752, 3492 DOI 10.17487/RFC7752, March 2016, 3493 . 3495 [RFC7923] Voit, E., Clemm, A., and A. Gonzalez Prieto, "Requirements 3496 for Subscription to YANG Datastores", RFC 7923, 3497 DOI 10.17487/RFC7923, June 2016, 3498 . 3500 [RFC7926] Farrel, A., Ed., Drake, J., Bitar, N., Swallow, G., 3501 Ceccarelli, D., and X. Zhang, "Problem Statement and 3502 Architecture for Information Exchange between 3503 Interconnected Traffic-Engineered Networks", BCP 206, 3504 RFC 7926, DOI 10.17487/RFC7926, July 2016, 3505 . 3507 [RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", 3508 RFC 7950, DOI 10.17487/RFC7950, August 2016, 3509 . 3511 [RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF 3512 Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017, 3513 . 3515 [RFC8051] Zhang, X., Ed. and I. Minei, Ed., "Applicability of a 3516 Stateful Path Computation Element (PCE)", RFC 8051, 3517 DOI 10.17487/RFC8051, January 2017, 3518 . 3520 [RFC8189] Randriamasy, S., Roome, W., and N. Schwan, "Multi-Cost 3521 Application-Layer Traffic Optimization (ALTO)", RFC 8189, 3522 DOI 10.17487/RFC8189, October 2017, 3523 . 3525 [RFC8231] Crabbe, E., Minei, I., Medved, J., and R. Varga, "Path 3526 Computation Element Communication Protocol (PCEP) 3527 Extensions for Stateful PCE", RFC 8231, 3528 DOI 10.17487/RFC8231, September 2017, 3529 . 3531 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 3532 Interchange Format", STD 90, RFC 8259, 3533 DOI 10.17487/RFC8259, December 2017, 3534 . 3536 [RFC8281] Crabbe, E., Minei, I., Sivabalan, S., and R. Varga, "Path 3537 Computation Element Communication Protocol (PCEP) 3538 Extensions for PCE-Initiated LSP Setup in a Stateful PCE 3539 Model", RFC 8281, DOI 10.17487/RFC8281, December 2017, 3540 . 3542 [RFC8283] Farrel, A., Ed., Zhao, Q., Ed., Li, Z., and C. Zhou, "An 3543 Architecture for Use of PCE and the PCE Communication 3544 Protocol (PCEP) in a Network with Central Control", 3545 RFC 8283, DOI 10.17487/RFC8283, December 2017, 3546 . 3548 [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 3549 Decraene, B., Litkowski, S., and R. Shakir, "Segment 3550 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 3551 July 2018, . 3553 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for 3554 Abstraction and Control of TE Networks (ACTN)", RFC 8453, 3555 DOI 10.17487/RFC8453, August 2018, 3556 . 3558 [RFC8570] Ginsberg, L., Ed., Previdi, S., Ed., Giacalone, S., Ward, 3559 D., Drake, J., and Q. Wu, "IS-IS Traffic Engineering (TE) 3560 Metric Extensions", RFC 8570, DOI 10.17487/RFC8570, March 3561 2019, . 3563 [RFC8571] Ginsberg, L., Ed., Previdi, S., Wu, Q., Tantsura, J., and 3564 C. Filsfils, "BGP - Link State (BGP-LS) Advertisement of 3565 IGP Traffic Engineering Performance Metric Extensions", 3566 RFC 8571, DOI 10.17487/RFC8571, March 2019, 3567 . 3569 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 3570 "Deterministic Networking Architecture", RFC 8655, 3571 DOI 10.17487/RFC8655, October 2019, 3572 . 3574 [RFC8664] Sivabalan, S., Filsfils, C., Tantsura, J., Henderickx, W., 3575 and J. Hardwick, "Path Computation Element Communication 3576 Protocol (PCEP) Extensions for Segment Routing", RFC 8664, 3577 DOI 10.17487/RFC8664, December 2019, 3578 . 3580 [RFC8685] Zhang, F., Zhao, Q., Gonzalez de Dios, O., Casellas, R., 3581 and D. King, "Path Computation Element Communication 3582 Protocol (PCEP) Extensions for the Hierarchical Path 3583 Computation Element (H-PCE) Architecture", RFC 8685, 3584 DOI 10.17487/RFC8685, December 2019, 3585 . 3587 [RFC8795] Liu, X., Bryskin, I., Beeram, V., Saad, T., Shah, H., and 3588 O. Gonzalez de Dios, "YANG Data Model for Traffic 3589 Engineering (TE) Topologies", RFC 8795, 3590 DOI 10.17487/RFC8795, August 2020, 3591 . 3593 [RFC8896] Randriamasy, S., Yang, R., Wu, Q., Deng, L., and N. 3594 Schwan, "Application-Layer Traffic Optimization (ALTO) 3595 Cost Calendar", RFC 8896, DOI 10.17487/RFC8896, November 3596 2020, . 3598 [RFC8938] Varga, B., Ed., Farkas, J., Berger, L., Malis, A., and S. 3599 Bryant, "Deterministic Networking (DetNet) Data Plane 3600 Framework", RFC 8938, DOI 10.17487/RFC8938, November 2020, 3601 . 3603 [RFC8955] Loibl, C., Hares, S., Raszuk, R., McPherson, D., and M. 3604 Bacher, "Dissemination of Flow Specification Rules", 3605 RFC 8955, DOI 10.17487/RFC8955, December 2020, 3606 . 3608 [RR94] Rodrigues, M. and K. Ramakrishnan, "Optimal Routing in 3609 Shortest Path Networks", Proceedings ITS'94, Rio de 3610 Janeiro, Brazil, 1994. 3612 [SLDC98] Suter, B., Lakshman, T., Stiliadis, D., and A. Choudhury, 3613 "Design Considerations for Supporting TCP with Per-flow 3614 Queueing", Proceedings INFOCOM'98, p. 299-306, 1998. 3616 [WANG] Wang, Y., Wang, Z., and L. Zhang, "Internet traffic 3617 engineering without full mesh overlaying", 3618 Proceedings INFOCOM'2001, April 2001. 3620 [XIAO] Xiao, X., Hannan, A., Bailey, B., and L. Ni, "Traffic 3621 Engineering with MPLS in the Internet", Article IEEE 3622 Network Magazine, March 2000. 3624 [YARE95] Yang, C. and A. Reddy, "A Taxonomy for Congestion Control 3625 Algorithms in Packet Switching Networks", Article IEEE 3626 Network Magazine, p. 34-45, 1995. 3628 Appendix A. Historic Overview 3630 A.1. Traffic Engineering in Classical Telephone Networks 3632 This subsection presents a brief overview of traffic engineering in 3633 telephone networks which often relates to the way user traffic is 3634 steered from an originating node to the terminating node. This 3635 subsection presents a brief overview of this topic. A detailed 3636 description of the various routing strategies applied in telephone 3637 networks is included in the book by G. Ash [ASH2]. 3639 The early telephone network relied on static hierarchical routing, 3640 whereby routing patterns remained fixed independent of the state of 3641 the network or time of day. The hierarchy was intended to 3642 accommodate overflow traffic, improve network reliability via 3643 alternate routes, and prevent call looping by employing strict 3644 hierarchical rules. The network was typically over-provisioned since 3645 a given fixed route had to be dimensioned so that it could carry user 3646 traffic during a busy hour of any busy day. Hierarchical routing in 3647 the telephony network was found to be too rigid upon the advent of 3648 digital switches and stored program control which were able to manage 3649 more complicated traffic engineering rules. 3651 Dynamic routing was introduced to alleviate the routing inflexibility 3652 in the static hierarchical routing so that the network would operate 3653 more efficiently. This resulted in significant economic gains 3654 [HUSS87]. Dynamic routing typically reduces the overall loss 3655 probability by 10 to 20 percent (compared to static hierarchical 3656 routing). Dynamic routing can also improve network resilience by 3657 recalculating routes on a per-call basis and periodically updating 3658 routes. 3660 There are three main types of dynamic routing in the telephone 3661 network. They are time-dependent routing, state-dependent routing 3662 (SDR), and event dependent routing (EDR). 3664 In time-dependent routing, regular variations in traffic loads (such 3665 as time of day or day of week) are exploited in pre-planned routing 3666 tables. In state-dependent routing, routing tables are updated 3667 online according to the current state of the network (e.g., traffic 3668 demand, utilization, etc.). In event dependent routing, routing 3669 changes are triggers by events (such as call setups encountering 3670 congested or blocked links) whereupon new paths are searched out 3671 using learning models. EDR methods are real-time adaptive, but they 3672 do not require global state information as does SDR. Examples of EDR 3673 schemes include the dynamic alternate routing (DAR) from BT, the 3674 state-and-time dependent routing (STR) from NTT, and the success-to- 3675 the-top (STT) routing from AT&T. 3677 Dynamic non-hierarchical routing (DNHR) is an example of dynamic 3678 routing that was introduced in the AT&T toll network in the 1980's to 3679 respond to time-dependent information such as regular load variations 3680 as a function of time. Time-dependent information in terms of load 3681 may be divided into three timescales: hourly, weekly, and yearly. 3682 Correspondingly, three algorithms are defined to pre-plan the routing 3683 tables. The network design algorithm operates over a year-long 3684 interval while the demand servicing algorithm operates on a weekly 3685 basis to fine tune link sizes and routing tables to correct forecast 3686 errors on the yearly basis. At the smallest timescale, the routing 3687 algorithm is used to make limited adjustments based on daily traffic 3688 variations. Network design and demand servicing are computed using 3689 offline calculations. Typically, the calculations require extensive 3690 searches on possible routes. On the other hand, routing may need 3691 online calculations to handle crankback. DNHR adopts a "two-link" 3692 approach whereby a path can consist of two links at most. The 3693 routing algorithm presents an ordered list of route choices between 3694 an originating switch and a terminating switch. If a call overflows, 3695 a via switch (a tandem exchange between the originating switch and 3696 the terminating switch) would send a crankback signal to the 3697 originating switch. This switch would then select the next route, 3698 and so on, until there are no alternative routes available in which 3699 the call is blocked. 3701 A.2. Evolution of Traffic Engineering in Packet Networks 3703 This subsection reviews related prior work that was intended to 3704 improve the performance of data networks. Indeed, optimization of 3705 the performance of data networks started in the early days of the 3706 ARPANET. Other early commercial networks such as SNA also recognized 3707 the importance of performance optimization and service 3708 differentiation. 3710 In terms of traffic management, the Internet has been a best effort 3711 service environment until recently. In particular, very limited 3712 traffic management capabilities existed in IP networks to provide 3713 differentiated queue management and scheduling services to packets 3714 belonging to different classes. 3716 In terms of routing control, the Internet has employed distributed 3717 protocols for intra-domain routing. These protocols are highly 3718 scalable and resilient. However, they are based on simple algorithms 3719 for path selection which have very limited functionality to allow 3720 flexible control of the path selection process. 3722 In the following subsections, the evolution of practical traffic 3723 engineering mechanisms in IP networks and its predecessors are 3724 reviewed. 3726 A.2.1. Adaptive Routing in the ARPANET 3728 The early ARPANET recognized the importance of adaptive routing where 3729 routing decisions were based on the current state of the network 3730 [MCQ80]. Early minimum delay routing approaches forwarded each 3731 packet to its destination along a path for which the total estimated 3732 transit time was the smallest. Each node maintained a table of 3733 network delays, representing the estimated delay that a packet would 3734 experience along a given path toward its destination. The minimum 3735 delay table was periodically transmitted by a node to its neighbors. 3736 The shortest path, in terms of hop count, was also propagated to give 3737 the connectivity information. 3739 One drawback to this approach is that dynamic link metrics tend to 3740 create "traffic magnets" causing congestion to be shifted from one 3741 location of a network to another location, resulting in oscillation 3742 and network instability. 3744 A.2.2. Dynamic Routing in the Internet 3746 The Internet evolved from the ARPANET and adopted dynamic routing 3747 algorithms with distributed control to determine the paths that 3748 packets should take en-route to their destinations. The routing 3749 algorithms are adaptations of shortest path algorithms where costs 3750 are based on link metrics. The link metric can be based on static or 3751 dynamic quantities. The link metric based on static quantities may 3752 be assigned administratively according to local criteria. The link 3753 metric based on dynamic quantities may be a function of a network 3754 congestion measure such as delay or packet loss. 3756 It was apparent early that static link metric assignment was 3757 inadequate because it can easily lead to unfavorable scenarios in 3758 which some links become congested while others remain lightly loaded. 3759 One of the many reasons for the inadequacy of static link metrics is 3760 that link metric assignment was often done without considering the 3761 traffic matrix in the network. Also, the routing protocols did not 3762 take traffic attributes and capacity constraints into account when 3763 making routing decisions. This results in traffic concentration 3764 being localized in subsets of the network infrastructure and 3765 potentially causing congestion. Even if link metrics are assigned in 3766 accordance with the traffic matrix, unbalanced loads in the network 3767 can still occur due to a number factors including: 3769 o Resources may not be deployed in the most optimal locations from a 3770 routing perspective. 3772 o Forecasting errors in traffic volume and/or traffic distribution. 3774 o Dynamics in traffic matrix due to the temporal nature of traffic 3775 patterns, BGP policy change from peers, etc. 3777 The inadequacy of the legacy Internet interior gateway routing system 3778 is one of the factors motivating the interest in path oriented 3779 technology with explicit routing and constraint-based routing 3780 capability such as MPLS. 3782 A.2.3. ToS Routing 3784 Type-of-Service (ToS) routing involves different routes going to the 3785 same destination with selection dependent upon the ToS field of an IP 3786 packet [RFC2474]. The ToS classes may be classified as low delay and 3787 high throughput. Each link is associated with multiple link costs 3788 and each link cost is used to compute routes for a particular ToS. A 3789 separate shortest path tree is computed for each ToS. The shortest 3790 path algorithm must be run for each ToS resulting in very expensive 3791 computation. Classical ToS-based routing is now outdated as the IP 3792 header field has been replaced by a Diffserv field. Effective 3793 traffic engineering is difficult to perform in classical ToS-based 3794 routing because each class still relies exclusively on shortest path 3795 routing which results in localization of traffic concentration within 3796 the network. 3798 A.2.4. Equal Cost Multi-Path 3800 Equal Cost Multi-Path (ECMP) is another technique that attempts to 3801 address the deficiency in the Shortest Path First (SPF) interior 3802 gateway routing systems [RFC2328]. In the classical SPF algorithm, 3803 if two or more shortest paths exist to a given destination, the 3804 algorithm will choose one of them. The algorithm is modified 3805 slightly in ECMP so that if two or more equal cost shortest paths 3806 exist between two nodes, the traffic between the nodes is distributed 3807 among the multiple equal-cost paths. Traffic distribution across the 3808 equal-cost paths is usually performed in one of two ways: (1) packet- 3809 based in a round-robin fashion, or (2) flow-based using hashing on 3810 source and destination IP addresses and possibly other fields of the 3811 IP header. The first approach can easily cause out- of-order packets 3812 while the second approach is dependent upon the number and 3813 distribution of flows. Flow-based load sharing may be unpredictable 3814 in an enterprise network where the number of flows is relatively 3815 small and less heterogeneous (for example, hashing may not be 3816 uniform), but it is generally effective in core public networks where 3817 the number of flows is large and heterogeneous. 3819 In ECMP, link costs are static and bandwidth constraints are not 3820 considered, so ECMP attempts to distribute the traffic as equally as 3821 possible among the equal-cost paths independent of the congestion 3822 status of each path. As a result, given two equal-cost paths, it is 3823 possible that one of the paths will be more congested than the other. 3824 Another drawback of ECMP is that load sharing cannot be achieved on 3825 multiple paths which have non-identical costs. 3827 A.2.5. Nimrod 3829 Nimrod was a routing system developed to provide heterogeneous 3830 service specific routing in the Internet, while taking multiple 3831 constraints into account [RFC1992]. Essentially, Nimrod was a link 3832 state routing protocol to support path oriented packet forwarding. 3833 It used the concept of maps to represent network connectivity and 3834 services at multiple levels of abstraction. Mechanisms allowed 3835 restriction of the distribution of routing information. 3837 Even though Nimrod did not enjoy deployment in the public Internet, a 3838 number of key concepts incorporated into the Nimrod architecture, 3839 such as explicit routing which allows selection of paths at 3840 originating nodes, are beginning to find applications in some recent 3841 constraint-based routing initiatives. 3843 A.3. Development of Internet Traffic Engineering 3845 A.3.1. Overlay Model 3847 In the overlay model, a virtual-circuit network, such as Synchronous 3848 Optical Network / Synchronous Digital Hierarchy (SONET/SDH), Optical 3849 Transport Network (OTN), or Wavelength Division Multiplexing (WDM), 3850 provides virtual-circuit connectivity between routers that are 3851 located at the edges of a virtual-circuit cloud. In this mode, two 3852 routers that are connected through a virtual circuit see a direct 3853 adjacency between themselves independent of the physical route taken 3854 by the virtual circuit through the ATM, frame relay, or WDM network. 3855 Thus, the overlay model essentially decouples the logical topology 3856 that routers see from the physical topology that the ATM, frame 3857 relay, or WDM network manages. The overlay model based on ATM or 3858 frame relay enables a network administrator or an automaton to employ 3859 traffic engineering concepts to perform path optimization by re- 3860 configuring or rearranging the virtual circuits so that a virtual 3861 circuit on a congested or sub-optimal physical link can be re-routed 3862 to a less congested or more optimal one. In the overlay model, 3863 traffic engineering is also employed to establish relationships 3864 between the traffic management parameters (e.g., Peak Cell Rate, 3865 Sustained Cell Rate, and Maximum Burst Size for ATM) of the virtual- 3866 circuit technology and the actual traffic that traverses each 3867 circuit. These relationships can be established based upon known or 3868 projected traffic profiles, and some other factors. 3870 Appendix B. Overview of Traffic Engineering Related Work in Other SDOs 3872 B.1. Overview of ITU Activities Related to Traffic Engineering 3874 This section provides an overview of prior work within the ITU-T 3875 pertaining to traffic engineering in traditional telecommunications 3876 networks. 3878 ITU-T Recommendations E.600 [ITU-E600], E.701 [ITU-E701], and E.801 3879 [ITU-E801] address traffic engineering issues in traditional 3880 telecommunications networks. Recommendation E.600 provides a 3881 vocabulary for describing traffic engineering concepts, while E.701 3882 defines reference connections, Grade of Service (GoS), and traffic 3883 parameters for ISDN. Recommendation E.701 uses the concept of a 3884 reference connection to identify representative cases of different 3885 types of connections without describing the specifics of their actual 3886 realizations by different physical means. As defined in 3887 Recommendation E.600, "a connection is an association of resources 3888 providing means for communication between two or more devices in, or 3889 attached to, a telecommunication network." Also, E.600 defines "a 3890 resource as any set of physically or conceptually identifiable 3891 entities within a telecommunication network, the use of which can be 3892 unambiguously determined" [ITU-E600]. There can be different types 3893 of connections as the number and types of resources in a connection 3894 may vary. 3896 Typically, different network segments are involved in the path of a 3897 connection. For example, a connection may be local, national, or 3898 international. The purposes of reference connections are to clarify 3899 and specify traffic performance issues at various interfaces between 3900 different network domains. Each domain may consist of one or more 3901 service provider networks. 3903 Reference connections provide a basis to define grade of service 3904 (GoS) parameters related to traffic engineering within the ITU-T 3905 framework. As defined in E.600, "GoS refers to a number of traffic 3906 engineering variables which are used to provide a measure of the 3907 adequacy of a group of resources under specified conditions." These 3908 GoS variables may be probability of loss, dial tone, delay, etc. 3909 They are essential for network internal design and operation as well 3910 as for component performance specification. 3912 GoS is different from quality of service (QoS) in the ITU framework. 3913 QoS is the performance perceivable by a telecommunication service 3914 user and expresses the user's degree of satisfaction of the service. 3915 QoS parameters focus on performance aspects observable at the service 3916 access points and network interfaces, rather than their causes within 3917 the network. GoS, on the other hand, is a set of network oriented 3918 measures which characterize the adequacy of a group of resources 3919 under specified conditions. For a network to be effective in serving 3920 its users, the values of both GoS and QoS parameters must be related, 3921 with GoS parameters typically making a major contribution to the QoS. 3923 Recommendation E.600 stipulates that a set of GoS parameters must be 3924 selected and defined on an end-to-end basis for each major service 3925 category provided by a network to assist the network provider with 3926 improving efficiency and effectiveness of the network. Based on a 3927 selected set of reference connections, suitable target values are 3928 assigned to the selected GoS parameters under normal and high load 3929 conditions. These end-to-end GoS target values are then apportioned 3930 to individual resource components of the reference connections for 3931 dimensioning purposes. 3933 Appendix C. Summary of Changes Since RFC 3272 3935 The changes to this document since RFC 3272 are substantial and not 3936 easily summarized as section-by-section changes. The material in the 3937 document has been moved around considerably, some of it removed, and 3938 new text added. 3940 The approach taken here is to list the table of content of both the 3941 previous RFC and this document saying, respectively, where the text 3942 has been place and where the text came from. 3944 C.1. RFC 3272 3946 1.0 Introduction: Edited in place in Section 1. 3948 1.1 What is Internet Traffic Engineering?: Edited in place in 3949 Section 1.1. 3951 1.2 Scope: Moved to Section 1.3. 3953 1.3 Terminology: Moved to Section 1.4 with some obsolete terms 3954 removed and a little editing. 3956 2.0 Background: Retained as Section 2 with some text removed. 3958 2.1 Context of Internet Traffic Engineering: Retained as 3959 Section 2.1. 3961 2.2 Network Context: Rewritten as Section 2.2. 3963 2.3 Problem Context: Rewritten as Section 2.3. 3965 2.3.1 Congestion and its Ramifications: Retained as 3966 Section 2.3.1. 3968 2.4 Solution Context: Edited as Section 2.4. 3970 2.4.1 Combating the Congestion Problem: Reformatted as 3971 Section 2.4.1. 3973 2.5 Implementation and Operational Context: Retained as 3974 Section 2.5. 3976 3.0 Traffic Engineering Process Model: Retained as Section 3. 3978 3.1 Components of the Traffic Engineering Process Model: Retained 3979 as Section 3.1. 3981 3.2 Measurement: Merged into Section 3.1. 3983 3.3 Modeling, Analysis, and Simulation: Merged into Section 3.1. 3985 3.4 Optimization: Merged into Section 3.1. 3987 4.0 Historical Review and Recent Developments: Retained as 3988 Section 5, but the very historic aspects moved to Appendix A. 3990 4.1 Traffic Engineering in Classical Telephone Networks: Moved to 3991 Appendix A.1. 3993 4.2 Evolution of Traffic Engineering in the Internet: Moved to Ap 3994 pendix A.2. 3996 4.2.1 Adaptive Routing in ARPANET: Moved to Appendix A.2.1. 3998 4.2.2 Dynamic Routing in the Internet: Moved to 3999 Appendix A.2.2. 4001 4.2.3 ToS Routing: Moved to Appendix A.2.3. 4003 4.2.4 Equal Cost Multi-Path: Moved to Appendix A.2.4. 4005 4.2.5 Nimrod: Moved to Appendix A.2.5. 4007 4.3 Overlay Model: Moved to Appendix A.3.1. 4009 4.4 Constraint-Based Routing: Retained as Section 5.1.1, but 4010 moved into Section 5.1. 4012 4.5 Overview of Other IETF Projects Related to Traffic 4013 Engineering: 4014 Retained as Section 5.1 with many new subsections. 4016 4.5.1 Integrated Services: Retained as Section 5.1.2. 4018 4.5.2 RSVP: Retained as Section 5.1.3 with some edits. 4020 4.5.3 Differentiated Services: Retained as Section 5.1.4. 4022 4.5.4 MPLS: Retained as Section 5.1.6. 4024 4.5.5 IP Performance Metrics: Retained as Section 5.1.8. 4026 4.5.6 Flow Measurement: Retained as Section 5.1.9 with some 4027 reformatting. 4029 4.5.7 Endpoint Congestion Management: Retained as Section 5.1.10. 4031 4.6 Overview of ITU Activities Related to Traffic Engineering: Moved 4032 to Appendix B.1. 4034 4.7 Content Distribution: Retained as Section 5.2. 4036 5.0 Taxonomy of Traffic Engineering Systems: Retained as Section 4. 4038 5.1 Time-Dependent Versus State-Dependent: Retained as 4039 Section 4.1. 4041 5.2 Offline Versus Online: Retained as Section 4.2. 4043 5.3 Centralized Versus Distributed: Retained as Section 4.3 with 4044 additions. 4046 5.4 Local Versus Global: Retained as Section 4.4. 4048 5.5 Prescriptive Versus Descriptive: Retained as Section 4.5 with 4049 additions. 4051 5.6 Open-Loop Versus Closed-Loop: Retained as Section 4.6. 4053 5.7 Tactical vs Strategic: Retained as Section 4.7. 4055 6.0 Recommendations for Internet Traffic Engineering: Retained as 4056 Section 6. 4058 6.1 Generic Non-functional Recommendations: Retained as 4059 Section 6.1. 4061 6.2 Routing Recommendations: Retained as Section 6.2 with edits. 4063 6.3 Traffic Mapping Recommendations: Retained as Section 6.3. 4065 6.4 Measurement Recommendations: Retained as Section 6.4. 4067 6.5 Network Survivability: Retained as Section 6.6. 4069 6.5.1 Survivability in MPLS Based Networks: Retained as 4070 Section 6.6.1. 4072 6.5.2 Protection Option: Retained as Section 6.6.2. 4074 6.6 Traffic Engineering in Diffserv Environments: Retained as 4075 Section 6.8 with edits. 4077 6.7 Network Controllability: Retained as Section 6.9. 4079 7.0 Inter-Domain Considerations: Retained as Section 7. 4081 8.0 Overview of Contemporary TE Practices in Operational IP Networks: 4083 Retained as Section 8. 4085 9.0 Conclusion: Removed. 4087 10.0 Security Considerations: Retained as Section 9 with 4088 considerable new text. 4090 C.2. This Document 4092 o Section 1: Based on Section 1 of RFC 3272. 4094 * Section 1.1: Based on Section 1.1 of RFC 3272. 4096 * Section 1.2: New for this document. 4098 * Section 1.3: Based on Section 1.2 of RFC 3272. 4100 * Section 1.4: Based on Section 1.3 of RFC 3272. 4102 o Section 2: Based on Section 2. of RFC 3272. 4104 * Section 2.1: Based on Section 2.1 of RFC 3272. 4106 * Section 2.2: Based on Section 2.2 of RFC 3272. 4108 * Section 2.3: Based on Section 2.3 of RFC 3272. 4110 + Section 2.3.1: Based on Section 2.3.1 of RFC 3272. 4112 * Section 2.4: Based on Section 2.4 of RFC 3272. 4114 + Section 2.4.1: Based on Section 2.4.1 of RFC 327 4116 * Section 2.5: Based on Section 2.5 of RFC 3272. 4118 o Section 3: Based on Section 3 of RFC 3272. 4120 * Section 3.1: Based on Sections 3.1, 3.2, 3.3, and 3.4 of RFC 4121 3272. 4123 o Section 4: Based on Section 5 of RFC 3272. 4125 * Section 4.1: Based on Section 5.1 of RFC 3272. 4127 * Section 4.2: Based on Section 5.2 of RFC 3272. 4129 * Section 4.3: Based on Section 5.3 of RFC 3272. 4131 + Section 4.3.1: New for this document. 4133 + Section 4.3.2: New for this document. 4135 * Section 4.4: Based on Section 5.4 of RFC 3272. 4137 * Section 4.5: Based on Section 5.5 of RFC 3272. 4139 + Section 4.5.1: New for this document. 4141 * Section 4.6: Based on Section 5.6 of RFC 3272. 4143 * Section 4.7: Based on Section 5.7 of RFC 3272. 4145 o Section 5: Based on Section 4 of RFC 3272. 4147 * Section 5.1: Based on Section 4.5 of RFC 3272. 4149 + Section 5.1.1: Based on Section 4.4 of RFC 3272. 4151 - Section 5.1.1.1: New for this document. 4153 + Section 5.1.2: Based on Section 4.5.1 of RFC 3272. 4155 + Section 5.1.3: Based on Section 4.5.2 of RFC 3272. 4157 + Section 5.1.4: Based on Section 4.5.3 of RFC 3272. 4159 + Section 5.1.5: New for this document. 4161 + Section 5.1.6: Based on Section 4.5.4 of RFC 3272. 4163 + Section 5.1.7: New for this document. 4165 + Section 5.1.8: Based on Section 4.5.5 of RFC 3272. 4167 + Section 5.1.9: Based on Section 4.5.6 of RFC 3272. 4169 + Section 5.1.10: Based on Section 4.5.7 of RFC 3272. 4171 + Section 5.1.11: New for this document. 4173 + Section 5.1.12: New for this document. 4175 + Section 5.1.13: New for this document. 4177 + Section 5.1.14: New for this document. 4179 + Section 5.1.15: New for this document. 4181 + Section 5.1.16: New for this document. 4183 + Section 5.1.17: New for this document. 4185 + Section 5.1.18: New for this document. 4187 + Section 5.1.19: New for this document. 4189 + Section 5.1.20: New for this document. 4191 + Section 5.1.21: New for this document. 4193 * Section 5.2: Based on Section 4.7 of RFC 3272. 4195 o Section 6: Based on Section 6 of RFC 3272. 4197 * Section 6.1: Based on Section 6.1 of RFC 3272. 4199 * Section 6.2: Based on Section 6.2 of RFC 3272. 4201 * Section 6.3: Based on Section 6.3 of RFC 3272. 4203 * Section 6.4: Based on Section 6.4 of RFC 3272. 4205 * Section 6.5: New for this document. 4207 * Section 6.6: Based on Section 6.5 of RFC 3272. 4209 + Section 6.6.1: Based on Section 6.5.1 of RFC 3272. 4211 + Section 6.6.2: Based on Section 6.5.2 of RFC 3272. 4213 * Section 6.7: New for this document. 4215 * Section 6.8: Based on Section 6.6. of RFC 3272. 4217 * Section 6.9: Based on Section 6.7 of RFC 3272. 4219 o Section 7: Based on Section 7 of RFC 3272. 4221 o Section 8: Based on Section 8 of RFC 3272. 4223 o Section 9: Based on Section 10 of RFC 3272. 4225 o Appendix A: New for this document. 4227 * Appendix A.1: Based on Section 4.1 of RFC 3272. 4229 * Appendix A.2: Based on Section 4.2 of RFC 3272. 4231 + Appendix A.2.1: Based on Section 4.2.1 of RFC 3272. 4233 + Appendix A.2.2: Based on Section 4.2.2 of RFC 3272. 4235 + Appendix A.2.3: Based on Section 4.2.3 of RFC 3272. 4237 + Appendix A.2.4: Based on Section 4.2.4 of RFC 3272. 4239 + Appendix A.2.5: Based on Section 4.2.5 of RFC 3272. 4241 * Appendix A.3: New for this document. 4243 + Appendix A.3.1: Based on Section 4.3 of RFC 3272. 4245 o Appendix B: New for this document. 4247 * Appendix B.1: Based on Section 4.7 of RFC 3272. 4249 Author's Address 4251 Adrian Farrel (editor) 4252 Old Dog Consulting 4254 Email: adrian@olddog.co.uk