idnits 2.17.1 draft-ietf-teas-rfc3272bis-11.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (April 7, 2021) is 1115 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-28) exists of draft-ietf-alto-performance-metrics-14 == Outdated reference: A later version (-21) exists of draft-ietf-bess-evpn-unequal-lb-07 == Outdated reference: A later version (-07) exists of draft-ietf-detnet-ip-over-tsn-05 == Outdated reference: A later version (-26) exists of draft-ietf-idr-segment-routing-te-policy-11 == Outdated reference: A later version (-26) exists of draft-ietf-lsr-flex-algo-13 == Outdated reference: A later version (-17) exists of draft-ietf-lsr-ip-flexalgo-00 == Outdated reference: A later version (-22) exists of draft-ietf-spring-segment-routing-policy-09 == Outdated reference: A later version (-17) exists of draft-ietf-teas-enhanced-vpn-06 == Outdated reference: A later version (-09) exists of draft-irtf-nmrg-ibn-concepts-definitions-02 -- Obsolete informational reference (is this intentional?): RFC 3272 (Obsoleted by RFC 9522) -- Obsolete informational reference (is this intentional?): RFC 7752 (Obsoleted by RFC 9552) Summary: 0 errors (**), 0 flaws (~~), 10 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TEAS Working Group A. Farrel, Ed. 3 Internet-Draft Old Dog Consulting 4 Obsoletes: 3272 (if approved) April 7, 2021 5 Intended status: Informational 6 Expires: October 9, 2021 8 Overview and Principles of Internet Traffic Engineering 9 draft-ietf-teas-rfc3272bis-11 11 Abstract 13 This document describes the principles of traffic engineering (TE) in 14 the Internet. The document is intended to promote better 15 understanding of the issues surrounding traffic engineering in IP 16 networks and the networks that support IP networking, and to provide 17 a common basis for the development of traffic engineering 18 capabilities for the Internet. The principles, architectures, and 19 methodologies for performance evaluation and performance optimization 20 of operational networks are also discussed. 22 This work was first published as RFC 3272 in May 2002. This document 23 obsoletes RFC 3272 by making a complete update to bring the text in 24 line with best current practices for Internet traffic engineering and 25 to include references to the latest relevant work in the IETF. 27 Status of This Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at https://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on October 9, 2021. 44 Copyright Notice 46 Copyright (c) 2021 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents 51 (https://trustee.ietf.org/license-info) in effect on the date of 52 publication of this document. Please review these documents 53 carefully, as they describe your rights and restrictions with respect 54 to this document. Code Components extracted from this document must 55 include Simplified BSD License text as described in Section 4.e of 56 the Trust Legal Provisions and are provided without warranty as 57 described in the Simplified BSD License. 59 Table of Contents 61 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 62 1.1. What is Internet Traffic Engineering? . . . . . . . . . . 4 63 1.2. Components of Traffic Engineering . . . . . . . . . . . . 6 64 1.3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 8 65 1.4. Terminology . . . . . . . . . . . . . . . . . . . . . . . 8 66 2. Background . . . . . . . . . . . . . . . . . . . . . . . . . 11 67 2.1. Context of Internet Traffic Engineering . . . . . . . . . 11 68 2.2. Network Domain Context . . . . . . . . . . . . . . . . . 12 69 2.3. Problem Context . . . . . . . . . . . . . . . . . . . . . 14 70 2.3.1. Congestion and its Ramifications . . . . . . . . . . 15 71 2.4. Solution Context . . . . . . . . . . . . . . . . . . . . 16 72 2.4.1. Combating the Congestion Problem . . . . . . . . . . 18 73 2.5. Implementation and Operational Context . . . . . . . . . 21 74 3. Traffic Engineering Process Models . . . . . . . . . . . . . 21 75 3.1. Components of the Traffic Engineering Process Model . . . 22 76 4. Review of TE Techniques . . . . . . . . . . . . . . . . . . . 22 77 4.1. Overview of IETF Projects Related to Traffic Engineering 23 78 4.1.1. Constraint-Based Routing . . . . . . . . . . . . . . 23 79 4.1.2. Integrated Services . . . . . . . . . . . . . . . . . 25 80 4.1.3. RSVP . . . . . . . . . . . . . . . . . . . . . . . . 25 81 4.1.4. Differentiated Services . . . . . . . . . . . . . . . 26 82 4.1.5. QUIC . . . . . . . . . . . . . . . . . . . . . . . . 27 83 4.1.6. Multiprotocol Label Switching (MPLS) . . . . . . . . 28 84 4.1.7. Generalized MPLS (GMPLS) . . . . . . . . . . . . . . 29 85 4.1.8. IP Performance Metrics . . . . . . . . . . . . . . . 29 86 4.1.9. Flow Measurement . . . . . . . . . . . . . . . . . . 30 87 4.1.10. Endpoint Congestion Management . . . . . . . . . . . 30 88 4.1.11. TE Extensions to the IGPs . . . . . . . . . . . . . . 31 89 4.1.12. Link-State BGP . . . . . . . . . . . . . . . . . . . 31 90 4.1.13. Path Computation Element . . . . . . . . . . . . . . 32 91 4.1.14. Multi-Layer Traffic Engineering . . . . . . . . . . . 32 92 4.1.15. Application-Layer Traffic Optimization . . . . . . . 33 93 4.1.16. Segment Routing with MPLS Encapsulation (SR-MPLS) . . 34 94 4.1.17. Network Virtualization and Abstraction . . . . . . . 36 95 4.1.18. Network Slicing . . . . . . . . . . . . . . . . . . . 37 96 4.1.19. Deterministic Networking . . . . . . . . . . . . . . 38 97 4.1.20. Network TE State Definition and Presentation . . . . 38 98 4.1.21. System Management and Control Interfaces . . . . . . 39 99 4.2. Content Distribution . . . . . . . . . . . . . . . . . . 39 100 5. Taxonomy of Traffic Engineering Systems . . . . . . . . . . . 40 101 5.1. Time-Dependent Versus State-Dependent Versus Event- 102 Dependent . . . . . . . . . . . . . . . . . . . . . . . . 40 103 5.2. Offline Versus Online . . . . . . . . . . . . . . . . . . 42 104 5.3. Centralized Versus Distributed . . . . . . . . . . . . . 42 105 5.3.1. Hybrid Systems . . . . . . . . . . . . . . . . . . . 42 106 5.3.2. Considerations for Software Defined Networking . . . 43 107 5.4. Local Versus Global . . . . . . . . . . . . . . . . . . . 44 108 5.5. Prescriptive Versus Descriptive . . . . . . . . . . . . . 44 109 5.5.1. Intent-Based Networking . . . . . . . . . . . . . . . 44 110 5.6. Open-Loop Versus Closed-Loop . . . . . . . . . . . . . . 45 111 5.7. Tactical versus Strategic . . . . . . . . . . . . . . . . 46 112 6. Recommendations for Internet Traffic Engineering . . . . . . 46 113 6.1. Generic Non-functional Recommendations . . . . . . . . . 46 114 6.2. Routing Recommendations . . . . . . . . . . . . . . . . . 48 115 6.3. Traffic Mapping Recommendations . . . . . . . . . . . . . 51 116 6.4. Measurement Recommendations . . . . . . . . . . . . . . . 51 117 6.5. Network Survivability . . . . . . . . . . . . . . . . . . 52 118 6.5.1. Survivability in MPLS Based Networks . . . . . . . . 54 119 6.5.2. Protection Options . . . . . . . . . . . . . . . . . 55 120 6.6. Traffic Engineering in Diffserv Environments . . . . . . 56 121 6.7. Network Controllability . . . . . . . . . . . . . . . . . 58 122 7. Inter-Domain Considerations . . . . . . . . . . . . . . . . . 58 123 8. Overview of Contemporary TE Practices in Operational IP 124 Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 60 125 9. Security Considerations . . . . . . . . . . . . . . . . . . . 63 126 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 63 127 11. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 63 128 12. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 65 129 13. Informative References . . . . . . . . . . . . . . . . . . . 66 130 Appendix A. Historic Overview . . . . . . . . . . . . . . . . . 78 131 A.1. Traffic Engineering in Classical Telephone Networks . . . 78 132 A.2. Evolution of Traffic Engineering in Packet Networks . . . 80 133 A.2.1. Adaptive Routing in the ARPANET . . . . . . . . . . . 80 134 A.2.2. Dynamic Routing in the Internet . . . . . . . . . . . 81 135 A.2.3. ToS Routing . . . . . . . . . . . . . . . . . . . . . 82 136 A.2.4. Equal Cost Multi-Path . . . . . . . . . . . . . . . . 82 137 A.2.5. Nimrod . . . . . . . . . . . . . . . . . . . . . . . 83 138 A.3. Development of Internet Traffic Engineering . . . . . . . 83 139 A.3.1. Overlay Model . . . . . . . . . . . . . . . . . . . . 83 140 Appendix B. Overview of Traffic Engineering Related Work in 141 Other SDOs . . . . . . . . . . . . . . . . . . . . . 83 142 B.1. Overview of ITU Activities Related to Traffic Engineering 84 143 Appendix C. Summary of Changes Since RFC 3272 . . . . . . . . . 85 144 C.1. RFC 3272 . . . . . . . . . . . . . . . . . . . . . . . . 85 145 C.2. This Document . . . . . . . . . . . . . . . . . . . . . . 88 146 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 91 148 1. Introduction 150 This document describes the principles of Internet traffic 151 engineering (TE). The objective of the document is to articulate the 152 general issues and principles for Internet traffic engineering, and 153 where appropriate to provide recommendations, guidelines, and options 154 for the development of online and offline Internet traffic 155 engineering capabilities and support systems. 157 This document provides a terminology and taxonomy for describing and 158 understanding common Internet traffic engineering concepts. 160 Even though Internet traffic engineering is most effective when 161 applied end-to-end, the focus of this document is traffic engineering 162 within a given domain (such as an autonomous system). However, 163 because a preponderance of Internet traffic tends to originate in one 164 autonomous system and terminate in another, this document also 165 provides an overview of aspects pertaining to inter-domain traffic 166 engineering. 168 This work was first published as [RFC3272] in May 2002. This 169 document obsoletes [RFC3272] by making a complete update to bring the 170 text in line with best current practices for Internet traffic 171 engineering and to include references to the latest relevant work in 172 the IETF. It is worth noting around three fifths of the RFCs 173 referenced in this document post-date the publication of RFC 3272. 174 Appendix C provides a summary of changes between RFC 3272 and this 175 document. 177 1.1. What is Internet Traffic Engineering? 179 One of the most significant functions performed by the Internet is 180 the routing of traffic from ingress nodes to egress nodes. 181 Therefore, one of the most distinctive functions performed by 182 Internet traffic engineering is the control and optimization of the 183 routing function, to steer traffic through the network. 185 Internet traffic engineering is defined as that aspect of Internet 186 network engineering dealing with the issues of performance evaluation 187 and performance optimization of operational IP networks. Traffic 188 engineering encompasses the application of technology and scientific 189 principles to the measurement, characterization, modeling, and 190 control of Internet traffic [RFC2702], [AWD2]. 192 It is the performance of the network as seen by end users of network 193 services that is paramount. The characteristics visible to end users 194 are the emergent properties of the network, which are the 195 characteristics of the network when viewed as a whole. A central 196 goal of the service provider, therefore, is to enhance the emergent 197 properties of the network while taking economic considerations into 198 account. This is accomplished by addressing traffic oriented 199 performance requirements while utilizing network resources 200 economically and reliably. Traffic oriented performance measures 201 include delay, delay variation, packet loss, and throughput. 203 Internet traffic engineering responds to network events. Aspects of 204 capacity management respond at intervals ranging from days to years. 205 Routing control functions operate at intervals ranging from 206 milliseconds to days. Packet level processing functions operate at 207 very fine levels of temporal resolution, ranging from picoseconds to 208 milliseconds while reacting to the real-time statistical behavior of 209 traffic. 211 Thus, the optimization aspects of traffic engineering can be viewed 212 from a control perspective, and can be both pro-active and reactive. 213 In the pro-active case, the traffic engineering control system takes 214 preventive action to protect against predicted unfavorable future 215 network states, for example, by engineering backup paths. It may 216 also take action that will lead to a more desirable future network 217 state. In the reactive case, the control system responds to correct 218 issues and adapt to network events, such as routing after failure. 220 Another important objective of Internet traffic engineering is to 221 facilitate reliable network operations [RFC2702]. Reliable network 222 operations can be facilitated by providing mechanisms that enhance 223 network integrity and by embracing policies emphasizing network 224 survivability. This reduces the vulnerability of services to outages 225 arising from errors, faults, and failures occurring within the 226 network infrastructure. 228 The optimization aspects of traffic engineering can be achieved 229 through capacity management and traffic management. In this 230 document, capacity management includes capacity planning, routing 231 control, and resource management. Network resources of particular 232 interest include link bandwidth, buffer space, and computational 233 resources. In this document, traffic management includes: 235 1. nodal traffic control functions such as traffic conditioning, 236 queue management, scheduling 238 2. other functions that regulate traffic flow through the network or 239 that arbitrate access to network resources between different 240 packets or between different traffic streams. 242 One major challenge of Internet traffic engineering is the 243 realization of automated control capabilities that adapt quickly and 244 cost effectively to significant changes in network state, while still 245 maintaining stability of the network. Performance evaluation can 246 assess the effectiveness of traffic engineering methods, and the 247 results of this evaluation can be used to identify existing problems, 248 guide network re-optimization, and aid in the prediction of potential 249 future problems. However, this process can also be time consuming 250 and may not be suitable to act on short-lived changes in the network. 252 Performance evaluation can be achieved in many different ways. The 253 most notable techniques include analytical methods, simulation, and 254 empirical methods based on measurements. 256 Traffic engineering comes in two flavors: either a background process 257 that constantly monitors traffic and optimizes the use of resources 258 to improve performance; or a form of a pre-planned optimized traffic 259 distribution that is considered optimal. In the later case, any 260 deviation from the optimum distribution (e.g., caused by a fiber cut) 261 is reverted upon repair without further optimization. However, this 262 form of traffic engineering relies upon the notion that the planned 263 state of the network is optimal. Hence, in such a mode there are two 264 levels of traffic engineering: the TE-planning task to enable optimum 265 traffic distribution, and the routing task keeping traffic flows 266 attached to the pre-planned distribution. 268 As a general rule, traffic engineering concepts and mechanisms must 269 be sufficiently specific and well-defined to address known 270 requirements, but simultaneously flexible and extensible to 271 accommodate unforeseen future demands. 273 1.2. Components of Traffic Engineering 275 As mentioned in Section 1.1, Internet traffic engineering provides 276 performance optimization of operational IP networks while utilizing 277 network resources economically and reliably. Such optimization is 278 supported at the control/controller level and within the data/ 279 forwarding plane. 281 The key elements required in any TE solution are as follows: 283 1. Policy 285 2. Path steering 286 3. Resource management 288 Some TE solutions rely on these elements to a lesser or greater 289 extent. Debate remains about whether a solution can truly be called 290 traffic engineering if it does not include all of these elements. 291 For the sake of this document, we assert that all TE solutions must 292 include some aspects of all of these elements. Other solutions can 293 be classed as "partial TE" and also fall in scope of this document. 295 Policy allows for the selection of next hops and paths based on 296 information beyond basic reachability. Early definitions of routing 297 policy, e.g., [RFC1102] and [RFC1104], discuss routing policy being 298 applied to restrict access to network resources at an aggregate 299 level. BGP is an example of a commonly used mechanism for applying 300 such policies, see [RFC4271] and [RFC8955]. In the traffic 301 engineering context, policy decisions are made within the control 302 plane or by controllers, and govern the selection of paths. Examples 303 can be found in [RFC4655] and [RFC5394]. Standard TE solutions may 304 cover the mechanisms to distribute and/or enforce polices, but 305 specific policy definition is generally unspecified. 307 Path steering is the ability to forward packets using more 308 information than just knowledge of the next hop. Examples of path 309 steering include IPv4 source routes [RFC0791], RSVP-TE explicit 310 routes [RFC3209], and Segment Routing [RFC8402]. Path steering for 311 TE can be supported via control plane protocols, by encoding in the 312 data plane headers, or by a combination of the two. This includes 313 when control is provided by a controller using a southbound (i.e., 314 controller to router) control protocol. 316 Resource management provides resource aware control and forwarding. 317 Examples of resources are bandwidth, buffers, and queues, all of 318 which can be managed to control loss and latency. 320 Resource reservation is the control aspect of resource management. 321 It provides for domain-wide consensus about which network 322 resources are used by a particular flow. This determination may 323 be made at a very course or very fine level. Note that this 324 consensus exists at the network control or controller level, not 325 within the data plane. It may be composed purely of accounting/ 326 bookkeeping, but it typically includes an ability to admit, 327 reject, or reclassify a flow based on policy. Such accounting can 328 be done based on any combination of a static understanding of 329 resource requirements, and the use of dynamic mechanisms to 330 collect requirements (e.g., via [RFC3209]) and resource 331 availability (e.g., via [RFC4203]). 333 Resource allocation is the data plane aspect of resource 334 management. It provides for the allocation of specific node and 335 link resources to specific flows. Example resources include 336 buffers, policing, and rate-shaping mechanisms that are typically 337 supported via queuing. It also includes the matching of a flow 338 (i.e., flow classification) to a particular set of allocated 339 resources. The method of flow classification and granularity of 340 resource management is technology specific. Examples include 341 Diffserv with dropping and remarking [RFC4594], MPLS-TE [RFC3209], 342 and GMPLS based label switched paths [RFC3945], as well as 343 controller-based solutions [RFC8453]. This level of resource 344 control, while optional, is important in networks that wish to 345 support congestion management policies to control or regulate the 346 offered traffic to deliver different levels of service and 347 alleviate congestion problems, or those networks that wish to 348 control latencies experienced by specific traffic flows. 350 1.3. Scope 352 The scope of this document is intra-domain traffic engineering. That 353 is, traffic engineering within a given autonomous system in the 354 Internet. This document discusses concepts pertaining to intra- 355 domain traffic control, including such issues as routing control, 356 micro and macro resource allocation, and the control coordination 357 problems that arise consequently. 359 This document describes and characterizes techniques already in use 360 or in advanced development for Internet traffic engineering. The way 361 these techniques fit together is discussed and scenarios in which 362 they are useful will be identified. 364 Although the emphasis in this document is on intra-domain traffic 365 engineering, in Section 7, an overview of the high level 366 considerations pertaining to inter-domain traffic engineering will be 367 provided. Inter-domain Internet traffic engineering is crucial to 368 the performance enhancement of the global Internet infrastructure. 370 Whenever possible, relevant requirements from existing IETF documents 371 and other sources are incorporated by reference. 373 1.4. Terminology 375 This section provides terminology which is useful for Internet 376 traffic engineering. The definitions presented apply to this 377 document. These terms may have other meanings elsewhere. 379 Busy hour: A one hour period within a specified interval of time 380 (typically 24 hours) in which the traffic load in a network or 381 sub-network is greatest. 383 Congestion: A state of a network resource in which the traffic 384 incident on the resource exceeds its output capacity over an 385 interval of time. 387 Congestion avoidance: An approach to congestion management that 388 attempts to obviate the occurrence of congestion. 390 Congestion control: An approach to congestion management that 391 attempts to remedy congestion problems that have already occurred. 393 Constraint-based routing: A class of routing protocols that take 394 specified traffic attributes, network constraints, and policy 395 constraints into account when making routing decisions. 396 Constraint-based routing is applicable to traffic aggregates as 397 well as flows. It is a generalization of QoS routing. 399 Demand side congestion management: A congestion management scheme 400 that addresses congestion problems by regulating or conditioning 401 offered load. 403 Effective bandwidth: The minimum amount of bandwidth that can be 404 assigned to a flow or traffic aggregate in order to deliver 405 'acceptable service quality' to the flow or traffic aggregate. 407 Hot-spot: A network element or subsystem which is in a state of 408 congestion. 410 Inter-domain traffic: Traffic that originates in one Autonomous 411 system and terminates in another. 413 Metric: A parameter defined in terms of standard units of 414 measurement. 416 Measurement methodology: A repeatable measurement technique used to 417 derive one or more metrics of interest. 419 Network survivability: The capability to provide a prescribed level 420 of QoS for existing services after a given number of failures 421 occur within the network. 423 Offline traffic engineering: A traffic engineering system that 424 exists outside of the network. 426 Online traffic engineering: A traffic engineering system that exists 427 within the network, typically implemented on or as adjuncts to 428 operational network elements. 430 Performance measures: Metrics that provide quantitative or 431 qualitative measures of the performance of systems or subsystems 432 of interest. 434 Performance metric: A performance parameter defined in terms of 435 standard units of measurement. 437 Provisioning: The process of assigning or configuring network 438 resources to meet certain requests. 440 QoS routing: Class of routing systems that selects paths to be used 441 by a flow based on the QoS requirements of the flow. 443 Service Level Agreement (SLA): A contract between a provider and a 444 customer that guarantees specific levels of performance and 445 reliability at a certain cost. 447 Service Level Objective (SLO): A key element of an SLA between a 448 provider and a customer. SLOs are agreed upon as a means of 449 measuring the performance of the Service Provider and are outlined 450 as a way of avoiding disputes between the two parties based on 451 misunderstanding. 453 Stability: An operational state in which a network does not 454 oscillate in a disruptive manner from one mode to another mode. 456 Supply-side congestion management: A congestion management scheme 457 that provisions additional network resources to address existing 458 and/or anticipated congestion problems. 460 Traffic characteristic: A description of the temporal behavior or a 461 description of the attributes of a given traffic flow or traffic 462 aggregate. 464 Traffic engineering system: A collection of objects, mechanisms, and 465 protocols that are used together to accomplish traffic engineering 466 objectives. 468 Traffic flow: A stream of packets between two end-points that can be 469 characterized in a certain way. A micro-flow has a more specific 470 definition A micro-flow is a stream of packets with the same 471 source and destination addresses, source and destination ports, 472 and protocol ID. 474 Traffic matrix: A representation of the traffic demand between a set 475 of origin and destination abstract nodes. An abstract node can 476 consist of one or more network elements. 478 Traffic monitoring: The process of observing traffic characteristics 479 at a given point in a network and collecting the traffic 480 information for analysis and further action. 482 Traffic trunk: An aggregation of traffic flows belonging to the same 483 class which are forwarded through a common path. A traffic trunk 484 may be characterized by an ingress and egress node, and a set of 485 attributes which determine its behavioral characteristics and 486 requirements from the network. 488 2. Background 490 The Internet must convey IP packets from ingress nodes to egress 491 nodes efficiently, expeditiously, and economically. Furthermore, in 492 a multiclass service environment (e.g., Diffserv capable networks - 493 see Section 4.1.4), the resource sharing parameters of the network 494 must be appropriately determined and configured according to 495 prevailing policies and service models to resolve resource contention 496 issues arising from mutual interference between packets traversing 497 through the network. Thus, consideration must be given to resolving 498 competition for network resources between traffic streams belonging 499 to the same service class (intra-class contention resolution) and 500 traffic streams belonging to different classes (inter-class 501 contention resolution). 503 2.1. Context of Internet Traffic Engineering 505 The context of Internet traffic engineering includes: 507 1. A network domain context that defines the scope under 508 consideration, and in particular the situations in which the 509 traffic engineering problems occur. The network domain context 510 includes network structure, network policies, network 511 characteristics, network constraints, network quality attributes, 512 and network optimization criteria. 514 2. A problem context defining the general and concrete issues that 515 traffic engineering addresses. The problem context includes 516 identification, abstraction of relevant features, representation, 517 formulation, specification of the requirements on the solution 518 space, and specification of the desirable features of acceptable 519 solutions. 521 3. A solution context suggesting how to address the issues 522 identified by the problem context. The solution context includes 523 analysis, evaluation of alternatives, prescription, and 524 resolution. 526 4. An implementation and operational context in which the solutions 527 are instantiated. The implementation and operational context 528 includes planning, organization, and execution. 530 The context of Internet traffic engineering and the different problem 531 scenarios are discussed in the following subsections. 533 2.2. Network Domain Context 535 IP networks range in size from small clusters of routers situated 536 within a given location, to thousands of interconnected routers, 537 switches, and other components distributed all over the world. 539 At the most basic level of abstraction, an IP network can be 540 represented as a distributed dynamic system consisting of: 542 o a set of interconnected resources which provide transport services 543 for IP traffic subject to certain constraints 545 o a demand system representing the offered load to be transported 546 through the network 548 o a response system consisting of network processes, protocols, and 549 related mechanisms which facilitate the movement of traffic 550 through the network (see also [AWD2]). 552 The network elements and resources may have specific characteristics 553 restricting the manner in which the traffic demand is handled. 554 Additionally, network resources may be equipped with traffic control 555 mechanisms managing the way in which the demand is serviced. Traffic 556 control mechanisms may be used to: 558 o control packet processing activities within a given resource 560 o arbitrate contention for access to the resource by different 561 packets 563 o regulate traffic behavior through the resource. 565 A configuration management and provisioning system may allow the 566 settings of the traffic control mechanisms to be manipulated by 567 external or internal entities in order to exercise control over the 568 way in which the network elements respond to internal and external 569 stimuli. 571 The details of how the network carries packets are specified in the 572 policies of the network administrators and are installed through 573 network configuration management and policy based provisioning 574 systems. Generally, the types of service provided by the network 575 also depend upon the technology and characteristics of the network 576 elements and protocols, the prevailing service and utility models, 577 and the ability of the network administrators to translate policies 578 into network configurations. 580 Internet networks have three significant characteristics: 582 o they provide real-time services 584 o they are mission critical 586 o their operating environments are very dynamic. 588 The dynamic characteristics of IP and IP/MPLS networks can be 589 attributed in part to fluctuations in demand, to the interaction 590 between various network protocols and processes, to the rapid 591 evolution of the infrastructure which demands the constant inclusion 592 of new technologies and new network elements, and to transient and 593 persistent faults which occur within the system. 595 Packets contend for the use of network resources as they are conveyed 596 through the network. A network resource is considered to be 597 congested if, for an interval of time, the arrival rate of packets 598 exceed the output capacity of the resource. Congestion may result in 599 some of the arriving packets being delayed or even dropped. 601 Congestion increases transit delay, delay variation, may lead to 602 packet loss, and reduces the predictability of network services. 603 Clearly, congestion is highly undesirable. Combating congestion at a 604 reasonable cost is a major objective of Internet traffic engineering. 606 Efficient sharing of network resources by multiple traffic streams is 607 a basic operational premise for the Internet. A fundamental 608 challenge in network operation is to increase resource utilization 609 while minimizing the possibility of congestion. 611 The Internet has to function in the presence of different classes of 612 traffic with different service requirements. RFC 2475 provides an 613 architecture for Differentiated Services (Diffserv) and makes this 614 requirement clear [RFC2475]. The RFC allows packets to be grouped 615 into behavior aggregates such that each aggregate has a common set of 616 behavioral characteristics or a common set of delivery requirements. 617 Delivery requirements of a specific set of packets may be specified 618 explicitly or implicitly. Two of the most important traffic delivery 619 requirements are capacity constraints and QoS constraints. 621 Capacity constraints can be expressed statistically as peak rates, 622 mean rates, burst sizes, or as some deterministic notion of effective 623 bandwidth. QoS requirements can be expressed in terms of: 625 o integrity constraints such as packet loss 627 o temporal constraints such as timing restrictions for the delivery 628 of each packet (delay) and timing restrictions for the delivery of 629 consecutive packets belonging to the same traffic stream (delay 630 variation). 632 2.3. Problem Context 634 There are several large problems associated with operating a network 635 described in the previous section. This section analyzes the problem 636 context in relation to traffic engineering. The identification, 637 abstraction, representation, and measurement of network features 638 relevant to traffic engineering are significant issues. 640 A particular challenge is to formulate the problems that traffic 641 engineering attempts to solve. For example: 643 o how to identify the requirements on the solution space 645 o how to specify the desirable features of solutions 647 o how to actually solve the problems 649 o how to measure and characterize the effectiveness of solutions. 651 Another class of problems is how to measure and estimate relevant 652 network state parameters. Effective traffic engineering relies on a 653 good estimate of the offered traffic load as well as a view of the 654 underlying topology and associated resource constraints. A network- 655 wide view of the topology is also a must for offline planning. 657 Still another class of problem is how to characterize the state of 658 the network and how to evaluate its performance. The performance 659 evaluation problem is two-fold: one aspect relates to the evaluation 660 of the system-level performance of the network; the other aspect 661 relates to the evaluation of resource-level performance, which 662 restricts attention to the performance analysis of individual network 663 resources. 665 In this document, we refer to the system-level characteristics of the 666 network as the "macro-states" and the resource-level characteristics 667 as the "micro-states." The system-level characteristics are also 668 known as the emergent properties of the network. Correspondingly, we 669 refer to the traffic engineering schemes dealing with network 670 performance optimization at the systems level as "macro-TE" and the 671 schemes that optimize at the individual resource level as "micro-TE." 672 Under certain circumstances, the system-level performance can be 673 derived from the resource-level performance using appropriate rules 674 of composition, depending upon the particular performance measures of 675 interest. 677 Another fundamental class of problem concerns how to effectively 678 optimize network performance. Performance optimization may entail 679 translating solutions for specific traffic engineering problems into 680 network configurations. Optimization may also entail some degree of 681 resource management control, routing control, and capacity 682 augmentation. 684 2.3.1. Congestion and its Ramifications 686 Congestion is one of the most significant problems in an operational 687 IP context. A network element is said to be congested if it 688 experiences sustained overload over an interval of time. Congestion 689 almost always results in degradation of service quality to end users. 690 Congestion control schemes can include demand-side policies and 691 supply-side policies. Demand-side policies may restrict access to 692 congested resources or dynamically regulate the demand to alleviate 693 the overload situation. Supply-side policies may expand or augment 694 network capacity to better accommodate offered traffic. Supply-side 695 policies may also re-allocate network resources by redistributing 696 traffic over the infrastructure. Traffic redistribution and resource 697 re-allocation serve to increase the 'effective capacity' of the 698 network. 700 The emphasis of this document is primarily on congestion management 701 schemes falling within the scope of the network, rather than on 702 congestion management systems dependent upon sensitivity and 703 adaptivity from end-systems. That is, the aspects that are 704 considered in this document with respect to congestion management are 705 those solutions that can be provided by control entities operating on 706 the network and by the actions of network administrators and network 707 operations systems. 709 2.4. Solution Context 711 The solution context for Internet traffic engineering involves 712 analysis, evaluation of alternatives, and choice between alternative 713 courses of action. Generally the solution context is based on making 714 reasonable inferences about the current or future state of the 715 network, and making decisions that may involve a preference between 716 alternative sets of action. More specifically, the solution context 717 demands reasonable estimates of traffic workload, characterization of 718 network state, derivation of solutions which may be implicitly or 719 explicitly formulated, and possibly instantiating a set of control 720 actions. Control actions may involve the manipulation of parameters 721 associated with routing, control over tactical capacity acquisition, 722 and control over the traffic management functions. 724 The following list of instruments may be applicable to the solution 725 context of Internet traffic engineering. 727 o A set of policies, objectives, and requirements (which may be 728 context dependent) for network performance evaluation and 729 performance optimization. 731 o A collection of online and possibly offline tools and mechanisms 732 for measurement, characterization, modeling, and control traffic, 733 and control over the placement and allocation of network 734 resources, as well as control over the mapping or distribution of 735 traffic onto the infrastructure. 737 o A set of constraints on the operating environment, the network 738 protocols, and the traffic engineering system itself. 740 o A set of quantitative and qualitative techniques and methodologies 741 for abstracting, formulating, and solving traffic engineering 742 problems. 744 o A set of administrative control parameters which may be 745 manipulated through a Configuration Management (CM) system. The 746 CM system itself may include a configuration control subsystem, a 747 configuration repository, a configuration accounting subsystem, 748 and a configuration auditing subsystem. 750 o A set of guidelines for network performance evaluation, 751 performance optimization, and performance improvement. 753 Determining traffic characteristics through measurement or estimation 754 is very useful within the realm the traffic engineering solution 755 space. Traffic estimates can be derived from customer subscription 756 information, traffic projections, traffic models, and from actual 757 measurements. The measurements may be performed at different levels, 758 e.g., at the traffic-aggregate level or at the flow level. 759 Measurements at the flow level or on small traffic aggregates may be 760 performed at edge nodes, when traffic enters and leaves the network. 761 Measurements for large traffic-aggregates may be performed within the 762 core of the network. 764 To conduct performance studies and to support planning of existing 765 and future networks, a routing analysis may be performed to determine 766 the paths the routing protocols will choose for various traffic 767 demands, and to ascertain the utilization of network resources as 768 traffic is routed through the network. Routing analysis captures the 769 selection of paths through the network, the assignment of traffic 770 across multiple feasible routes, and the multiplexing of IP traffic 771 over traffic trunks (if such constructs exist) and over the 772 underlying network infrastructure. A model of network topology is 773 necessary to perform routing analysis. A network topology model may 774 be extracted from: 776 o network architecture documents 778 o network designs 780 o information contained in router configuration files 782 o routing databases 784 o routing tables 786 o automated tools that discover and collate network topology 787 information. 789 Topology information may also be derived from servers that monitor 790 network state, and from servers that perform provisioning functions. 792 Routing in operational IP networks can be administratively controlled 793 at various levels of abstraction including the manipulation of BGP 794 attributes and interior gateway protocol (IGP) metrics. For path 795 oriented technologies such as MPLS, routing can be further controlled 796 by the manipulation of relevant traffic engineering parameters, 797 resource parameters, and administrative policy constraints. Within 798 the context of MPLS, the path of an explicitly routed label switched 799 path (LSP) can be computed and established in various ways including: 801 o manually 803 o automatically, online using constraint-based routing processes 804 implemented on label switching routers 806 o automatically, offline using constraint-based routing entities 807 implemented on external traffic engineering support systems. 809 2.4.1. Combating the Congestion Problem 811 Minimizing congestion is a significant aspect of Internet traffic 812 engineering. This subsection gives an overview of the general 813 approaches that have been used or proposed to combat congestion. 815 Congestion management policies can be categorized based upon the 816 following criteria (see [YARE95] for a more detailed taxonomy of 817 congestion control schemes): 819 1. Congestion Management based on Response Time Scales 821 * Long (weeks to months): Expanding network capacity by adding 822 new equipment, routers, and links takes time and is 823 comparatively costly. Capacity planning needs to take this 824 into consideration. Network capacity is expanded based on 825 estimates or forecasts of future traffic development and 826 traffic distribution. These upgrades are typically carried 827 out over weeks or months, or maybe even years. 829 * Medium (minutes to days): Several control policies fall within 830 the medium timescale category. Examples include: 832 a. Adjusting routing protocol parameters to route traffic 833 away or towards certain segments of the network. 835 b. Setting up or adjusting explicitly routed LSPs in MPLS 836 networks to route traffic trunks away from possibly 837 congested resources or toward possibly more favorable 838 routes. 840 c. Re-configuring the logical topology of the network to make 841 it correlate more closely with the spatial traffic 842 distribution using, for example, an underlying path- 843 oriented technology such as MPLS LSPs or optical channel 844 trails. 846 Many of these adaptive schemes rely on measurement systems. A 847 measurement system monitors changes in traffic distribution, 848 traffic loads, and network resource utilization and then 849 provides feedback to the online or offline traffic engineering 850 mechanisms and tools so that they can trigger control actions 851 within the network. The traffic engineering mechanisms and 852 tools can be implemented in a distributed or centralized 853 fashion. A centralized scheme may have global visibility into 854 the network state and may produce more optimal solutions. 855 However, centralized schemes are prone to single points of 856 failure and may not scale as well as distributed schemes. 857 Moreover, the information utilized by a centralized scheme may 858 be stale and might not reflect the actual state of the 859 network. It is not an objective of this document to make a 860 recommendation between distributed and centralized schemes: 861 that is a choice that network administrators must make based 862 on their specific needs. 864 * Short (picoseconds to minutes): This category includes packet 865 level processing functions and events that are recorded on the 866 order of several round trip times. It also includes router 867 mechanisms such as passive and active buffer management. All 868 of these mechanisms are used to control congestion or signal 869 congestion to end systems so that they can adaptively regulate 870 the rate at which traffic is injected into the network. One 871 of the most popular active queue management schemes, 872 especially for TCP traffic, is Random Early Detection (RED) 873 [FLJA93]. During congestion (but before the queue is filled), 874 the RED scheme chooses arriving packets to "mark" according to 875 a probabilistic algorithm which takes into account the average 876 queue size. A router that does not utilize explicit 877 congestion notification (ECN) [FLOY94] can simply drop marked 878 packets to alleviate congestion and implicitly notify the 879 receiver about the congestion. On the other hand, if the 880 router supports ECN, it can set the ECN field in the packet 881 header. Several variations of RED have been proposed to 882 support different drop precedence levels in multi-class 883 environments [RFC2597]. RED provides congestion avoidance 884 which is not worse than traditional Tail-Drop (TD) queue 885 management (drop arriving packets only when the queue is 886 full). Importantly, RED reduces the possibility of global 887 synchronization where retransmission burst become synchronized 888 across the whole network, and improves fairness among 889 different TCP sessions. However, RED by itself cannot prevent 890 congestion and unfairness caused by sources unresponsive to 891 RED, e.g., UDP traffic and some misbehaved greedy connections. 892 Other schemes have been proposed to improve the performance 893 and fairness in the presence of unresponsive traffic. Some of 894 those schemes (such as Longest Queue Drop (LQD) and Dynamic 895 Soft Partitioning with Random Drop (RND) [SLDC98]) were 896 proposed as theoretical frameworks and are typically not 897 available in existing commercial products. 899 2. Congestion Management: Reactive Versus Preventive Schemes 900 * Reactive: Reactive (recovery) congestion management policies 901 react to existing congestion problems. All the policies 902 described above for the long and medium time scales can be 903 categorized as being reactive. They are based on monitoring 904 and identifying congestion problems that exist in the network, 905 and on the initiation of relevant actions to ease a situation. 907 * Preventive: Preventive (predictive/avoidance) policies take 908 proactive action to prevent congestion based on estimates and 909 predictions of future congestion problems. Some of the 910 policies described for the long and medium time scales fall 911 into this category. Preventive policies do not necessarily 912 respond immediately to existing congestion problems. Instead, 913 forecasts of traffic demand and workload distribution are 914 considered, and action may be taken to prevent potential 915 future congestion problems. The schemes described for the 916 short time scale can also be used for congestion avoidance 917 because dropping or marking packets before queues actually 918 overflow would trigger corresponding TCP sources to slow down. 920 3. Congestion Management: Supply-Side Versus Demand-Side Schemes 922 * Supply-side: Supply-side congestion management policies 923 increase the effective capacity available to traffic in order 924 to control or reduce congestion. This can be accomplished by 925 increasing capacity or by balancing distribution of traffic 926 over the network. Capacity planning aims to provide a 927 physical topology and associated link bandwidths that match or 928 exceed estimated traffic workload and traffic distribution 929 subject to traffic forecasts and budgetary or other 930 constraints. If the actual traffic distribution does not fit 931 the topology derived from capacity panning, then the traffic 932 can be mapped onto the topology by using routing control 933 mechanisms, by applying path oriented technologies (e.g., MPLS 934 LSPs and optical channel trails) to modify the logical 935 topology, or by employing some other load redistribution 936 mechanisms. 938 * Demand-side: Demand-side congestion management policies 939 control or regulate the offered traffic to alleviate 940 congestion problems. For example, some of the short time 941 scale mechanisms described earlier as well as policing and 942 rate-shaping mechanisms attempt to regulate the offered load 943 in various ways. 945 2.5. Implementation and Operational Context 947 The operational context of Internet traffic engineering is 948 characterized by constant changes that occur at multiple levels of 949 abstraction. The implementation context demands effective planning, 950 organization, and execution. The planning aspects may involve 951 determining prior sets of actions to achieve desired objectives. 952 Organizing involves arranging and assigning responsibility to the 953 various components of the traffic engineering system and coordinating 954 the activities to accomplish the desired TE objectives. Execution 955 involves measuring and applying corrective or perfective actions to 956 attain and maintain desired TE goals. 958 3. Traffic Engineering Process Models 960 This section describes a generic process model that captures the 961 high-level practical aspects of Internet traffic engineering in an 962 operational context. The process model is described as a sequence of 963 actions that must be carried out to optimize the performance of an 964 operational network (see also [RFC2702], [AWD2]). This process model 965 may be enacted explicitly or implicitly, by a software process or by 966 a human. 968 The traffic engineering process model is iterative [AWD2]. The four 969 phases of the process model described below are repeated as a 970 continual sequence. 972 o Define the relevant control policies that govern the operation of 973 the network. 975 o Acquire measurement data from the operational network. 977 o Analyze the network state and characterize the traffic workload. 978 Proactive analysis identifies potential problems that could 979 manifest in the future. Reactive analysis identifies existing 980 problems and determines their causes. 982 o Optimize the performance of the network. This involves a decision 983 process which selects and implements a set of actions from a set 984 of alternatives given the results of the three previous steps. 985 Optimization actions may include the use of techniques to control 986 the offered traffic and to control the distribution of traffic 987 across the network. 989 3.1. Components of the Traffic Engineering Process Model 991 The key components of the traffic engineering process model are as 992 follows. 994 1. Measurement is crucial to the traffic engineering function. The 995 operational state of a network can only be conclusively 996 determined through measurement. Measurement is also critical to 997 the optimization function because it provides feedback data which 998 is used by traffic engineering control subsystems. This data is 999 used to adaptively optimize network performance in response to 1000 events and stimuli originating within and outside the network. 1001 Measurement in support of the TE function can occur at different 1002 levels of abstraction. For example, measurement can be used to 1003 derive packet level characteristics, flow level characteristics, 1004 user or customer level characteristics, traffic aggregate 1005 characteristics, component level characteristics, and network 1006 wide characteristics. 1008 2. Modeling, analysis, and simulation are important aspects of 1009 Internet traffic engineering. Modeling involves constructing an 1010 abstract or physical representation which depicts relevant 1011 traffic characteristics and network attributes. A network model 1012 is an abstract representation of the network which captures 1013 relevant network features, attributes, and characteristic. 1014 Network simulation tools are extremely useful for traffic 1015 engineering. Because of the complexity of realistic quantitative 1016 analysis of network behavior, certain aspects of network 1017 performance studies can only be conducted effectively using 1018 simulation. 1020 3. Network performance optimization involves resolving network 1021 issues by transforming such issues into concepts that enable a 1022 solution, identification of a solution, and implementation of the 1023 solution. Network performance optimization can be corrective or 1024 perfective. In corrective optimization, the goal is to remedy a 1025 problem that has occurred or that is incipient. In perfective 1026 optimization, the goal is to improve network performance even 1027 when explicit problems do not exist and are not anticipated. 1029 4. Review of TE Techniques 1031 This section briefly reviews different traffic engineering approaches 1032 proposed and implemented in telecommunications and computer networks 1033 using IETF protocols and architectures. The discussion is not 1034 intended to be comprehensive. It is primarily intended to illuminate 1035 existing approaches to traffic engineering in the Internet. A 1036 historic overview of traffic engineering in telecommunications 1037 networks is provided in Appendix A, while Appendix B describes 1038 approaches in other standards bodies. 1040 4.1. Overview of IETF Projects Related to Traffic Engineering 1042 This subsection reviews a number of IETF activities pertinent to 1043 Internet traffic engineering. 1045 4.1.1. Constraint-Based Routing 1047 Constraint-based routing refers to a class of routing systems that 1048 compute routes through a network subject to the satisfaction of a set 1049 of constraints and requirements. In the most general case, 1050 constraint-based routing may also seek to optimize overall network 1051 performance while minimizing costs. 1053 The constraints and requirements may be imposed by the network itself 1054 or by administrative policies. Constraints may include bandwidth, 1055 hop count, delay, and policy instruments such as resource class 1056 attributes. Constraints may also include domain specific attributes 1057 of certain network technologies and contexts which impose 1058 restrictions on the solution space of the routing function. Path 1059 oriented technologies such as MPLS have made constraint-based routing 1060 feasible and attractive in public IP networks. 1062 The concept of constraint-based routing within the context of MPLS 1063 traffic engineering requirements in IP networks was first described 1064 in [RFC2702] and led to developments such as MPLS-TE [RFC3209] as 1065 described in Section 4.1.6. 1067 Unlike QoS routing (for example, see [RFC2386] and [MA]) which 1068 generally addresses the issue of routing individual traffic flows to 1069 satisfy prescribed flow-based QoS requirements subject to network 1070 resource availability, constraint-based routing is applicable to 1071 traffic aggregates as well as flows and may be subject to a wide 1072 variety of constraints which may include policy restrictions. 1074 4.1.1.1. IGP Flexible Algorithms (Flex-Algos) 1076 The traditional approach to routing in an IGP network relies on the 1077 IGPs deriving "shortest paths" over the network based solely on the 1078 IGP metric assigned to the links. Such an approach is often limited: 1079 traffic may tend to converge toward the destination, possibly causing 1080 congestion; and it is not possible to steer traffic onto paths 1081 depending on the end-to-end qualities demanded by the applications. 1083 To overcome this limitation, various sorts of traffic engineering 1084 have been widely deployed (as described in this document), where the 1085 TE component is responsible for computing the path based on 1086 additionalcmetrics and/or constraints. Such paths (or tunnels) need 1087 to be installed in the routers' forwarding tables in addition to, or 1088 as a replacement for the original paths computed by IGPs. The main 1089 drawback of these TE approaches is the additional complexity of 1090 protocols and management, and the state that may need to be 1091 maintained within the network. 1093 IGP flexible algorithms (flex-algos) [I-D.ietf-lsr-flex-algo] allow 1094 IGPs to construct constraint-based paths over the network by 1095 computing constraint- based next hops. The intent of flex-algos is 1096 to reduce TE complexity by letting an IGP perform some basic TE 1097 computation capabilities. Flex-algo includes a set of extensions to 1098 the IGPs that enable a router to send TLVs that: 1100 o describe a set of constraints on the topology 1102 o identify calculation-type 1104 o describe a metric-type that is to be used to compute the best 1105 paths through the constrained topology. 1107 A given combination of calculation-type, metric-type, and constraints 1108 is known as a "Flexible Algorithm Definition" (or FAD). A router 1109 that sends such a set of TLVs also assigns a specific identifier (the 1110 Flexible Algorithm) to the specified combination of calculation-type, 1111 metric-type, and constraints. 1113 There are two use cases for flex-algo: in IP networks 1114 [I-D.ietf-lsr-ip-flexalgo] and in segment routing networks 1115 [I-D.ietf-lsr-flex-algo]. In the first case, flex-algo computes 1116 paths to an IPv4 or IPv6 address, in the second case, flex-algo 1117 computes paths to a prefix SID (see Section 4.1.16). 1119 There are many use cases where flex-algo can bring big value, such 1120 as: 1122 o Expansion of functionality of IP Performance metrics [RFC5664] 1123 when points of interest could instantiate specific constraint- 1124 based routing (flex-algo) based on the measurement results. 1126 o Nested usage of flex-algo and TE extensions for IGP (see 1127 Section 4.1.11) when we can form 'underlay' by means of flex-algo 1128 and 'overlay' by TE. 1130 o Flex-algo in SR-MPLS (Section 4.1.16) is a base use case when we 1131 can easily benefit from TE-like topology that will be built 1132 without external TE component on routers or PCE (see 1133 Section 4.1.13). 1135 o Building of network slices 1136 [I-D.nsdt-teas-ietf-network-slice-definition] where particular 1137 IETF network slice SLO can be guaranteed by flex-algo. 1139 4.1.2. Integrated Services 1141 The IETF developed the Integrated Services (Intserv) model that 1142 requires resources, such as bandwidth and buffers, to be reserved a 1143 priori for a given traffic flow to ensure that the quality of service 1144 requested by the traffic flow is satisfied. The Integrated Services 1145 model includes additional components beyond those used in the best- 1146 effort model such as packet classifiers, packet schedulers, and 1147 admission control. A packet classifier is used to identify flows 1148 that are to receive a certain level of service. A packet scheduler 1149 handles the scheduling of service to different packet flows to ensure 1150 that QoS commitments are met. Admission control is used to determine 1151 whether a router has the necessary resources to accept a new flow. 1153 The main issue with the Integrated Services model has been 1154 scalability [RFC2998], especially in large public IP networks which 1155 may potentially have millions of active micro-flows in transit 1156 concurrently. 1158 A notable feature of the Integrated Services model is that it 1159 requires explicit signaling of QoS requirements from end systems to 1160 routers [RFC2753]. The Resource Reservation Protocol (RSVP) performs 1161 this signaling function and is a critical component of the Integrated 1162 Services model. RSVP is described in Section 4.1.3. 1164 4.1.3. RSVP 1166 RSVP is a soft state signaling protocol [RFC2205]. It supports 1167 receiver initiated establishment of resource reservations for both 1168 multicast and unicast flows. RSVP was originally developed as a 1169 signaling protocol within the Integrated Services framework (see 1170 Section 4.1.2) for applications to communicate QoS requirements to 1171 the network and for the network to reserve relevant resources to 1172 satisfy the QoS requirements [RFC2205]. 1174 In RSVP, the traffic sender or source node sends a PATH message to 1175 the traffic receiver with the same source and destination addresses 1176 as the traffic which the sender will generate. The PATH message 1177 contains: (1) a sender traffic specification describing the 1178 characteristics of the traffic, (2) a sender template specifying the 1179 format of the traffic, and (3) an optional advertisement 1180 specification which is used to support the concept of One Pass With 1181 Advertising (OPWA) [RFC2205]. Every intermediate router along the 1182 path forwards the PATH message to the next hop determined by the 1183 routing protocol. Upon receiving a PATH message, the receiver 1184 responds with a RESV message which includes a flow descriptor used to 1185 request resource reservations. The RESV message travels to the 1186 sender or source node in the opposite direction along the path that 1187 the PATH message traversed. Every intermediate router along the path 1188 can reject or accept the reservation request of the RESV message. If 1189 the request is rejected, the rejecting router will send an error 1190 message to the receiver and the signaling process will terminate. If 1191 the request is accepted, link bandwidth and buffer space are 1192 allocated for the flow and the related flow state information is 1193 installed in the router. 1195 One of the issues with the original RSVP specification was 1196 Scalability. This is because reservations were required for micro- 1197 flows, so that the amount of state maintained by network elements 1198 tends to increase linearly with the number of micro-flows. These 1199 issues are described in [RFC2961] which also modifies and extends 1200 RSVP to mitigate the scaling problems to make RSVP a versatile 1201 signaling protocol for the Internet. For example, RSVP has been 1202 extended to reserve resources for aggregation of flows, to set up 1203 MPLS explicit label switched paths (see Section 4.1.6), and to 1204 perform other signaling functions within the Internet. [RFC2961] 1205 also describes a mechanism to reduce the amount of Refresh messages 1206 required to maintain established RSVP sessions. 1208 4.1.4. Differentiated Services 1210 The goal of Differentiated Services (Diffserv) within the IETF was to 1211 devise scalable mechanisms for categorization of traffic into 1212 behavior aggregates, which ultimately allows each behavior aggregate 1213 to be treated differently, especially when there is a shortage of 1214 resources such as link bandwidth and buffer space [RFC2475]. One of 1215 the primary motivations for Diffserv was to devise alternative 1216 mechanisms for service differentiation in the Internet that mitigate 1217 the scalability issues encountered with the Intserv model. 1219 Diffserv uses the Differentiated Services field in the IP header (the 1220 DS field) consisting of six bits in what was formerly known as the 1221 Type of Service (TOS) octet. The DS field is used to indicate the 1222 forwarding treatment that a packet should receive at a transit node 1223 [RFC2474]. Diffserv includes the concept of Per-Hop Behavior (PHB) 1224 groups. Using the PHBs, several classes of services can be defined 1225 using different classification, policing, shaping, and scheduling 1226 rules. 1228 For an end-user of network services to utilize Differentiated 1229 Services provided by its Internet Service Provider (ISP), it may be 1230 necessary for the user to have an SLA with the ISP. An SLA may 1231 explicitly or implicitly specify a Traffic Conditioning Agreement 1232 (TCA) which defines classifier rules as well as metering, marking, 1233 discarding, and shaping rules. 1235 Packets are classified, and possibly policed and shaped at the 1236 ingress to a Diffserv network. When a packet traverses the boundary 1237 between different Diffserv domains, the DS field of the packet may be 1238 re-marked according to existing agreements between the domains. 1240 Differentiated Services allows only a finite number of service 1241 classes to be specified by the DS field. The main advantage of the 1242 Diffserv approach relative to the Intserv model is scalability. 1243 Resources are allocated on a per-class basis and the amount of state 1244 information is proportional to the number of classes rather than to 1245 the number of application flows. 1247 The Diffserv model deals with traffic management issues on a per hop 1248 basis. The Diffserv control model consists of a collection of micro- 1249 TE control mechanisms. Other traffic engineering capabilities, such 1250 as capacity management (including routing control), are also required 1251 in order to deliver acceptable service quality in Diffserv networks. 1252 The concept of Per Domain Behaviors has been introduced to better 1253 capture the notion of Differentiated Services across a complete 1254 domain [RFC3086]. 1256 4.1.5. QUIC 1258 QUIC [I-D.ietf-quic-transport] is a UDP-based multiplexed and secure 1259 transport protocol. QUIC provides applications with flow-controlled 1260 streams for structured communication, low-latency connection 1261 establishment, and network path migration. 1263 QUIC is a connection-oriented protocol that creates a stateful 1264 interaction between a client and server. QUIC uses a handshake 1265 procedure that combines negotiation of cryptographic and transport 1266 parameters. This is a key differentiation from other transport 1267 protocols. 1269 Endpoints communicate in QUIC by exchanging QUIC packets that use a 1270 customized framing for protection. Most QUIC packets contain frames, 1271 which carry control information and application data between 1272 endpoints. QUIC authenticates all packets and encrypts as much as is 1273 practical. QUIC packets are carried in UDP datagrams to better 1274 facilitate deployment within existing systems and networks. 1276 Application protocols exchange information over a QUIC connection via 1277 streams, which are ordered sequences of bytes. Two types of stream 1278 can be created: bidirectional streams, which allow both endpoints to 1279 send data; and unidirectional streams, which allow a single endpoint 1280 to send data. A credit-based scheme is used to limit stream creation 1281 and to bound the amount of data that can be sent. 1283 QUIC provides the necessary feedback to implement reliable delivery 1284 and congestion control to avoid network congestion. 1286 4.1.6. Multiprotocol Label Switching (MPLS) 1288 MPLS is an advanced forwarding scheme which also includes extensions 1289 to conventional IP control plane protocols. MPLS extends the 1290 Internet routing model and enhances packet forwarding and path 1291 control [RFC3031]. 1293 At the ingress to an MPLS domain, Label Switching Routers (LSRs) 1294 classify IP packets into Forwarding Equivalence Classes (FECs) based 1295 on a variety of factors, including, e.g., a combination of the 1296 information carried in the IP header of the packets and the local 1297 routing information maintained by the LSRs. An MPLS label stack 1298 entry is then prepended to each packet according to their forwarding 1299 equivalence classes. The MPLS label stack entry is 32 bits long and 1300 contains a 20-bit label field. 1302 An LSR makes forwarding decisions by using the label prepended to 1303 packets as the index into a local next hop label forwarding entry 1304 (NHLFE). The packet is then processed as specified in the NHLFE. 1305 The incoming label may be replaced by an outgoing label (label swap), 1306 and the packet may be forwarded to the next LSR. Before a packet 1307 leaves an MPLS domain, its MPLS label may be removed (label pop). A 1308 Label Switched Path (LSP) is the path between an ingress LSRs and an 1309 egress LSRs through which a labeled packet traverses. The path of an 1310 explicit LSP is defined at the originating (ingress) node of the LSP. 1311 MPLS can use a signaling protocol such as RSVP or LDP to set up LSPs. 1313 MPLS is a very powerful technology for Internet traffic engineering 1314 because it supports explicit LSPs which allow constraint-based 1315 routing to be implemented efficiently in IP networks [AWD2]. The 1316 requirements for traffic engineering over MPLS are described in 1317 [RFC2702]. Extensions to RSVP to support instantiation of explicit 1318 LSP are discussed in [RFC3209]. 1320 4.1.7. Generalized MPLS (GMPLS) 1322 GMPLS extends MPLS control protocols to encompass time-division 1323 (e.g., Synchronous Optical Network / Synchronous Digital Hierarchy 1324 (SONET/SDH), Plesiochronous Digital Hierarchy (PDH), Optical 1325 Transport Network (OTN)), wavelength (lambdas), and spatial switching 1326 (e.g., incoming port or fiber to outgoing port or fiber) as well as 1327 continuing to support packet switching. GMPLS provides a common set 1328 of control protocols for all of these layers (including some 1329 technology-specific extensions) each of which has a diverse data or 1330 forwarding plane. GMPLS covers both the signaling and the routing 1331 part of that control plane and is based on the Traffic Engineering 1332 extensions to MPLS (see Section 4.1.6). 1334 In GMPLS, the original MPLS architecture is extended to include LSRs 1335 whose forwarding planes rely on circuit switching, and therefore 1336 cannot forward data based on the information carried in either packet 1337 or cell headers. Specifically, such LSRs include devices where the 1338 switching is based on time slots, wavelengths, or physical ports. 1339 These additions impact basic LSP properties: how labels are requested 1340 and communicated, the unidirectional nature of MPLS LSPs, how errors 1341 are propagated, and information provided for synchronizing the 1342 ingress and egress LSRs. 1344 4.1.8. IP Performance Metrics 1346 The IETF IP Performance Metrics (IPPM) working group has developed a 1347 set of standard metrics that can be used to monitor the quality, 1348 performance, and reliability of Internet services. These metrics can 1349 be applied by network operators, end-users, and independent testing 1350 groups to provide users and service providers with a common 1351 understanding of the performance and reliability of the Internet 1352 component 'clouds' they use/provide [RFC2330]. The criteria for 1353 performance metrics developed by the IPPM working group are described 1354 in [RFC2330]. Examples of performance metrics include one-way packet 1355 loss [RFC7680], one-way delay [RFC7679], and connectivity measures 1356 between two nodes [RFC2678]. Other metrics include second-order 1357 measures of packet loss and delay. 1359 Some of the performance metrics specified by the IPPM working group 1360 are useful for specifying SLAs. SLAs are sets of service level 1361 objectives negotiated between users and service providers, wherein 1362 each objective is a combination of one or more performance metrics, 1363 possibly subject to certain constraints. 1365 4.1.9. Flow Measurement 1367 The IETF Real Time Flow Measurement (RTFM) working group produced an 1368 architecture that defines a method to specify traffic flows as well 1369 as a number of components for flow measurement (meters, meter 1370 readers, manager) [RFC2722]. A flow measurement system enables 1371 network traffic flows to be measured and analyzed at the flow level 1372 for a variety of purposes. As noted in RFC 2722, a flow measurement 1373 system can be very useful in the following contexts: 1375 o understanding the behavior of existing networks 1377 o planning for network development and expansion 1379 o quantification of network performance 1381 o verifying the quality of network service 1383 o attribution of network usage to users. 1385 A flow measurement system consists of meters, meter readers, and 1386 managers. A meter observes packets passing through a measurement 1387 point, classifies them into groups, accumulates usage data (such as 1388 the number of packets and bytes for each group), and stores the usage 1389 data in a flow table. A group may represent any collection of user 1390 applications, hosts, networks, etc. A meter reader gathers usage 1391 data from various meters so it can be made available for analysis. A 1392 manager is responsible for configuring and controlling meters and 1393 meter readers. The instructions received by a meter from a manager 1394 include flow specifications, meter control parameters, and sampling 1395 techniques. The instructions received by a meter reader from a 1396 manager include the address of the meter whose date is to be 1397 collected, the frequency of data collection, and the types of flows 1398 to be collected. 1400 4.1.10. Endpoint Congestion Management 1402 [RFC3124] provides a set of congestion control mechanisms for the use 1403 of transport protocols. It is also allows the development of 1404 mechanisms for unifying congestion control across a subset of an 1405 endpoint's active unicast connections (called a congestion group). A 1406 congestion manager continuously monitors the state of the path for 1407 each congestion group under its control. The manager uses that 1408 information to instruct a scheduler on how to partition bandwidth 1409 among the connections of that congestion group. 1411 4.1.11. TE Extensions to the IGPs 1413 [RFC5305] describes the extensions to the Intermediate System to 1414 Intermediate System (IS-IS) protocol to support TE, similarly 1415 [RFC3630] specifies TE extensions for OSPFv2 ([RFC5329] has the same 1416 description for OSPFv3). 1418 The idea of redistribution TE extensions such as link type and ID, 1419 local and remote IP addresses, TE metric, maximum bandwidth, maximum 1420 reservable bandwidth and unreserved bandwidth, admin group in IGP is 1421 a common for both IS-IS and OSPF. 1423 The difference is in the details of their transmission: IS-IS uses 1424 the Extended IS Reachability TLV (type 22) and Sub-TLVs for those TE 1425 parameters, OSPFv2 uses Opaque LSA [RFC5250] type 10 (OSPFv3 uses 1426 Intra-Area-TE-LSA) with two top-level TLV (Router Address and Link) 1427 also with Sub-TLVs for that purpose. 1429 IS-IS also uses the Extended IP Reachability TLV (type 135, which 1430 have the new 32 bit metric) and the TE Router ID TLV (type 134). 1431 Those Sub-TLV details are described in [RFC8570] for IS-IS and in 1432 [RFC7471] for OSPFv2 ([RFC5329] for OSPFv3). 1434 4.1.12. Link-State BGP 1436 In a number of environments, a component external to a network is 1437 called upon to perform computations based on the network topology and 1438 current state of the connections within the network, including 1439 traffic engineering information. This is information typically 1440 distributed by IGP routing protocols within the network (see 1441 Section 4.1.11. 1443 The Border Gateway Protocol (BGP) Section 7 is one of the essential 1444 routing protocols that glue the Internet together. BGP Link State 1445 (BGP-LS) [RFC7752] is a mechanism by which link-state and traffic 1446 engineering information can be collected from networks and shared 1447 with external components using the BGP routing protocol. The 1448 mechanism is applicable to physical and virtual IGP links, and is 1449 subject to policy control. 1451 Information collected by BGP-LS can be used to construct the Traffic 1452 Engineering Database (TED, see Section 4.1.20) for use by the Path 1453 Computation Element (PCE, see Section 4.1.13), or may be used by 1454 Application-Layer Traffic Optimization (ALTO) servers (see 1455 Section 4.1.15). 1457 4.1.13. Path Computation Element 1459 Constraint-based path computation is a fundamental building block for 1460 traffic engineering in MPLS and GMPLS networks. Path computation in 1461 large, multi-domain networks is complex and may require special 1462 computational components and cooperation between the elements in 1463 different domains. The Path Computation Element (PCE) [RFC4655] is 1464 an entity (component, application, or network node) that is capable 1465 of computing a network path or route based on a network graph and 1466 applying computational constraints. 1468 Thus, a PCE can provide a central component in a traffic engineering 1469 system operating on the Traffic Engineering Database (TED, see 1470 Section 4.1.20) with delegated responsibility for determining paths 1471 in MPLS, GMPLS, or Segment Routing networks. The PCE uses the Path 1472 Computation Element Communication Protocol (PCEP) [RFC5440] to 1473 communicate with Path Computation Clients (PCCs), such as MPLS LSRs, 1474 to answer their requests for computed paths or to instruct them to 1475 initiate new paths [RFC8281] and maintain state about paths already 1476 installed in the network [RFC8231]. 1478 PCEs form key components of a number of traffic engineering systems. 1479 More information about the applicability of PCE can be found in 1480 [RFC8051], while [RFC6805] describes the application of PCE to 1481 determining paths across multiple domains. PCE also has potential 1482 use in Abstraction and Control of TE Networks (ACTN) (see 1483 Section 4.1.17), Centralized Network Control [RFC8283], and Software 1484 Defined Networking (SDN) (see Section 5.3.2). 1486 4.1.14. Multi-Layer Traffic Engineering 1488 Networks are often arranged as layers. A layer relationship may 1489 represent the interaction between technologies (for example, an IP 1490 network operated over an optical network), or the relationship 1491 between different network operators (for example, a customer network 1492 operated over a service provider's network). Note that a multi-layer 1493 network does not imply the use of multiple technologies, although 1494 some form of encapsulation is often applied. 1496 Multi-layer traffic engineering presents a number of challenges 1497 associated with scalability and confidentiality. These issues are 1498 addressed in [RFC7926] which discusses the sharing of information 1499 between domains through policy filters, aggregation, abstraction, and 1500 virtualization. That document also discusses how existing protocols 1501 can support this scenario with special reference to BGP-LS (see 1502 Section 4.1.12). 1504 PCE (see Section 4.1.13) is also a useful tool for multi-layer 1505 networks as described in [RFC6805] and [RFC8685]. Signaling 1506 techniques for multi-layer traffic engineering are described in 1507 [RFC6107]. 1509 See also Appendix A.3.1 for a discussion of how the overlay model has 1510 been important in the development of traffic engineering. 1512 4.1.15. Application-Layer Traffic Optimization 1514 This document describes various TE mechanisms available in the 1515 network. However, distributed applications in general and, in 1516 particular, bandwidth-greedy P2P applications that are used, for 1517 example, for file sharing, cannot directly use those techniques. As 1518 per [RFC5693], applications could greatly improve traffic 1519 distribution and quality by cooperating with external services that 1520 are aware of the network topology. Addressing the Application-Layer 1521 Traffic Optimization (ALTO) problem means, on the one hand, deploying 1522 an ALTO service to provide applications with information regarding 1523 the underlying network (e.g., basic network location structure and 1524 preferences of network paths) and, on the other hand, enhancing 1525 applications in order to use such information to perform better-than- 1526 random selection of the endpoints with which they establish 1527 connections. 1529 The basic function of ALTO is based on abstract maps of a network. 1530 These maps provide a simplified view, yet enough information about a 1531 network for applications to effectively utilize them. Additional 1532 services are built on top of the maps. [RFC7285] describes a 1533 protocol implementing the ALTO services as an information-publishing 1534 interface that allows a network to publish its network information 1535 such as network locations, costs between them at configurable 1536 granularities, and end-host properties to network applications. The 1537 information published by the ALTO Protocol should benefit both the 1538 network and the applications. The ALTO Protocol uses a REST-ful 1539 design and encodes its requests and responses using JSON [RFC8259] 1540 with a modular design by dividing ALTO information publication into 1541 multiple ALTO services (e.g., the Map service, the Map-Filtering 1542 Service, the Endpoint Property Service, and the Endpoint Cost 1543 Service). 1545 [RFC8189] defines a new service that allows an ALTO Client to 1546 retrieve several cost metrics in a single request for an ALTO 1547 filtered cost map and endpoint cost map. [RFC8896] extends the ALTO 1548 cost information service so that applications decide not only 'where' 1549 to connect, but also 'when'. This is useful for applications that 1550 need to perform bulk data transfer and would like to schedule these 1551 transfers during an off-peak hour, for example. 1553 [I-D.ietf-alto-performance-metrics] introducing network performance 1554 metrics, including network delay, jitter, packet loss rate, hop 1555 count, and bandwidth. The ALTO server may derive and aggregate such 1556 performance metrics from BGP-LS (see Section 4.1.12) or IGP-TE (see 1557 Section 4.1.11), or management tools, and then expose the information 1558 to allow applications to determine 'where' to connect based on 1559 network performance criteria. ALTO WG is evaluating the use of 1560 network TE properties while making application decisions for new use- 1561 cases such as Edge computing and Datacenter interconnect. 1563 4.1.16. Segment Routing with MPLS Encapsulation (SR-MPLS) 1565 Segment Routing (SR) [RFC8402] leverages the source routing and 1566 tunneling paradigms. The path a packet takes is defined at the 1567 ingress and the packet is tunneled to the egress. A node steers a 1568 packet through a controlled set of instructions, called segments, by 1569 prepending the packet with an SR header: a label stack in MPLS case. 1571 A segment can represent any instruction, topological or service- 1572 based, thanks to the MPLS architecture [RFC3031]. Labels can be 1573 looked up in a global context (platform wide) as well as in some 1574 other context (see "context labels" in Section 3 of [RFC5331]). 1576 4.1.16.1. Base Segment Routing Identifier Types 1578 Segments are identified by Segment Identifiers (SIDs). There are 1579 four types of SID that are relevant for traffic engineering. 1581 Prefix SID: Uses the SR Global Block (SRGB), must be unique within 1582 the routing domain SRGB, and is advertised by an IGP. The Prefix- 1583 SID can be configured as an absolute value or an index. 1585 Node SID: A Prefix SID with the 'N' (node) bit set. It is 1586 associated with a host prefix (/32 or /128) that identifies the 1587 node. More than 1 Node SID can be configured per node. 1589 Adjacency SID: Locally significant by default, an Adjacency SID can 1590 be made globally significant through use of the 'L' flag. It 1591 identifies a unidirectional adjacency. In most implementations 1592 Adjacency SIDs are automatically allocated for each adjacency. 1593 They are always encoded as an absolute (not indexed) value. 1595 Binding SID: A Binding SID has two purposes: 1597 1. Mapping Server in ISIS 1599 The SID/Label Binding TLV is used to advertise the mappings 1600 of prefixes to SIDs/Labels. This functionality is called 1601 the Segment Routing Mapping Server (SRMS). The behavior of 1602 the SRMS is defined in [RFC8661] 1604 2. Cross-connect (label to FEC mapping) 1606 This is fundamental for multi-domain/multi-layer operation. 1607 The Binding SID identifies a new path available at the 1608 anchor point. It is always local to the originator, must 1609 not be present at the top of the stack, and must be looked 1610 up in the context of the Node SID. It could be provisioned 1611 through Netconf/Restconf, PCEP, BGP, or the CLI. 1613 4.1.16.2. Segment Routing Policy 1615 SR Policy [I-D.ietf-spring-segment-routing-policy] is an evolution of 1616 Segment Routing to enhance the TE capabilities. It is a framework 1617 that enables instantiation of an ordered list of segments on a node 1618 for implementing a source routing policy with a specific intent for 1619 traffic steering from that node. 1621 An SR Policy is identified through the tuple . The headend is the IP address of the node where the 1623 policy is instantiated. The endpoint is the IP address of the 1624 destination of the policy. The color is an index that associates the 1625 SR Policy with an intent (e.g., low-latency). 1627 The headend node is notified of SR Policies and associated SR paths 1628 via configuration or by a extensions to protocols such as PCEP 1629 [RFC8664] or BGP [I-D.ietf-idr-segment-routing-te-policy]. Each SR 1630 path consists of a Segment-List (an SR source-routed path), and the 1631 headend uses the endpoint and color parameters to classify packets to 1632 match the SR policy and so determine along which path to forward 1633 them. If an SR Policy is associated with a set of SR paths, each is 1634 associated with a weight for weighted load balancing. Furthermore, 1635 multiple SR Policies may be associated with a set of SR paths to 1636 allow multiple traffic flows to be placed on the same paths. 1638 An SR Binding SID (BSID) are also be associated with each candidate 1639 path associated with an SR Policy, or with the SR Policy itself. The 1640 headend node installs a BSID-keyed entry in the forwarding plane and 1641 assigns it the action of steering packets that match the entry to the 1642 selected path of the SR Policy. This steering can be done in various 1643 ways: 1645 o SID Steering: Incoming packets have an active SID matching a local 1646 BSID at the headend. 1648 o Per-destination Steering: Incoming packets match a BGP/Service 1649 route which indicates an SR Policy. 1651 o Per-flow Steering: Incoming packets match a forwarding array (for 1652 example, the classic 5-tuple) which indicates an SR Policies. 1654 o Policy-based Steering: Incoming packets match a routing policy 1655 which directs them to an SR Policy. 1657 4.1.17. Network Virtualization and Abstraction 1659 One of the main drivers for Software Defined Networking (SDN) 1660 [RFC7149] is a decoupling of the network control plane from the data 1661 plane. This separation has been achieved for TE networks with the 1662 development of MPLS/GMPLS (see Section 4.1.6 and Section 4.1.7) and 1663 the Path Computation Element (PCE) (Section 4.1.13). One of the 1664 advantages of SDN is its logically centralized control regime that 1665 allows a global view of the underlying networks. Centralized control 1666 in SDN helps improve network resource utilization compared with 1667 distributed network control. 1669 Abstraction and Control of TE Networks (ACTN) [RFC8453] defines a 1670 hierarchical SDN architecture which describes the functional entities 1671 and methods for the coordination of resources across multiple 1672 domains, to provide end-to-end traffic engineered services. ACTN 1673 facilitates end-to-end connections and provides them to the user. 1674 ACTN is focused on: 1676 o Abstraction of the underlying network resources and how they are 1677 provided to higher-layer applications and customers. 1679 o Virtualization of underlying resources for use by the customer, 1680 application, or service. The creation of a virtualized 1681 environment allows operators to view and control multi-domain 1682 networks as a single virtualized network. 1684 o Presentation to customers of networks as a virtual network via 1685 open and programmable interfaces. 1687 The ACTN managed infrastructure is built from traffic engineered 1688 network resources, which may include statistical packet bandwidth, 1689 physical forwarding plane sources (such as wavelengths and time 1690 slots), forwarding and cross-connect capabilities. The type of 1691 network virtualization seen in ACTN allows customers and applications 1692 (tenants) to utilize and independently control allocated virtual 1693 network resources as if resources as if they were physically their 1694 own resource. The ACTN network is "sliced", with tenants being given 1695 a different partial and abstracted topology view of the physical 1696 underlying network. 1698 4.1.18. Network Slicing 1700 An IETF Network Slice is a logical network topology connecting a 1701 number of endpoints using a set of shared or dedicated network 1702 resources [I-D.nsdt-teas-ietf-network-slice-definition]. The 1703 resources are used to satisfy specific Service Level Objectives 1704 (SLOs) specified by the consumer. 1706 IETF Network Slices are created and managed within the scope of one 1707 or more network technologies (e.g., IP, MPLS, optical). They are 1708 intended to enable a diverse set of applications that have different 1709 requirements to coexist on the same network infrastructure. IETF 1710 Network Slices are defined such that they are independent of the 1711 underlying infrastructure connectivity and technologies used. This 1712 is to allow an IETF Network Slice consumer to describe their network 1713 connectivity and relevant objectives in a common format, independent 1714 of the underlying technologies used. 1716 An IETF Network Slice is a well-defined composite of a set of 1717 endpoints, the connectivity requirements between subsets of these 1718 endpoints, and associated service requirements. The service 1719 requirements are expressed in terms of quantifiable characteristics 1720 or service level objectives (SLOs). SLOs along with terms Service 1721 Level Indicator (SLI) and Service Level Agreement (SLA) are used to 1722 define the performance of a service at different levels 1723 [I-D.nsdt-teas-ietf-network-slice-definition]. 1725 The concept of an IETF network slice is consistent with an enhanced 1726 VPN (VPN+) [I-D.ietf-teas-enhanced-vpn]. That is, from a consumer's 1727 perspective it looks like a VPN connectivity matrix with additional 1728 information about the level of service required between endpoints, 1729 while from an operator's perspective it looks like a set of routing 1730 or tunneling instructions with the network resource reservations 1731 necessary to provide the required service levels as specified by the 1732 SLOs. 1734 IETF network slices are not, of themselves, TE constructs. However, 1735 a network operator that offers IETF network slices is likely to use 1736 many TE tools in order to manage their network and provide the 1737 services. 1739 4.1.19. Deterministic Networking 1741 Deterministic Networking (DetNet) [RFC8655] is an architecture for 1742 applications with critical timing and reliability requirements. The 1743 layered architecture particularly focuses on developing DetNet 1744 service capabilities in the data plane [RFC8938]. The DetNet service 1745 sub-layer provides a set of Packet Replication, Elimination, and 1746 Ordering Functions (PREOF) functions to provide end-to-end service 1747 assurance. The DetNet forwarding sub-layer provides corresponding 1748 forwarding assurance (low packet loss, bounded latency, and in-order 1749 delivery) functions using resource allocations and explicit route 1750 mechanisms. 1752 The separation into two sub-layers allows a greater flexibility to 1753 adapt Detnet capability over a number of TE data plane mechanisms 1754 such as IP, MPLS, and Segment Routing. More importantly it 1755 interconnects IEEE 802.1 Time Sensitive Networking (TSN) 1756 [I-D.ietf-detnet-ip-over-tsn] deployed in Industry Control and 1757 Automation Systems (ICAS). 1759 DetNet can be seen as a specialized branch of TE, since it sets up 1760 explicit optimized paths with allocation of resources as requested. 1761 A DetNet application can express its QoS attributes or traffic 1762 behavior using any combination of DetNet functions described in sub- 1763 layers. They are then distributed and provisioned using well- 1764 established control and provisioning mechanisms adopted for traffic- 1765 engineering. 1767 In DetNet, a considerable state information is required to maintain 1768 per flow queuing disciplines and resource reservation for a large 1769 number of individual flows. This can be quite challenging for 1770 network operations during network events such as faults, change in 1771 traffic volume or re-provisioning. Therefore, DetNet recommends 1772 support for aggregated flows, however, it still requires large amount 1773 of control signaling to establish and maintain DetNet flows. 1775 4.1.20. Network TE State Definition and Presentation 1777 The network states that are relevant to the traffic engineering need 1778 to be stored in the system and presented to the user. The Traffic 1779 Engineering Database (TED) is a collection of all TE information 1780 about all TE nodes and TE links in the network, which is an essential 1781 component of a TE system, such as MPLS-TE [RFC2702] and GMPLS 1782 [RFC3945]. In order to formally define the data in the TED and to 1783 present the data to the user with high usability, the data modeling 1784 language YANG [RFC7950] can be used as described in [RFC8795]. 1786 4.1.21. System Management and Control Interfaces 1788 The traffic engineering control system needs to have a management 1789 interface that is human-friendly and a control interfaces that is 1790 programmable for automation. The Network Configuration Protocol 1791 (NETCONF) [RFC6241] or the RESTCONF Protocol [RFC8040] provide 1792 programmable interfaces that are also human-friendly. These 1793 protocols use XML or JSON encoded messages. When message compactness 1794 or protocol bandwidth consumption needs to be optimized for the 1795 control interface, other protocols, such as Group Communication for 1796 the Constrained Application Protocol (CoAP) [RFC7390] or gRPC, are 1797 available, especially when the protocol messages are encoded in a 1798 binary format. Along with any of these protocols, the data modeling 1799 language YANG [RFC7950] can be used to formally and precisely define 1800 the interface data. 1802 The Path Computation Element Communication Protocol (PCEP) [RFC5440] 1803 is another protocol that has evolved to be an option for the TE 1804 system control interface. The messages of PCEP are TLV-based, not 1805 defined by a data modeling language such as YANG. 1807 4.2. Content Distribution 1809 The Internet is dominated by client-server interactions, principally 1810 Web traffic although in the future, more sophisticated media servers 1811 may become dominant. The location and performance of major 1812 information servers has a significant impact on the traffic patterns 1813 within the Internet as well as on the perception of service quality 1814 by end users. 1816 A number of dynamic load balancing techniques have been devised to 1817 improve the performance of replicated information servers. These 1818 techniques can cause spatial traffic characteristics to become more 1819 dynamic in the Internet because information servers can be 1820 dynamically picked based upon the location of the clients, the 1821 location of the servers, the relative utilization of the servers, the 1822 relative performance of different networks, and the relative 1823 performance of different parts of a network. This process of 1824 assignment of distributed servers to clients is called traffic 1825 directing. It is an application layer function. 1827 Traffic directing schemes that allocate servers in multiple 1828 geographically dispersed locations to clients may require empirical 1829 network performance statistics to make more effective decisions. In 1830 the future, network measurement systems may need to provide this type 1831 of information. 1833 When congestion exists in the network, traffic directing and traffic 1834 engineering systems should act in a coordinated manner. This topic 1835 is for further study. 1837 The issues related to location and replication of information 1838 servers, particularly web servers, are important for Internet traffic 1839 engineering because these servers contribute a substantial proportion 1840 of Internet traffic. 1842 5. Taxonomy of Traffic Engineering Systems 1844 This section presents a short taxonomy of traffic engineering systems 1845 constructed based on traffic engineering styles and views as listed 1846 below and described in greater detail in the following subsections of 1847 this document. 1849 o Time-dependent versus State-dependent versus Event-dependent 1851 o Offline versus Online 1853 o Centralized versus Distributed 1855 o Local versus Global Information 1857 o Prescriptive versus Descriptive 1859 o Open Loop versus Closed Loop 1861 o Tactical versus Strategic 1863 5.1. Time-Dependent Versus State-Dependent Versus Event-Dependent 1865 Traffic engineering methodologies can be classified as time- 1866 dependent, state-dependent, or event-dependent. All TE schemes are 1867 considered to be dynamic in this document. Static TE implies that no 1868 traffic engineering methodology or algorithm is being applied - it is 1869 a feature of network planning, but lacks the reactive and flexible 1870 nature of traffic engineering. 1872 In time-dependent TE, historical information based on periodic 1873 variations in traffic (such as time of day) is used to pre-program 1874 routing and other TE control mechanisms. Additionally, customer 1875 subscription or traffic projection may be used. Pre-programmed 1876 routing plans typically change on a relatively long time scale (e.g., 1877 daily). Time-dependent algorithms do not attempt to adapt to short- 1878 term variations in traffic or changing network conditions. An 1879 example of a time-dependent algorithm is a global centralized 1880 optimizer where the input to the system is a traffic matrix and 1881 multi-class QoS requirements as described [MR99]. Another example of 1882 such a methodology is the application of data mining to Internet 1883 traffic [AJ19] which enables the use of various machine learning 1884 algorithms to identify patterns within historically collected 1885 datasets about Internet traffic, and to extract information in order 1886 to guide decision-making, and to improve efficiency and productivity 1887 of operational processes. 1889 State-dependent TE adapts the routing plans based on the current 1890 state of the network which provides additional information on 1891 variations in actual traffic (i.e., perturbations from regular 1892 variations) that could not be predicted using historical information. 1893 Constraint-based routing is an example of state-dependent TE 1894 operating in a relatively long time scale. An example operating in a 1895 relatively short timescale is a load-balancing algorithm described in 1896 [MATE]. The state of the network can be based on parameters flooded 1897 by the routers. Another approach is for a particular router 1898 performing adaptive TE to send probe packets along a path to gather 1899 the state of that path. [RFC6374] defines protocol extensions to 1900 collect performance measurements from MPLS networks. Another 1901 approach is for a management system to gather the relevant 1902 information directly from network elements using telemetry data 1903 collection "publication/subscription" techniques [RFC7923]. Timely 1904 gathering and distribution of state information is critical for 1905 adaptive TE. While time-dependent algorithms are suitable for 1906 predictable traffic variations, state-dependent algorithms may be 1907 applied to increase network efficiency and resilience to adapt to the 1908 prevailing network state. 1910 Event-dependent TE methods can also be used for TE path selection. 1911 Event-dependent TE methods are distinct from time-dependent and 1912 state-dependent TE methods in the manner in which paths are selected. 1913 These algorithms are adaptive and distributed in nature and typically 1914 use learning models to find good paths for TE in a network. While 1915 state-dependent TE models typically use available-link-bandwidth 1916 (ALB) flooding for TE path selection, event-dependent TE methods do 1917 not require ALB flooding. Rather, event-dependent TE methods 1918 typically search out capacity by learning models, as in the success- 1919 to-the-top (STT) method. ALB flooding can be resource intensive, 1920 since it requires link bandwidth to carry LSAs, processor capacity to 1921 process LSAs, and the overhead can limit area/Autonomous System (AS) 1922 size. Modeling results suggest that event-dependent TE methods could 1923 lead to a reduction in ALB flooding overhead without loss of network 1924 throughput performance [I-D.ietf-tewg-qos-routing]. 1926 5.2. Offline Versus Online 1928 Traffic engineering requires the computation of routing plans. The 1929 computation may be performed offline or online. The computation can 1930 be done offline for scenarios where routing plans need not be 1931 executed in real-time. For example, routing plans computed from 1932 forecast information may be computed offline. Typically, offline 1933 computation is also used to perform extensive searches on multi- 1934 dimensional solution spaces. 1936 Online computation is required when the routing plans must adapt to 1937 changing network conditions as in state-dependent algorithms. Unlike 1938 offline computation (which can be computationally demanding), online 1939 computation is geared toward relative simple and fast calculations to 1940 select routes, fine-tune the allocations of resources, and perform 1941 load balancing. 1943 5.3. Centralized Versus Distributed 1945 Under centralized control there is a central authority which 1946 determines routing plans and perhaps other TE control parameters on 1947 behalf of each router. The central authority periodically collects 1948 network-state information from all routers, and sends routing 1949 information to the routers. The update cycle for information 1950 exchange in both directions is a critical parameter directly 1951 impacting the performance of the network being controlled. 1952 Centralized control may need high processing power and high bandwidth 1953 control channels. 1955 Distributed control determines route selection by each router 1956 autonomously based on the router's view of the state of the network. 1957 The network state information may be obtained by the router using a 1958 probing method or distributed by other routers on a periodic basis 1959 using link state advertisements. Network state information may also 1960 be disseminated under exception conditions. Examples of protocol 1961 extensions used to advertise network link state information are 1962 defined in [RFC5305], [RFC6119], [RFC7471], [RFC8570], and [RFC8571]. 1963 See also Section 4.1.11. 1965 5.3.1. Hybrid Systems 1967 In practice, most TE systems will be a hybrid of central and 1968 distributed control. For example, a popular MPLS approach to TE is 1969 to use a central controller based on an active, stateful PCE, but to 1970 use routing and signaling protocols to make local decisions at 1971 routers within the network. Local decisions may be able to respond 1972 more quickly to network events, but may result in conflicts with 1973 decisions made by other routers. 1975 Network operations for TE systems may also use a hybrid of offline 1976 and online computation. TE paths may be precomputed based on stable- 1977 state network information and planned traffic demands, but may then 1978 be modified in the active network depending on variations in network 1979 state and traffic load. Furthermore, responses to network events may 1980 be precomputed offline to allow rapid reactions without further 1981 computation, or may be derived online depending on the nature of the 1982 events. 1984 Lastly, note that a fully functional TE system is likely to use all 1985 aspects of time-dependent, state-dependent, and event-dependent 1986 methodologies as described in Section 5.1. 1988 5.3.2. Considerations for Software Defined Networking 1990 As discussed in Section 4.1.17, one of the main drivers for SDN is a 1991 decoupling of the network control plane from the data plane 1992 [RFC7149]. However, SDN may also combine centralized control of 1993 resources, and facilitate application-to-network interaction via an 1994 application programming interface (API) such as [RFC8040]. Combining 1995 these features provides a flexible network architecture that can 1996 adapt to network requirements of a variety of higher-layer 1997 applications, a concept often referred to as the "programmable 1998 network" [RFC7426]. 2000 The centralized control aspect of SDN helps improve global network 2001 resource utilization compared with distributed network control, where 2002 local policy may often override global optimization goals. In an SDN 2003 environment, the data plane forwards traffic to its desired 2004 destination. However, before traffic reaches the data plane, the 2005 logically centralized SDN control plane often determines the end-to- 2006 end path the application traffic will take in the network. 2007 Therefore, the SDN control plane needs to be aware of the underlying 2008 network topology, capabilities and current node and link resource 2009 state. 2011 Using a PCE-based SDN control framework [RFC7491], the available 2012 network topology may be discovered by running a passive instance of 2013 OSPF or IS-IS, or via BGP-LS [RFC7752], to generate a TED (see 2014 Section 4.1.20). The PCE is used to compute a path (see 2015 Section 4.1.13) based on the TED and available bandwidth, and further 2016 path optimization may be based on requested objective functions 2017 [RFC5541]. When a suitable path has been computed the programming of 2018 the explicit network path may be performed using either end-to-end 2019 signaling protocol [RFC3209] or per-hop with each node being directly 2020 programmed [RFC8283] by the SDN controller. 2022 By utilizing a centralized approach to network control, additional 2023 network benefits are also available, including Global Concurrent 2024 Optimization (GCO) [RFC5557]. A GCO path computation request will 2025 simultaneously use the network topology and set of new end-to-end 2026 path requests, along with their respective constraints, for optimal 2027 placement in the network. Correspondingly, a GCO-based computation 2028 may be applied to recompute existing network paths to groom traffic 2029 and to mitigate congestion. 2031 5.4. Local Versus Global 2033 Traffic engineering algorithms may require local and global network- 2034 state information. 2036 Local information is the state of a portion of the domain. Examples 2037 include the bandwidth and packet loss rate of a particular path, or 2038 the state and capabilities of a network link. Local state 2039 information may be sufficient for certain instances of distributed 2040 control TE. 2042 Global information is the state of the entire TE domain. Examples 2043 include a global traffic matrix, and loading information on each link 2044 throughout the domain of interest. Global state information is 2045 typically required with centralized control. Distributed TE systems 2046 may also need global information in some cases. 2048 5.5. Prescriptive Versus Descriptive 2050 TE systems may also be classified as prescriptive or descriptive. 2052 Prescriptive traffic engineering evaluates alternatives and 2053 recommends a course of action. Prescriptive traffic engineering can 2054 be further categorized as either corrective or perfective. 2055 Corrective TE prescribes a course of action to address an existing or 2056 predicted anomaly. Perfective TE prescribes a course of action to 2057 evolve and improve network performance even when no anomalies are 2058 evident. 2060 Descriptive traffic engineering, on the other hand, characterizes the 2061 state of the network and assesses the impact of various policies 2062 without recommending any particular course of action. 2064 5.5.1. Intent-Based Networking 2066 Intent is defined in [I-D.irtf-nmrg-ibn-concepts-definitions] as a 2067 set of operational goals (that a network should meet) and outcomes 2068 (that a network is supposed to deliver), defined in a declarative 2069 manner without specifying how to achieve or implement them. This 2070 definition is based on [RFC7575] where, in the context of Autonomic 2071 Networks, it is described as "an abstract, high-level policy used to 2072 operate a network." 2074 Thus, intent-based management or Intent-Based Networking (IBN) is the 2075 concept of operating a network based on the concept of intent. 2077 Intent-Based Networking aims to produce networks that are simpler to 2078 manage and operate, requiring only minimal intervention. Networks 2079 have no way of automatically knowing operational goals nor which 2080 instances of networking services to support, thus the operator's 2081 intent needs to be communicated to the network. 2083 More specifically, intent is a declaration of operational goals that 2084 a network should meet and outcomes that the network is supposed to 2085 deliver, without specifying how to achieve them. Those goals and 2086 outcomes are defined in a purely declarative way: they specify what 2087 to accomplish, not how to achieve it. Intent applies two concepts: 2089 o It provides data abstraction: users and operators do not need to 2090 be concerned with low-level device configuration. 2092 o It provides functional abstraction: users and operators do not 2093 need to be concerned with how to achieve a given intent. What is 2094 specified is the desired outcome which is converted by the 2095 management system into the actions that will achieve the outcome. 2097 Intent-Based Networking is applicable to traffic engineering because 2098 many of the high-level objectives may be expressed as "intent." For 2099 example, load balancing, delivery of services, and robustness against 2100 failures. The intent is converted by the management system into 2101 traffic engineering actions within the network. 2103 5.6. Open-Loop Versus Closed-Loop 2105 Open-loop traffic engineering control is where control action does 2106 not use feedback information from the current network state. The 2107 control action may use its own local information for accounting 2108 purposes, however. 2110 Closed-loop traffic engineering control is where control action 2111 utilizes feedback information from the network state. The feedback 2112 information may be in the form of historical information or current 2113 measurement. 2115 5.7. Tactical versus Strategic 2117 Tactical traffic engineering aims to address specific performance 2118 problems (such as hot-spots) that occur in the network from a 2119 tactical perspective, without consideration of overall strategic 2120 imperatives. Without proper planning and insights, tactical TE tends 2121 to be ad hoc in nature. 2123 Strategic traffic engineering approaches the TE problem from a more 2124 organized and systematic perspective, taking into consideration the 2125 immediate and longer term consequences of specific policies and 2126 actions. 2128 6. Recommendations for Internet Traffic Engineering 2130 This section describes high-level recommendations for traffic 2131 engineering in the Internet in general terms. 2133 The recommendations describe the capabilities needed to solve a 2134 traffic engineering problem or to achieve a traffic engineering 2135 objective. Broadly speaking, these recommendations can be 2136 categorized as either functional or non-functional recommendations. 2138 o Functional recommendations describe the functions that a traffic 2139 engineering system should perform. These functions are needed to 2140 realize traffic engineering objectives by addressing traffic 2141 engineering problems. 2143 o Non-functional recommendations relate to the quality attributes or 2144 state characteristics of a traffic engineering system. These 2145 recommendations may contain conflicting assertions and may 2146 sometimes be difficult to quantify precisely. 2148 6.1. Generic Non-functional Recommendations 2150 The generic non-functional recommendations for Internet traffic 2151 engineering are listed in the paragraphs that follow. In a given 2152 context, some of these recommendations may be critical while others 2153 may be optional. Therefore, prioritization may be required during 2154 the development phase of a traffic engineering system to tailor it to 2155 a specific operational context. 2157 Usability: Usability is a human aspect of traffic engineering 2158 systems. It refers to the ease with which a traffic engineering 2159 system can be deployed and operated. In general, it is desirable 2160 to have a TE system that can be readily deployed in an existing 2161 network. It is also desirable to have a TE system that is easy to 2162 operate and maintain. 2164 Automation: Whenever feasible, a TE system should automate as many 2165 TE functions as possible to minimize the amount of human effort 2166 needed to analyze and control operational networks. Automation is 2167 particularly important in large-scale public networks because of 2168 the high cost of the human aspects of network operations and the 2169 high risk of network problems caused by human errors. Automation 2170 may entail the incorporation of automatic feedback and 2171 intelligence into some components of the TE system. 2173 Scalability: Public networks continue to grow rapidly with respect 2174 to network size and traffic volume. Therefore, to remain 2175 applicable as the network evolves, a TE system should be scalable. 2176 In particular, a TE system should remain functional as the network 2177 expands with regard to the number of routers and links, and with 2178 respect to the traffic volume. A TE system should have a scalable 2179 architecture, should not adversely impair other functions and 2180 processes in a network element, and should not consume too many 2181 network resources when collecting and distributing state 2182 information, or when exerting control. 2184 Stability: Stability is a very important consideration in TE systems 2185 that respond to changes in the state of the network. State- 2186 dependent TE methodologies typically include a trade-off between 2187 responsiveness and stability. It is strongly recommended that 2188 when a trade-off between responsiveness and stability is needed, 2189 it should be made in favor of stability (especially in public IP 2190 backbone networks). 2192 Flexibility: A TE system should allow for changes in optimization 2193 policy. In particular, a TE system should provide sufficient 2194 configuration options so that a network administrator can tailor 2195 the system to a particular environment. It may also be desirable 2196 to have both online and offline TE subsystems which can be 2197 independently enabled and disabled. TE systems that are used in 2198 multi-class networks should also have options to support class 2199 based performance evaluation and optimization. 2201 Visibility: Mechanisms should exist as part of the TE system to 2202 collect statistics from the network and to analyze these 2203 statistics to determine how well the network is functioning. 2204 Derived statistics such as traffic matrices, link utilization, 2205 latency, packet loss, and other performance measures of interest 2206 which are determined from network measurements can be used as 2207 indicators of prevailing network conditions. The capabilities of 2208 the various components of the routing system are other examples of 2209 status information which should be observable. 2211 Simplicity: A TE system should be as simple as possible and easy to 2212 use (i.e., have clean, convenient, and intuitive user interfaces). 2213 Simplicity in user interface does not necessarily imply that the 2214 TE system will use naive algorithms. When complex algorithms and 2215 internal structures are used, the user interface should hide such 2216 complexities from the network administrator as much as possible. 2218 Interoperability: Whenever feasible, TE systems and their components 2219 should be developed with open standards-based interfaces to allow 2220 interoperation with other systems and components. 2222 Security: Security is a critical consideration in TE systems. Such 2223 systems typically exert control over functional aspects of the 2224 network to achieve the desired performance objectives. Therefore, 2225 adequate measures must be taken to safeguard the integrity of the 2226 TE system. Adequate measures must also be taken to protect the 2227 network from vulnerabilities that originate from security breaches 2228 and other impairments within the TE system. 2230 The remaining subsections of this section focus on some of the high- 2231 level functional recommendations for traffic engineering. 2233 6.2. Routing Recommendations 2235 Routing control is a significant aspect of Internet traffic 2236 engineering. Routing impacts many of the key performance measures 2237 associated with networks, such as throughput, delay, and utilization. 2238 Generally, it is very difficult to provide good service quality in a 2239 wide area network without effective routing control. A desirable TE 2240 routing system is one that takes traffic characteristics and network 2241 constraints into account during route selection while maintaining 2242 stability. 2244 Shortest path first (SPF) IGPs are based on shortest path algorithms 2245 and have limited control capabilities for TE [RFC2702], [AWD2]. 2246 These limitations include: 2248 1. Pure SPF protocols do not take network constraints and traffic 2249 characteristics into account during route selection. For 2250 example, IGPs always select the shortest paths based on link 2251 metrics assigned by administrators) so load sharing cannot be 2252 performed across paths of different costs. Using shortest paths 2253 to forward traffic may cause the following problems: 2255 * If traffic from a source to a destination exceeds the capacity 2256 of a link along the shortest path, the link (and hence the 2257 shortest path) becomes congested while a longer path between 2258 these two nodes may be under-utilized 2260 * The shortest paths from different sources can overlap at some 2261 links. If the total traffic from the sources exceeds the 2262 capacity of any of these links, congestion will occur. 2264 * Problems can also occur because traffic demand changes over 2265 time, but network topology and routing configuration cannot be 2266 changed as rapidly. This causes the network topology and 2267 routing configuration to become sub-optimal over time, which 2268 may result in persistent congestion problems. 2270 2. The Equal-Cost Multi-Path (ECMP) capability of SPF IGPs supports 2271 sharing of traffic among equal cost paths between two nodes. 2272 However, ECMP attempts to divide the traffic as equally as 2273 possible among the equal cost shortest paths. Generally, ECMP 2274 does not support configurable load sharing ratios among equal 2275 cost paths. The result is that one of the paths may carry 2276 significantly more traffic than other paths because it may also 2277 carry traffic from other sources. This situation can result in 2278 congestion along the path that carries more traffic. Weighted 2279 ECMP (WECMP) (see, for example, [I-D.ietf-bess-evpn-unequal-lb]) 2280 provides some mitigation. 2282 3. Modifying IGP metrics to control traffic routing tends to have 2283 network-wide effects. Consequently, undesirable and 2284 unanticipated traffic shifts can be triggered as a result. Work 2285 described in Section 8 may be capable of better control [FT00], 2286 [FT01]. 2288 Because of these limitations, new capabilities are needed to enhance 2289 the routing function in IP networks. Some of these capabilities are 2290 summarized below. 2292 o Constraint-based routing computes routes to fulfill requirements 2293 subject to constraints. This can be useful in public IP backbones 2294 with complex topologies. Constraints may include bandwidth, hop 2295 count, delay, and administrative policy instruments such as 2296 resource class attributes [RFC2702], [RFC2386]. This makes it 2297 possible to select routes that satisfy a given set of 2298 requirements. Routes computed by constraint-based routing are not 2299 necessarily the shortest paths. Constraint-based routing works 2300 best with path-oriented technologies that support explicit 2301 routing, such as MPLS. 2303 Constraint-based routing can also be used as a way to distribute 2304 traffic onto the infrastructure, including for best effort 2305 traffic. For example, congestion problems caused by uneven 2306 traffic distribution may be avoided or reduced by knowing the 2307 reservable bandwidth attributes of the network links and by 2308 specifying the bandwidth requirements for path selection. 2310 o A number of enhancements to the link state IGPs are needed to 2311 allow them to distribute additional state information required for 2312 constraint-based routing. The extensions to OSPF are described in 2313 [RFC3630], and to IS-IS in [RFC5305]. Some of the additional 2314 topology state information includes link attributes such as 2315 reservable bandwidth and link resource class attribute (an 2316 administratively specified property of the link). The resource 2317 class attribute concept is defined in [RFC2702]. The additional 2318 topology state information is carried in new TLVs and sub-TLVs in 2319 IS-IS, or in the Opaque LSA in OSPF [RFC5305], [RFC3630]. 2321 An enhanced link-state IGP may flood information more frequently 2322 than a normal IGP. This is because even without changes in 2323 topology, changes in reservable bandwidth or link affinity can 2324 trigger the enhanced IGP to initiate flooding. A trade-off 2325 between the timeliness of the information flooded and the flooding 2326 frequency is typically implemented using a threshold based on the 2327 percentage change of the advertised resources to avoid excessive 2328 consumption of link bandwidth and computational resources, and to 2329 avoid instability in the TED. 2331 o In a TE system, it is also desirable for the routing subsystem to 2332 make the load splitting ratio among multiple paths (with equal 2333 cost or different cost) configurable. This capability gives 2334 network administrators more flexibility in the control of traffic 2335 distribution across the network. It can be very useful for 2336 avoiding/relieving congestion in certain situations. Examples can 2337 be found in [XIAO] and [I-D.ietf-bess-evpn-unequal-lb]. 2339 o The routing system should also have the capability to control the 2340 routes of subsets of traffic without affecting the routes of other 2341 traffic if sufficient resources exist for this purpose. This 2342 capability allows a more refined control over the distribution of 2343 traffic across the network. For example, the ability to move 2344 traffic away from its original path to another path (without 2345 affecting other traffic paths) allows the traffic to be moved from 2346 resource-poor network segments to resource-rich segments. Path 2347 oriented technologies such as MPLS-TE inherently support this 2348 capability as discussed in [AWD2]. 2350 o Additionally, the routing subsystem should be able to select 2351 different paths for different classes of traffic (or for different 2352 traffic behavior aggregates) if the network supports multiple 2353 classes of service (different behavior aggregates). 2355 6.3. Traffic Mapping Recommendations 2357 Traffic mapping is the assignment of traffic workload onto pre- 2358 established paths to meet certain requirements. Thus, while 2359 constraint-based routing deals with path selection, traffic mapping 2360 deals with the assignment of traffic to established paths which may 2361 have been generated by constraint-based routing or by some other 2362 means. Traffic mapping can be performed by time-dependent or state- 2363 dependent mechanisms, as described in Section 5.1. 2365 An important aspect of the traffic mapping function is the ability to 2366 establish multiple paths between an originating node and a 2367 destination node, and the capability to distribute the traffic 2368 between the two nodes across the paths according to some policies. A 2369 pre-condition for this scheme is the existence of flexible mechanisms 2370 to partition traffic and then assign the traffic partitions onto the 2371 parallel paths as noted in [RFC2702]. When traffic is assigned to 2372 multiple parallel paths, it is recommended that special care should 2373 be taken to ensure proper ordering of packets belonging to the same 2374 application (or micro-flow) at the destination node of the parallel 2375 paths. 2377 Mechanisms that perform the traffic mapping functions should aim to 2378 map the traffic onto the network infrastructure to minimize 2379 congestion. If the total traffic load cannot be accommodated, or if 2380 the routing and mapping functions cannot react fast enough to 2381 changing traffic conditions, then a traffic mapping system may use 2382 short time scale congestion control mechanisms (such as queue 2383 management, scheduling, etc.) to mitigate congestion. Thus, 2384 mechanisms that perform the traffic mapping functions complement 2385 existing congestion control mechanisms. In an operational network, 2386 traffic should be mapped onto the infrastructure such that intra- 2387 class and inter-class resource contention are minimized (see 2388 Section 2). 2390 When traffic mapping techniques that depend on dynamic state feedback 2391 (e.g., MATE [MATE] and such like) are used, special care must be 2392 taken to guarantee network stability. 2394 6.4. Measurement Recommendations 2396 The importance of measurement in traffic engineering has been 2397 discussed throughout this document. A TE system should include 2398 mechanisms to measure and collect statistics from the network to 2399 support the TE function. Additional capabilities may be needed to 2400 help in the analysis of the statistics. The actions of these 2401 mechanisms should not adversely affect the accuracy and integrity of 2402 the statistics collected. The mechanisms for statistical data 2403 acquisition should also be able to scale as the network evolves. 2405 Traffic statistics may be classified according to long-term or short- 2406 term timescales. Long-term traffic statistics are very useful for 2407 traffic engineering. Long-term traffic statistics may periodicity 2408 record network workload (such as hourly, daily, and weekly variations 2409 in traffic profiles) as well as traffic trends. Aspects of the 2410 traffic statistics may also describe class of service characteristics 2411 for a network supporting multiple classes of service. Analysis of 2412 the long-term traffic statistics may yield other information such as 2413 busy hour characteristics, traffic growth patterns, persistent 2414 congestion problems, hot-spot, and imbalances in link utilization 2415 caused by routing anomalies. 2417 A mechanism for constructing traffic matrices for both long-term and 2418 short-term traffic statistics should be in place. In multi-service 2419 IP networks, the traffic matrices may be constructed for different 2420 service classes. Each element of a traffic matrix represents a 2421 statistic about the traffic flow between a pair of abstract nodes. 2422 An abstract node may represent a router, a collection of routers, or 2423 a site in a VPN. 2425 Traffic statistics should provide reasonable and reliable indicators 2426 of the current state of the network on the short-term scale. Some 2427 short term traffic statistics may reflect link utilization and link 2428 congestion status. Examples of congestion indicators include 2429 excessive packet delay, packet loss, and high resource utilization. 2430 Examples of mechanisms for distributing this kind of information 2431 include SNMP, probing tools, FTP, IGP link state advertisements, and 2432 Netconf/Restconf, etc. 2434 6.5. Network Survivability 2436 Network survivability refers to the capability of a network to 2437 maintain service continuity in the presence of faults. This can be 2438 accomplished by promptly recovering from network impairments and 2439 maintaining the required QoS for existing services after recovery. 2440 Survivability is an issue of great concern within the Internet 2441 community due to the demand to carry mission critical traffic, real- 2442 time traffic, and other high priority traffic over the Internet. 2443 Survivability can be addressed at the device level by developing 2444 network elements that are more reliable; and at the network level by 2445 incorporating redundancy into the architecture, design, and operation 2446 of networks. It is recommended that a philosophy of robustness and 2447 survivability should be adopted in the architecture, design, and 2448 operation of traffic engineering that control IP networks (especially 2449 public IP networks). Because different contexts may demand different 2450 levels of survivability, the mechanisms developed to support network 2451 survivability should be flexible so that they can be tailored to 2452 different needs. A number of tools and techniques have been 2453 developed to enable network survivability including MPLS Fast Reroute 2454 [RFC4090], RSVP-TE Extensions in Support of End-to-End GMPLS Recovery 2455 [RFC4872], and GMPLS Segment Recovery [RFC4873]. 2457 The impact of service outages varies significantly for different 2458 service classes depending on the duration of the outage which can 2459 vary from milliseconds (with minor service impact) to seconds (with 2460 possible call drops for IP telephony and session time-outs for 2461 connection oriented transactions) to minutes and hours (with 2462 potentially considerable social and business impact). Different 2463 duration outages have different impacts depending on the throughput 2464 of the traffic flows that are interrupted. 2466 Failure protection and restoration capabilities are available in 2467 multiple layers as network technologies have continued to evolve. 2468 Optical networks are capable of providing dynamic ring and mesh 2469 restoration functionality at the wavelength level. At the SONET/SDH 2470 layer survivability capability is provided with Automatic Protection 2471 Switching (APS) as well as self-healing ring and mesh architectures. 2472 Similar functionality is provided by layer 2 technologies such as 2473 Ethernet. 2475 Rerouting is used at the IP layer to restore service following link 2476 and node outages. Rerouting at the IP layer occurs after a period of 2477 routing convergence which may require seconds to minutes to complete. 2478 Path-oriented technologies such a MPLS ([RFC3469]) can be used to 2479 enhance the survivability of IP networks in a potentially cost 2480 effective manner. 2482 An important of multi-layer survivability is that technologies at 2483 different layers may provide protection and restoration capabilities 2484 at different granularities in terms of time scales and at different 2485 bandwidth granularity (from packet-level to wavelength level). 2486 Protection and restoration capabilities can also be sensitive to 2487 different service classes and different network utility models. 2488 Coordinating different protection and restoration capabilities across 2489 multiple layers in a cohesive manner to ensure network survivability 2490 is maintained at reasonable cost is a challenging task. Protection 2491 and restoration coordination across layers may not always be 2492 feasible, because networks at different layers may belong to 2493 different administrative domains. 2495 The following paragraphs present some of the general recommendations 2496 for protection and restoration coordination. 2498 o Protection and restoration capabilities from different layers 2499 should be coordinated to provide network survivability in a 2500 flexible and cost effective manner. Avoiding duplication of 2501 functions in different layers is one way to achieve the 2502 coordination. Escalation of alarms and other fault indicators 2503 from lower to higher layers may also be performed in a coordinated 2504 manner. The order of timing of restoration triggers from 2505 different layers is another way to coordinate multi-layer 2506 protection/restoration. 2508 o Network capacity reserved in one layer to provide protection and 2509 restoration is not available to carry traffic in a higher layer: 2510 it is not visible as spare capacity in the higher layer. Placing 2511 protection/restoration functions in many layers may increase 2512 redundancy and robustness, but it can result in significant 2513 inefficiencies in network resource utilization. Careful planning 2514 is needed to balance the trade-off between the desire for 2515 survivablity and the optimal use of resources. 2517 o It is generally desirable to have protection and restoration 2518 schemes that are intrinsically bandwidth efficient. 2520 o Failure notifications throughout the network should be timely and 2521 reliable if they are to be acted on as triggers for effective 2522 protection and restoration actions. 2524 o Alarms and other fault monitoring and reporting capabilities 2525 should be provided at the right network layers so that the 2526 protection and restoration actions can be taken in those layers. 2528 6.5.1. Survivability in MPLS Based Networks 2530 Because MPLS is path-oriented, it has the potential to provide faster 2531 and more predictable protection and restoration capabilities than 2532 conventional hop by hop routed IP systems. Protection types for MPLS 2533 networks can be divided into four categories. 2535 o Link Protection: The objective of link protection is to protect an 2536 LSP from the failure of a given link. Under link protection, a 2537 protection or backup LSP (the secondary LSP) follows a path that 2538 is disjoint from the path of the working or operational LSP (the 2539 primary LSP) at the particular link where link protection is 2540 required. When the protected link fails, traffic on the working 2541 LSP is switched to the protection LSP at the head-end of the 2542 failed link. As a local repair method, link protection can be 2543 fast. This form of protection may be most appropriate in 2544 situations where some network elements along a given path are 2545 known to be less reliable than others. 2547 o Node Protection: The objective of node protection is to protect an 2548 LSP from the failure of a given node. Under node protection, the 2549 secondary LSP follows a path that is disjoint from the path of the 2550 primary LSP at the particular node where node protection is 2551 required. The secondary LSP is also disjoint from the primary LSP 2552 at all links attached to the node to be protected. When the 2553 protected node fails, traffic on the working LSP is switched over 2554 to the protection LSP at the upstream LSR directly connected to 2555 the failed node. Node protection covers a slightly larger part of 2556 the network compared to link protection, but is otherwise 2557 fundamentally the same. 2559 o Path Protection: The goal of LSP path protection (or end-to-end 2560 protection) is to protect an LSP from any failure along its routed 2561 path. Under path protection, the path of the protection LSP is 2562 completely disjoint from the path of the working LSP. The 2563 advantage of path protection is that the backup LSP protects the 2564 working LSP from all possible link and node failures along the 2565 path, except for failures of ingress or egress LSR. Additionally, 2566 path protection may be more efficient in terms of resource usage 2567 than link or node protection applied at every jop along the path. 2568 However, path protection may be slower than link and node 2569 protection because the fault notifications have to be propagated 2570 further. 2572 o Segment Protection: An MPLS domain may be partitioned into 2573 multiple subdomains (protection domains). Path protection is 2574 applied to the path of each LSP as it crosses the domain from its 2575 ingress to the domain to where it egresses the domain. In cases 2576 where an LSP traverses multiple protection domains, a protection 2577 mechanism within a domain only needs to protect the segment of the 2578 LSP that lies within the domain. Segment protection will 2579 generally be faster than end-to-end path protection because 2580 recovery generally occurs closer to the fault and the notification 2581 doesn't have to propagate as far. 2583 See [RFC3469] and [RFC6372] for a more comprehensive discussion of 2584 MPLS based recovery. 2586 6.5.2. Protection Options 2588 Another issue to consider is the concept of protection options. We 2589 use notation such as "m:n protection", where m is the number of 2590 protection LSPs used to protect n working LSPs. In all cases except 2591 1+1 protection, the resources associated with the protection LSPs can 2592 be used to carry preemptable best-effort traffic when the working LSP 2593 is functioning correctly. 2595 o 1:1 protection: One working LSP is protected/restored by one 2596 protection LSP. 2598 o 1:n protection: One protection LSP is used to protect/restore n 2599 working LSPs. Only one failed LSP can be restored at any time. 2601 o n:1 protection: One working LSP is protected/restored by n 2602 protection LSPs, possibly with load splitting across the 2603 protection LSPs. This may be especially useful when it is not 2604 feasible to find one path for the backup that can satisfy the 2605 bandwidth requirement of the primary LSP. 2607 o 1+1 protection: Traffic is sent concurrently on both the working 2608 LSP and a protection LSP. The egress LSR selects one of the two 2609 LSPs based on local policy (usually based on traffic integrity). 2610 When a fault disrupts the traffic on one LSP, the egress switches 2611 to receive traffic from the other LSP. This approach is expensive 2612 in how it consumes network but recovers from failures most 2613 rapidly. 2615 6.6. Traffic Engineering in Diffserv Environments 2617 Increasing requirements to support multiple classes of traffic in the 2618 Internet, such as best effort and mission critical data, calls for IP 2619 networks to differentiate traffic according to some criteria and to 2620 give preferential treatment to certain types of traffic. Large 2621 numbers of flows can be aggregated into a few behavior aggregates 2622 based on some criteria based on common performance requirements in 2623 terms of packet loss ratio, delay, and jitter, or in terms of common 2624 fields within the IP packet headers. 2626 Differentiated Services (Diffserv) [RFC2475] can be used to ensure 2627 that SLAs defined to differentiate between traffic flows are met. 2628 Classes of service (CoS) can be supported in a Diffserv environment 2629 by concatenating per-hop behaviors (PHBs) along the routing path. A 2630 PHB is the forwarding behavior that a packet receives at a Diffserv- 2631 compliant node, and it can be configured at each router. PHBs are 2632 delivered using buffer management and packet scheduling mechanisms 2633 and require that the ingress nodes use traffic classification, 2634 marking, policing, and shaping. 2636 Traffic engineering can compliment Diffserv to improve utilization of 2637 network resources. Traffic engineering can be operated on an 2638 aggregated basis across all service classes [RFC3270], or on a per 2639 service class basis. The former is used to provide better 2640 distribution of the traffic load over the network resources (see 2641 [RFC3270] for detailed mechanisms to support aggregate traffic 2642 engineering). The latter case is discussed below since it is 2643 specific to the Diffserv environment, with so called Diffserv-aware 2644 traffic engineering [RFC4124]. 2646 For some Diffserv networks, it may be desirable to control the 2647 performance of some service classes by enforcing relationships 2648 between the traffic workload contributed by each service class and 2649 the amount of network resources allocated or provisioned for that 2650 service class. Such relationships between demand and resource 2651 allocation can be enforced using a combination of, for example: 2653 o TE mechanisms on a per service class basis that enforce the 2654 relationship between the amount of traffic contributed by a given 2655 service class and the resources allocated to that class. 2657 o Mechanisms that dynamically adjust the resources allocated to a 2658 given service class to relate to the amount of traffic contributed 2659 by that service class. 2661 It may also be desirable to limit the performance impact of high 2662 priority traffic on relatively low priority traffic. This can be 2663 achieved, for example, by controlling the percentage of high priority 2664 traffic that is routed through a given link. Another way to 2665 accomplish this is to increase link capacities appropriately so that 2666 lower priority traffic can still enjoy adequate service quality. 2667 When the ratio of traffic workload contributed by different service 2668 classes varies significantly from router to router, it may not be 2669 enough to rely on conventional IGP routing protocols or on TE 2670 mechanisms that are not sensitive to different service classes. 2671 Instead, it may be desirable to perform traffic engineering, 2672 especially routing control and mapping functions, on a per service 2673 class basis. One way to accomplish this in a domain that supports 2674 both MPLS and Diffserv is to define class specific LSPs and to map 2675 traffic from each class onto one or more LSPs that correspond to that 2676 service class. An LSP corresponding to a given service class can 2677 then be routed and protected/restored in a class dependent manner, 2678 according to specific policies. 2680 Performing traffic engineering on a per class basis may require per- 2681 class parameters to be distributed. It is common to have some 2682 classes share some aggregate constraints (e.g., maximum bandwidth 2683 requirement) without enforcing the constraint on each individual 2684 class. These classes can be grouped into class-types, and per-class- 2685 type parameters can be distributed to improve scalability. This also 2686 allows better bandwidth sharing between classes in the same class- 2687 type. A class-type is a set of classes that satisfy the following 2688 two conditions: 2690 o Classes in the same class-type have common aggregate requirements 2691 to satisfy required performance levels. 2693 o There is no requirement to be enforced at the level of an 2694 individual class in the class-type. Note that it is, 2695 nevertheless, still possible to implement some priority policies 2696 for classes in the same class-type to permit preferential access 2697 to the class-type bandwidth through the use of preemption 2698 priorities. 2700 See [RFC4124] for detailed requirements on Diffserv-aware traffic 2701 engineering. 2703 6.7. Network Controllability 2705 Offline and online (see Section 5.2) TE considerations are of limited 2706 utility if the network cannot be controlled effectively to implement 2707 the results of TE decisions and to achieve the desired network 2708 performance objectives. 2710 Capacity augmentation is a coarse-grained solution to TE issues. 2711 However, it is simple and may be advantageous if bandwidth is 2712 abundant and cheap. However, bandwidth is not always abundant and 2713 cheap, and additional capacity might not always be the best solution. 2714 Adjustments of administrative weights and other parameters associated 2715 with routing protocols provide finer-grained control, but this 2716 approach is difficult to use and imprecise because of the the way the 2717 routing protocols interact occur across the network. 2719 Control mechanisms can be manual (e.g., static configuration), 2720 partially-automated (e.g., scripts), or fully-automated (e.g., policy 2721 based management systems). Automated mechanisms are particularly 2722 useful in large scale networks. Multi-vendor interoperability can be 2723 facilitated by standardized management systems (e.g., YANG models) to 2724 support the control functions required to address TE objectives. 2726 Network control functions should be secure, reliable, and stable as 2727 these are often needed to operate correctly in times of network 2728 impairments (e.g., during network congestion or security attacks). 2730 7. Inter-Domain Considerations 2732 Inter-domain TE is concerned with performance optimization for 2733 traffic that originates in one administrative domain and terminates 2734 in a different one. 2736 BGP [RFC4271] is the standard exterior gateway protocol used to 2737 exchange routing information between autonomous systems (ASes) in the 2738 Internet. BGP includes a sequential decision process that calculates 2739 the preference for routes to a given destination network. There are 2740 two fundamental aspects to inter-domain TE using BGP: 2742 o Route Redistribution: Controlling the import and export of routes 2743 between ASes, and controlling the redistribution of routes between 2744 BGP and other protocols within an AS. 2746 o Best path selection: Selecting the best path when there are 2747 multiple candidate paths to a given destination network. This is 2748 performed by the BGP decision process, selecting preferred exit 2749 points out of an AS towards specific destination networks taking a 2750 number of different considerations into account. The BGP path 2751 selection process can be influenced by manipulating the attributes 2752 associated with the process, including NEXT-HOP, WEIGHT, LOCAL- 2753 PREFERENCE, AS-PATH, ROUTE-ORIGIN, MULTI-EXIT-DESCRIMINATOR (MED), 2754 IGP METRIC, etc. 2756 Route-maps provide the flexibility to implement complex BGP policies 2757 based on pre-configured logical conditions. They can be used to 2758 control import and export policies for incoming and outgoing routes, 2759 control the redistribution of routes between BGP and other protocols, 2760 and influence the selection of best paths by manipulating the 2761 attributes associated with the BGP decision process. Very complex 2762 logical expressions that implement various types of policies can be 2763 implemented using a combination of Route-maps, BGP-attributes, 2764 Access-lists, and Community attributes. 2766 When considering inter-domain TE with BGP, note that the outbound 2767 traffic exit point is controllable, whereas the interconnection point 2768 where inbound traffic is received typically is not. Therefore, it is 2769 up to each individual network to implement TE strategies that deal 2770 with the efficient delivery of outbound traffic from its customers to 2771 its peering points. The vast majority of TE policy is based on a 2772 "closest exit" strategy, which offloads interdomain traffic at the 2773 nearest outbound peering point towards the destination AS. Most 2774 methods of manipulating the point at which inbound traffic enters a 2775 are either ineffective, or not accepted in the peering community. 2777 Inter-domain TE with BGP is generally effective, but it is usually 2778 applied in a trial-and-error fashion because a TE system usually only 2779 has a view of the available network resources within one domain (an 2780 AS in this case). A systematic approach for inter-domain TE requires 2781 cooperation between the domains. Further, what may be considered a 2782 good solution in one domain may not necessarily be a good solution in 2783 another. Moreover, it is generally considered inadvisable for one 2784 domain to permit a control process from another domain to influence 2785 the routing and management of traffic in its network. 2787 MPLS TE-tunnels (LSPs) can add a degree of flexibility in the 2788 selection of exit points for inter-domain routing by applying rhe 2789 concept of relative and absolute metrics. If BGP attributes are 2790 defined such that the BGP decision process depends on IGP metrics to 2791 select exit points for inter-domain traffic, then some inter-domain 2792 traffic destined to a given peer network can be made to prefer a 2793 specific exit point by establishing a TE-tunnel between the router 2794 making the selection and the peering point via a TE-tunnel and 2795 assigning the TE-tunnel a metric which is smaller than the IGP cost 2796 to all other peering points. 2798 Similarly to intra-domain TE, inter-domain TE is best accomplished 2799 when a traffic matrix can be derived to depict the volume of traffic 2800 from one AS to another. 2802 8. Overview of Contemporary TE Practices in Operational IP Networks 2804 This section provides an overview of some traffic engineering 2805 practices in IP networks. The focus is on aspects of control of the 2806 routing function in operational contexts. The intent here is to 2807 provide an overview of the commonly used practices: the discussion is 2808 not intended to be exhaustive. 2810 Service providers apply many of the traffic engineering mechanisms 2811 described in this document to optimize the performance of their IP 2812 networks. These techniques include capacity planning for long 2813 timescales; routing control using IGP metrics and MPLS, as well as 2814 path planning and path control using MPLS and Segment Routing for 2815 medium timescales; and traffic management mechanisms for short 2816 timescale. 2818 Capacity planning is an important component of how a service provider 2819 plans an effective IP network. These plans may take the following 2820 aspects into account: location of and new links or nodes, existing 2821 and predicted traffic patterns, costs, link capacity, topology, 2822 routing design, and survivability. 2824 Performance optimization of operational networks is usually an 2825 ongoing process in which traffic statistics, performance parameters, 2826 and fault indicators are continually collected from the network. 2827 This empirical data is analyzed and used to trigger TE mechanisms. 2828 Tools that perform what-if analysis can also be used to assist the TE 2829 process by reviewing scenarios before a new set of configurations are 2830 implemented in the operational network. 2832 Real-time intra-domain TE using the IGP is done by increasing the 2833 OSPF or IS-IS metric of a congested link until enough traffic has 2834 been diverted away from that link. This approach has some 2835 limitations as discussed in Section 6.2. Intra-domain TE approaches 2836 ([RR94] [FT00] [FT01] [WANG]) take traffic matrix, network topology, 2837 and network performance objectives as input, and produce link metrics 2838 and load-sharing ratios. These processes open the possibility for 2839 intra-domain TE with IGP to be done in a more systematic way. 2841 Administrators of MPLS-TE networks specify and configure link 2842 attributes and resource constraints such as maximum reservable 2843 bandwidth and resource class attributes for the links in the domain. 2844 A link state IGP that supports TE extensions (IS-IS-TE or OSPF-TE) is 2845 used to propagate information about network topology and link 2846 attributes to all routers in the domain. Network administrators 2847 specify the LSPs that are to originate at each router. For each LSP, 2848 the network administrator specifies the destination node and the 2849 attributes of the LSP which indicate the requirements that are to be 2850 satisfied during the path selection process. The attributes may 2851 include and explicit path for the LSP to follow, or originating 2852 router uses a local constraint-based routing process to compute the 2853 path of the LSP. RSVP-TE is used as a signaling protocol to 2854 instantiate the LSPs. By assigning proper bandwidth values to links 2855 and LSPs, congestion caused by uneven traffic distribution can be 2856 avoided or mitigated. 2858 The bandwidth attributes of an LSP relates to the bandwidth 2859 requirements of traffic that flows through the LSP. The traffic 2860 attribute of an LSP can be modified to accommodate persistent shifts 2861 in demand (traffic growth or reduction). If network congestion 2862 occurs due to some unexpected events, existing LSPs can be rerouted 2863 to alleviate the situation or network administrator can configure new 2864 LSPs to divert some traffic to alternative paths. The reservable 2865 bandwidth of the congested links can also be reduced to force some 2866 LSPs to be rerouted to other paths. A traffic matrix in an MPLS 2867 domain can also be estimated by monitoring the traffic on LSPs. Such 2868 traffic statistics can be used for a variety of purposes including 2869 network planning and network optimization. 2871 Network management and planning systems have evolved and taken over a 2872 lot of the responsibility for determining traffic paths in TE 2873 networks. This allows a network-wide view of resources, and 2874 facilitates coordination of the use of resources for all traffic 2875 flows in the network. Initial solutions using a PCE to perform path 2876 computation on behalf of network routers have given way to an 2877 approach that follows the SDN architecture. A stateful PCE is able 2878 to track all of the LSPs in the network and can redistribute them to 2879 make better use of the available resources. Such a PCE can forms 2880 part of a network orchestrator that uses PCEP or some other 2881 southbound interface to instruct the signaling protocol or directly 2882 program the routers. 2884 Segment routing leverages a centralized TE controller and either an 2885 MPLS or IPv6 forwarding plane, but does not need to use a signaling 2886 protocol or management plane protocol to reserve resources in the 2887 routers. All resource reservation is logical within the controller, 2888 and not distributed to the routers. Packets are steered through the 2889 network using segment routing. 2891 As mentioned in Section 7, there is usually no direct control over 2892 the distribution of inbound traffic to a domain. Therefore, the main 2893 goal of inter-domain TE is to optimize the distribution of outbound 2894 traffic between multiple inter-domain links. When operating a global 2895 network, maintaining the ability to operate the network in a regional 2896 fashion where desired, while continuing to take advantage of the 2897 benefits of a global network, also becomes an important objective. 2899 Inter-domain TE with BGP begins with the placement of multiple 2900 peering interconnection points that are in close proximity to traffic 2901 sources/destination, and offer lowest cost paths across the network 2902 between the peering points and and the sources/destinations. Some 2903 location-decision problems that arise in association with inter- 2904 domain routing are discussed in [AWD5]. 2906 Once the locations of the peering interconnects have been determined 2907 and implemented, the network operator decides how best to handle the 2908 routes advertised by the peer, as well as how to propagate the peer's 2909 routes within their network. One way to engineer outbound traffic 2910 flows in a network with many peering interconnects is to create a 2911 hierarchy of peers. Generally, the shortest AS paths will be chosen 2912 to forward traffic but BGP metrics can be used to prefer some peers 2913 and so favor particular paths. Preferred peers are those peers 2914 attached through peering interconnects with the most available 2915 capacity. Changes may be needed, for example, to deal with a 2916 "problem peer" who is difficult to work with on upgrades or is 2917 charging high prices for connectivity to their network. In that 2918 case, the peer may be given a reduced preference. This type of 2919 change can affect a large amount of traffic, and is only used after 2920 other methods have failed to provide the desired results. 2922 When there are multiple exit points toward a given peer, and only one 2923 of them is congested, it is not necessary to shift traffic away from 2924 the peer entirely, but only from the one congested connections. This 2925 can be achieved by using passive IGP-metrics, AS-path filtering, or 2926 prefix filtering. 2928 9. Security Considerations 2930 This document does not introduce new security issues. 2932 Network security is, of course, an important issue. In general, TE 2933 mechanisms are security neutral: they may use tunnels which can 2934 slightly help protect traffic from inspection and which, in some 2935 cases, can be secured using encryption; they put traffic onto 2936 predictable paths within the network that may make it easier to find 2937 and attack; they increase the complexity or operation and management 2938 of the network; and they enable traffic to be steered onto more 2939 secure links or to more secure parts of the network. 2941 The consequences of attacks on the control and management protocols 2942 used to operate TE networks can be significant: traffic can be 2943 hijacked to pass through specific nodes that perform inspection, or 2944 even to be delivered to the wrong place; traffic can be steered onto 2945 paths that deliver quality that is below the desired quality; and, 2946 networks can be congested or have resources on key links consumed. 2947 Thus, it is important to use adequate protection mechanisms on all 2948 protocols used to deliver TE. 2950 Certain aspects of a network may be deduced from the details of the 2951 TE paths that are used. For example, the link connectivity of the 2952 network, and the quality and load on individual links may be assumed 2953 from knowing the paths of traffic and the requirements they place on 2954 the network (for example, by seeing the control messages or through 2955 path- trace techniques). Such knowledge can be used to launch 2956 targeted attacks (for example, taking down critical links) or can 2957 reveal commercially sensitive information (for example, whether a 2958 network is close to capacity). Network operators may, therefore, 2959 choose techniques that mask or hide information from within the 2960 network. 2962 10. IANA Considerations 2964 This draft makes no requests for IANA action. 2966 11. Acknowledgments 2968 Much of the text in this document is derived from RFC 3272. The 2969 authors of this document would like to express their gratitude to all 2970 involved in that work. Although the source text has been edited in 2971 the production of this document, the original authors should be 2972 considered as Contributors to this work. They were: 2974 Daniel O. Awduche 2975 Movaz Networks 2977 Angela Chiu 2978 Celion Networks 2980 Anwar Elwalid 2981 Lucent Technologies 2983 Indra Widjaja 2984 Bell Labs, Lucent Technologies 2986 XiPeng Xiao 2987 Redback Networks 2989 The acknowledgements in RFC3272 were as below. All people who helped 2990 in the production of that document also need to be thanked for the 2991 carry-over into this new document. 2993 The authors would like to thank Jim Boyle for inputs on the 2994 recommendations section, Francois Le Faucheur for inputs on 2995 Diffserv aspects, Blaine Christian for inputs on measurement, 2996 Gerald Ash for inputs on routing in telephone networks and for 2997 text on event-dependent TE methods, Steven Wright for inputs 2998 on network controllability, and Jonathan Aufderheide for 2999 inputs on inter-domain TE with BGP. Special thanks to 3000 Randy Bush for proposing the TE taxonomy based on "tactical versus 3001 strategic" methods. The subsection describing an "Overview of 3002 ITU Activities Related to Traffic Engineering" was adapted from 3003 a contribution by Waisum Lai. Useful feedback and pointers to 3004 relevant materials were provided by J. Noel Chiappa. 3005 Additional comments were provided by Glenn Grotefeld during 3006 the working last call process. Finally, the authors would like 3007 to thank Ed Kern, the TEWG co-chair, for his comments and 3008 support. 3010 The early versions of this document were produced by the TEAS Working 3011 Group's RFC3272bis Design Team. The full list of members of this 3012 team is: 3014 Acee Lindem 3015 Adrian Farrel 3016 Aijun Wang 3017 Daniele Ceccarelli 3018 Dieter Beller 3019 Jeff Tantsura 3020 Julien Meuric 3021 Liu Hua 3022 Loa Andersson 3023 Luis Miguel Contreras 3024 Martin Horneffer 3025 Tarek Saad 3026 Xufeng Liu 3028 The production of this document includes a fix to the original text 3029 resulting from an Errata Report by Jean-Michel Grimaldi. 3031 The author of this document would also like to thank Dhurv Dhody for 3032 review comments. 3034 12. Contributors 3036 The following people contributed substantive text to this document: 3038 Gert Grammel 3039 EMail: ggrammel@juniper.net 3041 Loa Andersson 3042 EMail: loa@pi.nu 3044 Xufeng Liu 3045 EMail: xufeng.liu.ietf@gmail.com 3047 Lou Berger 3048 EMail: lberger@labn.net 3050 Jeff Tantsura 3051 EMail: jefftant.ietf@gmail.com 3053 Daniel King 3054 EMail: daniel@olddog.co.uk 3056 Boris Hassanov 3057 EMail: bhassanov@yandex-team.ru 3059 Kiran Makhijani 3060 Email: kiranm@futurewei.com 3062 Dhruv Dhody 3063 Email: dhruv.ietf@gmail.com 3065 13. Informative References 3067 [AJ19] Adekitan, A., Abolade, J., and O. Shobayo, "Data mining 3068 approach for predicting the daily Internet data traffic of 3069 a smart university", Article Journal of Big Data, 2019, 3070 Volume 6, Number 1, Page 1, 1998. 3072 [ASH2] Ash, J., "Dynamic Routing in Telecommunications Networks", 3073 Book McGraw Hill, 1998. 3075 [AWD2] Awduche, D., "MPLS and Traffic Engineering in IP 3076 Networks", Article IEEE Communications Magazine, December 3077 1999. 3079 [AWD5] Awduche, D., "An Approach to Optimal Peering Between 3080 Autonomous Systems in the Internet", Paper International 3081 Conference on Computer Communications and Networks 3082 (ICCCN'98), October 1998. 3084 [FLJA93] Floyd, S. and V. Jacobson, "Random Early Detection 3085 Gateways for Congestion Avoidance", Article IEEE/ACM 3086 Transactions on Networking, Vol. 1, p. 387-413, November 3087 1993. 3089 [FLOY94] Floyd, S., "TCP and Explicit Congestion Notification", 3090 Article ACM Computer Communication Review, V. 24, No. 5, 3091 p. 10-23, October 1994. 3093 [FT00] Fortz, B. and M. Thorup, "Internet Traffic Engineering by 3094 Optimizing OSPF Weights", Article IEEE INFOCOM 2000, March 3095 2000. 3097 [FT01] Fortz, B. and M. Thorup, "Optimizing OSPF/IS-IS Weights in 3098 a Changing World", n.d., 3099 . 3101 [HUSS87] Hurley, B., Seidl, C., and W. Sewel, "A Survey of Dynamic 3102 Routing Methods for Circuit-Switched Traffic", 3103 Article IEEE Communication Magazine, September 1987. 3105 [I-D.ietf-alto-performance-metrics] 3106 WU, Q., Yang, Y., Lee, Y., Dhody, D., Randriamasy, S., and 3107 L. Contreras, "ALTO Performance Cost Metrics", draft-ietf- 3108 alto-performance-metrics-14 (work in progress), January 3109 2021. 3111 [I-D.ietf-bess-evpn-unequal-lb] 3112 Malhotra, N., Sajassi, A., Rabadan, J., Drake, J., 3113 Lingala, A., and S. Thoria, "Weighted Multi-Path 3114 Procedures for EVPN All-Active Multi-Homing", draft-ietf- 3115 bess-evpn-unequal-lb-07 (work in progress), October 2020. 3117 [I-D.ietf-detnet-ip-over-tsn] 3118 Varga, B., Farkas, J., Malis, A., and S. Bryant, "DetNet 3119 Data Plane: IP over IEEE 802.1 Time Sensitive Networking 3120 (TSN)", draft-ietf-detnet-ip-over-tsn-05 (work in 3121 progress), December 2020. 3123 [I-D.ietf-idr-segment-routing-te-policy] 3124 Previdi, S., Filsfils, C., Talaulikar, K., Mattes, P., 3125 Rosen, E., Jain, D., and S. Lin, "Advertising Segment 3126 Routing Policies in BGP", draft-ietf-idr-segment-routing- 3127 te-policy-11 (work in progress), November 2020. 3129 [I-D.ietf-lsr-flex-algo] 3130 Psenak, P., Hegde, S., Filsfils, C., Talaulikar, K., and 3131 A. Gulko, "IGP Flexible Algorithm", draft-ietf-lsr-flex- 3132 algo-13 (work in progress), October 2020. 3134 [I-D.ietf-lsr-ip-flexalgo] 3135 Britto, W., Hegde, S., Kaneriya, P., Shetty, R., Bonica, 3136 R., and P. Psenak, "IGP Flexible Algorithms (Flex- 3137 Algorithm) In IP Networks", draft-ietf-lsr-ip-flexalgo-00 3138 (work in progress), December 2020. 3140 [I-D.ietf-quic-transport] 3141 Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed 3142 and Secure Transport", draft-ietf-quic-transport-34 (work 3143 in progress), January 2021. 3145 [I-D.ietf-spring-segment-routing-policy] 3146 Filsfils, C., Talaulikar, K., Voyer, D., Bogdanov, A., and 3147 P. Mattes, "Segment Routing Policy Architecture", draft- 3148 ietf-spring-segment-routing-policy-09 (work in progress), 3149 November 2020. 3151 [I-D.ietf-teas-enhanced-vpn] 3152 Dong, J., Bryant, S., Li, Z., Miyasaka, T., and Y. Lee, "A 3153 Framework for Enhanced Virtual Private Networks (VPN+) 3154 Service", draft-ietf-teas-enhanced-vpn-06 (work in 3155 progress), July 2020. 3157 [I-D.ietf-tewg-qos-routing] 3158 Ash, G., "Traffic Engineering & QoS Methods for IP-, ATM-, 3159 & Based Multiservice Networks", draft-ietf-tewg-qos- 3160 routing-04 (work in progress), October 2001. 3162 [I-D.irtf-nmrg-ibn-concepts-definitions] 3163 Clemm, A., Ciavaglia, L., Granville, L., and J. Tantsura, 3164 "Intent-Based Networking - Concepts and Definitions", 3165 draft-irtf-nmrg-ibn-concepts-definitions-02 (work in 3166 progress), September 2020. 3168 [I-D.nsdt-teas-ietf-network-slice-definition] 3169 Rokui, R., Homma, S., Makhijani, K., Contreras, L., and J. 3170 Tantsura, "Definition of IETF Network Slices", draft-nsdt- 3171 teas-ietf-network-slice-definition-02 (work in progress), 3172 December 2020. 3174 [ITU-E600] 3175 "Terms and Definitions of Traffic Engineering", 3176 Recommendation ITU-T Recommendation E.600, March 1993. 3178 [ITU-E701] 3179 "Reference Connections for Traffic Engineering", 3180 Recommendation ITU-T Recommendation E.701, October 1993. 3182 [ITU-E801] 3183 "Framework for Service Quality Agreement", 3184 Recommendation ITU-T Recommendation E.801, October 1996. 3186 [MA] Ma, Q., "Quality of Service Routing in Integrated Services 3187 Networks", Ph.D. PhD Dissertation, CMU-CS-98-138, CMU, 3188 1998. 3190 [MATE] Elwalid, A., Jin, C., Low, S., and I. Widjaja, "MATE - 3191 MPLS Adaptive Traffic Engineering", 3192 Proceedings INFOCOM'01, April 2001. 3194 [MCQ80] McQuillan, J., Richer, I., and E. Rosen, "The New Routing 3195 Algorithm for the ARPANET", Transaction IEEE Transactions 3196 on Communications, vol. 28, no. 5, p. 711-719, May 1980. 3198 [MR99] Mitra, D. and K. Ramakrishnan, "A Case Study of 3199 Multiservice, Multipriority Traffic Engineering Design for 3200 Data Networks", Proceedings Globecom'99, December 1999. 3202 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 3203 DOI 10.17487/RFC0791, September 1981, 3204 . 3206 [RFC1102] Clark, D., "Policy routing in Internet protocols", 3207 RFC 1102, DOI 10.17487/RFC1102, May 1989, 3208 . 3210 [RFC1104] Braun, H., "Models of policy based routing", RFC 1104, 3211 DOI 10.17487/RFC1104, June 1989, 3212 . 3214 [RFC1992] Castineyra, I., Chiappa, N., and M. Steenstrup, "The 3215 Nimrod Routing Architecture", RFC 1992, 3216 DOI 10.17487/RFC1992, August 1996, 3217 . 3219 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 3220 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 3221 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 3222 September 1997, . 3224 [RFC2328] Moy, J., "OSPF Version 2", STD 54, RFC 2328, 3225 DOI 10.17487/RFC2328, April 1998, 3226 . 3228 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 3229 "Framework for IP Performance Metrics", RFC 2330, 3230 DOI 10.17487/RFC2330, May 1998, 3231 . 3233 [RFC2386] Crawley, E., Nair, R., Rajagopalan, B., and H. Sandick, "A 3234 Framework for QoS-based Routing in the Internet", 3235 RFC 2386, DOI 10.17487/RFC2386, August 1998, 3236 . 3238 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 3239 "Definition of the Differentiated Services Field (DS 3240 Field) in the IPv4 and IPv6 Headers", RFC 2474, 3241 DOI 10.17487/RFC2474, December 1998, 3242 . 3244 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 3245 and W. Weiss, "An Architecture for Differentiated 3246 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 3247 . 3249 [RFC2597] Heinanen, J., Baker, F., Weiss, W., and J. Wroclawski, 3250 "Assured Forwarding PHB Group", RFC 2597, 3251 DOI 10.17487/RFC2597, June 1999, 3252 . 3254 [RFC2678] Mahdavi, J. and V. Paxson, "IPPM Metrics for Measuring 3255 Connectivity", RFC 2678, DOI 10.17487/RFC2678, September 3256 1999, . 3258 [RFC2702] Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M., and J. 3259 McManus, "Requirements for Traffic Engineering Over MPLS", 3260 RFC 2702, DOI 10.17487/RFC2702, September 1999, 3261 . 3263 [RFC2722] Brownlee, N., Mills, C., and G. Ruth, "Traffic Flow 3264 Measurement: Architecture", RFC 2722, 3265 DOI 10.17487/RFC2722, October 1999, 3266 . 3268 [RFC2753] Yavatkar, R., Pendarakis, D., and R. Guerin, "A Framework 3269 for Policy-based Admission Control", RFC 2753, 3270 DOI 10.17487/RFC2753, January 2000, 3271 . 3273 [RFC2961] Berger, L., Gan, D., Swallow, G., Pan, P., Tommasi, F., 3274 and S. Molendini, "RSVP Refresh Overhead Reduction 3275 Extensions", RFC 2961, DOI 10.17487/RFC2961, April 2001, 3276 . 3278 [RFC2998] Bernet, Y., Ford, P., Yavatkar, R., Baker, F., Zhang, L., 3279 Speer, M., Braden, R., Davie, B., Wroclawski, J., and E. 3280 Felstaine, "A Framework for Integrated Services Operation 3281 over Diffserv Networks", RFC 2998, DOI 10.17487/RFC2998, 3282 November 2000, . 3284 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 3285 Label Switching Architecture", RFC 3031, 3286 DOI 10.17487/RFC3031, January 2001, 3287 . 3289 [RFC3086] Nichols, K. and B. Carpenter, "Definition of 3290 Differentiated Services Per Domain Behaviors and Rules for 3291 their Specification", RFC 3086, DOI 10.17487/RFC3086, 3292 April 2001, . 3294 [RFC3124] Balakrishnan, H. and S. Seshan, "The Congestion Manager", 3295 RFC 3124, DOI 10.17487/RFC3124, June 2001, 3296 . 3298 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 3299 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 3300 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 3301 . 3303 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 3304 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 3305 Protocol Label Switching (MPLS) Support of Differentiated 3306 Services", RFC 3270, DOI 10.17487/RFC3270, May 2002, 3307 . 3309 [RFC3272] Awduche, D., Chiu, A., Elwalid, A., Widjaja, I., and X. 3310 Xiao, "Overview and Principles of Internet Traffic 3311 Engineering", RFC 3272, DOI 10.17487/RFC3272, May 2002, 3312 . 3314 [RFC3469] Sharma, V., Ed. and F. Hellstrand, Ed., "Framework for 3315 Multi-Protocol Label Switching (MPLS)-based Recovery", 3316 RFC 3469, DOI 10.17487/RFC3469, February 2003, 3317 . 3319 [RFC3630] Katz, D., Kompella, K., and D. Yeung, "Traffic Engineering 3320 (TE) Extensions to OSPF Version 2", RFC 3630, 3321 DOI 10.17487/RFC3630, September 2003, 3322 . 3324 [RFC3945] Mannie, E., Ed., "Generalized Multi-Protocol Label 3325 Switching (GMPLS) Architecture", RFC 3945, 3326 DOI 10.17487/RFC3945, October 2004, 3327 . 3329 [RFC4090] Pan, P., Ed., Swallow, G., Ed., and A. Atlas, Ed., "Fast 3330 Reroute Extensions to RSVP-TE for LSP Tunnels", RFC 4090, 3331 DOI 10.17487/RFC4090, May 2005, 3332 . 3334 [RFC4124] Le Faucheur, F., Ed., "Protocol Extensions for Support of 3335 Diffserv-aware MPLS Traffic Engineering", RFC 4124, 3336 DOI 10.17487/RFC4124, June 2005, 3337 . 3339 [RFC4203] Kompella, K., Ed. and Y. Rekhter, Ed., "OSPF Extensions in 3340 Support of Generalized Multi-Protocol Label Switching 3341 (GMPLS)", RFC 4203, DOI 10.17487/RFC4203, October 2005, 3342 . 3344 [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A 3345 Border Gateway Protocol 4 (BGP-4)", RFC 4271, 3346 DOI 10.17487/RFC4271, January 2006, 3347 . 3349 [RFC4594] Babiarz, J., Chan, K., and F. Baker, "Configuration 3350 Guidelines for DiffServ Service Classes", RFC 4594, 3351 DOI 10.17487/RFC4594, August 2006, 3352 . 3354 [RFC4655] Farrel, A., Vasseur, J., and J. Ash, "A Path Computation 3355 Element (PCE)-Based Architecture", RFC 4655, 3356 DOI 10.17487/RFC4655, August 2006, 3357 . 3359 [RFC4872] Lang, J., Ed., Rekhter, Y., Ed., and D. Papadimitriou, 3360 Ed., "RSVP-TE Extensions in Support of End-to-End 3361 Generalized Multi-Protocol Label Switching (GMPLS) 3362 Recovery", RFC 4872, DOI 10.17487/RFC4872, May 2007, 3363 . 3365 [RFC4873] Berger, L., Bryskin, I., Papadimitriou, D., and A. Farrel, 3366 "GMPLS Segment Recovery", RFC 4873, DOI 10.17487/RFC4873, 3367 May 2007, . 3369 [RFC5250] Berger, L., Bryskin, I., Zinin, A., and R. Coltun, "The 3370 OSPF Opaque LSA Option", RFC 5250, DOI 10.17487/RFC5250, 3371 July 2008, . 3373 [RFC5305] Li, T. and H. Smit, "IS-IS Extensions for Traffic 3374 Engineering", RFC 5305, DOI 10.17487/RFC5305, October 3375 2008, . 3377 [RFC5329] Ishiguro, K., Manral, V., Davey, A., and A. Lindem, Ed., 3378 "Traffic Engineering Extensions to OSPF Version 3", 3379 RFC 5329, DOI 10.17487/RFC5329, September 2008, 3380 . 3382 [RFC5331] Aggarwal, R., Rekhter, Y., and E. Rosen, "MPLS Upstream 3383 Label Assignment and Context-Specific Label Space", 3384 RFC 5331, DOI 10.17487/RFC5331, August 2008, 3385 . 3387 [RFC5394] Bryskin, I., Papadimitriou, D., Berger, L., and J. Ash, 3388 "Policy-Enabled Path Computation Framework", RFC 5394, 3389 DOI 10.17487/RFC5394, December 2008, 3390 . 3392 [RFC5440] Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation 3393 Element (PCE) Communication Protocol (PCEP)", RFC 5440, 3394 DOI 10.17487/RFC5440, March 2009, 3395 . 3397 [RFC5541] Le Roux, JL., Vasseur, JP., and Y. Lee, "Encoding of 3398 Objective Functions in the Path Computation Element 3399 Communication Protocol (PCEP)", RFC 5541, 3400 DOI 10.17487/RFC5541, June 2009, 3401 . 3403 [RFC5557] Lee, Y., Le Roux, JL., King, D., and E. Oki, "Path 3404 Computation Element Communication Protocol (PCEP) 3405 Requirements and Protocol Extensions in Support of Global 3406 Concurrent Optimization", RFC 5557, DOI 10.17487/RFC5557, 3407 July 2009, . 3409 [RFC5664] Halevy, B., Welch, B., and J. Zelenka, "Object-Based 3410 Parallel NFS (pNFS) Operations", RFC 5664, 3411 DOI 10.17487/RFC5664, January 2010, 3412 . 3414 [RFC5693] Seedorf, J. and E. Burger, "Application-Layer Traffic 3415 Optimization (ALTO) Problem Statement", RFC 5693, 3416 DOI 10.17487/RFC5693, October 2009, 3417 . 3419 [RFC6107] Shiomoto, K., Ed. and A. Farrel, Ed., "Procedures for 3420 Dynamically Signaled Hierarchical Label Switched Paths", 3421 RFC 6107, DOI 10.17487/RFC6107, February 2011, 3422 . 3424 [RFC6119] Harrison, J., Berger, J., and M. Bartlett, "IPv6 Traffic 3425 Engineering in IS-IS", RFC 6119, DOI 10.17487/RFC6119, 3426 February 2011, . 3428 [RFC6241] Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed., 3429 and A. Bierman, Ed., "Network Configuration Protocol 3430 (NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011, 3431 . 3433 [RFC6372] Sprecher, N., Ed. and A. Farrel, Ed., "MPLS Transport 3434 Profile (MPLS-TP) Survivability Framework", RFC 6372, 3435 DOI 10.17487/RFC6372, September 2011, 3436 . 3438 [RFC6374] Frost, D. and S. Bryant, "Packet Loss and Delay 3439 Measurement for MPLS Networks", RFC 6374, 3440 DOI 10.17487/RFC6374, September 2011, 3441 . 3443 [RFC6805] King, D., Ed. and A. Farrel, Ed., "The Application of the 3444 Path Computation Element Architecture to the Determination 3445 of a Sequence of Domains in MPLS and GMPLS", RFC 6805, 3446 DOI 10.17487/RFC6805, November 2012, 3447 . 3449 [RFC7149] Boucadair, M. and C. Jacquenet, "Software-Defined 3450 Networking: A Perspective from within a Service Provider 3451 Environment", RFC 7149, DOI 10.17487/RFC7149, March 2014, 3452 . 3454 [RFC7285] Alimi, R., Ed., Penno, R., Ed., Yang, Y., Ed., Kiesel, S., 3455 Previdi, S., Roome, W., Shalunov, S., and R. Woundy, 3456 "Application-Layer Traffic Optimization (ALTO) Protocol", 3457 RFC 7285, DOI 10.17487/RFC7285, September 2014, 3458 . 3460 [RFC7390] Rahman, A., Ed. and E. Dijk, Ed., "Group Communication for 3461 the Constrained Application Protocol (CoAP)", RFC 7390, 3462 DOI 10.17487/RFC7390, October 2014, 3463 . 3465 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 3466 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 3467 Defined Networking (SDN): Layers and Architecture 3468 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 3469 2015, . 3471 [RFC7471] Giacalone, S., Ward, D., Drake, J., Atlas, A., and S. 3472 Previdi, "OSPF Traffic Engineering (TE) Metric 3473 Extensions", RFC 7471, DOI 10.17487/RFC7471, March 2015, 3474 . 3476 [RFC7491] King, D. and A. Farrel, "A PCE-Based Architecture for 3477 Application-Based Network Operations", RFC 7491, 3478 DOI 10.17487/RFC7491, March 2015, 3479 . 3481 [RFC7575] Behringer, M., Pritikin, M., Bjarnason, S., Clemm, A., 3482 Carpenter, B., Jiang, S., and L. Ciavaglia, "Autonomic 3483 Networking: Definitions and Design Goals", RFC 7575, 3484 DOI 10.17487/RFC7575, June 2015, 3485 . 3487 [RFC7679] Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, 3488 Ed., "A One-Way Delay Metric for IP Performance Metrics 3489 (IPPM)", STD 81, RFC 7679, DOI 10.17487/RFC7679, January 3490 2016, . 3492 [RFC7680] Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, 3493 Ed., "A One-Way Loss Metric for IP Performance Metrics 3494 (IPPM)", STD 82, RFC 7680, DOI 10.17487/RFC7680, January 3495 2016, . 3497 [RFC7752] Gredler, H., Ed., Medved, J., Previdi, S., Farrel, A., and 3498 S. Ray, "North-Bound Distribution of Link-State and 3499 Traffic Engineering (TE) Information Using BGP", RFC 7752, 3500 DOI 10.17487/RFC7752, March 2016, 3501 . 3503 [RFC7923] Voit, E., Clemm, A., and A. Gonzalez Prieto, "Requirements 3504 for Subscription to YANG Datastores", RFC 7923, 3505 DOI 10.17487/RFC7923, June 2016, 3506 . 3508 [RFC7926] Farrel, A., Ed., Drake, J., Bitar, N., Swallow, G., 3509 Ceccarelli, D., and X. Zhang, "Problem Statement and 3510 Architecture for Information Exchange between 3511 Interconnected Traffic-Engineered Networks", BCP 206, 3512 RFC 7926, DOI 10.17487/RFC7926, July 2016, 3513 . 3515 [RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", 3516 RFC 7950, DOI 10.17487/RFC7950, August 2016, 3517 . 3519 [RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF 3520 Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017, 3521 . 3523 [RFC8051] Zhang, X., Ed. and I. Minei, Ed., "Applicability of a 3524 Stateful Path Computation Element (PCE)", RFC 8051, 3525 DOI 10.17487/RFC8051, January 2017, 3526 . 3528 [RFC8189] Randriamasy, S., Roome, W., and N. Schwan, "Multi-Cost 3529 Application-Layer Traffic Optimization (ALTO)", RFC 8189, 3530 DOI 10.17487/RFC8189, October 2017, 3531 . 3533 [RFC8231] Crabbe, E., Minei, I., Medved, J., and R. Varga, "Path 3534 Computation Element Communication Protocol (PCEP) 3535 Extensions for Stateful PCE", RFC 8231, 3536 DOI 10.17487/RFC8231, September 2017, 3537 . 3539 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 3540 Interchange Format", STD 90, RFC 8259, 3541 DOI 10.17487/RFC8259, December 2017, 3542 . 3544 [RFC8281] Crabbe, E., Minei, I., Sivabalan, S., and R. Varga, "Path 3545 Computation Element Communication Protocol (PCEP) 3546 Extensions for PCE-Initiated LSP Setup in a Stateful PCE 3547 Model", RFC 8281, DOI 10.17487/RFC8281, December 2017, 3548 . 3550 [RFC8283] Farrel, A., Ed., Zhao, Q., Ed., Li, Z., and C. Zhou, "An 3551 Architecture for Use of PCE and the PCE Communication 3552 Protocol (PCEP) in a Network with Central Control", 3553 RFC 8283, DOI 10.17487/RFC8283, December 2017, 3554 . 3556 [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 3557 Decraene, B., Litkowski, S., and R. Shakir, "Segment 3558 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 3559 July 2018, . 3561 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for 3562 Abstraction and Control of TE Networks (ACTN)", RFC 8453, 3563 DOI 10.17487/RFC8453, August 2018, 3564 . 3566 [RFC8570] Ginsberg, L., Ed., Previdi, S., Ed., Giacalone, S., Ward, 3567 D., Drake, J., and Q. Wu, "IS-IS Traffic Engineering (TE) 3568 Metric Extensions", RFC 8570, DOI 10.17487/RFC8570, March 3569 2019, . 3571 [RFC8571] Ginsberg, L., Ed., Previdi, S., Wu, Q., Tantsura, J., and 3572 C. Filsfils, "BGP - Link State (BGP-LS) Advertisement of 3573 IGP Traffic Engineering Performance Metric Extensions", 3574 RFC 8571, DOI 10.17487/RFC8571, March 2019, 3575 . 3577 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 3578 "Deterministic Networking Architecture", RFC 8655, 3579 DOI 10.17487/RFC8655, October 2019, 3580 . 3582 [RFC8661] Bashandy, A., Ed., Filsfils, C., Ed., Previdi, S., 3583 Decraene, B., and S. Litkowski, "Segment Routing MPLS 3584 Interworking with LDP", RFC 8661, DOI 10.17487/RFC8661, 3585 December 2019, . 3587 [RFC8664] Sivabalan, S., Filsfils, C., Tantsura, J., Henderickx, W., 3588 and J. Hardwick, "Path Computation Element Communication 3589 Protocol (PCEP) Extensions for Segment Routing", RFC 8664, 3590 DOI 10.17487/RFC8664, December 2019, 3591 . 3593 [RFC8685] Zhang, F., Zhao, Q., Gonzalez de Dios, O., Casellas, R., 3594 and D. King, "Path Computation Element Communication 3595 Protocol (PCEP) Extensions for the Hierarchical Path 3596 Computation Element (H-PCE) Architecture", RFC 8685, 3597 DOI 10.17487/RFC8685, December 2019, 3598 . 3600 [RFC8795] Liu, X., Bryskin, I., Beeram, V., Saad, T., Shah, H., and 3601 O. Gonzalez de Dios, "YANG Data Model for Traffic 3602 Engineering (TE) Topologies", RFC 8795, 3603 DOI 10.17487/RFC8795, August 2020, 3604 . 3606 [RFC8896] Randriamasy, S., Yang, R., Wu, Q., Deng, L., and N. 3607 Schwan, "Application-Layer Traffic Optimization (ALTO) 3608 Cost Calendar", RFC 8896, DOI 10.17487/RFC8896, November 3609 2020, . 3611 [RFC8938] Varga, B., Ed., Farkas, J., Berger, L., Malis, A., and S. 3612 Bryant, "Deterministic Networking (DetNet) Data Plane 3613 Framework", RFC 8938, DOI 10.17487/RFC8938, November 2020, 3614 . 3616 [RFC8955] Loibl, C., Hares, S., Raszuk, R., McPherson, D., and M. 3617 Bacher, "Dissemination of Flow Specification Rules", 3618 RFC 8955, DOI 10.17487/RFC8955, December 2020, 3619 . 3621 [RR94] Rodrigues, M. and K. Ramakrishnan, "Optimal Routing in 3622 Shortest Path Networks", Proceedings ITS'94, Rio de 3623 Janeiro, Brazil, 1994. 3625 [SLDC98] Suter, B., Lakshman, T., Stiliadis, D., and A. Choudhury, 3626 "Design Considerations for Supporting TCP with Per-flow 3627 Queueing", Proceedings INFOCOM'98, p. 299-306, 1998. 3629 [WANG] Wang, Y., Wang, Z., and L. Zhang, "Internet traffic 3630 engineering without full mesh overlaying", 3631 Proceedings INFOCOM'2001, April 2001. 3633 [XIAO] Xiao, X., Hannan, A., Bailey, B., and L. Ni, "Traffic 3634 Engineering with MPLS in the Internet", Article IEEE 3635 Network Magazine, March 2000. 3637 [YARE95] Yang, C. and A. Reddy, "A Taxonomy for Congestion Control 3638 Algorithms in Packet Switching Networks", Article IEEE 3639 Network Magazine, p. 34-45, 1995. 3641 Appendix A. Historic Overview 3643 A.1. Traffic Engineering in Classical Telephone Networks 3645 This subsection presents a brief overview of traffic engineering in 3646 telephone networks which often relates to the way user traffic is 3647 steered from an originating node to the terminating node. This 3648 subsection presents a brief overview of this topic. A detailed 3649 description of the various routing strategies applied in telephone 3650 networks is included in the book by G. Ash [ASH2]. 3652 The early telephone network relied on static hierarchical routing, 3653 whereby routing patterns remained fixed independent of the state of 3654 the network or time of day. The hierarchy was intended to 3655 accommodate overflow traffic, improve network reliability via 3656 alternate routes, and prevent call looping by employing strict 3657 hierarchical rules. The network was typically over-provisioned since 3658 a given fixed route had to be dimensioned so that it could carry user 3659 traffic during a busy hour of any busy day. Hierarchical routing in 3660 the telephony network was found to be too rigid upon the advent of 3661 digital switches and stored program control which were able to manage 3662 more complicated traffic engineering rules. 3664 Dynamic routing was introduced to alleviate the routing inflexibility 3665 in the static hierarchical routing so that the network would operate 3666 more efficiently. This resulted in significant economic gains 3667 [HUSS87]. Dynamic routing typically reduces the overall loss 3668 probability by 10 to 20 percent (compared to static hierarchical 3669 routing). Dynamic routing can also improve network resilience by 3670 recalculating routes on a per-call basis and periodically updating 3671 routes. 3673 There are three main types of dynamic routing in the telephone 3674 network. They are time-dependent routing, state-dependent routing 3675 (SDR), and event dependent routing (EDR). 3677 In time-dependent routing, regular variations in traffic loads (such 3678 as time of day or day of week) are exploited in pre-planned routing 3679 tables. In state-dependent routing, routing tables are updated 3680 online according to the current state of the network (e.g., traffic 3681 demand, utilization, etc.). In event dependent routing, routing 3682 changes are triggers by events (such as call setups encountering 3683 congested or blocked links) whereupon new paths are searched out 3684 using learning models. EDR methods are real-time adaptive, but they 3685 do not require global state information as does SDR. Examples of EDR 3686 schemes include the dynamic alternate routing (DAR) from BT, the 3687 state-and-time dependent routing (STR) from NTT, and the success-to- 3688 the-top (STT) routing from AT&T. 3690 Dynamic non-hierarchical routing (DNHR) is an example of dynamic 3691 routing that was introduced in the AT&T toll network in the 1980's to 3692 respond to time-dependent information such as regular load variations 3693 as a function of time. Time-dependent information in terms of load 3694 may be divided into three timescales: hourly, weekly, and yearly. 3695 Correspondingly, three algorithms are defined to pre-plan the routing 3696 tables. The network design algorithm operates over a year-long 3697 interval while the demand servicing algorithm operates on a weekly 3698 basis to fine tune link sizes and routing tables to correct forecast 3699 errors on the yearly basis. At the smallest timescale, the routing 3700 algorithm is used to make limited adjustments based on daily traffic 3701 variations. Network design and demand servicing are computed using 3702 offline calculations. Typically, the calculations require extensive 3703 searches on possible routes. On the other hand, routing may need 3704 online calculations to handle crankback. DNHR adopts a "two-link" 3705 approach whereby a path can consist of two links at most. The 3706 routing algorithm presents an ordered list of route choices between 3707 an originating switch and a terminating switch. If a call overflows, 3708 a via switch (a tandem exchange between the originating switch and 3709 the terminating switch) would send a crankback signal to the 3710 originating switch. This switch would then select the next route, 3711 and so on, until there are no alternative routes available in which 3712 the call is blocked. 3714 A.2. Evolution of Traffic Engineering in Packet Networks 3716 This subsection reviews related prior work that was intended to 3717 improve the performance of data networks. Indeed, optimization of 3718 the performance of data networks started in the early days of the 3719 ARPANET. Other early commercial networks such as SNA also recognized 3720 the importance of performance optimization and service 3721 differentiation. 3723 In terms of traffic management, the Internet has been a best effort 3724 service environment until recently. In particular, very limited 3725 traffic management capabilities existed in IP networks to provide 3726 differentiated queue management and scheduling services to packets 3727 belonging to different classes. 3729 In terms of routing control, the Internet has employed distributed 3730 protocols for intra-domain routing. These protocols are highly 3731 scalable and resilient. However, they are based on simple algorithms 3732 for path selection which have very limited functionality to allow 3733 flexible control of the path selection process. 3735 In the following subsections, the evolution of practical traffic 3736 engineering mechanisms in IP networks and its predecessors are 3737 reviewed. 3739 A.2.1. Adaptive Routing in the ARPANET 3741 The early ARPANET recognized the importance of adaptive routing where 3742 routing decisions were based on the current state of the network 3743 [MCQ80]. Early minimum delay routing approaches forwarded each 3744 packet to its destination along a path for which the total estimated 3745 transit time was the smallest. Each node maintained a table of 3746 network delays, representing the estimated delay that a packet would 3747 experience along a given path toward its destination. The minimum 3748 delay table was periodically transmitted by a node to its neighbors. 3749 The shortest path, in terms of hop count, was also propagated to give 3750 the connectivity information. 3752 One drawback to this approach is that dynamic link metrics tend to 3753 create "traffic magnets" causing congestion to be shifted from one 3754 location of a network to another location, resulting in oscillation 3755 and network instability. 3757 A.2.2. Dynamic Routing in the Internet 3759 The Internet evolved from the ARPANET and adopted dynamic routing 3760 algorithms with distributed control to determine the paths that 3761 packets should take en-route to their destinations. The routing 3762 algorithms are adaptations of shortest path algorithms where costs 3763 are based on link metrics. The link metric can be based on static or 3764 dynamic quantities. The link metric based on static quantities may 3765 be assigned administratively according to local criteria. The link 3766 metric based on dynamic quantities may be a function of a network 3767 congestion measure such as delay or packet loss. 3769 It was apparent early that static link metric assignment was 3770 inadequate because it can easily lead to unfavorable scenarios in 3771 which some links become congested while others remain lightly loaded. 3772 One of the many reasons for the inadequacy of static link metrics is 3773 that link metric assignment was often done without considering the 3774 traffic matrix in the network. Also, the routing protocols did not 3775 take traffic attributes and capacity constraints into account when 3776 making routing decisions. This results in traffic concentration 3777 being localized in subsets of the network infrastructure and 3778 potentially causing congestion. Even if link metrics are assigned in 3779 accordance with the traffic matrix, unbalanced loads in the network 3780 can still occur due to a number factors including: 3782 o Resources may not be deployed in the most optimal locations from a 3783 routing perspective. 3785 o Forecasting errors in traffic volume and/or traffic distribution. 3787 o Dynamics in traffic matrix due to the temporal nature of traffic 3788 patterns, BGP policy change from peers, etc. 3790 The inadequacy of the legacy Internet interior gateway routing system 3791 is one of the factors motivating the interest in path oriented 3792 technology with explicit routing and constraint-based routing 3793 capability such as MPLS. 3795 A.2.3. ToS Routing 3797 Type-of-Service (ToS) routing involves different routes going to the 3798 same destination with selection dependent upon the ToS field of an IP 3799 packet [RFC2474]. The ToS classes may be classified as low delay and 3800 high throughput. Each link is associated with multiple link costs 3801 and each link cost is used to compute routes for a particular ToS. A 3802 separate shortest path tree is computed for each ToS. The shortest 3803 path algorithm must be run for each ToS resulting in very expensive 3804 computation. Classical ToS-based routing is now outdated as the IP 3805 header field has been replaced by a Diffserv field. Effective 3806 traffic engineering is difficult to perform in classical ToS-based 3807 routing because each class still relies exclusively on shortest path 3808 routing which results in localization of traffic concentration within 3809 the network. 3811 A.2.4. Equal Cost Multi-Path 3813 Equal Cost Multi-Path (ECMP) is another technique that attempts to 3814 address the deficiency in the Shortest Path First (SPF) interior 3815 gateway routing systems [RFC2328]. In the classical SPF algorithm, 3816 if two or more shortest paths exist to a given destination, the 3817 algorithm will choose one of them. The algorithm is modified 3818 slightly in ECMP so that if two or more equal cost shortest paths 3819 exist between two nodes, the traffic between the nodes is distributed 3820 among the multiple equal-cost paths. Traffic distribution across the 3821 equal-cost paths is usually performed in one of two ways: (1) packet- 3822 based in a round-robin fashion, or (2) flow-based using hashing on 3823 source and destination IP addresses and possibly other fields of the 3824 IP header. The first approach can easily cause out- of-order packets 3825 while the second approach is dependent upon the number and 3826 distribution of flows. Flow-based load sharing may be unpredictable 3827 in an enterprise network where the number of flows is relatively 3828 small and less heterogeneous (for example, hashing may not be 3829 uniform), but it is generally effective in core public networks where 3830 the number of flows is large and heterogeneous. 3832 In ECMP, link costs are static and bandwidth constraints are not 3833 considered, so ECMP attempts to distribute the traffic as equally as 3834 possible among the equal-cost paths independent of the congestion 3835 status of each path. As a result, given two equal-cost paths, it is 3836 possible that one of the paths will be more congested than the other. 3837 Another drawback of ECMP is that load sharing cannot be achieved on 3838 multiple paths which have non-identical costs. 3840 A.2.5. Nimrod 3842 Nimrod was a routing system developed to provide heterogeneous 3843 service specific routing in the Internet, while taking multiple 3844 constraints into account [RFC1992]. Essentially, Nimrod was a link 3845 state routing protocol to support path oriented packet forwarding. 3846 It used the concept of maps to represent network connectivity and 3847 services at multiple levels of abstraction. Mechanisms allowed 3848 restriction of the distribution of routing information. 3850 Even though Nimrod did not enjoy deployment in the public Internet, a 3851 number of key concepts incorporated into the Nimrod architecture, 3852 such as explicit routing which allows selection of paths at 3853 originating nodes, are beginning to find applications in some recent 3854 constraint-based routing initiatives. 3856 A.3. Development of Internet Traffic Engineering 3858 A.3.1. Overlay Model 3860 In the overlay model, a virtual-circuit network, such as Synchronous 3861 Optical Network / Synchronous Digital Hierarchy (SONET/SDH), Optical 3862 Transport Network (OTN), or Wavelength Division Multiplexing (WDM), 3863 provides virtual-circuit connectivity between routers that are 3864 located at the edges of a virtual-circuit cloud. In this mode, two 3865 routers that are connected through a virtual circuit see a direct 3866 adjacency between themselves independent of the physical route taken 3867 by the virtual circuit through the ATM, frame relay, or WDM network. 3868 Thus, the overlay model essentially decouples the logical topology 3869 that routers see from the physical topology that the ATM, frame 3870 relay, or WDM network manages. The overlay model based on ATM or 3871 frame relay enables a network administrator or an automaton to employ 3872 traffic engineering concepts to perform path optimization by re- 3873 configuring or rearranging the virtual circuits so that a virtual 3874 circuit on a congested or sub-optimal physical link can be re-routed 3875 to a less congested or more optimal one. In the overlay model, 3876 traffic engineering is also employed to establish relationships 3877 between the traffic management parameters (e.g., Peak Cell Rate, 3878 Sustained Cell Rate, and Maximum Burst Size for ATM) of the virtual- 3879 circuit technology and the actual traffic that traverses each 3880 circuit. These relationships can be established based upon known or 3881 projected traffic profiles, and some other factors. 3883 Appendix B. Overview of Traffic Engineering Related Work in Other SDOs 3884 B.1. Overview of ITU Activities Related to Traffic Engineering 3886 This section provides an overview of prior work within the ITU-T 3887 pertaining to traffic engineering in traditional telecommunications 3888 networks. 3890 ITU-T Recommendations E.600 [ITU-E600], E.701 [ITU-E701], and E.801 3891 [ITU-E801] address traffic engineering issues in traditional 3892 telecommunications networks. Recommendation E.600 provides a 3893 vocabulary for describing traffic engineering concepts, while E.701 3894 defines reference connections, Grade of Service (GoS), and traffic 3895 parameters for ISDN. Recommendation E.701 uses the concept of a 3896 reference connection to identify representative cases of different 3897 types of connections without describing the specifics of their actual 3898 realizations by different physical means. As defined in 3899 Recommendation E.600, "a connection is an association of resources 3900 providing means for communication between two or more devices in, or 3901 attached to, a telecommunication network." Also, E.600 defines "a 3902 resource as any set of physically or conceptually identifiable 3903 entities within a telecommunication network, the use of which can be 3904 unambiguously determined" [ITU-E600]. There can be different types 3905 of connections as the number and types of resources in a connection 3906 may vary. 3908 Typically, different network segments are involved in the path of a 3909 connection. For example, a connection may be local, national, or 3910 international. The purposes of reference connections are to clarify 3911 and specify traffic performance issues at various interfaces between 3912 different network domains. Each domain may consist of one or more 3913 service provider networks. 3915 Reference connections provide a basis to define grade of service 3916 (GoS) parameters related to traffic engineering within the ITU-T 3917 framework. As defined in E.600, "GoS refers to a number of traffic 3918 engineering variables which are used to provide a measure of the 3919 adequacy of a group of resources under specified conditions." These 3920 GoS variables may be probability of loss, dial tone, delay, etc. 3921 They are essential for network internal design and operation as well 3922 as for component performance specification. 3924 GoS is different from quality of service (QoS) in the ITU framework. 3925 QoS is the performance perceivable by a telecommunication service 3926 user and expresses the user's degree of satisfaction of the service. 3927 QoS parameters focus on performance aspects observable at the service 3928 access points and network interfaces, rather than their causes within 3929 the network. GoS, on the other hand, is a set of network oriented 3930 measures which characterize the adequacy of a group of resources 3931 under specified conditions. For a network to be effective in serving 3932 its users, the values of both GoS and QoS parameters must be related, 3933 with GoS parameters typically making a major contribution to the QoS. 3935 Recommendation E.600 stipulates that a set of GoS parameters must be 3936 selected and defined on an end-to-end basis for each major service 3937 category provided by a network to assist the network provider with 3938 improving efficiency and effectiveness of the network. Based on a 3939 selected set of reference connections, suitable target values are 3940 assigned to the selected GoS parameters under normal and high load 3941 conditions. These end-to-end GoS target values are then apportioned 3942 to individual resource components of the reference connections for 3943 dimensioning purposes. 3945 Appendix C. Summary of Changes Since RFC 3272 3947 The changes to this document since RFC 3272 are substantial and not 3948 easily summarized as section-by-section changes. The material in the 3949 document has been moved around considerably, some of it removed, and 3950 new text added. 3952 The approach taken here is to list the table of content of both the 3953 previous RFC and this document saying, respectively, where the text 3954 has been place and where the text came from. 3956 C.1. RFC 3272 3958 1.0 Introduction: Edited in place in Section 1. 3960 1.1 What is Internet Traffic Engineering?: Edited in place in 3961 Section 1.1. 3963 1.2 Scope: Moved to Section 1.3. 3965 1.3 Terminology: Moved to Section 1.4 with some obsolete terms 3966 removed and a little editing. 3968 2.0 Background: Retained as Section 2 with some text removed. 3970 2.1 Context of Internet Traffic Engineering: Retained as 3971 Section 2.1. 3973 2.2 Network Context: Rewritten as Section 2.2. 3975 2.3 Problem Context: Rewritten as Section 2.3. 3977 2.3.1 Congestion and its Ramifications: Retained as 3978 Section 2.3.1. 3980 2.4 Solution Context: Edited as Section 2.4. 3982 2.4.1 Combating the Congestion Problem: Reformatted as 3983 Section 2.4.1. 3985 2.5 Implementation and Operational Context: Retained as 3986 Section 2.5. 3988 3.0 Traffic Engineering Process Model: Retained as Section 3. 3990 3.1 Components of the Traffic Engineering Process Model: Retained 3991 as Section 3.1. 3993 3.2 Measurement: Merged into Section 3.1. 3995 3.3 Modeling, Analysis, and Simulation: Merged into Section 3.1. 3997 3.4 Optimization: Merged into Section 3.1. 3999 4.0 Historical Review and Recent Developments: Retained as 4000 Section 4, but the very historic aspects moved to Appendix A. 4002 4.1 Traffic Engineering in Classical Telephone Networks: Moved to 4003 Appendix A.1. 4005 4.2 Evolution of Traffic Engineering in the Internet: Moved to Ap 4006 pendix A.2. 4008 4.2.1 Adaptive Routing in ARPANET: Moved to Appendix A.2.1. 4010 4.2.2 Dynamic Routing in the Internet: Moved to 4011 Appendix A.2.2. 4013 4.2.3 ToS Routing: Moved to Appendix A.2.3. 4015 4.2.4 Equal Cost Multi-Path: Moved to Appendix A.2.4. 4017 4.2.5 Nimrod: Moved to Appendix A.2.5. 4019 4.3 Overlay Model: Moved to Appendix A.3.1. 4021 4.4 Constraint-Based Routing: Retained as Section 4.1.1, but 4022 moved into Section 4.1. 4024 4.5 Overview of Other IETF Projects Related to Traffic 4025 Engineering: 4026 Retained as Section 4.1 with many new subsections. 4028 4.5.1 Integrated Services: Retained as Section 4.1.2. 4030 4.5.2 RSVP: Retained as Section 4.1.3 with some edits. 4032 4.5.3 Differentiated Services: Retained as Section 4.1.4. 4034 4.5.4 MPLS: Retained as Section 4.1.6. 4036 4.5.5 IP Performance Metrics: Retained as Section 4.1.8. 4038 4.5.6 Flow Measurement: Retained as Section 4.1.9 with some 4039 reformatting. 4041 4.5.7 Endpoint Congestion Management: Retained as Section 4.1.10. 4043 4.6 Overview of ITU Activities Related to Traffic Engineering: Moved 4044 to Appendix B.1. 4046 4.7 Content Distribution: Retained as Section 4.2. 4048 5.0 Taxonomy of Traffic Engineering Systems: Retained as Section 5. 4050 5.1 Time-Dependent Versus State-Dependent: Retained as 4051 Section 5.1. 4053 5.2 Offline Versus Online: Retained as Section 5.2. 4055 5.3 Centralized Versus Distributed: Retained as Section 5.3 with 4056 additions. 4058 5.4 Local Versus Global: Retained as Section 5.4. 4060 5.5 Prescriptive Versus Descriptive: Retained as Section 5.5 with 4061 additions. 4063 5.6 Open-Loop Versus Closed-Loop: Retained as Section 5.6. 4065 5.7 Tactical vs Strategic: Retained as Section 5.7. 4067 6.0 Recommendations for Internet Traffic Engineering: Retained as 4068 Section 6. 4070 6.1 Generic Non-functional Recommendations: Retained as 4071 Section 6.1. 4073 6.2 Routing Recommendations: Retained as Section 6.2 with edits. 4075 6.3 Traffic Mapping Recommendations: Retained as Section 6.3. 4077 6.4 Measurement Recommendations: Retained as Section 6.4. 4079 6.5 Network Survivability: Retained as Section 6.5. 4081 6.5.1 Survivability in MPLS Based Networks: Retained as 4082 Section 6.5.1. 4084 6.5.2 Protection Option: Retained as Section 6.5.2. 4086 6.6 Traffic Engineering in Diffserv Environments: Retained as 4087 Section 6.6 with edits. 4089 6.7 Network Controllability: Retained as Section 6.7. 4091 7.0 Inter-Domain Considerations: Retained as Section 7. 4093 8.0 Overview of Contemporary TE Practices in Operational IP Networks: 4095 Retained as Section 8. 4097 9.0 Conclusion: Removed. 4099 10.0 Security Considerations: Retained as Section 9 with 4100 considerable new text. 4102 C.2. This Document 4104 o Section 1: Based on Section 1 of RFC 3272. 4106 * Section 1.1: Based on Section 1.1 of RFC 3272. 4108 * Section 1.2: New for this document. 4110 * Section 1.3: Based on Section 1.2 of RFC 3272. 4112 * Section 1.4: Based on Section 1.3 of RFC 3272. 4114 o Section 2: Based on Section 2. of RFC 3272. 4116 * Section 2.1: Based on Section 2.1 of RFC 3272. 4118 * Section 2.2: Based on Section 2.2 of RFC 3272. 4120 * Section 2.3: Based on Section 2.3 of RFC 3272. 4122 + Section 2.3.1: Based on Section 2.3.1 of RFC 3272. 4124 * Section 2.4: Based on Section 2.4 of RFC 3272. 4126 + Section 2.4.1: Based on Section 2.4.1 of RFC 327 4128 * Section 2.5: Based on Section 2.5 of RFC 3272. 4130 o Section 3: Based on Section 3 of RFC 3272. 4132 * Section 3.1: Based on Sections 3.1, 3.2, 3.3, and 3.4 of RFC 4133 3272. 4135 o Section 4: Based on Section 4 of RFC 3272. 4137 * Section 4.1: Based on Section 4.5 of RFC 3272. 4139 + Section 4.1.1: Based on Section 4.4 of RFC 3272. 4141 - Section 4.1.1.1: New for this document. 4143 + Section 4.1.2: Based on Section 4.5.1 of RFC 3272. 4145 + Section 4.1.3: Based on Section 4.5.2 of RFC 3272. 4147 + Section 4.1.4: Based on Section 4.5.3 of RFC 3272. 4149 + Section 4.1.5: New for this document. 4151 + Section 4.1.6: Based on Section 4.5.4 of RFC 3272. 4153 + Section 4.1.7: New for this document. 4155 + Section 4.1.8: Based on Section 4.5.5 of RFC 3272. 4157 + Section 4.1.9: Based on Section 4.5.6 of RFC 3272. 4159 + Section 4.1.10: Based on Section 4.5.7 of RFC 3272. 4161 + Section 4.1.11: New for this document. 4163 + Section 4.1.12: New for this document. 4165 + Section 4.1.13: New for this document. 4167 + Section 4.1.14: New for this document. 4169 + Section 4.1.15: New for this document. 4171 + Section 4.1.16: New for this document. 4173 - Section 4.1.16.1: New for this document. 4175 - Section 4.1.16.2: New for this document. 4177 + Section 4.1.17: New for this document. 4179 + Section 4.1.18: New for this document. 4181 + Section 4.1.19: New for this document. 4183 + Section 4.1.20: New for this document. 4185 + Section 4.1.21: New for this document. 4187 * Section 4.2: Based on Section 4.7 of RFC 3272. 4189 o Section 5: Based on Section 5 of RFC 3272. 4191 * Section 5.1: Based on Section 5.1 of RFC 3272. 4193 * Section 5.2: Based on Section 5.2 of RFC 3272. 4195 * Section 5.3: Based on Section 5.3 of RFC 3272. 4197 + Section 5.3.1: New for this document. 4199 + Section 5.3.2: New for this document. 4201 * Section 5.4: Based on Section 5.4 of RFC 3272. 4203 * Section 5.5: Based on Section 5.5 of RFC 3272. 4205 + Section 5.5.1: New for this document. 4207 * Section 5.6: Based on Section 5.6 of RFC 3272. 4209 * Section 5.7: Based on Section 5.7 of RFC 3272. 4211 o Section 6: Based on Section 6 of RFC 3272. 4213 * Section 6.1: Based on Section 6.1 of RFC 3272. 4215 * Section 6.2: Based on Section 6.2 of RFC 3272. 4217 * Section 6.3: Based on Section 6.3 of RFC 3272. 4219 * Section 6.4: Based on Section 6.4 of RFC 3272. 4221 * Section 6.5: Based on Section 6.5 of RFC 3272. 4223 + Section 6.5.1: Based on Section 6.5.1 of RFC 3272. 4225 + Section 6.5.2: Based on Section 6.5.2 of RFC 3272. 4227 * Section 6.6: Based on Section 6.6. of RFC 3272. 4229 * Section 6.7: Based on Section 6.7 of RFC 3272. 4231 o Section 7: Based on Section 7 of RFC 3272. 4233 o Section 8: Based on Section 8 of RFC 3272. 4235 o Section 9: Based on Section 10 of RFC 3272. 4237 o Appendix A: New for this document. 4239 * Appendix A.1: Based on Section 4.1 of RFC 3272. 4241 * Appendix A.2: Based on Section 4.2 of RFC 3272. 4243 + Appendix A.2.1: Based on Section 4.2.1 of RFC 3272. 4245 + Appendix A.2.2: Based on Section 4.2.2 of RFC 3272. 4247 + Appendix A.2.3: Based on Section 4.2.3 of RFC 3272. 4249 + Appendix A.2.4: Based on Section 4.2.4 of RFC 3272. 4251 + Appendix A.2.5: Based on Section 4.2.5 of RFC 3272. 4253 * Appendix A.3: New for this document. 4255 + Appendix A.3.1: Based on Section 4.3 of RFC 3272. 4257 o Appendix B: New for this document. 4259 * Appendix B.1: Based on Section 4.7 of RFC 3272. 4261 Author's Address 4263 Adrian Farrel (editor) 4264 Old Dog Consulting 4266 Email: adrian@olddog.co.uk