idnits 2.17.1 draft-ietf-teas-rfc3272bis-10.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (December 28, 2020) is 1214 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-28) exists of draft-ietf-alto-performance-metrics-12 == Outdated reference: A later version (-21) exists of draft-ietf-bess-evpn-unequal-lb-07 == Outdated reference: A later version (-07) exists of draft-ietf-detnet-ip-over-tsn-05 == Outdated reference: A later version (-26) exists of draft-ietf-idr-segment-routing-te-policy-11 == Outdated reference: A later version (-26) exists of draft-ietf-lsr-flex-algo-13 == Outdated reference: A later version (-34) exists of draft-ietf-quic-transport-33 == Outdated reference: A later version (-22) exists of draft-ietf-spring-segment-routing-policy-09 == Outdated reference: A later version (-17) exists of draft-ietf-teas-enhanced-vpn-06 -- Obsolete informational reference (is this intentional?): RFC 2370 (Obsoleted by RFC 5250) -- Obsolete informational reference (is this intentional?): RFC 3272 (Obsoleted by RFC 9522) -- Obsolete informational reference (is this intentional?): RFC 7159 (Obsoleted by RFC 8259) -- Obsolete informational reference (is this intentional?): RFC 7752 (Obsoleted by RFC 9552) -- Obsolete informational reference (is this intentional?): RFC 7810 (Obsoleted by RFC 8570) Summary: 0 errors (**), 0 flaws (~~), 9 warnings (==), 6 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TEAS Working Group A. Farrel, Ed. 3 Internet-Draft Old Dog Consulting 4 Obsoletes: 3272 (if approved) December 28, 2020 5 Intended status: Informational 6 Expires: July 1, 2021 8 Overview and Principles of Internet Traffic Engineering 9 draft-ietf-teas-rfc3272bis-10 11 Abstract 13 This document describes the principles of traffic engineering (TE) in 14 the Internet. The document is intended to promote better 15 understanding of the issues surrounding traffic engineering in IP 16 networks and the networks that support IP networking, and to provide 17 a common basis for the development of traffic engineering 18 capabilities for the Internet. The principles, architectures, and 19 methodologies for performance evaluation and performance optimization 20 of operational networks are also discussed. 22 This work was first published as RFC 3272 in May 2002. This document 23 obsoletes RFC 3272 by making a complete update to bring the text in 24 line with best current practices for Internet traffic engineering and 25 to include references to the latest relevant work in the IETF. 27 Status of This Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at https://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on July 1, 2021. 44 Copyright Notice 46 Copyright (c) 2020 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents 51 (https://trustee.ietf.org/license-info) in effect on the date of 52 publication of this document. Please review these documents 53 carefully, as they describe your rights and restrictions with respect 54 to this document. Code Components extracted from this document must 55 include Simplified BSD License text as described in Section 4.e of 56 the Trust Legal Provisions and are provided without warranty as 57 described in the Simplified BSD License. 59 Table of Contents 61 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 62 1.1. What is Internet Traffic Engineering? . . . . . . . . . . 4 63 1.2. Components of Traffic Engineering . . . . . . . . . . . . 6 64 1.3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 8 65 1.4. Terminology . . . . . . . . . . . . . . . . . . . . . . . 8 66 2. Background . . . . . . . . . . . . . . . . . . . . . . . . . 11 67 2.1. Context of Internet Traffic Engineering . . . . . . . . . 11 68 2.2. Network Domain Context . . . . . . . . . . . . . . . . . 12 69 2.3. Problem Context . . . . . . . . . . . . . . . . . . . . . 14 70 2.3.1. Congestion and its Ramifications . . . . . . . . . . 15 71 2.4. Solution Context . . . . . . . . . . . . . . . . . . . . 15 72 2.4.1. Combating the Congestion Problem . . . . . . . . . . 17 73 2.5. Implementation and Operational Context . . . . . . . . . 20 74 3. Traffic Engineering Process Models . . . . . . . . . . . . . 21 75 3.1. Components of the Traffic Engineering Process Model . . . 21 76 4. Review of TE Techniques . . . . . . . . . . . . . . . . . . . 22 77 4.1. Overview of IETF Projects Related to Traffic Engineering 22 78 4.1.1. Constraint-Based Routing . . . . . . . . . . . . . . 22 79 4.1.2. Integrated Services . . . . . . . . . . . . . . . . . 24 80 4.1.3. RSVP . . . . . . . . . . . . . . . . . . . . . . . . 25 81 4.1.4. Differentiated Services . . . . . . . . . . . . . . . 26 82 4.1.5. QUIC . . . . . . . . . . . . . . . . . . . . . . . . 27 83 4.1.6. Multiprotocol Label Switching (MPLS) . . . . . . . . 27 84 4.1.7. Generalized MPLS (GMPLS) . . . . . . . . . . . . . . 28 85 4.1.8. IP Performance Metrics . . . . . . . . . . . . . . . 29 86 4.1.9. Flow Measurement . . . . . . . . . . . . . . . . . . 29 87 4.1.10. Endpoint Congestion Management . . . . . . . . . . . 30 88 4.1.11. TE Extensions to the IGPs . . . . . . . . . . . . . . 30 89 4.1.12. Link-State BGP . . . . . . . . . . . . . . . . . . . 30 90 4.1.13. Path Computation Element . . . . . . . . . . . . . . 31 91 4.1.14. Application-Layer Traffic Optimization . . . . . . . 32 92 4.1.15. Segment Routing with MPLS Encapsulation (SR-MPLS) . . 33 93 4.1.16. Network Virtualization and Abstraction . . . . . . . 35 94 4.1.17. Network Slicing . . . . . . . . . . . . . . . . . . . 35 95 4.1.18. Deterministic Networking . . . . . . . . . . . . . . 36 96 4.1.19. Network TE State Definition and Presentation . . . . 37 97 4.1.20. System Management and Control Interfaces . . . . . . 37 98 4.2. Content Distribution . . . . . . . . . . . . . . . . . . 38 99 5. Taxonomy of Traffic Engineering Systems . . . . . . . . . . . 38 100 5.1. Time-Dependent Versus State-Dependent Versus Event- 101 Dependent . . . . . . . . . . . . . . . . . . . . . . . . 39 102 5.2. Offline Versus Online . . . . . . . . . . . . . . . . . . 40 103 5.3. Centralized Versus Distributed . . . . . . . . . . . . . 41 104 5.3.1. Hybrid Systems . . . . . . . . . . . . . . . . . . . 41 105 5.3.2. Considerations for Software Defined Networking . . . 42 106 5.4. Local Versus Global . . . . . . . . . . . . . . . . . . . 42 107 5.5. Prescriptive Versus Descriptive . . . . . . . . . . . . . 43 108 5.5.1. Intent-Based Networking . . . . . . . . . . . . . . . 43 109 5.6. Open-Loop Versus Closed-Loop . . . . . . . . . . . . . . 43 110 5.7. Tactical versus Strategic . . . . . . . . . . . . . . . . 44 111 6. Recommendations for Internet Traffic Engineering . . . . . . 44 112 6.1. Generic Non-functional Recommendations . . . . . . . . . 44 113 6.2. Routing Recommendations . . . . . . . . . . . . . . . . . 46 114 6.3. Traffic Mapping Recommendations . . . . . . . . . . . . . 49 115 6.4. Measurement Recommendations . . . . . . . . . . . . . . . 49 116 6.5. Network Survivability . . . . . . . . . . . . . . . . . . 50 117 6.5.1. Survivability in MPLS Based Networks . . . . . . . . 52 118 6.5.2. Protection Options . . . . . . . . . . . . . . . . . 53 119 6.6. Traffic Engineering in Diffserv Environments . . . . . . 54 120 6.7. Network Controllability . . . . . . . . . . . . . . . . . 56 121 7. Inter-Domain Considerations . . . . . . . . . . . . . . . . . 56 122 8. Overview of Contemporary TE Practices in Operational IP 123 Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 58 124 9. Security Considerations . . . . . . . . . . . . . . . . . . . 61 125 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 61 126 11. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 61 127 12. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 63 128 13. Informative References . . . . . . . . . . . . . . . . . . . 64 129 Appendix A. Historic Overview . . . . . . . . . . . . . . . . . 76 130 A.1. Traffic Engineering in Classical Telephone Networks . . . 76 131 A.2. Evolution of Traffic Engineering in Packet Networks . . . 77 132 A.2.1. Adaptive Routing in the ARPANET . . . . . . . . . . . 78 133 A.2.2. Dynamic Routing in the Internet . . . . . . . . . . . 78 134 A.2.3. ToS Routing . . . . . . . . . . . . . . . . . . . . . 79 135 A.2.4. Equal Cost Multi-Path . . . . . . . . . . . . . . . . 79 136 A.2.5. Nimrod . . . . . . . . . . . . . . . . . . . . . . . 80 137 A.3. Development of Internet Traffic Engineering . . . . . . . 80 138 A.3.1. Overlay Model . . . . . . . . . . . . . . . . . . . . 80 139 Appendix B. Overview of Traffic Engineering Related Work in 140 Other SDOs . . . . . . . . . . . . . . . . . . . . . 81 141 B.1. Overview of ITU Activities Related to Traffic Engineering 81 142 Appendix C. Summary of Changes Since RFC 3272 . . . . . . . . . 82 143 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 82 145 1. Introduction 147 This document describes the principles of Internet traffic 148 engineering (TE). The objective of the document is to articulate the 149 general issues and principles for Internet traffic engineering, and 150 where appropriate to provide recommendations, guidelines, and options 151 for the development of online and offline Internet traffic 152 engineering capabilities and support systems. 154 This document provides a terminology and taxonomy for describing and 155 understanding common Internet traffic engineering concepts. 157 Even though Internet traffic engineering is most effective when 158 applied end-to-end, the focus of this document is traffic engineering 159 within a given domain (such as an autonomous system). However, 160 because a preponderance of Internet traffic tends to originate in one 161 autonomous system and terminate in another, this document also 162 provides an overview of aspects pertaining to inter-domain traffic 163 engineering. 165 This work was first published as [RFC3272] in May 2002. This 166 document obsoletes [RFC3272] by making a complete update to bring the 167 text in line with best current practices for Internet traffic 168 engineering and to include references to the latest relevant work in 169 the IETF. It is worth noting around three fifths of the RFCs 170 referenced in this document post-date the publication of RFC 3272. 171 Appendix C provides a summary of changes between RFC 3272 and this 172 document. 174 1.1. What is Internet Traffic Engineering? 176 One of the most significant functions performed by the Internet is 177 the routing of traffic from ingress nodes to egress nodes. 178 Therefore, one of the most distinctive functions performed by 179 Internet traffic engineering is the control and optimization of the 180 routing function, to steer traffic through the network. 182 Internet traffic engineering is defined as that aspect of Internet 183 network engineering dealing with the issues of performance evaluation 184 and performance optimization of operational IP networks. Traffic 185 engineering encompasses the application of technology and scientific 186 principles to the measurement, characterization, modeling, and 187 control of Internet traffic [RFC2702], [AWD2]. 189 It is the performance of the network as seen by end users of network 190 services that is paramount. The characteristics visible to end users 191 are the emergent properties of the network, which are the 192 characteristics of the network when viewed as a whole. A central 193 goal of the service provider, therefore, is to enhance the emergent 194 properties of the network while taking economic considerations into 195 account. This is accomplished by addressing traffic oriented 196 performance requirements while utilizing network resources 197 economically and reliably. Traffic oriented performance measures 198 include delay, delay variation, packet loss, and throughput. 200 Internet traffic engineering responds to network events. Aspects of 201 capacity management respond at intervals ranging from days to years. 202 Routing control functions operate at intervals ranging from 203 milliseconds to days. Packet level processing functions operate at 204 very fine levels of temporal resolution, ranging from picoseconds to 205 milliseconds while reacting to the real-time statistical behavior of 206 traffic. 208 Thus, the optimization aspects of traffic engineering can be viewed 209 from a control perspective, and can be both pro-active and reactive. 210 In the pro-active case, the traffic engineering control system takes 211 preventive action to protect against predicted unfavorable future 212 network states, for example, by engineering backup paths. It may 213 also take action that will lead to a more desirable future network 214 state. In the reactive case, the control system responds to correct 215 issues and adapt to network events, such as routing after failure. 217 Another important objective of Internet traffic engineering is to 218 facilitate reliable network operations [RFC2702]. Reliable network 219 operations can be facilitated by providing mechanisms that enhance 220 network integrity and by embracing policies emphasizing network 221 survivability. This reduces the vulnerability of services to outages 222 arising from errors, faults, and failures occurring within the 223 network infrastructure. 225 The optimization aspects of traffic engineering can be achieved 226 through capacity management and traffic management. In this 227 document, capacity management includes capacity planning, routing 228 control, and resource management. Network resources of particular 229 interest include link bandwidth, buffer space, and computational 230 resources. In this document, traffic management includes: 232 1. nodal traffic control functions such as traffic conditioning, 233 queue management, scheduling 235 2. other functions that regulate traffic flow through the network or 236 that arbitrate access to network resources between different 237 packets or between different traffic streams. 239 One major challenge of Internet traffic engineering is the 240 realization of automated control capabilities that adapt quickly and 241 cost effectively to significant changes in network state, while still 242 maintaining stability of the network. Performance evaluation can 243 assess the effectiveness of traffic engineering methods, and the 244 results of this evaluation can be used to identify existing problems, 245 guide network re-optimization, and aid in the prediction of potential 246 future problems. However, this process can also be time consuming 247 and may not be suitable to act on short-lived changes in the network. 249 Performance evaluation can be achieved in many different ways. The 250 most notable techniques include analytical methods, simulation, and 251 empirical methods based on measurements. 253 Traffic engineering comes in two flavors: either a background process 254 that constantly monitors traffic and optimizes the use of resources 255 to improve performance; or a form of a pre-planned optimized traffic 256 distribution that is considered optimal. In the later case, any 257 deviation from the optimum distribution (e.g., caused by a fiber cut) 258 is reverted upon repair without further optimization. However, this 259 form of traffic engineering relies upon the notion that the planned 260 state of the network is optimal. Hence, in such a mode there are two 261 levels of traffic engineering: the TE-planning task to enable optimum 262 traffic distribution, and the routing task keeping traffic flows 263 attached to the pre-planned distribution. 265 As a general rule, traffic engineering concepts and mechanisms must 266 be sufficiently specific and well-defined to address known 267 requirements, but simultaneously flexible and extensible to 268 accommodate unforeseen future demands. 270 1.2. Components of Traffic Engineering 272 As mentioned in Section 1.1, Internet traffic engineering provides 273 performance optimization of operational IP networks while utilizing 274 network resources economically and reliably. Such optimization is 275 supported at the control/controller level and within the data/ 276 forwarding plane. 278 The key elements required in any TE solution are as follows: 280 1. Policy 282 2. Path steering 284 3. Resource management 286 Some TE solutions rely on these elements to a lesser or greater 287 extent. Debate remains about whether a solution can truly be called 288 traffic engineering if it does not include all of these elements. 290 For the sake of this document, we assert that all TE solutions must 291 include some aspects of all of these elements. Other solutions can 292 be classed as "partial TE" and also fall in scope of this document. 294 Policy allows for the selection of next hops and paths based on 295 information beyond basic reachability. Early definitions of routing 296 policy, e.g., [RFC1102] and [RFC1104], discuss routing policy being 297 applied to restrict access to network resources at an aggregate 298 level. BGP is an example of a commonly used mechanism for applying 299 such policies, see [RFC4271] and [I-D.ietf-idr-rfc5575bis]. In the 300 traffic engineering context, policy decisions are made within the 301 control plane or by controllers, and govern the selection of paths. 302 Examples can be found in [RFC4655] and [RFC5394]. Standard TE 303 solutions may cover the mechanisms to distribute and/or enforce 304 polices, but specific policy definition is generally unspecified. 306 Path steering is the ability to forward packets using more 307 information than just knowledge of the next hop. Examples of path 308 steering include IPv4 source routes [RFC0791], RSVP-TE explicit 309 routes [RFC3209], and Segment Routing [RFC8402]. Path steering for 310 TE can be supported via control plane protocols, by encoding in the 311 data plane headers, or by a combination of the two. This includes 312 when control is provided by a controller using a southbound (i.e., 313 controller to router) control protocol. 315 Resource management provides resource aware control and forwarding. 316 Examples of resources are bandwidth, buffers, and queues, all of 317 which can be managed to control loss and latency. 319 Resource reservation is the control aspect of resource management. 320 It provides for domain-wide consensus about which network 321 resources are used by a particular flow. This determination may 322 be made at a very course or very fine level. Note that this 323 consensus exists at the network control or controller level, not 324 within the data plane. It may be composed purely of accounting/ 325 bookkeeping, but it typically includes an ability to admit, 326 reject, or reclassify a flow based on policy. Such accounting can 327 be done based on any combination of a static understanding of 328 resource requirements, and the use of dynamic mechanisms to 329 collect requirements (e.g., via [RFC3209]) and resource 330 availability (e.g., via [RFC4203]). 332 Resource allocation is the data plane aspect of resource 333 management. It provides for the allocation of specific node and 334 link resources to specific flows. Example resources include 335 buffers, policing, and rate-shaping mechanisms that are typically 336 supported via queuing. It also includes the matching of a flow 337 (i.e., flow classification) to a particular set of allocated 338 resources. The method of flow classification and granularity of 339 resource management is technology specific. Examples include 340 Diffserv with dropping and remarking [RFC4594], MPLS-TE [RFC3209], 341 and GMPLS based label switched paths [RFC3945], as well as 342 controller-based solutions [RFC8453]. This level of resource 343 control, while optional, is important in networks that wish to 344 support congestion management policies to control or regulate the 345 offered traffic to deliver different levels of service and 346 alleviate congestion problems, or those networks that wish to 347 control latencies experienced by specific traffic flows. 349 1.3. Scope 351 The scope of this document is intra-domain traffic engineering. That 352 is, traffic engineering within a given autonomous system in the 353 Internet. This document discusses concepts pertaining to intra- 354 domain traffic control, including such issues as routing control, 355 micro and macro resource allocation, and the control coordination 356 problems that arise consequently. 358 This document describes and characterizes techniques already in use 359 or in advanced development for Internet traffic engineering. The way 360 these techniques fit together is discussed and scenarios in which 361 they are useful will be identified. 363 Although the emphasis in this document is on intra-domain traffic 364 engineering, in Section 7, an overview of the high level 365 considerations pertaining to inter-domain traffic engineering will be 366 provided. Inter-domain Internet traffic engineering is crucial to 367 the performance enhancement of the global Internet infrastructure. 369 Whenever possible, relevant requirements from existing IETF documents 370 and other sources are incorporated by reference. 372 1.4. Terminology 374 This section provides terminology which is useful for Internet 375 traffic engineering. The definitions presented apply to this 376 document. These terms may have other meanings elsewhere. 378 Busy hour: A one hour period within a specified interval of time 379 (typically 24 hours) in which the traffic load in a network or 380 sub-network is greatest. 382 Congestion: A state of a network resource in which the traffic 383 incident on the resource exceeds its output capacity over an 384 interval of time. 386 Congestion avoidance: An approach to congestion management that 387 attempts to obviate the occurrence of congestion. 389 Congestion control: An approach to congestion management that 390 attempts to remedy congestion problems that have already occurred. 392 Constraint-based routing: A class of routing protocols that take 393 specified traffic attributes, network constraints, and policy 394 constraints into account when making routing decisions. 395 Constraint-based routing is applicable to traffic aggregates as 396 well as flows. It is a generalization of QoS routing. 398 Demand side congestion management: A congestion management scheme 399 that addresses congestion problems by regulating or conditioning 400 offered load. 402 Effective bandwidth: The minimum amount of bandwidth that can be 403 assigned to a flow or traffic aggregate in order to deliver 404 'acceptable service quality' to the flow or traffic aggregate. 406 Hot-spot: A network element or subsystem which is in a state of 407 congestion. 409 Inter-domain traffic: Traffic that originates in one Autonomous 410 system and terminates in another. 412 Metric: A parameter defined in terms of standard units of 413 measurement. 415 Measurement methodology: A repeatable measurement technique used to 416 derive one or more metrics of interest. 418 Network survivability: The capability to provide a prescribed level 419 of QoS for existing services after a given number of failures 420 occur within the network. 422 Offline traffic engineering: A traffic engineering system that 423 exists outside of the network. 425 Online traffic engineering: A traffic engineering system that exists 426 within the network, typically implemented on or as adjuncts to 427 operational network elements. 429 Performance measures: Metrics that provide quantitative or 430 qualitative measures of the performance of systems or subsystems 431 of interest. 433 Performance metric: A performance parameter defined in terms of 434 standard units of measurement. 436 Provisioning: The process of assigning or configuring network 437 resources to meet certain requests. 439 QoS routing: Class of routing systems that selects paths to be used 440 by a flow based on the QoS requirements of the flow. 442 Service Level Agreement (SLA): A contract between a provider and a 443 customer that guarantees specific levels of performance and 444 reliability at a certain cost. 446 Service Level Objective (SLO): A key element of an SLA between a 447 provider and a customer. SLOs are agreed upon as a means of 448 measuring the performance of the Service Provider and are outlined 449 as a way of avoiding disputes between the two parties based on 450 misunderstanding. 452 Stability: An operational state in which a network does not 453 oscillate in a disruptive manner from one mode to another mode. 455 Supply-side congestion management: A congestion management scheme 456 that provisions additional network resources to address existing 457 and/or anticipated congestion problems. 459 Traffic characteristic: A description of the temporal behavior or a 460 description of the attributes of a given traffic flow or traffic 461 aggregate. 463 Traffic engineering system: A collection of objects, mechanisms, and 464 protocols that are used together to accomplish traffic engineering 465 objectives. 467 Traffic flow: A stream of packets between two end-points that can be 468 characterized in a certain way. A micro-flow has a more specific 469 definition A micro-flow is a stream of packets with the same 470 source and destination addresses, source and destination ports, 471 and protocol ID. 473 Traffic matrix: A representation of the traffic demand between a set 474 of origin and destination abstract nodes. An abstract node can 475 consist of one or more network elements. 477 Traffic monitoring: The process of observing traffic characteristics 478 at a given point in a network and collecting the traffic 479 information for analysis and further action. 481 Traffic trunk: An aggregation of traffic flows belonging to the same 482 class which are forwarded through a common path. A traffic trunk 483 may be characterized by an ingress and egress node, and a set of 484 attributes which determine its behavioral characteristics and 485 requirements from the network. 487 2. Background 489 The Internet must convey IP packets from ingress nodes to egress 490 nodes efficiently, expeditiously, and economically. Furthermore, in 491 a multiclass service environment (e.g., Diffserv capable networks - 492 see Section 4.1.4), the resource sharing parameters of the network 493 must be appropriately determined and configured according to 494 prevailing policies and service models to resolve resource contention 495 issues arising from mutual interference between packets traversing 496 through the network. Thus, consideration must be given to resolving 497 competition for network resources between traffic streams belonging 498 to the same service class (intra-class contention resolution) and 499 traffic streams belonging to different classes (inter-class 500 contention resolution). 502 2.1. Context of Internet Traffic Engineering 504 The context of Internet traffic engineering includes: 506 1. A network domain context that defines the scope under 507 consideration, and in particular the situations in which the 508 traffic engineering problems occur. The network domain context 509 includes network structure, network policies, network 510 characteristics, network constraints, network quality attributes, 511 and network optimization criteria. 513 2. A problem context defining the general and concrete issues that 514 traffic engineering addresses. The problem context includes 515 identification, abstraction of relevant features, representation, 516 formulation, specification of the requirements on the solution 517 space, and specification of the desirable features of acceptable 518 solutions. 520 3. A solution context suggesting how to address the issues 521 identified by the problem context. The solution context includes 522 analysis, evaluation of alternatives, prescription, and 523 resolution. 525 4. An implementation and operational context in which the solutions 526 are instantiated. The implementation and operational context 527 includes planning, organization, and execution. 529 The context of Internet traffic engineering and the different problem 530 scenarios are discussed in the following subsections. 532 2.2. Network Domain Context 534 IP networks range in size from small clusters of routers situated 535 within a given location, to thousands of interconnected routers, 536 switches, and other components distributed all over the world. 538 At the most basic level of abstraction, an IP network can be 539 represented as a distributed dynamic system consisting of: 541 o a set of interconnected resources which provide transport services 542 for IP traffic subject to certain constraints 544 o a demand system representing the offered load to be transported 545 through the network 547 o a response system consisting of network processes, protocols, and 548 related mechanisms which facilitate the movement of traffic 549 through the network (see also [AWD2]). 551 The network elements and resources may have specific characteristics 552 restricting the manner in which the traffic demand is handled. 553 Additionally, network resources may be equipped with traffic control 554 mechanisms managing the way in which the demand is serviced. Traffic 555 control mechanisms may be used to: 557 o control packet processing activities within a given resource 559 o arbitrate contention for access to the resource by different 560 packets 562 o regulate traffic behavior through the resource. 564 A configuration management and provisioning system may allow the 565 settings of the traffic control mechanisms to be manipulated by 566 external or internal entities in order to exercise control over the 567 way in which the network elements respond to internal and external 568 stimuli. 570 The details of how the network carries packets are specified in the 571 policies of the network administrators and are installed through 572 network configuration management and policy based provisioning 573 systems. Generally, the types of service provided by the network 574 also depend upon the technology and characteristics of the network 575 elements and protocols, the prevailing service and utility models, 576 and the ability of the network administrators to translate policies 577 into network configurations. 579 Internet networks have three significant characteristics: 581 o they provide real-time services 583 o they are mission critical 585 o their operating environments are very dynamic. 587 The dynamic characteristics of IP and IP/MPLS networks can be 588 attributed in part to fluctuations in demand, to the interaction 589 between various network protocols and processes, to the rapid 590 evolution of the infrastructure which demands the constant inclusion 591 of new technologies and new network elements, and to transient and 592 persistent faults which occur within the system. 594 Packets contend for the use of network resources as they are conveyed 595 through the network. A network resource is considered to be 596 congested if, for an interval of time, the arrival rate of packets 597 exceed the output capacity of the resource. Congestion may result in 598 some of the arriving packets being delayed or even dropped. 600 Congestion increases transit delay, delay variation, may lead to 601 packet loss, and reduces the predictability of network services. 602 Clearly, congestion is highly undesirable. Combating congestion at a 603 reasonable cost is a major objective of Internet traffic engineering. 605 Efficient sharing of network resources by multiple traffic streams is 606 a basic operational premise for the Internet. A fundamental 607 challenge in network operation is to increase resource utilization 608 while minimizing the possibility of congestion. 610 The Internet has to function in the presence of different classes of 611 traffic with different service requirements. RFC 2475 provides an 612 architecture for Differentiated Services (Diffserv) and makes this 613 requirement clear [RFC2475]. The RFC allows packets to be grouped 614 into behavior aggregates such that each aggregate has a common set of 615 behavioral characteristics or a common set of delivery requirements. 616 Delivery requirements of a specific set of packets may be specified 617 explicitly or implicitly. Two of the most important traffic delivery 618 requirements are capacity constraints and QoS constraints. 620 Capacity constraints can be expressed statistically as peak rates, 621 mean rates, burst sizes, or as some deterministic notion of effective 622 bandwidth. QoS requirements can be expressed in terms of: 624 o integrity constraints such as packet loss 626 o temporal constraints such as timing restrictions for the delivery 627 of each packet (delay) and timing restrictions for the delivery of 628 consecutive packets belonging to the same traffic stream (delay 629 variation). 631 2.3. Problem Context 633 There are several large problems associated with operating a network 634 described in the previous section. This section analyzes the problem 635 context in relation to traffic engineering. The identification, 636 abstraction, representation, and measurement of network features 637 relevant to traffic engineering are significant issues. 639 A particular challenge is to formulate the problems that traffic 640 engineering attempts to solve. For example: 642 o how to identify the requirements on the solution space 644 o how to specify the desirable features of solutions 646 o how to actually solve the problems 648 o how to measure and characterize the effectiveness of solutions. 650 Another class of problems is how to measure and estimate relevant 651 network state parameters. Effective traffic engineering relies on a 652 good estimate of the offered traffic load as well as a view of the 653 underlying topology and associated resource constraints. A network- 654 wide view of the topology is also a must for offline planning. 656 Still another class of problem is how to characterize the state of 657 the network and how to evaluate its performance. The performance 658 evaluation problem is two-fold: one aspect relates to the evaluation 659 of the system-level performance of the network; the other aspect 660 relates to the evaluation of resource-level performance, which 661 restricts attention to the performance analysis of individual network 662 resources. 664 In this document, we refer to the system-level characteristics of the 665 network as the "macro-states" and the resource-level characteristics 666 as the "micro-states." The system-level characteristics are also 667 known as the emergent properties of the network. Correspondingly, we 668 refer to the traffic engineering schemes dealing with network 669 performance optimization at the systems level as "macro-TE" and the 670 schemes that optimize at the individual resource level as "micro-TE." 671 Under certain circumstances, the system-level performance can be 672 derived from the resource-level performance using appropriate rules 673 of composition, depending upon the particular performance measures of 674 interest. 676 Another fundamental class of problem concerns how to effectively 677 optimize network performance. Performance optimization may entail 678 translating solutions for specific traffic engineering problems into 679 network configurations. Optimization may also entail some degree of 680 resource management control, routing control, and capacity 681 augmentation. 683 2.3.1. Congestion and its Ramifications 685 Congestion is one of the most significant problems in an operational 686 IP context. A network element is said to be congested if it 687 experiences sustained overload over an interval of time. Congestion 688 almost always results in degradation of service quality to end users. 689 Congestion control schemes can include demand-side policies and 690 supply-side policies. Demand-side policies may restrict access to 691 congested resources or dynamically regulate the demand to alleviate 692 the overload situation. Supply-side policies may expand or augment 693 network capacity to better accommodate offered traffic. Supply-side 694 policies may also re-allocate network resources by redistributing 695 traffic over the infrastructure. Traffic redistribution and resource 696 re-allocation serve to increase the 'effective capacity' of the 697 network. 699 The emphasis of this document is primarily on congestion management 700 schemes falling within the scope of the network, rather than on 701 congestion management systems dependent upon sensitivity and 702 adaptivity from end-systems. That is, the aspects that are 703 considered in this document with respect to congestion management are 704 those solutions that can be provided by control entities operating on 705 the network and by the actions of network administrators and network 706 operations systems. 708 2.4. Solution Context 710 The solution context for Internet traffic engineering involves 711 analysis, evaluation of alternatives, and choice between alternative 712 courses of action. Generally the solution context is based on making 713 reasonable inferences about the current or future state of the 714 network, and making decisions that may involve a preference between 715 alternative sets of action. More specifically, the solution context 716 demands reasonable estimates of traffic workload, characterization of 717 network state, derivation of solutions which may be implicitly or 718 explicitly formulated, and possibly instantiating a set of control 719 actions. Control actions may involve the manipulation of parameters 720 associated with routing, control over tactical capacity acquisition, 721 and control over the traffic management functions. 723 The following list of instruments may be applicable to the solution 724 context of Internet traffic engineering. 726 o A set of policies, objectives, and requirements (which may be 727 context dependent) for network performance evaluation and 728 performance optimization. 730 o A collection of online and possibly offline tools and mechanisms 731 for measurement, characterization, modeling, and control traffic, 732 and control over the placement and allocation of network 733 resources, as well as control over the mapping or distribution of 734 traffic onto the infrastructure. 736 o A set of constraints on the operating environment, the network 737 protocols, and the traffic engineering system itself. 739 o A set of quantitative and qualitative techniques and methodologies 740 for abstracting, formulating, and solving traffic engineering 741 problems. 743 o A set of administrative control parameters which may be 744 manipulated through a Configuration Management (CM) system. The 745 CM system itself may include a configuration control subsystem, a 746 configuration repository, a configuration accounting subsystem, 747 and a configuration auditing subsystem. 749 o A set of guidelines for network performance evaluation, 750 performance optimization, and performance improvement. 752 Determining traffic characteristics through measurement or estimation 753 is very useful within the realm the traffic engineering solution 754 space. Traffic estimates can be derived from customer subscription 755 information, traffic projections, traffic models, and from actual 756 measurements. The measurements may be performed at different levels, 757 e.g., at the traffic-aggregate level or at the flow level. 758 Measurements at the flow level or on small traffic aggregates may be 759 performed at edge nodes, when traffic enters and leaves the network. 760 Measurements for large traffic-aggregates may be performed within the 761 core of the network. 763 To conduct performance studies and to support planning of existing 764 and future networks, a routing analysis may be performed to determine 765 the paths the routing protocols will choose for various traffic 766 demands, and to ascertain the utilization of network resources as 767 traffic is routed through the network. Routing analysis captures the 768 selection of paths through the network, the assignment of traffic 769 across multiple feasible routes, and the multiplexing of IP traffic 770 over traffic trunks (if such constructs exist) and over the 771 underlying network infrastructure. A model of network topology is 772 necessary to perform routing analysis. A network topology model may 773 be extracted from: 775 o network architecture documents 777 o network designs 779 o information contained in router configuration files 781 o routing databases 783 o routing tables 785 o automated tools that discover and collate network topology 786 information. 788 Topology information may also be derived from servers that monitor 789 network state, and from servers that perform provisioning functions. 791 Routing in operational IP networks can be administratively controlled 792 at various levels of abstraction including the manipulation of BGP 793 attributes and interior gateway protocol (IGP) metrics. For path 794 oriented technologies such as MPLS, routing can be further controlled 795 by the manipulation of relevant traffic engineering parameters, 796 resource parameters, and administrative policy constraints. Within 797 the context of MPLS, the path of an explicitly routed label switched 798 path (LSP) can be computed and established in various ways including: 800 o manually 802 o automatically, online using constraint-based routing processes 803 implemented on label switching routers 805 o automatically, offline using constraint-based routing entities 806 implemented on external traffic engineering support systems. 808 2.4.1. Combating the Congestion Problem 810 Minimizing congestion is a significant aspect of Internet traffic 811 engineering. This subsection gives an overview of the general 812 approaches that have been used or proposed to combat congestion. 814 Congestion management policies can be categorized based upon the 815 following criteria (see [YARE95] for a more detailed taxonomy of 816 congestion control schemes): 818 1. Congestion Management based on Response Time Scales 820 * Long (weeks to months): Expanding network capacity by adding 821 new equipment, routers, and links takes time and is 822 comparatively costly. Capacity planning needs to take this 823 into consideration. Network capacity is expanded based on 824 estimates or forecasts of future traffic development and 825 traffic distribution. These upgrades are typically carried 826 out over weeks or months, or maybe even years. 828 * Medium (minutes to days): Several control policies fall within 829 the medium timescale category. Examples include: 831 a. Adjusting routing protocol parameters to route traffic 832 away or towards certain segments of the network. 834 b. Setting up or adjusting explicitly routed LSPs in MPLS 835 networks to route traffic trunks away from possibly 836 congested resources or toward possibly more favorable 837 routes. 839 c. Re-configuring the logical topology of the network to make 840 it correlate more closely with the spatial traffic 841 distribution using, for example, an underlying path- 842 oriented technology such as MPLS LSPs or optical channel 843 trails. 845 Many of these adaptive schemes rely on measurement systems. A 846 measurement system monitors changes in traffic distribution, 847 traffic loads, and network resource utilization and then 848 provides feedback to the online or offline traffic engineering 849 mechanisms and tools so that they can trigger control actions 850 within the network. The traffic engineering mechanisms and 851 tools can be implemented in a distributed or centralized 852 fashion. A centralized scheme may have global visibility into 853 the network state and may produce more optimal solutions. 854 However, centralized schemes are prone to single points of 855 failure and may not scale as well as distributed schemes. 856 Moreover, the information utilized by a centralized scheme may 857 be stale and might not reflect the actual state of the 858 network. It is not an objective of this document to make a 859 recommendation between distributed and centralized schemes: 860 that is a choice that network administrators must make based 861 on their specific needs. 863 * Short (picoseconds to minutes): This category includes packet 864 level processing functions and events that are recorded on the 865 order of several round trip times. It also includes router 866 mechanisms such as passive and active buffer management. All 867 of these mechanisms are used to control congestion or signal 868 congestion to end systems so that they can adaptively regulate 869 the rate at which traffic is injected into the network. One 870 of the most popular active queue management schemes, 871 especially for TCP traffic, is Random Early Detection (RED) 872 [FLJA93]. During congestion (but before the queue is filled), 873 the RED scheme chooses arriving packets to "mark" according to 874 a probabilistic algorithm which takes into account the average 875 queue size. A router that does not utilize explicit 876 congestion notification (ECN) [FLOY94] can simply drop marked 877 packets to alleviate congestion and implicitly notify the 878 receiver about the congestion. On the other hand, if the 879 router supports ECN, it can set the ECN field in the packet 880 header. Several variations of RED have been proposed to 881 support different drop precedence levels in multi-class 882 environments [RFC2597]. RED provides congestion avoidance 883 which is not worse than traditional Tail-Drop (TD) queue 884 management (drop arriving packets only when the queue is 885 full). Importantly, RED reduces the possibility of global 886 synchronization where retransmission burst become synchronized 887 across the whole network, and improves fairness among 888 different TCP sessions. However, RED by itself cannot prevent 889 congestion and unfairness caused by sources unresponsive to 890 RED, e.g., UDP traffic and some misbehaved greedy connections. 891 Other schemes have been proposed to improve the performance 892 and fairness in the presence of unresponsive traffic. Some of 893 those schemes (such as Longest Queue Drop (LQD) and Dynamic 894 Soft Partitioning with Random Drop (RND) [SLDC98]) were 895 proposed as theoretical frameworks and are typically not 896 available in existing commercial products. 898 2. Congestion Management: Reactive Versus Preventive Schemes 900 * Reactive: Reactive (recovery) congestion management policies 901 react to existing congestion problems. All the policies 902 described above for the long and medium time scales can be 903 categorized as being reactive. They are based on monitoring 904 and identifying congestion problems that exist in the network, 905 and on the initiation of relevant actions to ease a situation. 907 * Preventive: Preventive (predictive/avoidance) policies take 908 proactive action to prevent congestion based on estimates and 909 predictions of future congestion problems. Some of the 910 policies described for the long and medium time scales fall 911 into this category. Preventive policies do not necessarily 912 respond immediately to existing congestion problems. Instead, 913 forecasts of traffic demand and workload distribution are 914 considered, and action may be taken to prevent potential 915 future congestion problems. The schemes described for the 916 short time scale can also be used for congestion avoidance 917 because dropping or marking packets before queues actually 918 overflow would trigger corresponding TCP sources to slow down. 920 3. Congestion Management: Supply-Side Versus Demand-Side Schemes 922 * Supply-side: Supply-side congestion management policies 923 increase the effective capacity available to traffic in order 924 to control or reduce congestion. This can be accomplished by 925 increasing capacity or by balancing distribution of traffic 926 over the network. Capacity planning aims to provide a 927 physical topology and associated link bandwidths that match or 928 exceed estimated traffic workload and traffic distribution 929 subject to traffic forecasts and budgetary or other 930 constraints. If the actual traffic distribution does not fit 931 the topology derived from capacity panning, then the traffic 932 can be mapped onto the topology by using routing control 933 mechanisms, by applying path oriented technologies (e.g., MPLS 934 LSPs and optical channel trails) to modify the logical 935 topology, or by employing some other load redistribution 936 mechanisms. 938 * Demand-side: Demand-side congestion management policies 939 control or regulate the offered traffic to alleviate 940 congestion problems. For example, some of the short time 941 scale mechanisms described earlier as well as policing and 942 rate-shaping mechanisms attempt to regulate the offered load 943 in various ways. 945 2.5. Implementation and Operational Context 947 The operational context of Internet traffic engineering is 948 characterized by constant changes that occur at multiple levels of 949 abstraction. The implementation context demands effective planning, 950 organization, and execution. The planning aspects may involve 951 determining prior sets of actions to achieve desired objectives. 952 Organizing involves arranging and assigning responsibility to the 953 various components of the traffic engineering system and coordinating 954 the activities to accomplish the desired TE objectives. Execution 955 involves measuring and applying corrective or perfective actions to 956 attain and maintain desired TE goals. 958 3. Traffic Engineering Process Models 960 This section describes a generic process model that captures the 961 high-level practical aspects of Internet traffic engineering in an 962 operational context. The process model is described as a sequence of 963 actions that must be carried out to optimize the performance of an 964 operational network (see also [RFC2702], [AWD2]). This process model 965 may be enacted explicitly or implicitly, by a software process or by 966 a human. 968 The traffic engineering process model is iterative [AWD2]. The four 969 phases of the process model described below are repeated as a 970 continual sequence. 972 o Define the relevant control policies that govern the operation of 973 the network. 975 o Acquire measurement data from the operational network. 977 o Analyze the network state and characterize the traffic workload. 978 Proactive analysis identifies potential problems that could 979 manifest in the future. Reactive analysis identifies existing 980 problems and determines their causes. 982 o Optimize the performance of the network. This involves a decision 983 process which selects and implements a set of actions from a set 984 of alternatives given the results of the three previous steps. 985 Optimization actions may include the use of techniques to control 986 the offered traffic and to control the distribution of traffic 987 across the network. 989 3.1. Components of the Traffic Engineering Process Model 991 The key components of the traffic engineering process model are as 992 follows. 994 1. Measurement is crucial to the traffic engineering function. The 995 operational state of a network can only be conclusively 996 determined through measurement. Measurement is also critical to 997 the optimization function because it provides feedback data which 998 is used by traffic engineering control subsystems. This data is 999 used to adaptively optimize network performance in response to 1000 events and stimuli originating within and outside the network. 1001 Measurement in support of the TE function can occur at different 1002 levels of abstraction. For example, measurement can be used to 1003 derive packet level characteristics, flow level characteristics, 1004 user or customer level characteristics, traffic aggregate 1005 characteristics, component level characteristics, and network 1006 wide characteristics. 1008 2. Modeling, analysis, and simulation are important aspects of 1009 Internet traffic engineering. Modeling involves constructing an 1010 abstract or physical representation which depicts relevant 1011 traffic characteristics and network attributes. A network model 1012 is an abstract representation of the network which captures 1013 relevant network features, attributes, and characteristic. 1014 Network simulation tools are extremely useful for traffic 1015 engineering. Because of the complexity of realistic quantitative 1016 analysis of network behavior, certain aspects of network 1017 performance studies can only be conducted effectively using 1018 simulation. 1020 3. Network performance optimization involves resolving network 1021 issues by transforming such issues into concepts that enable a 1022 solution, identification of a solution, and implementation of the 1023 solution. Network performance optimization can be corrective or 1024 perfective. In corrective optimization, the goal is to remedy a 1025 problem that has occurred or that is incipient. In perfective 1026 optimization, the goal is to improve network performance even 1027 when explicit problems do not exist and are not anticipated. 1029 4. Review of TE Techniques 1031 This section briefly reviews different traffic engineering approaches 1032 proposed and implemented in telecommunications and computer networks 1033 using IETF protocols and architectures. The discussion is not 1034 intended to be comprehensive. It is primarily intended to illuminate 1035 existing approaches to traffic engineering in the Internet. A 1036 historic overview of traffic engineering in telecommunications 1037 networks is provided in Appendix A, while Appendix B describes 1038 approaches in other standards bodies. 1040 4.1. Overview of IETF Projects Related to Traffic Engineering 1042 This subsection reviews a number of IETF activities pertinent to 1043 Internet traffic engineering. 1045 4.1.1. Constraint-Based Routing 1047 Constraint-based routing refers to a class of routing systems that 1048 compute routes through a network subject to the satisfaction of a set 1049 of constraints and requirements. In the most general case, 1050 constraint-based routing may also seek to optimize overall network 1051 performance while minimizing costs. 1053 The constraints and requirements may be imposed by the network itself 1054 or by administrative policies. Constraints may include bandwidth, 1055 hop count, delay, and policy instruments such as resource class 1056 attributes. Constraints may also include domain specific attributes 1057 of certain network technologies and contexts which impose 1058 restrictions on the solution space of the routing function. Path 1059 oriented technologies such as MPLS have made constraint-based routing 1060 feasible and attractive in public IP networks. 1062 The concept of constraint-based routing within the context of MPLS 1063 traffic engineering requirements in IP networks was first described 1064 in [RFC2702] and led to developments such as MPLS-TE [RFC3209] as 1065 described in Section 4.1.6. 1067 Unlike QoS routing (for example, see [RFC2386] and [MA]) which 1068 generally addresses the issue of routing individual traffic flows to 1069 satisfy prescribed flow-based QoS requirements subject to network 1070 resource availability, constraint-based routing is applicable to 1071 traffic aggregates as well as flows and may be subject to a wide 1072 variety of constraints which may include policy restrictions. 1074 4.1.1.1. IGP Flexible Algortithms (Flex-Algos) 1076 The traditional approach to routing in an IGP network relies on the 1077 IGPs deriving "shortest paths" over the network based solely on the 1078 IGP metric assigned to the links. Such an approach is often limited: 1079 traffic may tend to converge toward the destination, possibly causing 1080 congestion; and it is not possible to steer traffic onto paths 1081 depending on the end-to-end qualities demanded by the applications. 1083 To overcome this limitation, various sorts of traffic engineering 1084 have been widely deployed (as described in this document), where the 1085 TE component is responsible for computing the path based on 1086 additionalcmetrics and/or constraints. Such paths (or tunnels) need 1087 to be installed in the routers' forwarding tables in addition to, or 1088 as a replacement for the original paths computed by IGPs. The main 1089 drawback of these TE approaches is the additional complexity of 1090 protocols and management, and the state that may need to be 1091 maintained within the network. 1093 IGP flexible algorithms (flex-algos) [I-D.ietf-lsr-flex-algo] allow 1094 IGPs to construct constraint-based paths over the network by 1095 computing constraint- based next hops. The intent of flex-algos is 1096 to reduce TE complexity by letting an IGP perform some basic TE 1097 computation capabilities. Flex-algo includes a set of extensions to 1098 the IGPs that enable a router to send TLVs that: 1100 o describe a set of constraints on the topology 1101 o identify calculation-type 1103 o describe a metric-type that is to be used to compute the best 1104 paths through the constrained topology. 1106 A given combination of calculation-type, metric-type, and constraints 1107 is known as a "Flexible Algorithm Definition" (or FAD). A router 1108 that sends such a set of TLVs also assigns a specific identifier (the 1109 Flexible Algorithm) to the specified combination of calculation-type, 1110 metric-type, and constraints. 1112 There are two use cases for flex-algo: in IP networks 1113 [I-D.bonica-lsr-ip-flexalgo] and in segment routing networks 1114 [I-D.ietf-lsr-flex-algo]. In the first case, flex-algo computes 1115 paths to an IPv4 or IPv6 address, in the second case, flex-algo 1116 computes paths to a prefix SID (see Section 4.1.15). 1118 There are many use cases where flex-algo can bring big value, such 1119 as: 1121 o Expansion of functionality of IP Performance metrics [RFC5664] 1122 when points of interest could instantiate specific constraint- 1123 based routing (flex-algo) based on the measurement results. 1125 o Nested usage of flex-algo and TE extsensions for IGP (see 1126 Section 4.1.11) when we can form 'underlay' by means of flex-algo 1127 and 'overlay' by TE. 1129 o Flex-algo in SR-MPLS (Section 4.1.15) is a base use case when we 1130 can easily benefit from TE-like topology that will be built 1131 without external TE component on routers or PCE (see 1132 Section 4.1.13). 1134 o Building of network slices 1135 [I-D.nsdt-teas-ietf-network-slice-definition] where particular 1136 IETF network slice SLO can be guaranteed by flex-algo. 1138 4.1.2. Integrated Services 1140 The IETF developed the Integrated Services (Intserv) model that 1141 requires resources, such as bandwidth and buffers, to be reserved a 1142 priori for a given traffic flow to ensure that the quality of service 1143 requested by the traffic flow is satisfied. The Integrated Services 1144 model includes additional components beyond those used in the best- 1145 effort model such as packet classifiers, packet schedulers, and 1146 admission control. A packet classifier is used to identify flows 1147 that are to receive a certain level of service. A packet scheduler 1148 handles the scheduling of service to different packet flows to ensure 1149 that QoS commitments are met. Admission control is used to determine 1150 whether a router has the necessary resources to accept a new flow. 1152 The main issue with the Integrated Services model has been 1153 scalability [RFC2998], especially in large public IP networks which 1154 may potentially have millions of active micro-flows in transit 1155 concurrently. 1157 A notable feature of the Integrated Services model is that it 1158 requires explicit signaling of QoS requirements from end systems to 1159 routers [RFC2753]. The Resource Reservation Protocol (RSVP) performs 1160 this signaling function and is a critical component of the Integrated 1161 Services model. RSVP is described in Section 4.1.3. 1163 4.1.3. RSVP 1165 RSVP is a soft state signaling protocol [RFC2205]. It supports 1166 receiver initiated establishment of resource reservations for both 1167 multicast and unicast flows. RSVP was originally developed as a 1168 signaling protocol within the Integrated Services framework (see 1169 Section 4.1.2) for applications to communicate QoS requirements to 1170 the network and for the network to reserve relevant resources to 1171 satisfy the QoS requirements [RFC2205]. 1173 In RSVP, the traffic sender or source node sends a PATH message to 1174 the traffic receiver with the same source and destination addresses 1175 as the traffic which the sender will generate. The PATH message 1176 contains: (1) a sender traffic specification describing the 1177 characteristics of the traffic, (2) a sender template specifying the 1178 format of the traffic, and (3) an optional advertisement 1179 specification which is used to support the concept of One Pass With 1180 Advertising (OPWA) [RFC2205]. Every intermediate router along the 1181 path forwards the PATH message to the next hop determined by the 1182 routing protocol. Upon receiving a PATH message, the receiver 1183 responds with a RESV message which includes a flow descriptor used to 1184 request resource reservations. The RESV message travels to the 1185 sender or source node in the opposite direction along the path that 1186 the PATH message traversed. Every intermediate router along the path 1187 can reject or accept the reservation request of the RESV message. If 1188 the request is rejected, the rejecting router will send an error 1189 message to the receiver and the signaling process will terminate. If 1190 the request is accepted, link bandwidth and buffer space are 1191 allocated for the flow and the related flow state information is 1192 installed in the router. 1194 One of the issues with the original RSVP specification was 1195 Scalability. This is because reservations were required for micro- 1196 flows, so that the amount of state maintained by network elements 1197 tends to increase linearly with the number of micro-flows. These 1198 issues are described in [RFC2961] which also modifies and extends 1199 RSVP to mitigate the scaling problems to make RSVP a versatile 1200 signaling protocol for the Internet. For example, RSVP has been 1201 extended to reserve resources for aggregation of flows, to set up 1202 MPLS explicit label switched paths (see Section 4.1.6), and to 1203 perform other signaling functions within the Internet. [RFC2961] 1204 also describes a mechanism to reduce the amount of Refresh messages 1205 required to maintain established RSVP sessions. 1207 4.1.4. Differentiated Services 1209 The goal of Differentiated Services (Diffserv) within the IETF was to 1210 devise scalable mechanisms for categorization of traffic into 1211 behavior aggregates, which ultimately allows each behavior aggregate 1212 to be treated differently, especially when there is a shortage of 1213 resources such as link bandwidth and buffer space [RFC2475]. One of 1214 the primary motivations for Diffserv was to devise alternative 1215 mechanisms for service differentiation in the Internet that mitigate 1216 the scalability issues encountered with the Intserv model. 1218 Diffserv uses the Differentiated Services field in the IP header (the 1219 DS field) consisting of six bits in what was formerly known as the 1220 Type of Service (TOS) octet. The DS field is used to indicate the 1221 forwarding treatment that a packet should receive at a transit node 1222 [RFC2474]. Diffserv includes the concept of Per-Hop Behavior (PHB) 1223 groups. Using the PHBs, several classes of services can be defined 1224 using different classification, policing, shaping, and scheduling 1225 rules. 1227 For an end-user of network services to utilize Differentiated 1228 Services provided by its Internet Service Provider (ISP), it may be 1229 necessary for the user to have an SLA with the ISP. An SLA may 1230 explicitly or implicitly specify a Traffic Conditioning Agreement 1231 (TCA) which defines classifier rules as well as metering, marking, 1232 discarding, and shaping rules. 1234 Packets are classified, and possibly policed and shaped at the 1235 ingress to a Diffserv network. When a packet traverses the boundary 1236 between different Diffserv domains, the DS field of the packet may be 1237 re-marked according to existing agreements between the domains. 1239 Differentiated Services allows only a finite number of service 1240 classes to be specified by the DS field. The main advantage of the 1241 Diffserv approach relative to the Intserv model is scalability. 1242 Resources are allocated on a per-class basis and the amount of state 1243 information is proportional to the number of classes rather than to 1244 the number of application flows. 1246 The Diffserv model deals with traffic management issues on a per hop 1247 basis. The Diffserv control model consists of a collection of micro- 1248 TE control mechanisms. Other traffic engineering capabilities, such 1249 as capacity management (including routing control), are also required 1250 in order to deliver acceptable service quality in Diffserv networks. 1251 The concept of Per Domain Behaviors has been introduced to better 1252 capture the notion of Differentiated Services across a complete 1253 domain [RFC3086]. 1255 4.1.5. QUIC 1257 QUIC [I-D.ietf-quic-transport] is a UDP-based multiplexed and secure 1258 transport protocol. QUIC provides applications with flow-controlled 1259 streams for structured communication, low-latency connection 1260 establishment, and network path migration. 1262 QUIC is a connection-oriented protocol that creates a stateful 1263 interaction between a client and server. QUIC uses a handshake 1264 procedure that combines negotiation of cryptographic and transport 1265 parameters. This is a key differentiation from other transport 1266 protocols. 1268 Endpoints communicate in QUIC by exchanging QUIC packets that use a 1269 customized framing for protectiong. Most QUIC packets contain 1270 frames, which carry control information and application data between 1271 endpoints. QUIC authenticates all packets and encrypts as much as is 1272 practical. QUIC packets are carried in UDP datagrams to better 1273 facilitate deployment within existing systems and networks. 1275 Application protocols exchange information over a QUIC connection via 1276 streams, which are ordered sequences of bytes. Two types of stream 1277 can be created: bidirectional streams, which allow both endpoints to 1278 send data; and unidirectional streams, which allow a single endpoint 1279 to send data. A credit-based scheme is used to limit stream creation 1280 and to bound the amount of data that can be sent. 1282 QUIC provides the necessary feedback to implement reliable delivery 1283 and congestion control to avoid network congestion. 1285 4.1.6. Multiprotocol Label Switching (MPLS) 1287 MPLS is an advanced forwarding scheme which also includes extensions 1288 to conventional IP control plane protocols. MPLS extends the 1289 Internet routing model and enhances packet forwarding and path 1290 control [RFC3031]. 1292 At the ingress to an MPLS domain, Label Switching Routers (LSRs) 1293 classify IP packets into Forwarding Equivalence Classes (FECs) based 1294 on a variety of factors, including, e.g., a combination of the 1295 information carried in the IP header of the packets and the local 1296 routing information maintained by the LSRs. An MPLS label stack 1297 entry is then prepended to each packet according to their forwarding 1298 equivalence classes. The MPLS label stack entry is 32 bits long and 1299 contains a 20-bit label field. 1301 An LSR makes forwarding decisions by using the label prepended to 1302 packets as the index into a local next hop label forwarding entry 1303 (NHLFE). The packet is then processed as specified in the NHLFE. 1304 The incoming label may be replaced by an outgoing label (label swap), 1305 and the packet may be forwarded to the next LSR. Before a packet 1306 leaves an MPLS domain, its MPLS label may be removed (label pop). A 1307 Label Switched Path (LSP) is the path between an ingress LSRs and an 1308 egress LSRs through which a labeled packet traverses. The path of an 1309 explicit LSP is defined at the originating (ingress) node of the LSP. 1310 MPLS can use a signaling protocol such as RSVP or LDP to set up LSPs. 1312 MPLS is a very powerful technology for Internet traffic engineering 1313 because it supports explicit LSPs which allow constraint-based 1314 routing to be implemented efficiently in IP networks [AWD2]. The 1315 requirements for traffic engineering over MPLS are described in 1316 [RFC2702]. Extensions to RSVP to support instantiation of explicit 1317 LSP are discussed in [RFC3209]. 1319 4.1.7. Generalized MPLS (GMPLS) 1321 GMPLS extends MPLS control protocols to encompass time-division 1322 (e.g., SONET/SDH, PDH, G.709), wavelength (lambdas), and spatial 1323 switching (e.g., incoming port or fiber to outgoing port or fiber) as 1324 well as continuing to support packet switching. GMPLS provides a 1325 common set of control protocols for all of these layers (including 1326 some technology-specific extensions) each of which has a diverse data 1327 or forwarding plane. GMPLS covers both the signaling and the routing 1328 part of that control plane and is based on the Traffic Engineering 1329 extensions to MPLS (see Section 4.1.6). 1331 In GMPLS, the original MPLS architecture is extended to include LSRs 1332 whose forwarding planes rely on circuit switching, and therefore 1333 cannot forward data based on the information carried in either packet 1334 or cell headers. Specifically, such LSRs include devices where the 1335 switching is based on time slots, wavelengths, or physical ports. 1336 These additions impact basic LSP properties: how labels are requested 1337 and communicated, the unidirectional nature of MPLS LSPs, how errors 1338 are propagated, and information provided for synchronizing the 1339 ingress and egress LSRs. 1341 4.1.8. IP Performance Metrics 1343 The IETF IP Performance Metrics (IPPM) working group has developed a 1344 set of standard metrics that can be used to monitor the quality, 1345 performance, and reliability of Internet services. These metrics can 1346 be applied by network operators, end-users, and independent testing 1347 groups to provide users and service providers with a common 1348 understanding of the performance and reliability of the Internet 1349 component 'clouds' they use/provide [RFC2330]. The criteria for 1350 performance metrics developed by the IPPM working group are described 1351 in [RFC2330]. Examples of performance metrics include one-way packet 1352 loss [RFC7680], one-way delay [RFC7679], and connectivity measures 1353 between two nodes [RFC2678]. Other metrics include second-order 1354 measures of packet loss and delay. 1356 Some of the performance metrics specified by the IPPM working group 1357 are useful for specifying SLAs. SLAs are sets of service level 1358 objectives negotiated between users and service providers, wherein 1359 each objective is a combination of one or more performance metrics, 1360 possibly subject to certain constraints. 1362 4.1.9. Flow Measurement 1364 The IETF Real Time Flow Measurement (RTFM) working group produced an 1365 architecture that defines a method to specify traffic flows as well 1366 as a number of components for flow measurement (meters, meter 1367 readers, manager) [RFC2722]. A flow measurement system enables 1368 network traffic flows to be measured and analyzed at the flow level 1369 for a variety of purposes. As noted in RFC 2722, a flow measurement 1370 system can be very useful in the following contexts: 1372 o understanding the behavior of existing networks 1374 o planning for network development and expansion 1376 o quantification of network performance 1378 o verifying the quality of network service 1380 o attribution of network usage to users. 1382 A flow measurement system consists of meters, meter readers, and 1383 managers. A meter observes packets passing through a measurement 1384 point, classifies them into groups, accumulates usage data (such as 1385 the number of packets and bytes for each group), and stores the usage 1386 data in a flow table. A group may represent any collection of user 1387 applications, hosts, networks, etc. A meter reader gathers usage 1388 data from various meters so it can be made available for analysis. A 1389 manager is responsible for configuring and controlling meters and 1390 meter readers. The instructions received by a meter from a manager 1391 include flow specifications, meter control parameters, and sampling 1392 techniques. The instructions received by a meter reader from a 1393 manager include the address of the meter whose date is to be 1394 collected, the frequency of data collection, and the types of flows 1395 to be collected. 1397 4.1.10. Endpoint Congestion Management 1399 [RFC3124] provides a set of congestion control mechanisms for the use 1400 of transport protocols. It is also allows the development of 1401 mechanisms for unifying congestion control across a subset of an 1402 endpoint's active unicast connections (called a congestion group). A 1403 congestion manager continuously monitors the state of the path for 1404 each congestion group under its control. The manager uses that 1405 information to instruct a scheduler on how to partition bandwidth 1406 among the connections of that congestion group. 1408 4.1.11. TE Extensions to the IGPs 1410 [RFC5305] describes the extensions to the Intermediate System to 1411 Intermediate System (IS-IS) protocol to support TE, similarly 1412 [RFC3630] specifies TE extensions for OSPFv2 ([RFC5329] has the same 1413 description for OSPFv3). 1415 The idea of redistribution TE extensions such as link type and ID, 1416 local and remote IP addresses, TE metric, maximum bandwidth, maximum 1417 reservable bandwidth and unreserved bandwidth, admin group in IGP is 1418 a common for both IS-IS and OSPF. 1420 The difference is in the details of their transmission: IS-IS uses 1421 the Extended IS Reachability TLV (type 22) and Sub-TLVs for those TE 1422 parameters, OSPFv2 uses Opaque LSA [RFC2370] type 10 (OSPFv3 uses 1423 Intra-Area-TE-LSA) with two top-level TLV (Router Address and Link) 1424 also with Sub-TLVs for that purpose. 1426 IS-IS also uses the Extended IP Reachability TLV (type 135, which 1427 have the new 32 bit metric) and the TE Router ID TLV (type 134). 1428 Those Sub-TLV details are described in [RFC7810] for IS-IS and in 1429 [RFC7471] for OSPFv2 ([RFC5329] for OSPFv3). 1431 4.1.12. Link-State BGP 1433 In a number of environments, a component external to a network is 1434 called upon to perform computations based on the network topology and 1435 current state of the connections within the network, including 1436 traffic engineering information. This is information typically 1437 distributed by IGP routing protocols within the network (see 1438 Section 4.1.11. 1440 The Border Gateway Protocol (BGP) Section 7 is one of the essential 1441 routing protocols that glue the Internet together. BGP Link State 1442 (BGP-LS) [RFC7752] is a mechanism by which link-state and traffic 1443 engineering information can be collected from networks and shared 1444 with external components using the BGP routing protocol. The 1445 mechanism is applicable to physical and virtual IGP links, and is 1446 subject to policy control. 1448 Information collected by BGP-LS can be used to construct the Traffic 1449 Engineering Database (TED, see Section 4.1.19) for use by the Path 1450 Computation Element (PCE, see Section 4.1.13), or may be used by 1451 Application-Layer Traffic Optimization (ALTO) servers (see 1452 Section 4.1.14). 1454 4.1.13. Path Computation Element 1456 Constraint-based path computation is a fundamental building block for 1457 traffic engineering in MPLS and GMPLS networks. Path computation in 1458 large, multi-domain networks is complex and may require special 1459 computational components and cooperation between the elements in 1460 different domains. The Path Computation Element (PCE) [RFC4655] is 1461 an entity (component, application, or network node) that is capable 1462 of computing a network path or route based on a network graph and 1463 applying computational constraints. 1465 Thus, a PCE can provide a central component in a traffic engineering 1466 system operating on the Traffic Engineering Database (TED, see 1467 Section 4.1.19) with delegated responsibility for determining paths 1468 in MPLS, GMPLS, or Segment Routing networks. The PCE uses the Path 1469 Computation Element Communication Protocol (PCEP) [RFC5440] to 1470 communicate with Path Computation Clients (PCCs), such as MPLS LSRs, 1471 to answer their requests for computed paths or to instruct them to 1472 initiate new paths [RFC8281] and maintain state about paths already 1473 installed in the network [RFC8231]. 1475 PCEs form key components of a number of traffic engineering systems, 1476 such as the Application of the Path Computation Element Architecture 1477 [RFC6805], the Applicability of a Stateful Path Computation Element 1478 ([RFC8051]), Abstraction and Control of TE Networks (ACTN) 1479 (Section 4.1.16), Centralized Network Control [RFC8283], and Software 1480 Defined Networking (SDN) (Section 5.3.2). 1482 4.1.14. Application-Layer Traffic Optimization 1484 This document describes various TE mechanisms available in the 1485 network. However, distributed applications in general and, in 1486 particular, bandwidth-greedy P2P applications that are used, for 1487 example, for file sharing, cannot directly use those techniques. As 1488 per [RFC5693], applications could greatly improve traffic 1489 distribution and quality by cooperating with external services that 1490 are aware of the network topology. Addressing the Application-Layer 1491 Traffic Optimization (ALTO) problem means, on the one hand, deploying 1492 an ALTO service to provide applications with information regarding 1493 the underlying network (e.g., basic network location structure and 1494 preferences of network paths) and, on the other hand, enhancing 1495 applications in order to use such information to perform better-than- 1496 random selection of the endpoints with which they establish 1497 connections. 1499 The basic function of ALTO is based on abstract maps of a network. 1500 These maps provide a simplified view, yet enough information about a 1501 network for applications to effectively utilize them. Additional 1502 services are built on top of the maps. [RFC7285] describes a 1503 protocol implementing the ALTO services as an information-publishing 1504 interface that allows a network to publish its network information 1505 such as network locations, costs between them at configurable 1506 granularities, and end-host properties to network applications. The 1507 information published by the ALTO Protocol should benefit both the 1508 network and the applications. The ALTO Protocol uses a REST-ful 1509 design and encodes its requests and responses using JSON [RFC7159] 1510 with a modular design by dividing ALTO information publication into 1511 multiple ALTO services (e.g., the Map service, the Map-Filtering 1512 Service, the Endpoint Property Service, and the Endpoint Cost 1513 Service). 1515 [RFC8189] defines a new service that allows an ALTO Client to 1516 retrieve several cost metrics in a single request for an ALTO 1517 filtered cost map and endpoint cost map. [RFC8896] extends the ALTO 1518 cost information service so that applications decide not only 'where' 1519 to connect, but also 'when'. This is useful for applications that 1520 need to perform bulk data transfer and would like to schedule these 1521 transfers during an off-peak hour, for example. 1522 [I-D.ietf-alto-performance-metrics] introducing network performance 1523 metrics, including network delay, jitter, packet loss rate, hop 1524 count, and bandwidth. The ALTO server may derive and aggregate such 1525 performance metrics from BGP-LS (see Section 4.1.12) or IGP-TE (see 1526 Section 4.1.11), or management tools, and then expose the information 1527 to allow applications to determine 'where' to connect based on 1528 network performance criteria. ALTO WG is evaluating the use of 1529 network TE properties while making application decisions for new use- 1530 cases such as Edge computing and Datacenter interconnect. 1532 4.1.15. Segment Routing with MPLS Encapsulation (SR-MPLS) 1534 Segment Routing (SR) [RFC8402] leverages the source routing and 1535 tunneling paradigms. The path a packet takes is defined at the 1536 ingress and the packet is tunneled to the egress. A node steers a 1537 packet through a controlled set of instructions, called segments, by 1538 prepending the packet with an SR header: a label stack in MPLS case. 1540 A segment can represent any instruction, topological or service- 1541 based, thanks to the MPLS architecture [RFC3031]. Labels can be 1542 looked up in a global context (platform wide) as well as in some 1543 other context (see "context labels" in Section 3 of [RFC5331]). 1545 4.1.15.1. Base Segment Routing Identifier Types 1547 Segments are identified by Segment Identifiers (SIDs). There are 1548 four types of SID that are relevant for traffic engineering. 1550 Prefix SID: Uses the SR Global Block (SRGB), must be unique within 1551 the routing domain SRGB, and is advertised by an IGP. The Prefix- 1552 SID can be configured as an absolute value or an index. 1554 Node SID: A Prefix SID with the 'N' (node) bit set. It is 1555 associated with a host prefix (/32 or /128) that identifies the 1556 node. More than 1 Node SID can be configured per node. 1558 Adjacency SID: Locally significant by default, an Adjacency SID can 1559 be made globally significant through use of the 'L' flag. It 1560 identifies a unidirectional adjacency. In most implementations 1561 Adjacency SIDs are automatically allocated for each adjacency. 1562 They are always encoded as an absolute (not indexed) value. 1564 Binding SID: A Binding SID has two purposes: 1566 1. Mapping Server in ISIS 1568 The SID/Label Binding TLV is used to advertise the mappings 1569 of prefixes to SIDs/Labels. This functionality is called 1570 the Segment Routing Mapping Server (SRMS). The behavior of 1571 the SRMS is defined in [RFC8661] 1573 2. Cross-connect (label to FEC mapping) 1575 This is fundamental for multi-domain/multi-layer operation. 1576 The Binding SID identifies a new path available at the 1577 anchor point. It is always local to the originator, must 1578 not be present at the top of the stack, and must be looked 1579 up in the context of the Node SID. It could be provisioned 1580 through Netconf/Restconf, PCEP, BGP, or the CLI. 1582 4.1.15.2. Segment Routing Policy 1584 SR Policy [I-D.ietf-spring-segment-routing-policy] is an evolution of 1585 Segment Routing to enhance the TE capabilities. It is a framework 1586 that enables instantiation of an ordered list of segments on a node 1587 for implementing a source routing policy with a specific intent for 1588 traffic steering from that node. 1590 An SR Policy is identified through the tuple . The headend is the IP address of the node where the 1592 policy is instantiated. The endpoint is the IP address of the 1593 destination of the policy. The color is an index that associates the 1594 SR Policy with an intent (e.g., low-latency). 1596 The headend node is notified of SR Policies and associated SR paths 1597 via configuration or by a extensions to protocols such as PCEP 1598 [RFC8664] or BGP [I-D.ietf-idr-segment-routing-te-policy]. Each SR 1599 path consists of a Segment-List (an SR source-routed path), and the 1600 headend uses the endpoint and color parameters to classify packets to 1601 match the SR policy and so determine along which path to forward 1602 them. If an SR Policy is associated with a set of SR paths, each is 1603 associated with a weight for weighted load balancing. Furthermore, 1604 multiple SR Policies may be associated with a set of SR paths to 1605 allow multiple traffic flows to be placed on the same paths. 1607 An SR Binding SID (BSID) are also be associated with each candidate 1608 path associated with an SR Policy, or with the SR Policy itself. The 1609 headend node installs a BSID-keyed entry in the forwarding plane and 1610 assigns it the action of steering packets that match the entry to the 1611 selected path of the SR Policy. This steering can be done in various 1612 ways: 1614 o SID Steering: Incoming packets have an active SID matching a local 1615 BSID at the headend. 1617 o Per-destination Steering: Incoming packets match a BGP/Service 1618 route which indicates an SR Policy. 1620 o Per-flow Steering: Incoming packets match a forwarding array (for 1621 example, the classic 5-tuple) which indicates an SR Policies. 1623 o Policy-based Steering: Incoming packets match a routing policy 1624 which directs them to an SR Policy. 1626 4.1.16. Network Virtualization and Abstraction 1628 One of the main drivers for Software Defined Networking (SDN) 1629 [RFC7149] is a decoupling of the network control plane from the data 1630 plane. This separation has been achieved for TE networks with the 1631 development of MPLS/GMPLS (see Section 4.1.6 and Section 4.1.7) and 1632 the Path Computation Element (PCE) (Section 4.1.13). One of the 1633 advantages of SDN is its logically centralized control regime that 1634 allows a global view of the underlying networks. Centralized control 1635 in SDN helps improve network resource utilization compared with 1636 distributed network control. 1638 Abstraction and Control of TE Networks (ACTN) [RFC8453] defines a 1639 hierarchical SDN architecture which describes the functional entities 1640 and methods for the coordination of resources across multiple 1641 domains, to provide end-to-end traffic engineered services. ACTN 1642 facilitates end-to-end connections and provides them to the user. 1643 ACTN is focused on: 1645 o Abstraction of the underlying network resources and how they are 1646 provided to higher-layer applications and customers. 1648 o Virtualization of underlying resources for use by the customer, 1649 application, or service. The creation of a virtualized 1650 environment allows operators to view and control multi-domain 1651 networks as a single virtualized network. 1653 o Presentation to customers of networks as a virtual network via 1654 open and programmable interfaces. 1656 The ACTN managed infrastructure is built from traffic engineered 1657 network resources, which may include statistical packet bandwidth, 1658 physical forwarding plane sources (such as wavelengths and time 1659 slots), forwarding and cross-connect capabilities. The type of 1660 network virtualization seen in ACTN allows customers and applications 1661 (tenants) to utilize and independently control allocated virtual 1662 network resources as if resources as if they were physically their 1663 own resource. The ACTN network is "sliced", with tenants being given 1664 a different partial and abstracted topology view of the physical 1665 underlying network. 1667 4.1.17. Network Slicing 1669 An IETF Network Slice is a logical network topology connecting a 1670 number of endpoints using a set of shared or dedicated network 1671 resources [I-D.nsdt-teas-ietf-network-slice-definition]. The 1672 resources are used to satisfy specific Service Level Objectives 1673 (SLOs) specified by the consumer. 1675 IETF Network Slices are created and managed within the scope of one 1676 or more network technologies (e.g., IP, MPLS, optical). They are 1677 intended to enable a diverse set of applications that have different 1678 requirements to coexist on the same network infrastructure. IETF 1679 Network Slices are defined such that they are independent of the 1680 underlying infrastructure connectivity and technologies used. This 1681 is to allow an IETF Network Slice consumer to describe their network 1682 connectivity and relevant objectives in a common format, independent 1683 of the underlying technologies used. 1685 An IETF Network Slice is a well-defined composite of a set of 1686 endpoints, the connectivity requirements between subsets of these 1687 endpoints, and associated service requirements. The service 1688 requirements are expressed in terms of quantifiable characteristics 1689 or service level objectives (SLOs). SLOs along with terms Service 1690 Level Indicator (SLI) and Service Level Agreement (SLA) are used to 1691 define the performance of a service at different levels 1692 [I-D.nsdt-teas-ietf-network-slice-definition]. 1694 The concept of an IETF network slice is consistent with an enhanced 1695 VPN (VPN+) [I-D.ietf-teas-enhanced-vpn]. That is, from a consumer's 1696 perspective it looks like a VPN connectivity matrix with additional 1697 information about the level of service required between endpoints, 1698 while from an operator's perspective it looks like a set of routing 1699 or tunneling instructions with the network resource reservations 1700 necessary to provide the required service levels as specified by the 1701 SLOs. 1703 IETF network slices are not, of themselves, TE constructs. However, 1704 a network operator that offers IETF network slices is likely to use 1705 many TE tools in order to manage their network and provide the 1706 services. 1708 4.1.18. Deterministic Networking 1710 Deterministic Networking (DetNet) [RFC8655] is an architecture for 1711 applications with critical timing and reliability requirements. The 1712 layered architecture particularly focuses on developing DetNet 1713 service capabilities in the data plane 1714 [I-D.ietf-detnet-data-plane-framework]. The DetNet service sub-layer 1715 provides a set of Packet Replication, Elimination, and Ordering 1716 Functions (PREOF) functions to provide end-to-end service assurance. 1717 The DetNet forwarding sub-layer provides corresponding forwarding 1718 assurance (low packet loss, bounded latency, and in-order delivery) 1719 functions using resource allocations and explicit route mechanisms. 1721 The separation into two sub-layers allows a greater flexibility to 1722 adapt Detnet capability over a number of TE data plane mechanisms 1723 such as IP, MPLS, and Segment Routing. More importantly it 1724 interconnects IEEE 802.1 Time Sensitive Networking (TSN) 1725 [I-D.ietf-detnet-ip-over-tsn] deployed in Industry Control and 1726 Automation Systems (ICAS). 1728 DetNet can be seen as a specialized branch of TE, since it sets up 1729 explicit optimized paths with allocation of resources as requested. 1730 A DetNet application can express its QoS attributes or traffic 1731 behavior using any combination of DetNet functions described in sub- 1732 layers. They are then distributed and provisioned using well- 1733 established control and provisioning mechanisms adopted for traffic- 1734 engineering. 1736 In DetNet, a considerable state information is required to maintain 1737 per flow queuing disciplines and resource reservation for a large 1738 number of individual flows. This can be quite challenging for 1739 network operations during network events such as faults, change in 1740 traffic volume or re-provisioning. Therefore, DetNet recommends 1741 support for aggregated flows, however, it still requires large amount 1742 of control signaling to establish and maintain DetNet flows. 1744 4.1.19. Network TE State Definition and Presentation 1746 The network states that are relevant to the traffic engineering need 1747 to be stored in the system and presented to the user. The Traffic 1748 Engineering Database (TED) is a collection of all TE information 1749 about all TE nodes and TE links in the network, which is an essential 1750 component of a TE system, such as MPLS-TE [RFC2702] and GMPLS 1751 [RFC3945]. In order to formally define the data in the TED and to 1752 present the data to the user with high usability, the data modeling 1753 language YANG [RFC7950] can be used as described in [RFC8795]. 1755 4.1.20. System Management and Control Interfaces 1757 The traffic engineering control system needs to have a management 1758 interface that is human-friendly and a control interfaces that is 1759 programmable for automation. The Network Configuration Protocol 1760 (NETCONF) [RFC6241] or the RESTCONF Protocol [RFC8040] provide 1761 programmable interfaces that are also human-friendly. These 1762 protocols use XML or JSON encoded messages. When message compactness 1763 or protocol bandwidth consumption needs to be optimized for the 1764 control interface, other protocols, such as Group Communication for 1765 the Constrained Application Protocol (CoAP) [RFC7390] or gRPC, are 1766 available, especially when the protocol messages are encoded in a 1767 binary format. Along with any of these protocols, the data modeling 1768 language YANG [RFC7950] can be used to formally and precisely define 1769 the interface data. 1771 The Path Computation Element Communication Protocol (PCEP) [RFC5440] 1772 is another protocol that has evolved to be an option for the TE 1773 system control interface. The messages of PCEP are TLV-based, not 1774 defined by a data modeling language such as YANG. 1776 4.2. Content Distribution 1778 The Internet is dominated by client-server interactions, principally 1779 Web traffic although in the future, more sophisticated media servers 1780 may become dominant. The location and performance of major 1781 information servers has a significant impact on the traffic patterns 1782 within the Internet as well as on the perception of service quality 1783 by end users. 1785 A number of dynamic load balancing techniques have been devised to 1786 improve the performance of replicated information servers. These 1787 techniques can cause spatial traffic characteristics to become more 1788 dynamic in the Internet because information servers can be 1789 dynamically picked based upon the location of the clients, the 1790 location of the servers, the relative utilization of the servers, the 1791 relative performance of different networks, and the relative 1792 performance of different parts of a network. This process of 1793 assignment of distributed servers to clients is called traffic 1794 directing. It is an application layer function. 1796 Traffic directing schemes that allocate servers in multiple 1797 geographically dispersed locations to clients may require empirical 1798 network performance statistics to make more effective decisions. In 1799 the future, network measurement systems may need to provide this type 1800 of information. 1802 When congestion exists in the network, traffic directing and traffic 1803 engineering systems should act in a coordinated manner. This topic 1804 is for further study. 1806 The issues related to location and replication of information 1807 servers, particularly web servers, are important for Internet traffic 1808 engineering because these servers contribute a substantial proportion 1809 of Internet traffic. 1811 5. Taxonomy of Traffic Engineering Systems 1813 This section presents a short taxonomy of traffic engineering systems 1814 constructed based on traffic engineering styles and views as listed 1815 below and described in greater detail in the following subsections of 1816 this document. 1818 o Time-dependent versus State-dependent versus Event-dependent 1819 o Offline versus Online 1821 o Centralized versus Distributed 1823 o Local versus Global Information 1825 o Prescriptive versus Descriptive 1827 o Open Loop versus Closed Loop 1829 o Tactical versus Strategic 1831 5.1. Time-Dependent Versus State-Dependent Versus Event-Dependent 1833 Traffic engineering methodologies can be classified as time- 1834 dependent, state-dependent, or event-dependent. All TE schemes are 1835 considered to be dynamic in this document. Static TE implies that no 1836 traffic engineering methodology or algorithm is being applied - it is 1837 a feature of network planning, but lacks the reactive and flexible 1838 nature of traffic engineering. 1840 In time-dependent TE, historical information based on periodic 1841 variations in traffic (such as time of day) is used to pre-program 1842 routing and other TE control mechanisms. Additionally, customer 1843 subscription or traffic projection may be used. Pre-programmed 1844 routing plans typically change on a relatively long time scale (e.g., 1845 daily). Time-dependent algorithms do not attempt to adapt to short- 1846 term variations in traffic or changing network conditions. An 1847 example of a time-dependent algorithm is a global centralized 1848 optimizer where the input to the system is a traffic matrix and 1849 multi-class QoS requirements as described [MR99]. Another example of 1850 such a methodology is the application of data mining to Internet 1851 traffic [AJ19] which enables the use of various machine learning 1852 algorithms to identify patterns within historically collected 1853 datasets about Internet traffic, and to extract information in order 1854 to guide decision-making, and to improve efficiency and productivity 1855 of operational processes. 1857 State-dependent TE adapts the routing plans based on the current 1858 state of the network which provides additional information on 1859 variations in actual traffic (i.e., perturbations from regular 1860 variations) that could not be predicted using historical information. 1861 Constraint-based routing is an example of state-dependent TE 1862 operating in a relatively long time scale. An example operating in a 1863 relatively short timescale is a load-balancing algorithm described in 1864 [MATE]. The state of the network can be based on parameters flooded 1865 by the routers. Another approach is for a particular router 1866 performing adaptive TE to send probe packets along a path to gather 1867 the state of that path. [RFC6374] defines protocol extensions to 1868 collect performance measurements from MPLS networks. Another 1869 approach is for a management system to gather the relevant 1870 information directly from network elements using telemetry data 1871 collection "publication/subscription" techniques [RFC7923]. Timely 1872 gathering and distribution of state information is critical for 1873 adaptive TE. While time-dependent algorithms are suitable for 1874 predictable traffic variations, state-dependent algorithms may be 1875 applied to increase network efficiency and resilience to adapt to the 1876 prevailing network state. 1878 Event-dependent TE methods can also be used for TE path selection. 1879 Event-dependent TE methods are distinct from time-dependent and 1880 state-dependent TE methods in the manner in which paths are selected. 1881 These algorithms are adaptive and distributed in nature and typically 1882 use learning models to find good paths for TE in a network. While 1883 state-dependent TE models typically use available-link-bandwidth 1884 (ALB) flooding for TE path selection, event-dependent TE methods do 1885 not require ALB flooding. Rather, event-dependent TE methods 1886 typically search out capacity by learning models, as in the success- 1887 to-the-top (STT) method. ALB flooding can be resource intensive, 1888 since it requires link bandwidth to carry LSAs, processor capacity to 1889 process LSAs, and the overhead can limit area/Autonomous System (AS) 1890 size. Modeling results suggest that event-dependent TE methods could 1891 lead to a reduction in ALB flooding overhead without loss of network 1892 throughput performance [I-D.ietf-tewg-qos-routing]. 1894 5.2. Offline Versus Online 1896 Traffic engineering requires the computation of routing plans. The 1897 computation may be performed offline or online. The computation can 1898 be done offline for scenarios where routing plans need not be 1899 executed in real-time. For example, routing plans computed from 1900 forecast information may be computed offline. Typically, offline 1901 computation is also used to perform extensive searches on multi- 1902 dimensional solution spaces. 1904 Online computation is required when the routing plans must adapt to 1905 changing network conditions as in state-dependent algorithms. Unlike 1906 offline computation (which can be computationally demanding), online 1907 computation is geared toward relative simple and fast calculations to 1908 select routes, fine-tune the allocations of resources, and perform 1909 load balancing. 1911 5.3. Centralized Versus Distributed 1913 Under centralized control there is a central authority which 1914 determines routing plans and perhaps other TE control parameters on 1915 behalf of each router. The central authority periodically collects 1916 network-state information from all routers, and sends routing 1917 information to the routers. The update cycle for information 1918 exchange in both directions is a critical parameter directly 1919 impacting the performance of the network being controlled. 1920 Centralized control may need high processing power and high bandwidth 1921 control channels. 1923 Distributed control determines route selection by each router 1924 autonomously based on the router's view of the state of the network. 1925 The network state information may be obtained by the router using a 1926 probing method or distributed by other routers on a periodic basis 1927 using link state advertisements. Network state information may also 1928 be disseminated under exception conditions. Examples of protocol 1929 extensions used to advertise network link state information are 1930 defined in [RFC5305], [RFC6119], [RFC7471], [RFC8570], and [RFC8571]. 1931 See also Section 4.1.11. 1933 5.3.1. Hybrid Systems 1935 In practice, most TE systems will be a hybrid of central and 1936 distributed control. For example, a popular MPLS approach to TE is 1937 to use a central controller based on an active, stateful PCE, but to 1938 use routing and signaling protocols to make local decisions at 1939 routers within the network. Local decisions may be able to respond 1940 more quickly to network events, but may result in conflicts with 1941 decisions made by other routers. 1943 Network operations for TE systems may also use a hybrid of offline 1944 and online computation. TE paths may be precomputed based on stable- 1945 state network information and planned traffic demands, but may then 1946 be modified in the active network depending on variations in network 1947 state and traffic load. Furthermore, responses to network events may 1948 be precomputed offline to allow rapid reactions without further 1949 computation, or may be derived online depending on the nature of the 1950 events. 1952 Lastly, note that a fully functional TE system is likely to use all 1953 aspects of time-dependent, state-dependent, and event-dependent 1954 methodologies as described in Section 5.1. 1956 5.3.2. Considerations for Software Defined Networking 1958 As discussed in Section 4.1.16, one of the main drivers for SDN is a 1959 decoupling of the network control plane from the data plane 1960 [RFC7149]. However, SDN may also combine centralized control of 1961 resources, and facilitate application-to-network interaction via an 1962 application programming interface (API) such as [RFC8040]. Combining 1963 these features provides a flexible network architecture that can 1964 adapt to network requirements of a variety of higher-layer 1965 applications, a concept often referred to as the "programmable 1966 network" [RFC7426]. 1968 The centralized control aspect of SDN helps improve global network 1969 resource utilization compared with distributed network control, where 1970 local policy may often override global optimization goals. In an SDN 1971 environment, the data plane forwards traffic to its desired 1972 destination. However, before traffic reaches the data plane, the 1973 logically centralized SDN control plane often determines the end-to- 1974 end path the application traffic will take in the network. 1975 Therefore, the SDN control plane needs to be aware of the underlying 1976 network topology, capabilities and current node and link resource 1977 state. 1979 Using a PCE-based SDN control framework [RFC7491], the available 1980 network topology may be discovered by running a passive instance of 1981 OSPF or IS-IS, or via BGP-LS [RFC7752], to generate a TED (see 1982 Section 4.1.19). The PCE is used to compute a path (see 1983 Section 4.1.13) based on the TED and available bandwidth, and further 1984 path optimization may be based on requested objective functions 1985 [RFC5541]. When a suitable path has been computed the programming of 1986 the explicit network path may be performed using either end-to-end 1987 signaling protocol [RFC3209] or per-hop with each node being directly 1988 programmed [RFC8283] by the SDN controller. 1990 By utilizing a centralized approach to network control, additional 1991 network benefits are also available, including Global Concurrent 1992 Optimization (GCO) [RFC5557]. A GCO path computation request will 1993 simultaneously use the network topology and set of new end-to-end 1994 path requests, along with their respective constraints, for optimal 1995 placement in the network. Correspondingly, a GCO-based computation 1996 may be applied to recompute existing network paths to groom traffic 1997 and to mitigate congestion. 1999 5.4. Local Versus Global 2001 Traffic engineering algorithms may require local and global network- 2002 state information. 2004 Local information is the state of a portion of the domain. Examples 2005 include the bandwidth and packet loss rate of a particular path, or 2006 the state and capabilities of a network link. Local state 2007 information may be sufficient for certain instances of distributed 2008 control TE. 2010 Global information is the state of the entire TE domain. Examples 2011 include a global traffic matrix, and loading information on each link 2012 throughout the domain of interest. Global state information is 2013 typically required with centralized control. Distributed TE systems 2014 may also need global information in some cases. 2016 5.5. Prescriptive Versus Descriptive 2018 TE systems may also be classified as prescriptive or descriptive. 2020 Prescriptive traffic engineering evaluates alternatives and 2021 recommends a course of action. Prescriptive traffic engineering can 2022 be further categorized as either corrective or perfective. 2023 Corrective TE prescribes a course of action to address an existing or 2024 predicted anomaly. Perfective TE prescribes a course of action to 2025 evolve and improve network performance even when no anomalies are 2026 evident. 2028 Descriptive traffic engineering, on the other hand, characterizes the 2029 state of the network and assesses the impact of various policies 2030 without recommending any particular course of action. 2032 5.5.1. Intent-Based Networking 2034 TBD : Jeff Tantsura 2036 5.6. Open-Loop Versus Closed-Loop 2038 Open-loop traffic engineering control is where control action does 2039 not use feedback information from the current network state. The 2040 control action may use its own local information for accounting 2041 purposes, however. 2043 Closed-loop traffic engineering control is where control action 2044 utilizes feedback information from the network state. The feedback 2045 information may be in the form of historical information or current 2046 measurement. 2048 5.7. Tactical versus Strategic 2050 Tactical traffic engineering aims to address specific performance 2051 problems (such as hot-spots) that occur in the network from a 2052 tactical perspective, without consideration of overall strategic 2053 imperatives. Without proper planning and insights, tactical TE tends 2054 to be ad hoc in nature. 2056 Strategic traffic engineering approaches the TE problem from a more 2057 organized and systematic perspective, taking into consideration the 2058 immediate and longer term consequences of specific policies and 2059 actions. 2061 6. Recommendations for Internet Traffic Engineering 2063 This section describes high-level recommendations for traffic 2064 engineering in the Internet in general terms. 2066 The recommendations describe the capabilities needed to solve a 2067 traffic engineering problem or to achieve a traffic engineering 2068 objective. Broadly speaking, these recommendations can be 2069 categorized as either functional or non-functional recommendations. 2071 o Functional recommendations describe the functions that a traffic 2072 engineering system should perform. These functions are needed to 2073 realize traffic engineering objectives by addressing traffic 2074 engineering problems. 2076 o Non-functional recommendations relate to the quality attributes or 2077 state characteristics of a traffic engineering system. These 2078 recommendations may contain conflicting assertions and may 2079 sometimes be difficult to quantify precisely. 2081 6.1. Generic Non-functional Recommendations 2083 The generic non-functional recommendations for Internet traffic 2084 engineering are listed in the paragraphs that follow. In a given 2085 context, some of these recommendations may be critical while others 2086 may be optional. Therefore, prioritization may be required during 2087 the development phase of a traffic engineering system to tailor it to 2088 a specific operational context. 2090 Usability: Usability is a human aspect of traffic engineering 2091 systems. It refers to the ease with which a traffic engineering 2092 system can be deployed and operated. In general, it is desirable 2093 to have a TE system that can be readily deployed in an existing 2094 network. It is also desirable to have a TE system that is easy to 2095 operate and maintain. 2097 Automation: Whenever feasible, a TE system should automate as many 2098 TE functions as possible to minimize the amount of human effort 2099 needed to analyze and control operational networks. Automation is 2100 particularly important in large-scale public networks because of 2101 the high cost of the human aspects of network operations and the 2102 high risk of network problems caused by human errors. Automation 2103 may entail the incorporation of automatic feedback and 2104 intelligence into some components of the TE system. 2106 Scalability: Public networks continue to grow rapidly with respect 2107 to network size and traffic volume. Therefore, to remain 2108 applicable as the network evolves, a TE system should be scalable. 2109 In particular, a TE system should remain functional as the network 2110 expands with regard to the number of routers and links, and with 2111 respect to the traffic volume. A TE system should have a scalable 2112 architecture, should not adversely impair other functions and 2113 processes in a network element, and should not consume too many 2114 network resources when collecting and distributing state 2115 information, or when exerting control. 2117 Stability: Stability is a very important consideration in TE systems 2118 that respond to changes in the state of the network. State- 2119 dependent TE methodologies typically include a trade-off between 2120 responsiveness and stability. It is strongly recommended that 2121 when a trade-off between responsiveness and stability is needed, 2122 it should be made in favor of stability (especially in public IP 2123 backbone networks). 2125 Flexibility: A TE system should allow for changes in optimization 2126 policy. In particular, a TE system should provide sufficient 2127 configuration options so that a network administrator can tailor 2128 the system to a particular environment. It may also be desirable 2129 to have both online and offline TE subsystems which can be 2130 independently enabled and disabled. TE systems that are used in 2131 multi-class networks should also have options to support class 2132 based performance evaluation and optimization. 2134 Visibility: Mechanisms should exist as part of the TE system to 2135 collect statistics from the network and to analyze these 2136 statistics to determine how well the network is functioning. 2137 Derived statistics such as traffic matrices, link utilization, 2138 latency, packet loss, and other performance measures of interest 2139 which are determined from network measurements can be used as 2140 indicators of prevailing network conditions. The capabilities of 2141 the various components of the routing system are other examples of 2142 status information which should be observable. 2144 Simplicity: A TE system should be as simple as possible and easy to 2145 use (i.e., have clean, convenient, and intuitive user interfaces). 2146 Simplicity in user interface does not necessarily imply that the 2147 TE system will use naive algorithms. When complex algorithms and 2148 internal structures are used, the user interface should hide such 2149 complexities from the network administrator as much as possible. 2151 Interoperability: Whenever feasible, TE systems and their components 2152 should be developed with open standards-based interfaces to allow 2153 interoperation with other systems and components. 2155 Security: Security is a critical consideration in TE systems. Such 2156 systems typically exert control over functional aspects of the 2157 network to achieve the desired performance objectives. Therefore, 2158 adequate measures must be taken to safeguard the integrity of the 2159 TE system. Adequate measures must also be taken to protect the 2160 network from vulnerabilities that originate from security breaches 2161 and other impairments within the TE system. 2163 The remaining subsections of this section focus on some of the high- 2164 level functional recommendations for traffic engineering. 2166 6.2. Routing Recommendations 2168 Routing control is a significant aspect of Internet traffic 2169 engineering. Routing impacts many of the key performance measures 2170 associated with networks, such as throughput, delay, and utilization. 2171 Generally, it is very difficult to provide good service quality in a 2172 wide area network without effective routing control. A desirable TE 2173 routing system is one that takes traffic characteristics and network 2174 constraints into account during route selection while maintaining 2175 stability. 2177 Shortest path first (SPF) IGPs are based on shortest path algorithms 2178 and have limited control capabilities for TE [RFC2702], [AWD2]. 2179 These limitations include: 2181 1. Pure SPF protocols do not take network constraints and traffic 2182 characteristics into account during route selection. For 2183 example, IGPs always select the shortest paths based on link 2184 metrics assigned by administrators) so load sharing cannot be 2185 performed across paths of different costs. Using shortest paths 2186 to forward traffic may cause the following problems: 2188 * If traffic from a source to a destination exceeds the capacity 2189 of a link along the shortest path, the link (and hence the 2190 shortest path) becomes congested while a longer path between 2191 these two nodes may be under-utilized 2193 * The shortest paths from different sources can overlap at some 2194 links. If the total traffic from the sources exceeds the 2195 capacity of any of these links, congestion will occur. 2197 * Problems can also occur because traffic demand changes over 2198 time, but network topology and routing configuration cannot be 2199 changed as rapidly. This causes the network topology and 2200 routing configuration to become sub-optimal over time, which 2201 may result in persistent congestion problems. 2203 2. The Equal-Cost Multi-Path (ECMP) capability of SPF IGPs supports 2204 sharing of traffic among equal cost paths between two nodes. 2205 However, ECMP attempts to divide the traffic as equally as 2206 possible among the equal cost shortest paths. Generally, ECMP 2207 does not support configurable load sharing ratios among equal 2208 cost paths. The result is that one of the paths may carry 2209 significantly more traffic than other paths because it may also 2210 carry traffic from other sources. This situation can result in 2211 congestion along the path that carries more traffic. Weighted 2212 ECMP (WECMP) (see, for example, [I-D.ietf-bess-evpn-unequal-lb]) 2213 provides some mitigation. 2215 3. Modifying IGP metrics to control traffic routing tends to have 2216 network-wide effects. Consequently, undesirable and 2217 unanticipated traffic shifts can be triggered as a result. Work 2218 described in Section 8 may be capable of better control [FT00], 2219 [FT01]. 2221 Because of these limitations, new capabilities are needed to enhance 2222 the routing function in IP networks. Some of these capabilities are 2223 summarized below. 2225 o Constraint-based routing computes routes to fulfill requirements 2226 subject to constraints. This can be useful in public IP backbones 2227 with complex topologies. Constraints may include bandwidth, hop 2228 count, delay, and administrative policy instruments such as 2229 resource class attributes [RFC2702], [RFC2386]. This makes it 2230 possible to select routes that satisfy a given set of 2231 requirements. Routes computed by constraint-based routing are not 2232 necessarily the shortest paths. Constraint-based routing works 2233 best with path-oriented technologies that support explicit 2234 routing, such as MPLS. 2236 Constraint-based routing can also be used as a way to distribute 2237 traffic onto the infrastructure, including for best effort 2238 traffic. For example, congestion problems caused by uneven 2239 traffic distribution may be avoided or reduced by knowing the 2240 reservable bandwidth attributes of the network links and by 2241 specifying the bandwidth requirements for path selection. 2243 o A number of enhancements to the link state IGPs are needed to 2244 allow them to distribute additional state information required for 2245 constraint-based routing. The extensions to OSPF are described in 2246 [RFC3630], and to IS-IS in [RFC5305]. Some of the additional 2247 topology state information includes link attributes such as 2248 reservable bandwidth and link resource class attribute (an 2249 administratively specified property of the link). The resource 2250 class attribute concept is defined in [RFC2702]. The additional 2251 topology state information is carried in new TLVs and sub-TLVs in 2252 IS-IS, or in the Opaque LSA in OSPF [RFC5305], [RFC3630]. 2254 An enhanced link-state IGP may flood information more frequently 2255 than a normal IGP. This is because even without changes in 2256 topology, changes in reservable bandwidth or link affinity can 2257 trigger the enhanced IGP to initiate flooding. A trade-off 2258 between the timeliness of the information flooded and the flooding 2259 frequency is typically implemented using a threshold based on the 2260 percentage change of the advertised resources to avoid excessive 2261 consumption of link bandwidth and computational resources, and to 2262 avoid instability in the TED. 2264 o In a TE system, it is also desirable for the routing subsystem to 2265 make the load splitting ratio among multiple paths (with equal 2266 cost or different cost) configurable. This capability gives 2267 network administrators more flexibility in the control of traffic 2268 distribution across the network. It can be very useful for 2269 avoiding/relieving congestion in certain situations. Examples can 2270 be found in [XIAO] and [I-D.ietf-bess-evpn-unequal-lb]. 2272 o The routing system should also have the capability to control the 2273 routes of subsets of traffic without affecting the routes of other 2274 traffic if sufficient resources exist for this purpose. This 2275 capability allows a more refined control over the distribution of 2276 traffic across the network. For example, the ability to move 2277 traffic away from its original path to another path (without 2278 affecting other traffic paths) allows the traffic to be moved from 2279 resource-poor network segments to resource-rich segments. Path 2280 oriented technologies such as MPLS-TE inherently support this 2281 capability as discussed in [AWD2]. 2283 o Additionally, the routing subsystem should be able to select 2284 different paths for different classes of traffic (or for different 2285 traffic behavior aggregates) if the network supports multiple 2286 classes of service (different behavior aggregates). 2288 6.3. Traffic Mapping Recommendations 2290 Traffic mapping is the assignment of traffic workload onto pre- 2291 established paths to meet certain requirements. Thus, while 2292 constraint-based routing deals with path selection, traffic mapping 2293 deals with the assignment of traffic to established paths which may 2294 have been generated by constraint-based routing or by some other 2295 means. Traffic mapping can be performed by time-dependent or state- 2296 dependent mechanisms, as described in Section 5.1. 2298 An important aspect of the traffic mapping function is the ability to 2299 establish multiple paths between an originating node and a 2300 destination node, and the capability to distribute the traffic 2301 between the two nodes across the paths according to some policies. A 2302 pre-condition for this scheme is the existence of flexible mechanisms 2303 to partition traffic and then assign the traffic partitions onto the 2304 parallel paths as noted in [RFC2702]. When traffic is assigned to 2305 multiple parallel paths, it is recommended that special care should 2306 be taken to ensure proper ordering of packets belonging to the same 2307 application (or micro-flow) at the destination node of the parallel 2308 paths. 2310 Mechanisms that perform the traffic mapping functions should aim to 2311 map the traffic onto the network infrastructure to minimize 2312 congestion. If the total traffic load cannot be accommodated, or if 2313 the routing and mapping functions cannot react fast enough to 2314 changing traffic conditions, then a traffic mapping system may use 2315 short time scale congestion control mechanisms (such as queue 2316 management, scheduling, etc.) to mitigate congestion. Thus, 2317 mechanisms that perform the traffic mapping functions complement 2318 existing congestion control mechanisms. In an operational network, 2319 traffic should be mapped onto the infrastructure such that intra- 2320 class and inter-class resource contention are minimized (see 2321 Section 2). 2323 When traffic mapping techniques that depend on dynamic state feedback 2324 (e.g., MATE [MATE] and such like) are used, special care must be 2325 taken to guarantee network stability. 2327 6.4. Measurement Recommendations 2329 The importance of measurement in traffic engineering has been 2330 discussed throughout this document. A TE system should include 2331 mechanisms to measure and collect statistics from the network to 2332 support the TE function. Additional capabilities may be needed to 2333 help in the analysis of the statistics. The actions of these 2334 mechanisms should not adversely affect the accuracy and integrity of 2335 the statistics collected. The mechanisms for statistical data 2336 acquisition should also be able to scale as the network evolves. 2338 Traffic statistics may be classified according to long-term or short- 2339 term timescales. Long-term traffic statistics are very useful for 2340 traffic engineering. Long-term traffic statistics may periodicity 2341 record network workload (such as hourly, daily, and weekly variations 2342 in traffic profiles) as well as traffic trends. Aspects of the 2343 traffic statistics may also describe class of service characteristics 2344 for a network supporting multiple classes of service. Analysis of 2345 the long-term traffic statistics may yield other information such as 2346 busy hour characteristics, traffic growth patterns, persistent 2347 congestion problems, hot-spot, and imbalances in link utilization 2348 caused by routing anomalies. 2350 A mechanism for constructing traffic matrices for both long-term and 2351 short-term traffic statistics should be in place. In multi-service 2352 IP networks, the traffic matrices may be constructed for different 2353 service classes. Each element of a traffic matrix represents a 2354 statistic about the traffic flow between a pair of abstract nodes. 2355 An abstract node may represent a router, a collection of routers, or 2356 a site in a VPN. 2358 Traffic statistics should provide reasonable and reliable indicators 2359 of the current state of the network on the short-term scale. Some 2360 short term traffic statistics may reflect link utilization and link 2361 congestion status. Examples of congestion indicators include 2362 excessive packet delay, packet loss, and high resource utilization. 2363 Examples of mechanisms for distributing this kind of information 2364 include SNMP, probing tools, FTP, IGP link state advertisements, and 2365 Netconf/Restconf, etc. 2367 6.5. Network Survivability 2369 Network survivability refers to the capability of a network to 2370 maintain service continuity in the presence of faults. This can be 2371 accomplished by promptly recovering from network impairments and 2372 maintaining the required QoS for existing services after recovery. 2373 Survivability is an issue of great concern within the Internet 2374 community due to the demand to carry mission critical traffic, real- 2375 time traffic, and other high priority traffic over the Internet. 2376 Survivability can be addressed at the device level by developing 2377 network elements that are more reliable; and at the network level by 2378 incorporating redundancy into the architecture, design, and operation 2379 of networks. It is recommended that a philosophy of robustness and 2380 survivability should be adopted in the architecture, design, and 2381 operation of traffic engineering that control IP networks (especially 2382 public IP networks). Because different contexts may demand different 2383 levels of survivability, the mechanisms developed to support network 2384 survivability should be flexible so that they can be tailored to 2385 different needs. A number of tools and techniques have been 2386 developed to enable network survivability including MPLS Fast Reroute 2387 [RFC4090], RSVP-TE Extensions in Support of End-to-End GMPLS Recovery 2388 [RFC4872], and GMPLS Segment Recovery [RFC4873]. 2390 The impact of service outages varies significantly for different 2391 service classes depending on the duration of the outage which can 2392 vary from milliseconds (with minor service impact) to seconds (with 2393 possible call drops for IP telephony and session time-outs for 2394 connection oriented transactions) to minutes and hours (with 2395 potentially considerable social and business impact). Different 2396 duration outages have different impacts depending on the throughput 2397 of the traffic flows that are interrupted. 2399 Failure protection and restoration capabilities are available in 2400 multiple layers as network technologies have continued to evolve. 2401 Optical networks are capable of providing dynamic ring and mesh 2402 restoration functionality at the wavelength level. At the SONET/SDH 2403 layer survivability capability is provided with Automatic Protection 2404 Switching (APS) as well as self-healing ring and mesh architectures. 2405 Similar functionality is provided by layer 2 technologies such as 2406 Ethernet. 2408 Rerouting is used at the IP layer to restore service following link 2409 and node outages. Rerouting at the IP layer occurs after a period of 2410 routing convergence which may require seconds to minutes to complete. 2411 Path-oriented technologies such a MPLS ([RFC3469]) can be used to 2412 enhance the survivability of IP networks in a potentially cost 2413 effective manner. 2415 An important of multi-layer survivability is that technologies at 2416 different layers may provide protection and restoration capabilities 2417 at different granularities in terms of time scales and at different 2418 bandwidth granularity (from packet-level to wavelength level). 2419 Protection and restoration capabilities can also be sensitive to 2420 different service classes and different network utility models. 2421 Coordinating different protection and restoration capabilities across 2422 multiple layers in a cohesive manner to ensure network survivability 2423 is maintained at reasonable cost is a challenging task. Protection 2424 and restoration coordination across layers may not always be 2425 feasible, because networks at different layers may belong to 2426 different administrative domains. 2428 The following paragraphs present some of the general recommendations 2429 for protection and restoration coordination. 2431 o Protection and restoration capabilities from different layers 2432 should be coordinated to provide network survivability in a 2433 flexible and cost effective manner. Avoiding duplication of 2434 functions in different layers is one way to achieve the 2435 coordination. Escalation of alarms and other fault indicators 2436 from lower to higher layers may also be performed in a coordinated 2437 manner. The order of timing of restoration triggers from 2438 different layers is another way to coordinate multi-layer 2439 protection/restoration. 2441 o Network capacity reserved in one layer to provide protection and 2442 restoration is not available to carry traffic in a higher layer: 2443 it is not visible as spare capacity in the higher layer. Placing 2444 protection/restoration functions in many layers may increase 2445 redundancy and robustness, but it can result in significant 2446 inefficiencies in network resource utilization. Careful planning 2447 is needed to balance the trade-off between the desire for 2448 survivablity and the optimal use of resources. 2450 o It is generally desirable to have protection and restoration 2451 schemes that are intrinsically bandwidth efficient. 2453 o Failure notifications throughout the network should be timely and 2454 reliable if they are to be acted on as triggers for effective 2455 protection and restoration actions. 2457 o Alarms and other fault monitoring and reporting capabilities 2458 should be provided at the right network layers so that the 2459 protection and restoration actions can be taken in those layers. 2461 6.5.1. Survivability in MPLS Based Networks 2463 Because MPLS is path-oriented, it has the potential to provide faster 2464 and more predictable protection and restoration capabilities than 2465 conventional hop by hop routed IP systems. Protection types for MPLS 2466 networks can be divided into four categories. 2468 o Link Protection: The objective of link protection is to protect an 2469 LSP from the failure of a given link. Under link protection, a 2470 protection or backup LSP (the secondary LSP) follows a path that 2471 is disjoint from the path of the working or operational LSP (the 2472 primary LSP) at the particular link where link protection is 2473 required. When the protected link fails, traffic on the working 2474 LSP is switched to the protection LSP at the head-end of the 2475 failed link. As a local repair method, link protection can be 2476 fast. This form of protection may be most appropriate in 2477 situations where some network elements along a given path are 2478 known to be less reliable than others. 2480 o Node Protection: The objective of node protection is to protect an 2481 LSP from the failure of a given node. Under node protection, the 2482 secondary LSP follows a path that is disjoint from the path of the 2483 primary LSP at the particular node where node protection is 2484 required. The secondary LSP is also disjoint from the primary LSP 2485 at all links attached to the node to be protected. When the 2486 protected node fails, traffic on the working LSP is switched over 2487 to the protection LSP at the upstream LSR directly connected to 2488 the failed node. Node protection covers a slightly larger part of 2489 the network compared to link protection, but is otherwise 2490 fundamentally the same. 2492 o Path Protection: The goal of LSP path protection (or end-to-end 2493 protection) is to protect an LSP from any failure along its routed 2494 path. Under path protection, the path of the protection LSP is 2495 completely disjoint from the path of the working LSP. The 2496 advantage of path protection is that the backup LSP protects the 2497 working LSP from all possible link and node failures along the 2498 path, except for failures of ingress or egress LSR. Additionally, 2499 path protection may be more efficient in terms of resource usage 2500 than link or node protection applied at every jop along the path. 2501 However, path protection may be slower than link and node 2502 protection because the fault notifications have to be propagated 2503 further. 2505 o Segment Protection: An MPLS domain may be partitioned into 2506 multiple subdomains (protection domains). Path protection is 2507 applied to the path of each LSP as it crosses the domain from its 2508 ingress to the domain to where it egresses the domain. In cases 2509 where an LSP traverses multiple protection domains, a protection 2510 mechanism within a domain only needs to protect the segment of the 2511 LSP that lies within the domain. Segment protection will 2512 generally be faster than end-to-end path protection because 2513 recovery generally occurs closer to the fault and the notification 2514 doesn't have to propagate as far. 2516 See [RFC3469] and [RFC6372] for a more comprehensive discussion of 2517 MPLS based recovery. 2519 6.5.2. Protection Options 2521 Another issue to consider is the concept of protection options. We 2522 use notation such as "m:n protection", where m is the number of 2523 protection LSPs used to protect n working LSPs. In all cases except 2524 1+1 protection, the resources associated with the protection LSPs can 2525 be used to carry preemptable best-effort traffic when the working LSP 2526 is functioning correctly. 2528 o 1:1 protection: One working LSP is protected/restored by one 2529 protection LSP. 2531 o 1:n protection: One protection LSP is used to protect/restore n 2532 working LSPs. Only one failed LSP can be restored at any time. 2534 o n:1 protection: One working LSP is protected/restored by n 2535 protection LSPs, possibly with load splitting across the 2536 protection LSPs. This may be especially useful when it is not 2537 feasible to find one path for the backup that can satisfy the 2538 bandwidth requirement of the primary LSP. 2540 o 1+1 protection: Traffic is sent concurrently on both the working 2541 LSP and a protection LSP. The egress LSR selects one of the two 2542 LSPs based on local policy (usually based on traffic integrity). 2543 When a fault disrupts the traffic on one LSP, the egress switches 2544 to receive traffic from the other LSP. This approach is expensive 2545 in how it consumes network but recovers from failures most 2546 rapidly. 2548 6.6. Traffic Engineering in Diffserv Environments 2550 Increasing requirements to support multiple classes of traffic in the 2551 Internet, such as best effort and mission critical data, calls for IP 2552 networks to differentiate traffic according to some criteria and to 2553 give preferential treatment to certain types of traffic. Large 2554 numbers of flows can be aggregated into a few behavior aggregates 2555 based on some criteria based on common performance requirements in 2556 terms of packet loss ratio, delay, and jitter, or in terms of common 2557 fields within the IP packet headers. 2559 Differentiated Services (Diffserv) [RFC2475] can be used to ensure 2560 that SLAs defined to differentiate between traffic flows are met. 2561 Classes of service (CoS) can be supported in a Diffserv environment 2562 by concatenating per-hop behaviors (PHBs) along the routing path. A 2563 PHB is the forwarding behavior that a packet receives at a Diffserv- 2564 compliant node, and it can be configured at each router. PHBs are 2565 delivered using buffer management and packet scheduling mechanisms 2566 and require that the ingress nodes use traffic classification, 2567 marking, policing, and shaping. 2569 Traffic engineering can compliment Diffserv to improve utilization of 2570 network resources. Traffic engineering can be operated on an 2571 aggregated basis across all service classes [RFC3270], or on a per 2572 service class basis. The former is used to provide better 2573 distribution of the traffic load over the network resources (see 2574 [RFC3270] for detailed mechanisms to support aggregate traffic 2575 engineering). The latter case is discussed below since it is 2576 specific to the Diffserv environment, with so called Diffserv-aware 2577 traffic engineering [RFC4124]. 2579 For some Diffserv networks, it may be desirable to control the 2580 performance of some service classes by enforcing relationships 2581 between the traffic workload contributed by each service class and 2582 the amount of network resources allocated or provisioned for that 2583 service class. Such relationships between demand and resource 2584 allocation can be enforced using a combination of, for example: 2586 o TE mechanisms on a per service class basis that enforce the 2587 relationship between the amount of traffic contributed by a given 2588 service class and the resources allocated to that class. 2590 o Mechanisms that dynamically adjust the resources allocated to a 2591 given service class to relate to the amount of traffic contributed 2592 by that service class. 2594 It may also be desirable to limit the performance impact of high 2595 priority traffic on relatively low priority traffic. This can be 2596 achieved, for example, by controlling the percentage of high priority 2597 traffic that is routed through a given link. Another way to 2598 accomplish this is to increase link capacities appropriately so that 2599 lower priority traffic can still enjoy adequate service quality. 2600 When the ratio of traffic workload contributed by different service 2601 classes varies significantly from router to router, it may not be 2602 enough to rely on conventional IGP routing protocols or on TE 2603 mechanisms that are not sensitive to different service classes. 2604 Instead, it may be desirable to perform traffic engineering, 2605 especially routing control and mapping functions, on a per service 2606 class basis. One way to accomplish this in a domain that supports 2607 both MPLS and Diffserv is to define class specific LSPs and to map 2608 traffic from each class onto one or more LSPs that correspond to that 2609 service class. An LSP corresponding to a given service class can 2610 then be routed and protected/restored in a class dependent manner, 2611 according to specific policies. 2613 Performing traffic engineering on a per class basis may require per- 2614 class parameters to be distributed. It is common to have some 2615 classes share some aggregate constraints (e.g., maximum bandwidth 2616 requirement) without enforcing the constraint on each individual 2617 class. These classes can be grouped into class-types, and per-class- 2618 type parameters can be distributed to improve scalability. This also 2619 allows better bandwidth sharing between classes in the same class- 2620 type. A class-type is a set of classes that satisfy the following 2621 two conditions: 2623 o Classes in the same class-type have common aggregate requirements 2624 to satisfy required performance levels. 2626 o There is no requirement to be enforced at the level of an 2627 individual class in the class-type. Note that it is, 2628 nevertheless, still possible to implement some priority policies 2629 for classes in the same class-type to permit preferential access 2630 to the class-type bandwidth through the use of preemption 2631 priorities. 2633 See [RFC4124] for detailed requirements on Diffserv-aware traffic 2634 engineering. 2636 6.7. Network Controllability 2638 Offline and online (see Section 5.2) TE considerations are of limited 2639 utility if the network cannot be controlled effectively to implement 2640 the results of TE decisions and to achieve the desired network 2641 performance objectives. 2643 Capacity augmentation is a coarse-grained solution to TE issues. 2644 However, it is simple and may be advantageous if bandwidth is 2645 abundant and cheap. However, bandwidth is not always abundant and 2646 cheap, and additional capacity might not always be the best solution. 2647 Adjustments of administrative weights and other parameters associated 2648 with routing protocols provide finer-grained control, but this 2649 approach is difficult to use and imprecise because of the the way the 2650 routing protocols interact occur across the network. 2652 Control mechanisms can be manual (e.g., static configuration), 2653 partially-automated (e.g., scripts), or fully-automated (e.g., policy 2654 based management systems). Automated mechanisms are particularly 2655 useful in large scale networks. Multi-vendor interoperability can be 2656 facilitated by standardized management systems (e.g., YANG models) to 2657 support the control functions required to address TE objectives. 2659 Network control functions should be secure, reliable, and stable as 2660 these are often needed to operate correctly in times of network 2661 impairments (e.g., during network congestion or security attacks). 2663 7. Inter-Domain Considerations 2665 Inter-domain TE is concerned with performance optimization for 2666 traffic that originates in one administrative domain and terminates 2667 in a different one. 2669 BGP [RFC4271] is the standard exterior gateway protocol used to 2670 exchange routing information between autonomous systems (ASes) in the 2671 Internet. BGP includes a sequential decision process that calculates 2672 the preference for routes to a given destination network. There are 2673 two fundamental aspects to inter-domain TE using BGP: 2675 o Route Redistribution: Controlling the import and export of routes 2676 between ASes, and controlling the redistribution of routes between 2677 BGP and other protocols within an AS. 2679 o Best path selection: Selecting the best path when there are 2680 multiple candidate paths to a given destination network. This is 2681 performed by the BGP decision process, selecting preferred exit 2682 points out of an AS towards specific destination networks taking a 2683 number of different considerations into account. The BGP path 2684 selection process can be influenced by manipulating the attributes 2685 associated with the process, including NEXT-HOP, WEIGHT, LOCAL- 2686 PREFERENCE, AS-PATH, ROUTE-ORIGIN, MULTI-EXIT-DESCRIMINATOR (MED), 2687 IGP METRIC, etc. 2689 Route-maps provide the flexibility to implement complex BGP policies 2690 based on pre-configured logical conditions. They can be used to 2691 control import and export policies for incoming and outgoing routes, 2692 control the redistribution of routes between BGP and other protocols, 2693 and influence the selection of best paths by manipulating the 2694 attributes associated with the BGP decision process. Very complex 2695 logical expressions that implement various types of policies can be 2696 implemented using a combination of Route-maps, BGP-attributes, 2697 Access-lists, and Community attributes. 2699 When considering inter-domain TE with BGP, note that the outbound 2700 traffic exit point is controllable, whereas the interconnection point 2701 where inbound traffic is received typically is not. Therefore, it is 2702 up to each individual network to implement TE strategies that deal 2703 with the efficient delivery of outbound traffic from its customers to 2704 its peering points. The vast majority of TE policy is based on a 2705 "closest exit" strategy, which offloads interdomain traffic at the 2706 nearest outbound peering point towards the destination AS. Most 2707 methods of manipulating the point at which inbound traffic enters a 2708 are either ineffective, or not accepted in the peering community. 2710 Inter-domain TE with BGP is generally effective, but it is usually 2711 applied in a trial-and-error fashion because a TE system usually only 2712 has a view of the available network resources within one domain (an 2713 AS in this case). A systematic approach for inter-domain TE requires 2714 cooperation between the domains. Further, what may be considered a 2715 good solution in one domain may not necessarily be a good solution in 2716 another. Moreover, it is generally considered inadvisable for one 2717 domain to permit a control process from another domain to influence 2718 the routing and management of traffic in its network. 2720 MPLS TE-tunnels (LSPs) can add a degree of flexibility in the 2721 selection of exit points for inter-domain routing by applying rhe 2722 concept of relative and absolute metrics. If BGP attributes are 2723 defined such that the BGP decision process depends on IGP metrics to 2724 select exit points for inter-domain traffic, then some inter-domain 2725 traffic destined to a given peer network can be made to prefer a 2726 specific exit point by establishing a TE-tunnel between the router 2727 making the selection and the peering point via a TE-tunnel and 2728 assigning the TE-tunnel a metric which is smaller than the IGP cost 2729 to all other peering points. 2731 Similarly to intra-domain TE, inter-domain TE is best accomplished 2732 when a traffic matrix can be derived to depict the volume of traffic 2733 from one AS to another. 2735 8. Overview of Contemporary TE Practices in Operational IP Networks 2737 This section provides an overview of some traffic engineering 2738 practices in IP networks. The focus is on aspects of control of the 2739 routing function in operational contexts. The intent here is to 2740 provide an overview of the commonly used practices: the discussion is 2741 not intended to be exhaustive. 2743 Service providers apply many of the traffic engineering mechanisms 2744 described in this document to optimize the performance of their IP 2745 networks. These techniques include capacity planning for long 2746 timescales; routing control using IGP metrics and MPLS, as well as 2747 path planning and path control using MPLS and Segment Routing for 2748 medium timescales; and traffic management mechanisms for short 2749 timescale. 2751 Capacity planning is an important component of how a service provider 2752 plans an effective IP network. These plans may take the following 2753 aspects into account: location of and new links or nodes, existing 2754 and predicted traffic patterns, costs, link capacity, topology, 2755 routing design, and survivability. 2757 Performance optimization of operational networks is usually an 2758 ongoing process in which traffic statistics, performance parameters, 2759 and fault indicators are continually collected from the network. 2760 This empirical data is analyzed and used to trigger TE mechanisms. 2761 Tools that perform what-if analysis can also be used to assist the TE 2762 process by reviewing scenarios before a new set of configurations are 2763 implemented in the operational network. 2765 Real-time intra-domain TE using the IGP is done by increasing the 2766 OSPF or IS-IS metric of a congested link until enough traffic has 2767 been diverted away from that link. This approach has some 2768 limitations as discussed in Section 6.2. Intra-domain TE approaches 2769 ([RR94] [FT00] [FT01] [WANG]) take traffic matrix, network topology, 2770 and network performance objectives as input, and produce link metrics 2771 and load-sharing ratios. These processes open the possibility for 2772 intra-domain TE with IGP to be done in a more systematic way. 2774 Administrators of MPLS-TE networks specify and configure link 2775 attributes and resource constraints such as maximum reservable 2776 bandwidth and resource class attributes for the links in the domain. 2777 A link state IGP that supports TE extensions (IS-IS-TE or OSPF-TE) is 2778 used to propagate information about network topology and link 2779 attributes to all routers in the domain. Network administrators 2780 specify the LSPs that are to originate at each router. For each LSP, 2781 the network administrator specifies the destination node and the 2782 attributes of the LSP which indicate the requirements that are to be 2783 satisfied during the path selection process. The attributes may 2784 include and explicit path for the LSP to follow, or originating 2785 router uses a local constraint-based routing process to compute the 2786 path of the LSP. RSVP-TE is used as a signaling protocol to 2787 instantiate the LSPs. By assigning proper bandwidth values to links 2788 and LSPs, congestion caused by uneven traffic distribution can be 2789 avoided or mitigated. 2791 The bandwidth attributes of an LSP relates to the bandwidth 2792 requirements of traffic that flows through the LSP. The traffic 2793 attribute of an LSP can be modified to accommodate persistent shifts 2794 in demand (traffic growth or reduction). If network congestion 2795 occurs due to some unexpected events, existing LSPs can be rerouted 2796 to alleviate the situation or network administrator can configure new 2797 LSPs to divert some traffic to alternative paths. The reservable 2798 bandwidth of the congested links can also be reduced to force some 2799 LSPs to be rerouted to other paths. A traffic matrix in an MPLS 2800 domain can also be estimated by monitoring the traffic on LSPs. Such 2801 traffic statistics can be used for a variety of purposes including 2802 network planning and network optimization. 2804 Network management and planning systems have evolved and taken over a 2805 lot of the responsibility for determining traffic paths in TE 2806 networks. This allows a network-wide view of resources, and 2807 facilitates coordination of the use of resources for all traffic 2808 flows in the network. Initial solutions using a PCE to perform path 2809 computation on behalf of network routers have given way to an 2810 approach that follows the SDN architecture. A stateful PCE is able 2811 to track all of the LSPs in the network and can redistribute them to 2812 make better use of the available resources. Such a PCE can forms 2813 part of a network orchestrator that uses PCEP or some other 2814 southbound interface to instruct the signaling protocol or directly 2815 program the routers. 2817 Segment routing leverages a centralized TE controller and either an 2818 MPLS or IPv6 forwarding plane, but does not need to use a signaling 2819 protocol or management plane protocol to reserve resources in the 2820 routers. All resource reservation is logical within the controller, 2821 and not distributed to the routers. Packets are steered through the 2822 network using segment routing. 2824 As mentioned in Section 7, there is usually no direct control over 2825 the distribution of inbound traffic to a domain. Therefore, the main 2826 goal of inter-domain TE is to optimize the distribution of outbound 2827 traffic between multiple inter-domain links. When operating a global 2828 network, maintaining the ability to operate the network in a regional 2829 fashion where desired, while continuing to take advantage of the 2830 benefits of a global network, also becomes an important objective. 2832 Inter-domain TE with BGP begins with the placement of multiple 2833 peering interconnection points that are in close proximity to traffic 2834 sources/destination, and offer lowest cost paths across the network 2835 between the peering points and and the sources/destinations. Some 2836 location-decision problems that arise in association with inter- 2837 domain routing are discussed in [AWD5]. 2839 Once the locations of the peering interconnects have been determined 2840 and implemented, the network operator decides how best to handle the 2841 routes advertised by the peer, as well as how to propagate the peer's 2842 routes within their network. One way to engineer outbound traffic 2843 flows in a network with many peering interconnects is to create a 2844 hierarchy of peers. Generally, the shortest AS paths will be chosen 2845 to forward traffic but BGP metrics can be used to prefer some peers 2846 and so favor particular paths. Preferred peers are those peers 2847 attached through peering interconnects with the most available 2848 capacity. Changes may be needed, for example, to deal with a 2849 "problem peer" who is difficult to work with on upgrades or is 2850 charging high prices for connectivity to their network. In that 2851 case, the peer may be given a reduced preference. This type of 2852 change can affect a large amount of traffic, and is only used after 2853 other methods have failed to provide the desired results. 2855 When there are multiple exit points toward a given peer, and only one 2856 of them is congested, it is not necessary to shift traffic away from 2857 the peer entirely, but only from the one congested connections. This 2858 can be achieved by using passive IGP-metrics, AS-path filtering, or 2859 prefix filtering. 2861 9. Security Considerations 2863 This document does not introduce new security issues. 2865 Network security is, of course, an important issue. In general, TE 2866 mechanisms are security neutral: they may use tunnels which can 2867 slightly help protect traffic from inspection and which, in some 2868 cases, can be secured using encryption; they put traffic onto 2869 predictable paths within the network that may make it easier to find 2870 and attack; they increase the complexity or operation and management 2871 of the network; and they enable traffic to be steered onto more 2872 secure links or to more secure parts of the network. 2874 The consequences of attacks on the control and management protocols 2875 used to operate TE networks can be significant: traffic can be 2876 hijacked to pass through specific nodes that perform inspection, or 2877 even to be delivered to the wrong place; traffic can be steered onto 2878 paths that deliver quality that is below the desired quality; and, 2879 networks can be congested or have resources on key links consumed. 2880 Thus, it is important to use adequate protection mechanisms on all 2881 protocols used to deliver TE. 2883 Certain aspects of a network may be deduced from the details of the 2884 TE paths that are used. For example, the link connectivity of the 2885 network, and the quality and load on individual links may be assumed 2886 from knowing the paths of traffic and the requirements they place on 2887 the network (for example, by seeing the control messages or through 2888 path- trace techniques). Such knowledge can be used to launch 2889 targeted attacks (for example, taking down critical links) or can 2890 reveal commercially sensistive information (for example, whether a 2891 network is close to capacity). Network operators may, therefore, 2892 choose techniques that mask or hide information from within the 2893 network. 2895 10. IANA Considerations 2897 This draft makes no requests for IANA action. 2899 11. Acknowledgments 2901 Much of the text in this document is derived from RFC 3272. The 2902 authors of this document would like to express their gratitude to all 2903 involved in that work. Although the source text has been edited in 2904 the production of this document, the orginal authors should be 2905 considered as Contributors to this work. They were: 2907 Daniel O. Awduche 2908 Movaz Networks 2910 Angela Chiu 2911 Celion Networks 2913 Anwar Elwalid 2914 Lucent Technologies 2916 Indra Widjaja 2917 Bell Labs, Lucent Technologies 2919 XiPeng Xiao 2920 Redback Networks 2922 The acknowledgements in RFC3272 were as below. All people who helped 2923 in the production of that document also need to be thanked for the 2924 carry-over into this new document. 2926 The authors would like to thank Jim Boyle for inputs on the 2927 recommendations section, Francois Le Faucheur for inputs on 2928 Diffserv aspects, Blaine Christian for inputs on measurement, 2929 Gerald Ash for inputs on routing in telephone networks and for 2930 text on event-dependent TE methods, Steven Wright for inputs 2931 on network controllability, and Jonathan Aufderheide for 2932 inputs on inter-domain TE with BGP. Special thanks to 2933 Randy Bush for proposing the TE taxonomy based on "tactical versus 2934 strategic" methods. The subsection describing an "Overview of 2935 ITU Activities Related to Traffic Engineering" was adapted from 2936 a contribution by Waisum Lai. Useful feedback and pointers to 2937 relevant materials were provided by J. Noel Chiappa. 2938 Additional comments were provided by Glenn Grotefeld during 2939 the working last call process. Finally, the authors would like 2940 to thank Ed Kern, the TEWG co-chair, for his comments and 2941 support. 2943 The early versions of this document were produced by the TEAS Working 2944 Group's RFC3272bis Design Team. The full list of members of this 2945 team is: 2947 Acee Lindem 2948 Adrian Farrel 2949 Aijun Wang 2950 Daniele Ceccarelli 2951 Dieter Beller 2952 Jeff Tantsura 2953 Julien Meuric 2954 Liu Hua 2955 Loa Andersson 2956 Luis Miguel Contreras 2957 Martin Horneffer 2958 Tarek Saad 2959 Xufeng Liu 2961 The production of this document includes a fix to the orriginal text 2962 resulting from an Errata Report by Jean-Michel Grimaldi. 2964 The authors of this document would also like to thank Dhurv Dhody for 2965 review comments. 2967 12. Contributors 2969 The following people contributed substantive text to this document: 2971 Gert Grammel 2972 EMail: ggrammel@juniper.net 2974 Loa Andersson 2975 EMail: loa@pi.nu 2977 Xufeng Liu 2978 EMail: xufeng.liu.ietf@gmail.com 2980 Lou Berger 2981 EMail: lberger@labn.net 2983 Jeff Tantsura 2984 EMail: jefftant.ietf@gmail.com 2986 Daniel King 2987 EMail: daniel@olddog.co.uk 2989 Boris Hassanov 2990 EMail: bhassanov@yandex-team.ru 2992 Kiran Makhijani 2993 Email: kiranm@futurewei.com 2995 Dhruv Dhody 2996 Email: dhruv.ietf@gmail.com 2998 13. Informative References 3000 [AJ19] Adekitan, A., Abolade, J., and O. Shobayo, "Data mining 3001 approach for predicting the daily Internet data traffic of 3002 a smart university", Article Journal of Big Data, 2019, 3003 Volume 6, Number 1, Page 1, 1998. 3005 [ASH2] Ash, J., "Dynamic Routing in Telecommunications Networks", 3006 Book McGraw Hill, 1998. 3008 [AWD2] Awduche, D., "MPLS and Traffic Engineering in IP 3009 Networks", Article IEEE Communications Magazine, December 3010 1999. 3012 [AWD5] Awduche, D., "An Approach to Optimal Peering Between 3013 Autonomous Systems in the Internet", Paper International 3014 Conference on Computer Communications and Networks 3015 (ICCCN'98), October 1998. 3017 [FLJA93] Floyd, S. and V. Jacobson, "Random Early Detection 3018 Gateways for Congestion Avoidance", Article IEEE/ACM 3019 Transactions on Networking, Vol. 1, p. 387-413, November 3020 1993. 3022 [FLOY94] Floyd, S., "TCP and Explicit Congestion Notification", 3023 Article ACM Computer Communication Review, V. 24, No. 5, 3024 p. 10-23, October 1994. 3026 [FT00] Fortz, B. and M. Thorup, "Internet Traffic Engineering by 3027 Optimizing OSPF Weights", Article IEEE INFOCOM 2000, March 3028 2000. 3030 [FT01] Fortz, B. and M. Thorup, "Optimizing OSPF/IS-IS Weights in 3031 a Changing World", n.d., 3032 . 3034 [HUSS87] Hurley, B., Seidl, C., and W. Sewel, "A Survey of Dynamic 3035 Routing Methods for Circuit-Switched Traffic", 3036 Article IEEE Communication Magazine, September 1987. 3038 [I-D.bonica-lsr-ip-flexalgo] 3039 Britto, W., Hegde, S., Kaneriya, P., Shetty, R., Bonica, 3040 R., and P. Psenak, "IGP Flexible Algorithms (Flex- 3041 Algorithm) In IP Networks", draft-bonica-lsr-ip- 3042 flexalgo-01 (work in progress), November 2020. 3044 [I-D.ietf-alto-performance-metrics] 3045 WU, Q., Yang, Y., Lee, Y., Dhody, D., Randriamasy, S., and 3046 L. Contreras, "ALTO Performance Cost Metrics", draft-ietf- 3047 alto-performance-metrics-12 (work in progress), July 2020. 3049 [I-D.ietf-bess-evpn-unequal-lb] 3050 Malhotra, N., Sajassi, A., Rabadan, J., Drake, J., 3051 Lingala, A., and S. Thoria, "Weighted Multi-Path 3052 Procedures for EVPN All-Active Multi-Homing", draft-ietf- 3053 bess-evpn-unequal-lb-07 (work in progress), October 2020. 3055 [I-D.ietf-detnet-data-plane-framework] 3056 Varga, B., Farkas, J., Berger, L., Malis, A., and S. 3057 Bryant, "DetNet Data Plane Framework", draft-ietf-detnet- 3058 data-plane-framework-06 (work in progress), May 2020. 3060 [I-D.ietf-detnet-ip-over-tsn] 3061 Varga, B., Farkas, J., Malis, A., and S. Bryant, "DetNet 3062 Data Plane: IP over IEEE 802.1 Time Sensitive Networking 3063 (TSN)", draft-ietf-detnet-ip-over-tsn-05 (work in 3064 progress), December 2020. 3066 [I-D.ietf-idr-rfc5575bis] 3067 Loibl, C., Hares, S., Raszuk, R., McPherson, D., and M. 3068 Bacher, "Dissemination of Flow Specification Rules", 3069 draft-ietf-idr-rfc5575bis-27 (work in progress), October 3070 2020. 3072 [I-D.ietf-idr-segment-routing-te-policy] 3073 Previdi, S., Filsfils, C., Talaulikar, K., Mattes, P., 3074 Rosen, E., Jain, D., and S. Lin, "Advertising Segment 3075 Routing Policies in BGP", draft-ietf-idr-segment-routing- 3076 te-policy-11 (work in progress), November 2020. 3078 [I-D.ietf-lsr-flex-algo] 3079 Psenak, P., Hegde, S., Filsfils, C., Talaulikar, K., and 3080 A. Gulko, "IGP Flexible Algorithm", draft-ietf-lsr-flex- 3081 algo-13 (work in progress), October 2020. 3083 [I-D.ietf-quic-transport] 3084 Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed 3085 and Secure Transport", draft-ietf-quic-transport-33 (work 3086 in progress), December 2020. 3088 [I-D.ietf-spring-segment-routing-policy] 3089 Filsfils, C., Talaulikar, K., Voyer, D., Bogdanov, A., and 3090 P. Mattes, "Segment Routing Policy Architecture", draft- 3091 ietf-spring-segment-routing-policy-09 (work in progress), 3092 November 2020. 3094 [I-D.ietf-teas-enhanced-vpn] 3095 Dong, J., Bryant, S., Li, Z., Miyasaka, T., and Y. Lee, "A 3096 Framework for Enhanced Virtual Private Networks (VPN+) 3097 Service", draft-ietf-teas-enhanced-vpn-06 (work in 3098 progress), July 2020. 3100 [I-D.ietf-tewg-qos-routing] 3101 Ash, G., "Traffic Engineering & QoS Methods for IP-, ATM-, 3102 & Based Multiservice Networks", draft-ietf-tewg-qos- 3103 routing-04 (work in progress), October 2001. 3105 [I-D.nsdt-teas-ietf-network-slice-definition] 3106 Rokui, R., Homma, S., Makhijani, K., Contreras, L., and J. 3107 Tantsura, "Definition of IETF Network Slices", draft-nsdt- 3108 teas-ietf-network-slice-definition-02 (work in progress), 3109 December 2020. 3111 [ITU-E600] 3112 "Terms and Definitions of Traffic Engineering", 3113 Recommendation ITU-T Recommendation E.600, March 1993. 3115 [ITU-E701] 3116 "Reference Connections for Traffic Engineering", 3117 Recommendation ITU-T Recommendation E.701, October 1993. 3119 [ITU-E801] 3120 "Framework for Service Quality Agreement", 3121 Recommendation ITU-T Recommendation E.801, October 1996. 3123 [MA] Ma, Q., "Quality of Service Routing in Integrated Services 3124 Networks", Ph.D. PhD Dissertation, CMU-CS-98-138, CMU, 3125 1998. 3127 [MATE] Elwalid, A., Jin, C., Low, S., and I. Widjaja, "MATE - 3128 MPLS Adaptive Traffic Engineering", 3129 Proceedings INFOCOM'01, April 2001. 3131 [MCQ80] McQuillan, J., Richer, I., and E. Rosen, "The New Routing 3132 Algorithm for the ARPANET", Transaction IEEE Transactions 3133 on Communications, vol. 28, no. 5, p. 711-719, May 1980. 3135 [MR99] Mitra, D. and K. Ramakrishnan, "A Case Study of 3136 Multiservice, Multipriority Traffic Engineering Design for 3137 Data Networks", Proceedings Globecom'99, December 1999. 3139 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 3140 DOI 10.17487/RFC0791, September 1981, 3141 . 3143 [RFC1102] Clark, D., "Policy routing in Internet protocols", 3144 RFC 1102, DOI 10.17487/RFC1102, May 1989, 3145 . 3147 [RFC1104] Braun, H., "Models of policy based routing", RFC 1104, 3148 DOI 10.17487/RFC1104, June 1989, 3149 . 3151 [RFC1992] Castineyra, I., Chiappa, N., and M. Steenstrup, "The 3152 Nimrod Routing Architecture", RFC 1992, 3153 DOI 10.17487/RFC1992, August 1996, 3154 . 3156 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 3157 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 3158 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 3159 September 1997, . 3161 [RFC2328] Moy, J., "OSPF Version 2", STD 54, RFC 2328, 3162 DOI 10.17487/RFC2328, April 1998, 3163 . 3165 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 3166 "Framework for IP Performance Metrics", RFC 2330, 3167 DOI 10.17487/RFC2330, May 1998, 3168 . 3170 [RFC2370] Coltun, R., "The OSPF Opaque LSA Option", RFC 2370, 3171 DOI 10.17487/RFC2370, July 1998, 3172 . 3174 [RFC2386] Crawley, E., Nair, R., Rajagopalan, B., and H. Sandick, "A 3175 Framework for QoS-based Routing in the Internet", 3176 RFC 2386, DOI 10.17487/RFC2386, August 1998, 3177 . 3179 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 3180 "Definition of the Differentiated Services Field (DS 3181 Field) in the IPv4 and IPv6 Headers", RFC 2474, 3182 DOI 10.17487/RFC2474, December 1998, 3183 . 3185 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 3186 and W. Weiss, "An Architecture for Differentiated 3187 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 3188 . 3190 [RFC2597] Heinanen, J., Baker, F., Weiss, W., and J. Wroclawski, 3191 "Assured Forwarding PHB Group", RFC 2597, 3192 DOI 10.17487/RFC2597, June 1999, 3193 . 3195 [RFC2678] Mahdavi, J. and V. Paxson, "IPPM Metrics for Measuring 3196 Connectivity", RFC 2678, DOI 10.17487/RFC2678, September 3197 1999, . 3199 [RFC2702] Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M., and J. 3200 McManus, "Requirements for Traffic Engineering Over MPLS", 3201 RFC 2702, DOI 10.17487/RFC2702, September 1999, 3202 . 3204 [RFC2722] Brownlee, N., Mills, C., and G. Ruth, "Traffic Flow 3205 Measurement: Architecture", RFC 2722, 3206 DOI 10.17487/RFC2722, October 1999, 3207 . 3209 [RFC2753] Yavatkar, R., Pendarakis, D., and R. Guerin, "A Framework 3210 for Policy-based Admission Control", RFC 2753, 3211 DOI 10.17487/RFC2753, January 2000, 3212 . 3214 [RFC2961] Berger, L., Gan, D., Swallow, G., Pan, P., Tommasi, F., 3215 and S. Molendini, "RSVP Refresh Overhead Reduction 3216 Extensions", RFC 2961, DOI 10.17487/RFC2961, April 2001, 3217 . 3219 [RFC2998] Bernet, Y., Ford, P., Yavatkar, R., Baker, F., Zhang, L., 3220 Speer, M., Braden, R., Davie, B., Wroclawski, J., and E. 3221 Felstaine, "A Framework for Integrated Services Operation 3222 over Diffserv Networks", RFC 2998, DOI 10.17487/RFC2998, 3223 November 2000, . 3225 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 3226 Label Switching Architecture", RFC 3031, 3227 DOI 10.17487/RFC3031, January 2001, 3228 . 3230 [RFC3086] Nichols, K. and B. Carpenter, "Definition of 3231 Differentiated Services Per Domain Behaviors and Rules for 3232 their Specification", RFC 3086, DOI 10.17487/RFC3086, 3233 April 2001, . 3235 [RFC3124] Balakrishnan, H. and S. Seshan, "The Congestion Manager", 3236 RFC 3124, DOI 10.17487/RFC3124, June 2001, 3237 . 3239 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 3240 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 3241 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 3242 . 3244 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 3245 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 3246 Protocol Label Switching (MPLS) Support of Differentiated 3247 Services", RFC 3270, DOI 10.17487/RFC3270, May 2002, 3248 . 3250 [RFC3272] Awduche, D., Chiu, A., Elwalid, A., Widjaja, I., and X. 3251 Xiao, "Overview and Principles of Internet Traffic 3252 Engineering", RFC 3272, DOI 10.17487/RFC3272, May 2002, 3253 . 3255 [RFC3469] Sharma, V., Ed. and F. Hellstrand, Ed., "Framework for 3256 Multi-Protocol Label Switching (MPLS)-based Recovery", 3257 RFC 3469, DOI 10.17487/RFC3469, February 2003, 3258 . 3260 [RFC3630] Katz, D., Kompella, K., and D. Yeung, "Traffic Engineering 3261 (TE) Extensions to OSPF Version 2", RFC 3630, 3262 DOI 10.17487/RFC3630, September 2003, 3263 . 3265 [RFC3945] Mannie, E., Ed., "Generalized Multi-Protocol Label 3266 Switching (GMPLS) Architecture", RFC 3945, 3267 DOI 10.17487/RFC3945, October 2004, 3268 . 3270 [RFC4090] Pan, P., Ed., Swallow, G., Ed., and A. Atlas, Ed., "Fast 3271 Reroute Extensions to RSVP-TE for LSP Tunnels", RFC 4090, 3272 DOI 10.17487/RFC4090, May 2005, 3273 . 3275 [RFC4124] Le Faucheur, F., Ed., "Protocol Extensions for Support of 3276 Diffserv-aware MPLS Traffic Engineering", RFC 4124, 3277 DOI 10.17487/RFC4124, June 2005, 3278 . 3280 [RFC4203] Kompella, K., Ed. and Y. Rekhter, Ed., "OSPF Extensions in 3281 Support of Generalized Multi-Protocol Label Switching 3282 (GMPLS)", RFC 4203, DOI 10.17487/RFC4203, October 2005, 3283 . 3285 [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A 3286 Border Gateway Protocol 4 (BGP-4)", RFC 4271, 3287 DOI 10.17487/RFC4271, January 2006, 3288 . 3290 [RFC4594] Babiarz, J., Chan, K., and F. Baker, "Configuration 3291 Guidelines for DiffServ Service Classes", RFC 4594, 3292 DOI 10.17487/RFC4594, August 2006, 3293 . 3295 [RFC4655] Farrel, A., Vasseur, J., and J. Ash, "A Path Computation 3296 Element (PCE)-Based Architecture", RFC 4655, 3297 DOI 10.17487/RFC4655, August 2006, 3298 . 3300 [RFC4872] Lang, J., Ed., Rekhter, Y., Ed., and D. Papadimitriou, 3301 Ed., "RSVP-TE Extensions in Support of End-to-End 3302 Generalized Multi-Protocol Label Switching (GMPLS) 3303 Recovery", RFC 4872, DOI 10.17487/RFC4872, May 2007, 3304 . 3306 [RFC4873] Berger, L., Bryskin, I., Papadimitriou, D., and A. Farrel, 3307 "GMPLS Segment Recovery", RFC 4873, DOI 10.17487/RFC4873, 3308 May 2007, . 3310 [RFC5305] Li, T. and H. Smit, "IS-IS Extensions for Traffic 3311 Engineering", RFC 5305, DOI 10.17487/RFC5305, October 3312 2008, . 3314 [RFC5329] Ishiguro, K., Manral, V., Davey, A., and A. Lindem, Ed., 3315 "Traffic Engineering Extensions to OSPF Version 3", 3316 RFC 5329, DOI 10.17487/RFC5329, September 2008, 3317 . 3319 [RFC5331] Aggarwal, R., Rekhter, Y., and E. Rosen, "MPLS Upstream 3320 Label Assignment and Context-Specific Label Space", 3321 RFC 5331, DOI 10.17487/RFC5331, August 2008, 3322 . 3324 [RFC5394] Bryskin, I., Papadimitriou, D., Berger, L., and J. Ash, 3325 "Policy-Enabled Path Computation Framework", RFC 5394, 3326 DOI 10.17487/RFC5394, December 2008, 3327 . 3329 [RFC5440] Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation 3330 Element (PCE) Communication Protocol (PCEP)", RFC 5440, 3331 DOI 10.17487/RFC5440, March 2009, 3332 . 3334 [RFC5541] Le Roux, JL., Vasseur, JP., and Y. Lee, "Encoding of 3335 Objective Functions in the Path Computation Element 3336 Communication Protocol (PCEP)", RFC 5541, 3337 DOI 10.17487/RFC5541, June 2009, 3338 . 3340 [RFC5557] Lee, Y., Le Roux, JL., King, D., and E. Oki, "Path 3341 Computation Element Communication Protocol (PCEP) 3342 Requirements and Protocol Extensions in Support of Global 3343 Concurrent Optimization", RFC 5557, DOI 10.17487/RFC5557, 3344 July 2009, . 3346 [RFC5664] Halevy, B., Welch, B., and J. Zelenka, "Object-Based 3347 Parallel NFS (pNFS) Operations", RFC 5664, 3348 DOI 10.17487/RFC5664, January 2010, 3349 . 3351 [RFC5693] Seedorf, J. and E. Burger, "Application-Layer Traffic 3352 Optimization (ALTO) Problem Statement", RFC 5693, 3353 DOI 10.17487/RFC5693, October 2009, 3354 . 3356 [RFC6119] Harrison, J., Berger, J., and M. Bartlett, "IPv6 Traffic 3357 Engineering in IS-IS", RFC 6119, DOI 10.17487/RFC6119, 3358 February 2011, . 3360 [RFC6241] Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed., 3361 and A. Bierman, Ed., "Network Configuration Protocol 3362 (NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011, 3363 . 3365 [RFC6372] Sprecher, N., Ed. and A. Farrel, Ed., "MPLS Transport 3366 Profile (MPLS-TP) Survivability Framework", RFC 6372, 3367 DOI 10.17487/RFC6372, September 2011, 3368 . 3370 [RFC6374] Frost, D. and S. Bryant, "Packet Loss and Delay 3371 Measurement for MPLS Networks", RFC 6374, 3372 DOI 10.17487/RFC6374, September 2011, 3373 . 3375 [RFC6805] King, D., Ed. and A. Farrel, Ed., "The Application of the 3376 Path Computation Element Architecture to the Determination 3377 of a Sequence of Domains in MPLS and GMPLS", RFC 6805, 3378 DOI 10.17487/RFC6805, November 2012, 3379 . 3381 [RFC7149] Boucadair, M. and C. Jacquenet, "Software-Defined 3382 Networking: A Perspective from within a Service Provider 3383 Environment", RFC 7149, DOI 10.17487/RFC7149, March 2014, 3384 . 3386 [RFC7159] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 3387 Interchange Format", RFC 7159, DOI 10.17487/RFC7159, March 3388 2014, . 3390 [RFC7285] Alimi, R., Ed., Penno, R., Ed., Yang, Y., Ed., Kiesel, S., 3391 Previdi, S., Roome, W., Shalunov, S., and R. Woundy, 3392 "Application-Layer Traffic Optimization (ALTO) Protocol", 3393 RFC 7285, DOI 10.17487/RFC7285, September 2014, 3394 . 3396 [RFC7390] Rahman, A., Ed. and E. Dijk, Ed., "Group Communication for 3397 the Constrained Application Protocol (CoAP)", RFC 7390, 3398 DOI 10.17487/RFC7390, October 2014, 3399 . 3401 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 3402 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 3403 Defined Networking (SDN): Layers and Architecture 3404 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 3405 2015, . 3407 [RFC7471] Giacalone, S., Ward, D., Drake, J., Atlas, A., and S. 3408 Previdi, "OSPF Traffic Engineering (TE) Metric 3409 Extensions", RFC 7471, DOI 10.17487/RFC7471, March 2015, 3410 . 3412 [RFC7491] King, D. and A. Farrel, "A PCE-Based Architecture for 3413 Application-Based Network Operations", RFC 7491, 3414 DOI 10.17487/RFC7491, March 2015, 3415 . 3417 [RFC7679] Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, 3418 Ed., "A One-Way Delay Metric for IP Performance Metrics 3419 (IPPM)", STD 81, RFC 7679, DOI 10.17487/RFC7679, January 3420 2016, . 3422 [RFC7680] Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, 3423 Ed., "A One-Way Loss Metric for IP Performance Metrics 3424 (IPPM)", STD 82, RFC 7680, DOI 10.17487/RFC7680, January 3425 2016, . 3427 [RFC7752] Gredler, H., Ed., Medved, J., Previdi, S., Farrel, A., and 3428 S. Ray, "North-Bound Distribution of Link-State and 3429 Traffic Engineering (TE) Information Using BGP", RFC 7752, 3430 DOI 10.17487/RFC7752, March 2016, 3431 . 3433 [RFC7810] Previdi, S., Ed., Giacalone, S., Ward, D., Drake, J., and 3434 Q. Wu, "IS-IS Traffic Engineering (TE) Metric Extensions", 3435 RFC 7810, DOI 10.17487/RFC7810, May 2016, 3436 . 3438 [RFC7923] Voit, E., Clemm, A., and A. Gonzalez Prieto, "Requirements 3439 for Subscription to YANG Datastores", RFC 7923, 3440 DOI 10.17487/RFC7923, June 2016, 3441 . 3443 [RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", 3444 RFC 7950, DOI 10.17487/RFC7950, August 2016, 3445 . 3447 [RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF 3448 Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017, 3449 . 3451 [RFC8051] Zhang, X., Ed. and I. Minei, Ed., "Applicability of a 3452 Stateful Path Computation Element (PCE)", RFC 8051, 3453 DOI 10.17487/RFC8051, January 2017, 3454 . 3456 [RFC8189] Randriamasy, S., Roome, W., and N. Schwan, "Multi-Cost 3457 Application-Layer Traffic Optimization (ALTO)", RFC 8189, 3458 DOI 10.17487/RFC8189, October 2017, 3459 . 3461 [RFC8231] Crabbe, E., Minei, I., Medved, J., and R. Varga, "Path 3462 Computation Element Communication Protocol (PCEP) 3463 Extensions for Stateful PCE", RFC 8231, 3464 DOI 10.17487/RFC8231, September 2017, 3465 . 3467 [RFC8281] Crabbe, E., Minei, I., Sivabalan, S., and R. Varga, "Path 3468 Computation Element Communication Protocol (PCEP) 3469 Extensions for PCE-Initiated LSP Setup in a Stateful PCE 3470 Model", RFC 8281, DOI 10.17487/RFC8281, December 2017, 3471 . 3473 [RFC8283] Farrel, A., Ed., Zhao, Q., Ed., Li, Z., and C. Zhou, "An 3474 Architecture for Use of PCE and the PCE Communication 3475 Protocol (PCEP) in a Network with Central Control", 3476 RFC 8283, DOI 10.17487/RFC8283, December 2017, 3477 . 3479 [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 3480 Decraene, B., Litkowski, S., and R. Shakir, "Segment 3481 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 3482 July 2018, . 3484 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for 3485 Abstraction and Control of TE Networks (ACTN)", RFC 8453, 3486 DOI 10.17487/RFC8453, August 2018, 3487 . 3489 [RFC8570] Ginsberg, L., Ed., Previdi, S., Ed., Giacalone, S., Ward, 3490 D., Drake, J., and Q. Wu, "IS-IS Traffic Engineering (TE) 3491 Metric Extensions", RFC 8570, DOI 10.17487/RFC8570, March 3492 2019, . 3494 [RFC8571] Ginsberg, L., Ed., Previdi, S., Wu, Q., Tantsura, J., and 3495 C. Filsfils, "BGP - Link State (BGP-LS) Advertisement of 3496 IGP Traffic Engineering Performance Metric Extensions", 3497 RFC 8571, DOI 10.17487/RFC8571, March 2019, 3498 . 3500 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 3501 "Deterministic Networking Architecture", RFC 8655, 3502 DOI 10.17487/RFC8655, October 2019, 3503 . 3505 [RFC8661] Bashandy, A., Ed., Filsfils, C., Ed., Previdi, S., 3506 Decraene, B., and S. Litkowski, "Segment Routing MPLS 3507 Interworking with LDP", RFC 8661, DOI 10.17487/RFC8661, 3508 December 2019, . 3510 [RFC8664] Sivabalan, S., Filsfils, C., Tantsura, J., Henderickx, W., 3511 and J. Hardwick, "Path Computation Element Communication 3512 Protocol (PCEP) Extensions for Segment Routing", RFC 8664, 3513 DOI 10.17487/RFC8664, December 2019, 3514 . 3516 [RFC8795] Liu, X., Bryskin, I., Beeram, V., Saad, T., Shah, H., and 3517 O. Gonzalez de Dios, "YANG Data Model for Traffic 3518 Engineering (TE) Topologies", RFC 8795, 3519 DOI 10.17487/RFC8795, August 2020, 3520 . 3522 [RFC8896] Randriamasy, S., Yang, R., Wu, Q., Deng, L., and N. 3523 Schwan, "Application-Layer Traffic Optimization (ALTO) 3524 Cost Calendar", RFC 8896, DOI 10.17487/RFC8896, November 3525 2020, . 3527 [RR94] Rodrigues, M. and K. Ramakrishnan, "Optimal Routing in 3528 Shortest Path Networks", Proceedings ITS'94, Rio de 3529 Janeiro, Brazil, 1994. 3531 [SLDC98] Suter, B., Lakshman, T., Stiliadis, D., and A. Choudhury, 3532 "Design Considerations for Supporting TCP with Per-flow 3533 Queueing", Proceedings INFOCOM'98, p. 299-306, 1998. 3535 [WANG] Wang, Y., Wang, Z., and L. Zhang, "Internet traffic 3536 engineering without full mesh overlaying", 3537 Proceedings INFOCOM'2001, April 2001. 3539 [XIAO] Xiao, X., Hannan, A., Bailey, B., and L. Ni, "Traffic 3540 Engineering with MPLS in the Internet", Article IEEE 3541 Network Magazine, March 2000. 3543 [YARE95] Yang, C. and A. Reddy, "A Taxonomy for Congestion Control 3544 Algorithms in Packet Switching Networks", Article IEEE 3545 Network Magazine, p. 34-45, 1995. 3547 Appendix A. Historic Overview 3549 A.1. Traffic Engineering in Classical Telephone Networks 3551 This subsection presents a brief overview of traffic engineering in 3552 telephone networks which often relates to the way user traffic is 3553 steered from an originating node to the terminating node. This 3554 subsection presents a brief overview of this topic. A detailed 3555 description of the various routing strategies applied in telephone 3556 networks is included in the book by G. Ash [ASH2]. 3558 The early telephone network relied on static hierarchical routing, 3559 whereby routing patterns remained fixed independent of the state of 3560 the network or time of day. The hierarchy was intended to 3561 accommodate overflow traffic, improve network reliability via 3562 alternate routes, and prevent call looping by employing strict 3563 hierarchical rules. The network was typically over-provisioned since 3564 a given fixed route had to be dimensioned so that it could carry user 3565 traffic during a busy hour of any busy day. Hierarchical routing in 3566 the telephony network was found to be too rigid upon the advent of 3567 digital switches and stored program control which were able to manage 3568 more complicated traffic engineering rules. 3570 Dynamic routing was introduced to alleviate the routing inflexibility 3571 in the static hierarchical routing so that the network would operate 3572 more efficiently. This resulted in significant economic gains 3573 [HUSS87]. Dynamic routing typically reduces the overall loss 3574 probability by 10 to 20 percent (compared to static hierarchical 3575 routing). Dynamic routing can also improve network resilience by 3576 recalculating routes on a per-call basis and periodically updating 3577 routes. 3579 There are three main types of dynamic routing in the telephone 3580 network. They are time-dependent routing, state-dependent routing 3581 (SDR), and event dependent routing (EDR). 3583 In time-dependent routing, regular variations in traffic loads (such 3584 as time of day or day of week) are exploited in pre-planned routing 3585 tables. In state-dependent routing, routing tables are updated 3586 online according to the current state of the network (e.g., traffic 3587 demand, utilization, etc.). In event dependent routing, routing 3588 changes are triggers by events (such as call setups encountering 3589 congested or blocked links) whereupon new paths are searched out 3590 using learning models. EDR methods are real-time adaptive, but they 3591 do not require global state information as does SDR. Examples of EDR 3592 schemes include the dynamic alternate routing (DAR) from BT, the 3593 state-and-time dependent routing (STR) from NTT, and the success-to- 3594 the-top (STT) routing from AT&T. 3596 Dynamic non-hierarchical routing (DNHR) is an example of dynamic 3597 routing that was introduced in the AT&T toll network in the 1980's to 3598 respond to time-dependent information such as regular load variations 3599 as a function of time. Time-dependent information in terms of load 3600 may be divided into three timescales: hourly, weekly, and yearly. 3601 Correspondingly, three algorithms are defined to pre-plan the routing 3602 tables. The network design algorithm operates over a year-long 3603 interval while the demand servicing algorithm operates on a weekly 3604 basis to fine tune link sizes and routing tables to correct forecast 3605 errors on the yearly basis. At the smallest timescale, the routing 3606 algorithm is used to make limited adjustments based on daily traffic 3607 variations. Network design and demand servicing are computed using 3608 offline calculations. Typically, the calculations require extensive 3609 searches on possible routes. On the other hand, routing may need 3610 online calculations to handle crankback. DNHR adopts a "two-link" 3611 approach whereby a path can consist of two links at most. The 3612 routing algorithm presents an ordered list of route choices between 3613 an originating switch and a terminating switch. If a call overflows, 3614 a via switch (a tandem exchange between the originating switch and 3615 the terminating switch) would send a crankback signal to the 3616 originating switch. This switch would then select the next route, 3617 and so on, until there are no alternative routes available in which 3618 the call is blocked. 3620 A.2. Evolution of Traffic Engineering in Packet Networks 3622 This subsection reviews related prior work that was intended to 3623 improve the performance of data networks. Indeed, optimization of 3624 the performance of data networks started in the early days of the 3625 ARPANET. Other early commercial networks such as SNA also recognized 3626 the importance of performance optimization and service 3627 differentiation. 3629 In terms of traffic management, the Internet has been a best effort 3630 service environment until recently. In particular, very limited 3631 traffic management capabilities existed in IP networks to provide 3632 differentiated queue management and scheduling services to packets 3633 belonging to different classes. 3635 In terms of routing control, the Internet has employed distributed 3636 protocols for intra-domain routing. These protocols are highly 3637 scalable and resilient. However, they are based on simple algorithms 3638 for path selection which have very limited functionality to allow 3639 flexible control of the path selection process. 3641 In the following subsections, the evolution of practical traffic 3642 engineering mechanisms in IP networks and its predecessors are 3643 reviewed. 3645 A.2.1. Adaptive Routing in the ARPANET 3647 The early ARPANET recognized the importance of adaptive routing where 3648 routing decisions were based on the current state of the network 3649 [MCQ80]. Early minimum delay routing approaches forwarded each 3650 packet to its destination along a path for which the total estimated 3651 transit time was the smallest. Each node maintained a table of 3652 network delays, representing the estimated delay that a packet would 3653 experience along a given path toward its destination. The minimum 3654 delay table was periodically transmitted by a node to its neighbors. 3655 The shortest path, in terms of hop count, was also propagated to give 3656 the connectivity information. 3658 One drawback to this approach is that dynamic link metrics tend to 3659 create "traffic magnets" causing congestion to be shifted from one 3660 location of a network to another location, resulting in oscillation 3661 and network instability. 3663 A.2.2. Dynamic Routing in the Internet 3665 The Internet evolved from the ARPANET and adopted dynamic routing 3666 algorithms with distributed control to determine the paths that 3667 packets should take en-route to their destinations. The routing 3668 algorithms are adaptations of shortest path algorithms where costs 3669 are based on link metrics. The link metric can be based on static or 3670 dynamic quantities. The link metric based on static quantities may 3671 be assigned administratively according to local criteria. The link 3672 metric based on dynamic quantities may be a function of a network 3673 congestion measure such as delay or packet loss. 3675 It was apparent early that static link metric assignment was 3676 inadequate because it can easily lead to unfavorable scenarios in 3677 which some links become congested while others remain lightly loaded. 3678 One of the many reasons for the inadequacy of static link metrics is 3679 that link metric assignment was often done without considering the 3680 traffic matrix in the network. Also, the routing protocols did not 3681 take traffic attributes and capacity constraints into account when 3682 making routing decisions. This results in traffic concentration 3683 being localized in subsets of the network infrastructure and 3684 potentially causing congestion. Even if link metrics are assigned in 3685 accordance with the traffic matrix, unbalanced loads in the network 3686 can still occur due to a number factors including: 3688 o Resources may not be deployed in the most optimal locations from a 3689 routing perspective. 3691 o Forecasting errors in traffic volume and/or traffic distribution. 3693 o Dynamics in traffic matrix due to the temporal nature of traffic 3694 patterns, BGP policy change from peers, etc. 3696 The inadequacy of the legacy Internet interior gateway routing system 3697 is one of the factors motivating the interest in path oriented 3698 technology with explicit routing and constraint-based routing 3699 capability such as MPLS. 3701 A.2.3. ToS Routing 3703 Type-of-Service (ToS) routing involves different routes going to the 3704 same destination with selection dependent upon the ToS field of an IP 3705 packet [RFC2474]. The ToS classes may be classified as low delay and 3706 high throughput. Each link is associated with multiple link costs 3707 and each link cost is used to compute routes for a particular ToS. A 3708 separate shortest path tree is computed for each ToS. The shortest 3709 path algorithm must be run for each ToS resulting in very expensive 3710 computation. Classical ToS-based routing is now outdated as the IP 3711 header field has been replaced by a Diffserv field. Effective 3712 traffic engineering is difficult to perform in classical ToS-based 3713 routing because each class still relies exclusively on shortest path 3714 routing which results in localization of traffic concentration within 3715 the network. 3717 A.2.4. Equal Cost Multi-Path 3719 Equal Cost Multi-Path (ECMP) is another technique that attempts to 3720 address the deficiency in the Shortest Path First (SPF) interior 3721 gateway routing systems [RFC2328]. In the classical SPF algorithm, 3722 if two or more shortest paths exist to a given destination, the 3723 algorithm will choose one of them. The algorithm is modified 3724 slightly in ECMP so that if two or more equal cost shortest paths 3725 exist between two nodes, the traffic between the nodes is distributed 3726 among the multiple equal-cost paths. Traffic distribution across the 3727 equal-cost paths is usually performed in one of two ways: (1) packet- 3728 based in a round-robin fashion, or (2) flow-based using hashing on 3729 source and destination IP addresses and possibly other fields of the 3730 IP header. The first approach can easily cause out- of-order packets 3731 while the second approach is dependent upon the number and 3732 distribution of flows. Flow-based load sharing may be unpredictable 3733 in an enterprise network where the number of flows is relatively 3734 small and less heterogeneous (for example, hashing may not be 3735 uniform), but it is generally effective in core public networks where 3736 the number of flows is large and heterogeneous. 3738 In ECMP, link costs are static and bandwidth constraints are not 3739 considered, so ECMP attempts to distribute the traffic as equally as 3740 possible among the equal-cost paths independent of the congestion 3741 status of each path. As a result, given two equal-cost paths, it is 3742 possible that one of the paths will be more congested than the other. 3743 Another drawback of ECMP is that load sharing cannot be achieved on 3744 multiple paths which have non-identical costs. 3746 A.2.5. Nimrod 3748 Nimrod was a routing system developed to provide heterogeneous 3749 service specific routing in the Internet, while taking multiple 3750 constraints into account [RFC1992]. Essentially, Nimrod was a link 3751 state routing protocol to support path oriented packet forwarding. 3752 It used the concept of maps to represent network connectivity and 3753 services at multiple levels of abstraction. Mechanisms allowed 3754 restriction of the distribution of routing information. 3756 Even though Nimrod did not enjoy deployment in the public Internet, a 3757 number of key concepts incorporated into the Nimrod architecture, 3758 such as explicit routing which allows selection of paths at 3759 originating nodes, are beginning to find applications in some recent 3760 constraint-based routing initiatives. 3762 A.3. Development of Internet Traffic Engineering 3764 A.3.1. Overlay Model 3766 In the overlay model, a virtual-circuit network, such as Sonet/SDH, 3767 OTN, or WDM, provides virtual-circuit connectivity between routers 3768 that are located at the edges of a virtual-circuit cloud. In this 3769 mode, two routers that are connected through a virtual circuit see a 3770 direct adjacency between themselves independent of the physical route 3771 taken by the virtual circuit through the ATM, frame relay, or WDM 3772 network. Thus, the overlay model essentially decouples the logical 3773 topology that routers see from the physical topology that the ATM, 3774 frame relay, or WDM network manages. The overlay model based on ATM 3775 or frame relay enables a network administrator or an automaton to 3776 employ traffic engineering concepts to perform path optimization by 3777 re-configuring or rearranging the virtual circuits so that a virtual 3778 circuit on a congested or sub-optimal physical link can be re-routed 3779 to a less congested or more optimal one. In the overlay model, 3780 traffic engineering is also employed to establish relationships 3781 between the traffic management parameters (e.g., PCR, SCR, and MBS 3782 for ATM) of the virtual-circuit technology and the actual traffic 3783 that traverses each circuit. These relationships can be established 3784 based upon known or projected traffic profiles, and some other 3785 factors. 3787 Appendix B. Overview of Traffic Engineering Related Work in Other SDOs 3789 B.1. Overview of ITU Activities Related to Traffic Engineering 3791 This section provides an overview of prior work within the ITU-T 3792 pertaining to traffic engineering in traditional telecommunications 3793 networks. 3795 ITU-T Recommendations E.600 [ITU-E600], E.701 [ITU-E701], and E.801 3796 [ITU-E801] address traffic engineering issues in traditional 3797 telecommunications networks. Recommendation E.600 provides a 3798 vocabulary for describing traffic engineering concepts, while E.701 3799 defines reference connections, Grade of Service (GOS), and traffic 3800 parameters for ISDN. Recommendation E.701 uses the concept of a 3801 reference connection to identify representative cases of different 3802 types of connections without describing the specifics of their actual 3803 realizations by different physical means. As defined in 3804 Recommendation E.600, "a connection is an association of resources 3805 providing means for communication between two or more devices in, or 3806 attached to, a telecommunication network." Also, E.600 defines "a 3807 resource as any set of physically or conceptually identifiable 3808 entities within a telecommunication network, the use of which can be 3809 unambiguously determined" [ITU-E600]. There can be different types 3810 of connections as the number and types of resources in a connection 3811 may vary. 3813 Typically, different network segments are involved in the path of a 3814 connection. For example, a connection may be local, national, or 3815 international. The purposes of reference connections are to clarify 3816 and specify traffic performance issues at various interfaces between 3817 different network domains. Each domain may consist of one or more 3818 service provider networks. 3820 Reference connections provide a basis to define grade of service 3821 (GoS) parameters related to traffic engineering within the ITU-T 3822 framework. As defined in E.600, "GoS refers to a number of traffic 3823 engineering variables which are used to provide a measure of the 3824 adequacy of a group of resources under specified conditions." These 3825 GoS variables may be probability of loss, dial tone, delay, etc. 3826 They are essential for network internal design and operation as well 3827 as for component performance specification. 3829 GoS is different from quality of service (QoS) in the ITU framework. 3830 QoS is the performance perceivable by a telecommunication service 3831 user and expresses the user's degree of satisfaction of the service. 3832 QoS parameters focus on performance aspects observable at the service 3833 access points and network interfaces, rather than their causes within 3834 the network. GoS, on the other hand, is a set of network oriented 3835 measures which characterize the adequacy of a group of resources 3836 under specified conditions. For a network to be effective in serving 3837 its users, the values of both GoS and QoS parameters must be related, 3838 with GoS parameters typically making a major contribution to the QoS. 3840 Recommendation E.600 stipulates that a set of GoS parameters must be 3841 selected and defined on an end-to-end basis for each major service 3842 category provided by a network to assist the network provider with 3843 improving efficiency and effectiveness of the network. Based on a 3844 selected set of reference connections, suitable target values are 3845 assigned to the selected GoS parameters under normal and high load 3846 conditions. These end-to-end GoS target values are then apportioned 3847 to individual resource components of the reference connections for 3848 dimensioning purposes. 3850 Appendix C. Summary of Changes Since RFC 3272 3852 This section is a place-holder. It is expected that once work on 3853 this document is nearly complete, this section will be updated to 3854 provide an overview of the structural and substantive changed from 3855 RFC 3272. 3857 TBD 3859 Author's Address 3861 Adrian Farrel (editor) 3862 Old Dog Consulting 3864 Email: adrian@olddog.co.uk