idnits 2.17.1 draft-ietf-teas-scheduled-resources-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (April 10, 2018) is 2207 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'T0' is mentioned on line 470, but not defined == Missing Reference: 'B0' is mentioned on line 470, but not defined == Missing Reference: 'T1' is mentioned on line 470, but not defined == Missing Reference: 'B1' is mentioned on line 470, but not defined == Missing Reference: 'T2' is mentioned on line 470, but not defined == Missing Reference: 'B2' is mentioned on line 470, but not defined == Missing Reference: 'T3' is mentioned on line 470, but not defined == Missing Reference: 'B3' is mentioned on line 470, but not defined -- Obsolete informational reference (is this intentional?): RFC 7752 (Obsoleted by RFC 9552) Summary: 0 errors (**), 0 flaws (~~), 9 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TEAS Working Group Y. Zhuang 3 Internet-Draft Q. Wu 4 Intended status: Informational H. Chen 5 Expires: October 12, 2018 Huawei 6 A. Farrel 7 Juniper Networks 8 April 10, 2018 10 Framework for Scheduled Use of Resources 11 draft-ietf-teas-scheduled-resources-07 13 Abstract 15 Time-scheduled reservation of traffic engineering (TE) resources can 16 be used to provide resource booking for TE Label Switched Paths so as 17 to better guarantee services for customers and to improve the 18 efficiency of network resource usage at any moment in time including 19 future planned network usage. This document provides a framework 20 that describes and discusses the architecture for supporting 21 scheduled reservation of TE resources. This document does not 22 describe specific protocols or protocol extensions needed to realize 23 this service. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on October 12, 2018. 42 Copyright Notice 44 Copyright (c) 2018 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (https://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 60 2. Problem Statement . . . . . . . . . . . . . . . . . . . . . . 3 61 2.1. Provisioning TE-LSPs and TE Resources . . . . . . . . . . 3 62 2.2. Selecting the Path of an LSP . . . . . . . . . . . . . . 4 63 2.3. Planning Future LSPs . . . . . . . . . . . . . . . . . . 5 64 2.4. Looking at Future Demands on TE Resources . . . . . . . . 6 65 2.4.1. Interaction Between Time-Scheduled and Ad Hoc 66 Reservations . . . . . . . . . . . . . . . . . . . . 6 67 2.5. Requisite State Information . . . . . . . . . . . . . . . 6 68 3. Architectural Concepts . . . . . . . . . . . . . . . . . . . 8 69 3.1. Where is Scheduling State Held? . . . . . . . . . . . . . 8 70 3.2. What State is Held? . . . . . . . . . . . . . . . . . . . 10 71 3.3. Enforcement of Operator Policy . . . . . . . . . . . . . 11 72 4. Architecture Overview . . . . . . . . . . . . . . . . . . . . 12 73 4.1. Service Request . . . . . . . . . . . . . . . . . . . . . 13 74 4.1.1. Reoptimization After TED Updates . . . . . . . . . . 14 75 4.2. Initialization and Recovery . . . . . . . . . . . . . . . 15 76 4.3. Synchronization Between PCEs . . . . . . . . . . . . . . 15 77 5. Multi-Domain Considerations . . . . . . . . . . . . . . . . . 16 78 6. Security Considerations . . . . . . . . . . . . . . . . . . . 18 79 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 80 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19 81 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 19 82 10. Informative References . . . . . . . . . . . . . . . . . . . 20 83 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 85 1. Introduction 87 Traffic Engineering Label Switched Paths (TE-LSPs) are connection 88 oriented tunnels in packet and non-packet networks [RFC3209], 89 [RFC3945]. TE-LSPs may reserve network resources for use by the 90 traffic they carry, thus providing some guarantees of service 91 delivery and allowing a network operator to plan the use of the 92 resources across the whole network. 94 In some technologies (such as wavelength switched optical networks) 95 the resource is synonymous with the label that is switched on the 96 path of the LSP so that it is not possible to establish an LSP that 97 can carry traffic without assigning a physical resource to the LSP. 98 In other technologies (such as packet switched networks) the 99 resources assigned to an LSP are a measure of the capacity of a link 100 that is dedicated for use by the traffic on the LSP. 102 In all cases, network planning consists of selecting paths for LSPs 103 through the network so that there will be no contention for 104 resources. LSP establishment is the act of setting up an LSP and 105 reserving resources within the network. Network optimization or re- 106 optimization is the process of re-positioning LSPs in the network to 107 make the unreserved network resources more useful for potential 108 future LSPs while ensuring that the established LSPs continue to 109 fulfill their objectives. 111 It is often the case that it is known that an LSP will be needed at 112 some specific time in the future. While a path for that LSP could be 113 computed using knowledge of the currently established LSPs and the 114 currently available resources, this does not give any degree of 115 certainty that the necessary resources will be available when it is 116 time to set up the new LSP. Yet setting up the LSP ahead of the time 117 when it is needed (which would guarantee the availability of the 118 resources) is wasteful since the network resources could be used for 119 some other purpose in the meantime. 121 Similarly, it may be known that an LSP will no longer be needed after 122 some future time and that it will be torn down releasing the network 123 resources that were assigned to it. This information can be helpful 124 in planning how a future LSP is placed in the network. 126 Time-Scheduled (TS) reservation of TE resources can be used to 127 provide resource booking for TE-LSPs so as to better guarantee 128 services for customers and to improve the efficiency of network 129 resource usage into the future. This document provides a framework 130 that describes the problem and discusses the architecture for the 131 scheduled reservation of TE resources. This document does not 132 describe specific protocols or protocol extensions needed to realize 133 this service. 135 2. Problem Statement 137 2.1. Provisioning TE-LSPs and TE Resources 139 TE-LSPs in existing networks are provisioned using a variety of 140 techniques. They may be set up using RSVP-TE as a signaling protocol 141 [RFC3209] [RFC3473]. Alternatively, they could be established by 142 direct control of network elements such as in the Software Defined 143 Networking (SDN) paradigm. They could also be provisioned using the 144 PCE Communication Protocol (PCEP) [RFC5440] as a control protocol to 145 communicate with the network elements. 147 TE resources are reserved at the point of use. That is, the 148 resources (wavelengths, timeslots, bandwidth, etc.) are reserved for 149 use on a specific link and are tracked by the Label Switching Routers 150 (LSRs) at the end points of the link. Those LSRs learn which 151 resources to reserve during the LSP setup process. 153 The use of TE resources can be varied by changing the parameters of 154 the LSP that uses them, and the resources can be released by tearing 155 down the LSP. 157 Resources that have been reserved in the network for use by one LSP 158 may be pre-empted for use by another LSP. If RSVP-TE signaling is in 159 use, a holding priority and a pre-emption priority are used to 160 determine which LSPs may pre-empted the resources in use for which 161 other LSPs. If direct (central) control is in use, the controller is 162 able to make pre-emption decisions. In either case, operator policy 163 forms a key part of pre-emption since there is a trade between 164 disrupting existing LSPs and enabling new LSPs. 166 2.2. Selecting the Path of an LSP 168 Although TE-LSPs can determine their paths hop-by-hop using the 169 shortest path toward the destination to route the signaling protocol 170 messages [RFC3209], in practice this option is not applied because it 171 does not look far enough ahead into the network to verify that the 172 desired resources are available. Instead, the full length of the 173 path of an LSP is usually computed ahead of time either by the head- 174 end LSR of a signaled LSP, or by Path Computation Element (PCE) 175 functionality in a dedicated server or built into network management 176 software [RFC4655]. 178 Such full-path computation is applied in order that an end-to-end 179 view of the available resources in the network can be used to 180 determine the best likelihood of establishing a viable LSP that meets 181 the service requirements. Even in this situation, however, it is 182 possible that two LSPs being set up at the same time will compete for 183 scarce network resources meaning that one or both of them will fail 184 to be established. This situation is avoided by using a centralized 185 PCE that is aware of the LSP setup requests that are in progress. 187 Path selection may make allowance for pre-emption as described in 188 Section 2.1. That is, when selecting a path, the decision may be 189 made to choose a path that will result in the pre-emption of an 190 existing LSP. The trade-off between selecting a less optimal path, 191 failing to select any path at all, and pre-empting an existing LSP 192 must be subject to operator policy. 194 Path computation is subject to "objective functions" that define what 195 criteria are to be met when the LSP is placed [RFC4655]. These can 196 be criteria that apply to the LSP itself (such as shortest path to 197 destination) or to the network state after the LSP is set up (such as 198 maximized residual link bandwidth). The objective functions may be 199 requested by the application requesting the LSP, and may be filtered 200 and enhanced by the computation engine according to operator policy. 202 2.3. Planning Future LSPs 204 LSPs may be established "on demand" when the requester determines 205 that a new LSP is needed. In this case, the path of the LSP is 206 computed as described in Section 2.2. 208 However, in many situations, the requester knows in advance that an 209 LSP will be needed at a particular time in the future. For example, 210 the requester may be aware of a large traffic flow that will start at 211 a well-known time, perhaps for a database synchronization or for the 212 exchange of content between streaming sites. Furthermore, the 213 requester may also know for how long the LSP is required before it 214 can be torn down. 216 The set of requests for future LSPs could be collected and held in a 217 central database (such as at a Network Management System - NMS): when 218 the time comes for each LSP to be set up the NMS can ask the PCE to 219 compute a path and can then request the LSP to be provisioned. This 220 approach has a number of drawbacks because it is not possible to 221 determine in advance whether it will be possible to deliver the LSP 222 since the resources it needs might be used by other LSPs in the 223 network. Thus, at the time the requester asks for the future LSP, 224 the NMS can only make a best-effort guarantee that the LSP will be 225 set up at the desired time. 227 A better solution, therefore, is for the requests for future LSPs to 228 be serviced at once. The paths of the LSPs can be computed ahead of 229 time and converted into reservations of network resources during 230 specific windows in the future. That is, while the path of the LSP 231 is computed and the network resources are reserved, the LSP is not 232 established in the network until the time for which it is scheduled. 234 There is a need to take into account items that need to be subject to 235 operator policy such as the amount of capacity available for 236 scheduling future reservations and the operator preference for the 237 measures which are used with respect to the use of scheduled 238 resources during rapid changes in traffic demand events or a complex 239 (multiple nodes/links) failure event so as to protect against network 240 destabilization. Operator policy is discussed further in 241 Section 3.3. 243 2.4. Looking at Future Demands on TE Resources 245 While path computation as described in Section 2.2 takes account of 246 the currently available network resources, and can act to place LSPs 247 in the network so that there is the best possibility of future LSPs 248 being accommodated, it cannot handle all eventualities. It is simple 249 to construct scenarios where LSPs that are placed one at a time lead 250 to future LSPs being blocked, but where foreknowledge of all of the 251 LSPs would have made it possible for them all to be set up. 253 If, therefore, we were able to know in advance what LSPs were going 254 to be requested, we could plan for them and ensure resources were 255 available. Furthermore, such an approach enables a commitment to be 256 made to a service user that an LSP will be set up and available at a 257 specific time. 259 A reservation service can be achieved by tracking the current use of 260 network resources and also having a future view of the resource 261 usage. We call this Time-Scheduled TE (TS-TE) resource reservation. 263 2.4.1. Interaction Between Time-Scheduled and Ad Hoc Reservations 265 There will, of course, be a mixture of resource uses in a network. 266 For example, normal unplanned LSPs may be requested alongside TS-TE 267 LSPs. When an unplanned LSP is requested no prior accommodation can 268 be made to arrange resource availability, so the LSP can be placed no 269 better than would be the case without TS-TE. However, the new LSP 270 can be placed considering the future demands of TS-TE LSPs that have 271 already been requested. Of course, the unplanned LSP has no known 272 end time and so any network planning must assume that it will consume 273 resources for ever. 275 2.5. Requisite State Information 277 In order to achieve the TS-TE resource reservation, the use of 278 resources on the path needs to be scheduled. Scheduling state is 279 used to indicate when resources are reserved and when they are 280 available for use. 282 A simple information model for one piece of scheduling state is as 283 follows: 285 { 286 link id; 287 resource id or reserved capacity; 288 reservation start time; 289 reservation end time 290 } 292 The resource that is scheduled could be link capacity, physical 293 resources on a link, buffers on an interfaces, etc., and could 294 include advanced considerations such as CPU utilization and the 295 availability of memory at nodes within the network. The resource- 296 related information might also include the maximal unreserved 297 bandwidth of the link over a time interval. That is, the intention 298 is to book (reserve) a percentage of the residual (unreserved) 299 bandwidth of the link. This could be used, for example, to reserve 300 bandwidth for a particular class of traffic (such as IP) that doesn't 301 have a provisioned LSP. 303 For any one resource there could be multiple pieces of scheduling 304 state, and for any one link, the timing windows might overlap. 306 There are multiple ways to realize this information model and 307 different ways to store the data. The resource state could be 308 expressed as a start time and an end time as shown above, or could be 309 expressed as a start time and a duration. Multiple reservation 310 periods, possibly of different lengths, may need to be recorded for 311 each resource. Furthermore, the current state of network reservation 312 could be kept separate from the scheduled usage, or everything could 313 be merged into a single TS database. 315 An application may make a reservation request for immediate resource 316 usage, or to book resources for future use so as to maximize the 317 chance of services being delivered and to avoid contention for 318 resources in the future. A single reservation request may book 319 resources for multiple periods and might request a reservation that 320 repeats on a regular cycle. 322 A computation engine (that is, a PCE) may use the scheduling state 323 information to help optimize the use of resources into the future and 324 reduce contention or blocking when the resources are actually needed. 326 Note that it is also necessary to store the information about future 327 LSPs as distinct from the specific resource scheduling. This 328 information is held to allow the LSPs to be instantiated when they 329 are due and using the paths/resources that have been computed for 330 them, but also to provide correlation with the TS-TE resource 331 reservations so that it is clear why resources were reserved, 332 allowing pre-emption, and handling release of reserved resources in 333 the event of cancellation of future LSPs. See Section 3.2 for 334 further discussion of the distinction between scheduled resource 335 state and scheduled LSP state. 337 Network performance factors (such as maximum link utilization and the 338 residual capacity of the network) with respect to supporting 339 scheduled reservations need to be supported and are subject to 340 operator policy. 342 3. Architectural Concepts 344 This section examines several important architectural concepts to 345 understand the design decisions reached in this document to achieve 346 TS-TE in a scalable and robust manner. 348 3.1. Where is Scheduling State Held? 350 The scheduling state information described in Section 2.5 has to be 351 held somewhere. There are two places where this makes sense: 353 o In the network nodes where the resources exist; 355 o In a central scheduling controller where decisions about resource 356 allocation are made. 358 The first of these makes policing of resource allocation easier. It 359 means that many points in the network can request immediate or 360 scheduled LSPs with the associated resource reservation and that all 361 such requests can be correlated at the point where the resources are 362 allocated. However, this approach has some scaling and technical 363 problems: 365 o The most obvious issue is that each network node must retain the 366 full time-based state for all of its resources. In a busy network 367 with a high arrival rate of new LSPs and a low hold time for each 368 LSP, this could be a lot of state. Network nodes are normally 369 implemented with minimal spare memory. 371 o In order that path computation can be performed, the computing 372 entity normally known as a Path Computation Element (PCE) 373 [RFC4655] needs access to a database of available links and nodes 374 in the network, and of the TE properties of the links. This 375 database is known as the Traffic Engineering Database (TED) and is 376 usually populated from information advertised in the IGP by each 377 of the network nodes or exported using BGP-LS [RFC7752]. To be 378 able to compute a path for a future LSP the PCE needs to populate 379 the TED with all of the future resource availability: if this 380 information is held on the network nodes it must also be 381 advertised in the IGP. This could be a significant scaling issue 382 for the IGP and the network nodes as all of the advertised 383 information is held at every network node and must be periodically 384 refreshed by the IGP. 386 o When a normal node restarts it can recover resource reservation 387 state from the forwarding hardware, from Non-Volatile Random- 388 Access Memory (NVRAM), or from adjacent nodes through the 389 signaling protocol [RFC5063]. If scheduling state is held at the 390 network nodes it must also be recovered after the restart of a 391 network node. This cannot be achieved from the forwarding 392 hardware because the reservation will not have been made, could 393 require additional expensive NVRAM, or might require that all 394 adjacent nodes also have the scheduling state in order to re- 395 install it on the restarting node. This is potentially complex 396 processing with scaling and cost implications. 398 Conversely, if the scheduling state is held centrally it is easily 399 available at the point of use. That is, the PCE can utilize the 400 state to plan future LSPs and can update that stored information with 401 the scheduled reservation of resources for those future LSPs. This 402 approach also has several issues: 404 o If there are multiple controllers then they must synchronize their 405 stored scheduling state as they each plan future LSPs, and must 406 have a mechanism to resolve resource contention. This is 407 relatively simple and is mitigated by the fact that there is ample 408 processing time to re-plan future LSPs in the case of resource 409 contention. 411 o If other sources of immediate LSPs are allowed (for example, other 412 controllers or autonomous action by head-end LSRs) then the 413 changes in resource availability caused by the setup or tear down 414 of these LSPs must be reflected in the TED (by use of the IGP as 415 currently) and may have an impact of planned future LSPs. This 416 impact can be mitigated by re-planning future LSPs or through LSP 417 preemption. 419 o If the scheduling state is held centrally at a PCE, the state must 420 be held and restored after a system restart. This is relatively 421 easy to achieve on a central server that can have access to non- 422 volatile storage. The PCE could also synchronize the scheduling 423 state with other PCEs after restart. See Section 4.2 for details. 425 o Of course, a centralized system must store information about all 426 of the resources in the network. In a busy network with a high 427 arrival rate of new LSPs and a low hold time for each LSP, this 428 could be a lot of state. This is multiplied by the size of the 429 network measured both by the number of links and nodes, and by the 430 number of trackable resources on each link or at each node. This 431 challenge may be mitigated by the centralized server being 432 dedicated hardware, but there remains the problem of collecting 433 the information from the network in a timely way when there is 434 potentially a very large amount of information to be collected and 435 when the rate of change of that information is high. This latter 436 challenge is only solved if the central server has full control of 437 the booking of resources and the establishment of new LSPs so that 438 the information from the network only serves to confirm what the 439 central server expected. 441 Thus, considering these tradeoffs, the architectural conclusion is 442 that scheduling state should be held centrally at the point of use 443 and not in the network devices. 445 3.2. What State is Held? 447 As already described, the PCE needs access to an enhanced, time-based 448 TED. It stores the traffic engineering (TE) information such as 449 bandwidth for every link for a series of time intervals. There are a 450 few ways to store the TE information in the TED. For example, 451 suppose that the amount of the unreserved bandwidth at a priority 452 level for a link is Bj in a time interval from time Tj to Tk (k = 453 j+1), where j = 0, 1, 2, .... 455 Bandwidth 456 ^ 457 | B3 458 | B1 ___________ 459 | __________ 460 |B0 B4 461 |__________ B2 _________ 462 | ________________ 463 | 464 -+-------------------------------------------------------> Time 465 |T0 T1 T2 T3 T4 467 Figure 1: A Plot of Bandwidth Usage against Time 469 The unreserved bandwidth for the link can be represented and stored 470 in the TED as [T0, B0], [T1, B1], [T2, B2], [T3, B3], ... as shown in 471 Figure 1. 473 But it must be noted that service requests for future LSPs are known 474 in terms of the LSPs whose paths are computed and for which resources 475 are scheduled. For example, if the requester of a future LSP decides 476 to cancel the request or to modify the request, the PCE must be able 477 to map this to the resources that were reserved. When the LSP or the 478 request for the LSP with a number of time intervals is cancelled, the 479 PCE must release the resources that were reserved on each of the 480 links along the path of the LSP in every time intervals from the TED. 481 If the bandwidth that had been reserved for the LSP on a link was B 482 from time T2 to T3 and the unreserved bandwidth on the link was B2 483 from T2 to T3, then B is added back to the link for the time interval 484 from T2 to T3 and the unreserved bandwidth on the link from T2 to T3 485 will be seen to be B2 + B. 487 This suggests that the PCE needs an LSP Database (LSP-DB) [RFC8231] 488 that contains information not only about LSPs that are active in the 489 network, but also those that are planned. The information for an LSP 490 stored in the LSP-DB includes for each time interval that applies to 491 the LSP: the time interval, the paths computed for the LSP satisfying 492 the constraints in the time interval, and the resources such as 493 bandwidth reserved for the LSP in the time interval. See also 494 Section 2.3 496 It is an implementation choice how the TED and LSP-DB are stored both 497 for dynamic use and for recovery after failure or restart, but it may 498 be noted that all of the information in the scheduled TED can be 499 recovered from the active network state and from the scheduled LSP- 500 DB. 502 3.3. Enforcement of Operator Policy 504 Computation requests for LSPs are serviced according to operator 505 policy. For example, a PCE may refuse a computation request because 506 the application making the request does not have sufficient 507 permissions, or because servicing the request might take specific 508 resource usage over a given threshold. 510 Furthermore, the pre-emption and holding priorities of any particular 511 computation request may be subject to the operator's policies. The 512 request could be rejected if it does not conform to the operator's 513 policies, or (possibly more likely) the priorities could be set/ 514 overwritten according to the operator's policies. 516 Additionally, the objective functions (OFs) of computation request 517 (such as maximizing residual bandwidth) are also subject to operator 518 policies. It is highly likely that the choice of OFs is not 519 available to an application and is selected by the PCE or management 520 system subject to operator policies and knowledge of the application. 522 None of these statements is new to scheduled resources. They apply 523 to stateless, stateful, passive, and active PCEs, and they continue 524 to apply to scheduling of resources. 526 An operator may choose to configure special behavior for a PCE that 527 handles resource scheduling. For example, an operator might want 528 only a certain percentage of any resource to be bookable. And an 529 operator might want the pre-emption of booked resources to be an 530 inverse function of how far in the future the resources are needed 531 for the first time. 533 It is a general assumption about the architecture described in 534 Section 4 that a PCE is under the operational control of the operator 535 that owns the resources that the PCE manipulates. Thus the operator 536 may configure any amount of (potentially complex) policy at the PCE. 537 This configuration would also include policy points surrounding re- 538 optimization of existing and planned LSPs in the event of changes in 539 the current and future (planned) resource availability. 541 The granularity of the timing window offered to an application will 542 depend on an operator's policy as well as the implementation in the 543 PCE, and goes to define the operator' service offerings. Different 544 granularities and different lengths of pre-booking may be offered to 545 different applications. 547 4. Architecture Overview 549 The architectural considerations and conclusions described in the 550 previous section lead to the architecture described in this section 551 and illustrated in Figure 2. The interfaces and interactions shown 552 on the figure and labeled (a) through (f) are described in 553 Section 4.1. 555 ------------------- 556 | Service Requester | 557 ------------------- 558 ^ 559 a| 560 v 561 ------- b -------- 562 | |<--->| LSP-DB | 563 | | -------- 564 | PCE | 565 | | c ----- 566 | |<---->| TED | 567 ------- ----- 568 ^ ^ 569 | | 570 d| |e 571 | | 572 ------+-----+-------------------- 573 | | Network 574 | -------- 575 | | Router | 576 v -------- 577 ----- ----- 578 | LSR |<------>| LSR | 579 ----- f ----- 581 Figure 2: Reference Architecture for Scheduled Use of Resources 583 4.1. Service Request 585 As shown in Figure 2, some component in the network requests a 586 service. This may be an application, an NMS, an LSR, or any 587 component that qualifies as a Path Computation Client (PCC). We show 588 this on the figure as the "Service Requester" and it sends a request 589 to the PCE for an LSP to be set up at some time (either now or in the 590 future). The request, indicated on Figure 2 by the arrow (a), 591 includes all of the parameters of the LSP that the requester wishes 592 to supply such as priority, bandwidth, start time, and end time. 593 Note that the requester in this case may be the LSR shown in the 594 figure or may be a distinct system. 596 The PCE enters the LSP request in its LSP-DB (b), and uses 597 information from its TED (c) to compute a path that satisfies the 598 constraints (such as bandwidth) for the LSP in the time interval from 599 the start time to the end time. It updates the future resource 600 availability in the TED so that further path computations can take 601 account of the scheduled resource usage. It stores the path for the 602 LSP into the LSP-DB (b). 604 When it is time (i.e., at the start time) for the LSP to be set up, 605 the PCE sends a PCEP Initiate request to the head end LSR (d) 606 providing the path to be signaled as well as other parameters such as 607 the bandwidth of the LSP. 609 As the LSP is signaled between LSRs (f) the use of resources in the 610 network is updated and distributed using the IGP. This information 611 is shared with the PCE either through the IGP or using BGP-LS (e), 612 and the PCE updates the information stored in its TED (c). 614 After the LSP is set up, the head end LSR sends a PCEP LSP State 615 Report (PCRpt message) to the PCE (d). The report contains the 616 resources such as bandwidth usage for the LSP. The PCE updates the 617 status of the LSP in the LSP-DB according to the report. 619 When an LSP is no longer required (either because the Service 620 Requester has cancelled the request, or because the LSP's scheduled 621 lifetime has expired) the PCE can remove it. If the LSP is currently 622 active, the PCE instructs the head-end LSR to tear it down (d), and 623 the network resource usage will be updated by the IGP and advertised 624 back to the PCE through the IGP or BGP-LS (e). Once the LSP is no 625 longer active, the PCE can remove it from the LSP-DB (b). 627 4.1.1. Reoptimization After TED Updates 629 When the TED is updated as indicated in Section 4.1, the PCE may 630 perform reoptimization of the LSPs for which it has computed paths 631 depending on operator policy so as to minimize network perturbations. 632 These LSPs may be already provisioned in which case the PCE issues 633 PCEP Update request messages for the LSPs that should be adjusted. 634 Additionally, the LSPs being reoptimized may be scheduled LSPs that 635 have not yet been provisioned, in which case reoptimization involves 636 updating the store of scheduled LSPs and resources. 638 In all cases, the purpose of reoptimization is to take account of the 639 resource usage and availability in the network and to compute paths 640 for the current and future LSPs that best satisfy the objectives of 641 those LSPs while keeping the network as clear as possible to support 642 further LSPs. Since reoptimization may perturb established LSPs, it 643 is subject to operator oversight and policy. As the stability of the 644 network will be impacted by frequent changes, the extent and impact 645 of any reoptimization needs to be subject to operator policy. 647 Additionally, the status of the reserved resources (alarms) can 648 enhance the computation and planning for future LSPs, and may 649 influence repair and reoptimization. Control of recalculations based 650 on failures and notifications to the operator is also subject to 651 policy. 653 See Section 3.3 for further discussion of operator policy. 655 4.2. Initialization and Recovery 657 When a PCE in the architecture shown in Figure 2 is initialized, it 658 must learn state from the network, from its stored databases, and 659 potentially from other PCEs in the network. 661 The first step is to get an accurate view of the topology and 662 resource availability in the network. This would normally involve 663 reading the state direct from the network via the IGP or BGP-LS (e), 664 but might include receiving a copy of the TED from another PCE. Note 665 that a TED stored from a previous instantiation of the PCE is 666 unlikely to be valid. 668 Next, the PCE must construct a time-based TED to show scheduled 669 resource usage. How it does this is implementation specific and this 670 document does not dictate any particular mechanism: it may recover a 671 time-based TED previously saved to non-volatile storage, or it may 672 reconstruct the time-based TED from information retrieved from the 673 LSP-DB previously saved to non-volatile storage. If there is more 674 than one PCE active in the network, the recovering PCE will need to 675 synchronize the LSP-DB and time-based TED with other PCEs (see 676 Section 4.3). 678 Note that the stored LSP-DB needs to include the intended state and 679 actual state of the LSPs so that when a PCE recovers it is able to 680 determine what actions are necessary. 682 4.3. Synchronization Between PCEs 684 If there is more than one PCE that supports scheduling active in the 685 network, it is important to achieve some consistency between the 686 scheduled TED and scheduled LSP-DB held by the PCEs. 688 [RFC7399] answers various questions around synchronization between 689 the PCEs. It should be noted that the time-based "scheduled" 690 information adds another dimension to the issue of synchronization 691 between PCEs. It should also be noted that a deployment may use a 692 primary PCE and the have other PCEs as backup, where a backup PCE can 693 take over only in the event of a failure of the primary PCE. 694 Alternatively, the PCEs may share the load at all times. The choice 695 of the synchronization technique is largely dependent on the 696 deployment of PCEs in the network. 698 One option for ensuring that multiple PCEs use the same scheduled 699 information is simply to have the PCEs driven from the same shared 700 database, but it is likely to be inefficient and interoperation 701 between multiple implementations will be harder. 703 Another option is for each PCE to be responsible for its own 704 scheduled database and to utilize some distributed database 705 synchronization mechanism to have consistent information. Depending 706 on the implementation, this could be efficient, but interoperation 707 between heterogeneous implementations is still hard. 709 A further approach is to utilize PCEP messages to synchronize the 710 scheduled state between PCEs. This approach would work well if the 711 number of PCEs which support scheduling is small, but as the number 712 increases considerable message exchange needs to happen to keep the 713 scheduled databases synchronized. Future solutions could also 714 utilize some synchronization optimization techniques for efficiency. 715 Another variation would be to request information from other PCEs for 716 a particular time slice, but this might have impact on the 717 optimization algorithm. 719 5. Multi-Domain Considerations 721 Multi-domain path computation usually requires some form of 722 cooperation between PCEs each of which has responsibility for 723 determining a segment of the end-to-end path in the domain for which 724 it has computational responsibility. When computing a scheduled 725 path, resources need to be booked in all of the domains that the path 726 will cross so that they are available when the LSP is finally 727 signalled. 729 Per-domain path computation [RFC5152] is not an appropriate mechanism 730 when a scheduled LSP is being computed because the computation 731 requests at downstream PCEs are only triggered by signaling. 732 However, a similar mechanism could be used where cooperating PCEs 733 exchange PCReq messages for a scheduled LSP as shown in Figure 3. In 734 this case the service requester asks for a scheduled LSP that will 735 span two domains (a). PCE1 computes a path across Domain 1 and 736 reserves the resources, and also asks PCE2 to compute and reserve in 737 Domain 2 (b). PCE2 may return a full path, or could return a path 738 key [RFC5520]. When it is time for LSP setup PCE1 triggers the head- 739 end LSR (c) and the LSP is signaled (d). If a path key is used, the 740 entry LSR in Domain 2 will consult PCE2 for the path expansion (e) 741 before completing signaling (f). 743 ------------------- 744 | Service Requester | 745 ------------------- 746 ^ 747 a| 748 v 749 ------ b ------ 750 | |<---------------->| | 751 | PCE1 | | PCE2 | 752 | | | | 753 ------ ------ 754 ^ ^ 755 | | 756 c| e| 757 | | 758 ----+----------------- ----+----------------- 759 | | Domain 1 | | | Domain 2 | 760 | v | | v | 761 | ----- d ----- | | ----- f ----- | 762 | | LSR |<--->| LSR |<-+--+->| LSR |<--->| LSR | | 763 | ----- ----- | | ----- ----- | 764 ---------------------- ---------------------- 766 Figure 3: Per-Domain Path Computation for Scheduled LSPs 768 Another mechanism for PCE cooperation in multi-domain LSP setup is 769 Backward- Recursive Path Computation (BRPC) [RFC5441]. This approach 770 relies on the downstream domain supply a variety of potential paths 771 to the upstream domain. Although BRPC can arrive at a more optimal 772 end-to-end path than per-domain path computation, it is not well 773 suited to LSP scheduling because the downstream PCE would need to 774 reserve resources on all of the potential paths and then release 775 those that the upstream PCE announced it did not plan to use. 777 Finally we should consider hierarchical PCE (H-PCE) [RFC6805]. This 778 mode of operation is similar to that shown in Figure 3, but a parent 779 PCE is used to coordinate the requests to the child PCEs resulting in 780 better visibility of the end-to-end path and better coordination of 781 the resource booking. The sequenced flow of control is shown in 782 Figure 4. 784 ------------------- 785 | Service Requester | 786 ------------------- 787 ^ 788 a| 789 v 790 -------- 791 | | 792 | Parent | 793 | PCE | 794 | | 795 -------- 796 ^ ^ b 797 b| |_______________________ 798 | | 799 v v 800 ------ ------ 801 | | | | 802 | PCE1 | | PCE2 | 803 | | | | 804 ------ ------ 805 ^ ^ 806 | | 807 c| e| 808 | | 809 ----+----------------- ----+----------------- 810 | | Domain 1 | | | Domain 2 | 811 | v | | v | 812 | ----- d ----- | | ----- f ----- | 813 | | LSR |<--->| LSR |<-+--+->| LSR |<--->| LSR | | 814 | ----- ----- | | ----- ----- | 815 ---------------------- ---------------------- 817 Figure 4: Hierarchical PCE for Path Computation for Scheduled LSPs 819 6. Security Considerations 821 The protocol implications of scheduled resources are unchanged from 822 "on-demand" LSP computation and setup. A discussion of securing PCEP 823 is found in [RFC5440] and work to extend that security is provided in 824 [RFC8253]. Furthermore, the path key mechanism described in 825 [RFC5520] can be used to enhance privacy and security. 827 Similarly, there is no change to the security implications for the 828 signaling of scheduled LSPs. A discussion of the security of the 829 signaling protocols that would be used is found in [RFC5920]. 831 However, the use of scheduled LSPs extends the attack surface for a 832 PCE-enabled TE system by providing a larger (logically infinite) 833 window during which an attack can be initiated or planned. That is, 834 if bogus scheduled LSPs can be requested and entered into the LSP-DB, 835 then a large number of LSPs could be launched, and significant 836 network resources could be blocked. Control of scheduling requests 837 needs to be subject to operator policy and additional authorization 838 needs to be applied for access to LSP scheduling. Diagnostic tools 839 need to be provided to inspect the LSP DB to spot attacks. 841 7. IANA Considerations 843 This architecture document makes no request for IANA action. 845 8. Acknowledgements 847 This work has benefited from the discussions of resource scheduling 848 over the years. In particular the DRAGON project [DRAGON] and 849 [I-D.yong-ccamp-ason-gmpls-autobw-service] both of which provide 850 approaches to auto-bandwidth services in GMPLS networks. 852 Mehmet Toy, Lei Liu, and Khuzema Pithewan contributed to an earlier 853 version of [I-D.chen-teas-frmwk-tts]. We would like to thank the 854 authors of that draft on Temporal Tunnel Services for material that 855 assisted in thinking about this document. 857 Thanks to Michael Scharf and Daniele Ceccarelli for useful comments 858 on this work. 860 Jonathan Hardwick provided a helpful Routing Directorate review. 862 Deborah Brungard, Mirja Kuehlewind, and Benjamin Kaduk suggested many 863 changes during their Area Director reviews. 865 9. Contributors 867 The following people contributed to discussions that led to the 868 development of this document: 870 Dhruv Dhody 871 Email: dhruv.dhody@huawei.com 873 10. Informative References 875 [DRAGON] National Science Foundation, "http://www.maxgigapop.net/ 876 wp-content/uploads/The-DRAGON-Project.pdf". 878 [I-D.chen-teas-frmwk-tts] 879 Chen, H., Toy, M., Liu, L., and K. Pithewan, "Framework 880 for Temporal Tunnel Services", draft-chen-teas-frmwk- 881 tts-01 (work in progress), March 2016. 883 [I-D.yong-ccamp-ason-gmpls-autobw-service] 884 Yong, L. and Y. Lee, "ASON/GMPLS Extension for Reservation 885 and Time Based Automatic Bandwidth Service", draft-yong- 886 ccamp-ason-gmpls-autobw-service-00 (work in progress), 887 October 2006. 889 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 890 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 891 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 892 . 894 [RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label 895 Switching (GMPLS) Signaling Resource ReserVation Protocol- 896 Traffic Engineering (RSVP-TE) Extensions", RFC 3473, 897 DOI 10.17487/RFC3473, January 2003, 898 . 900 [RFC3945] Mannie, E., Ed., "Generalized Multi-Protocol Label 901 Switching (GMPLS) Architecture", RFC 3945, 902 DOI 10.17487/RFC3945, October 2004, 903 . 905 [RFC4655] Farrel, A., Vasseur, J., and J. Ash, "A Path Computation 906 Element (PCE)-Based Architecture", RFC 4655, 907 DOI 10.17487/RFC4655, August 2006, 908 . 910 [RFC5063] Satyanarayana, A., Ed. and R. Rahman, Ed., "Extensions to 911 GMPLS Resource Reservation Protocol (RSVP) Graceful 912 Restart", RFC 5063, DOI 10.17487/RFC5063, October 2007, 913 . 915 [RFC5152] Vasseur, JP., Ed., Ayyangar, A., Ed., and R. Zhang, "A 916 Per-Domain Path Computation Method for Establishing Inter- 917 Domain Traffic Engineering (TE) Label Switched Paths 918 (LSPs)", RFC 5152, DOI 10.17487/RFC5152, February 2008, 919 . 921 [RFC5440] Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation 922 Element (PCE) Communication Protocol (PCEP)", RFC 5440, 923 DOI 10.17487/RFC5440, March 2009, 924 . 926 [RFC5441] Vasseur, JP., Ed., Zhang, R., Bitar, N., and JL. Le Roux, 927 "A Backward-Recursive PCE-Based Computation (BRPC) 928 Procedure to Compute Shortest Constrained Inter-Domain 929 Traffic Engineering Label Switched Paths", RFC 5441, 930 DOI 10.17487/RFC5441, April 2009, 931 . 933 [RFC5520] Bradford, R., Ed., Vasseur, JP., and A. Farrel, 934 "Preserving Topology Confidentiality in Inter-Domain Path 935 Computation Using a Path-Key-Based Mechanism", RFC 5520, 936 DOI 10.17487/RFC5520, April 2009, 937 . 939 [RFC5920] Fang, L., Ed., "Security Framework for MPLS and GMPLS 940 Networks", RFC 5920, DOI 10.17487/RFC5920, July 2010, 941 . 943 [RFC6805] King, D., Ed. and A. Farrel, Ed., "The Application of the 944 Path Computation Element Architecture to the Determination 945 of a Sequence of Domains in MPLS and GMPLS", RFC 6805, 946 DOI 10.17487/RFC6805, November 2012, 947 . 949 [RFC7399] Farrel, A. and D. King, "Unanswered Questions in the Path 950 Computation Element Architecture", RFC 7399, 951 DOI 10.17487/RFC7399, October 2014, 952 . 954 [RFC7752] Gredler, H., Ed., Medved, J., Previdi, S., Farrel, A., and 955 S. Ray, "North-Bound Distribution of Link-State and 956 Traffic Engineering (TE) Information Using BGP", RFC 7752, 957 DOI 10.17487/RFC7752, March 2016, 958 . 960 [RFC8231] Crabbe, E., Minei, I., Medved, J., and R. Varga, "Path 961 Computation Element Communication Protocol (PCEP) 962 Extensions for Stateful PCE", RFC 8231, 963 DOI 10.17487/RFC8231, September 2017, 964 . 966 [RFC8253] Lopez, D., Gonzalez de Dios, O., Wu, Q., and D. Dhody, 967 "PCEPS: Usage of TLS to Provide a Secure Transport for the 968 Path Computation Element Communication Protocol (PCEP)", 969 RFC 8253, DOI 10.17487/RFC8253, October 2017, 970 . 972 Authors' Addresses 974 Yan Zhuang 975 Huawei 976 101 Software Avenue, Yuhua District 977 Nanjing, Jiangsu 210012 978 China 980 Email: zhuangyan.zhuang@huawei.com 982 Qin Wu 983 Huawei 984 101 Software Avenue, Yuhua District 985 Nanjing, Jiangsu 210012 986 China 988 Email: bill.wu@huawei.com 990 Huaimo Chen 991 Huawei 992 Boston, MA 993 US 995 Email: huaimo.chen@huawei.com 997 Adrian Farrel 998 Juniper Networks 1000 Email: afarrel@juniper.net