idnits 2.17.1 draft-so-yong-rtgwg-cl-framework-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 7, 2012) is 4433 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-06) exists of draft-ietf-mpls-entropy-label-01 == Outdated reference: A later version (-09) exists of draft-ietf-mpls-tp-security-framework-02 == Outdated reference: A later version (-16) exists of draft-ietf-rtgwg-cl-requirement-05 == Outdated reference: A later version (-06) exists of draft-kompella-mpls-rsvp-ecmp-01 == Outdated reference: A later version (-01) exists of draft-symmvo-rtgwg-cl-use-cases-00 == Outdated reference: A later version (-03) exists of draft-villamizar-mpls-tp-multipath-01 == Outdated reference: A later version (-02) exists of draft-villamizar-mpls-tp-multipath-te-extn-00 -- Obsolete informational reference (is this intentional?): RFC 5316 (Obsoleted by RFC 9346) Summary: 1 error (**), 0 flaws (~~), 8 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 RTGWG N. So 3 Internet-Draft D. McDysan 4 Intended status: Informational Verizon 5 Expires: September 8, 2012 E. Osborne 6 Cisco 7 L. Yong 8 Huawei USA 9 C. Villamizar 10 Outer Cape Cod Network 11 Consulting 12 March 7, 2012 14 Composite Link Framework in Multi Protocol Label Switching (MPLS) 15 draft-so-yong-rtgwg-cl-framework-05 17 Abstract 19 This document specifies a framework for support of composite link in 20 MPLS networks. A composite link consists of a group of homogenous or 21 non-homogenous links that have the same forward adjacency and can be 22 considered as a single TE link or an IP link in routing. A composite 23 link relies on its component links to carry the traffic over the 24 composite link. Applicability is described for a single pair of 25 MPLS-capable nodes, a sequence of MPLS-capable nodes, or a set of 26 layer networks connecting MPLS-capable nodes. 28 Status of this Memo 30 This Internet-Draft is submitted in full conformance with the 31 provisions of BCP 78 and BCP 79. 33 Internet-Drafts are working documents of the Internet Engineering 34 Task Force (IETF). Note that other groups may also distribute 35 working documents as Internet-Drafts. The list of current Internet- 36 Drafts is at http://datatracker.ietf.org/drafts/current/. 38 Internet-Drafts are draft documents valid for a maximum of six months 39 and may be updated, replaced, or obsoleted by other documents at any 40 time. It is inappropriate to use Internet-Drafts as reference 41 material or to cite them other than as "work in progress." 43 This Internet-Draft will expire on September 8, 2012. 45 Copyright Notice 47 Copyright (c) 2012 IETF Trust and the persons identified as the 48 document authors. All rights reserved. 50 This document is subject to BCP 78 and the IETF Trust's Legal 51 Provisions Relating to IETF Documents 52 (http://trustee.ietf.org/license-info) in effect on the date of 53 publication of this document. Please review these documents 54 carefully, as they describe your rights and restrictions with respect 55 to this document. Code Components extracted from this document must 56 include Simplified BSD License text as described in Section 4.e of 57 the Trust Legal Provisions and are provided without warranty as 58 described in the Simplified BSD License. 60 Table of Contents 62 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 63 1.1. Architecture Summary . . . . . . . . . . . . . . . . . . . 4 64 1.2. Conventions used in this document . . . . . . . . . . . . 5 65 1.2.1. Terminology . . . . . . . . . . . . . . . . . . . . . 5 66 2. Composite Link Key Characteristics . . . . . . . . . . . . . . 5 67 2.1. Flow Identification . . . . . . . . . . . . . . . . . . . 6 68 2.2. Composite Link in Control Plane . . . . . . . . . . . . . 7 69 2.3. Composite Link in Data Plane . . . . . . . . . . . . . . . 10 70 3. Architecture Tradeoffs . . . . . . . . . . . . . . . . . . . . 11 71 3.1. Scalability Motivations . . . . . . . . . . . . . . . . . 11 72 3.2. Reducing Routing Information and Exchange . . . . . . . . 11 73 3.3. Reducing Signaling Load . . . . . . . . . . . . . . . . . 12 74 3.3.1. Reducing Signaling Load using LDP . . . . . . . . . . 12 75 3.3.2. Reducing Signaling Load using Hierarchy . . . . . . . 13 76 3.3.3. Using Both LDP and RSVP-TE Hierarchy . . . . . . . . . 13 77 3.4. Reducing Forwarding State . . . . . . . . . . . . . . . . 13 78 3.5. Avoiding Route Oscillation . . . . . . . . . . . . . . . . 14 79 4. New Challenges . . . . . . . . . . . . . . . . . . . . . . . . 14 80 4.1. Control Plane Challenges . . . . . . . . . . . . . . . . . 15 81 4.1.1. Delay and Jitter Sensitive Routing . . . . . . . . . . 15 82 4.1.2. Local Control of Traffic Distribution . . . . . . . . 16 83 4.1.3. Path Symmetry Requirements . . . . . . . . . . . . . . 16 84 4.1.4. Requirements for Contained LSP . . . . . . . . . . . . 17 85 4.1.5. Retaining Backwards Compatibility . . . . . . . . . . 17 86 4.2. Data Plane Challenges . . . . . . . . . . . . . . . . . . 18 87 4.2.1. Very Large LSP . . . . . . . . . . . . . . . . . . . . 19 88 4.2.2. Very Large Microflows . . . . . . . . . . . . . . . . 19 89 4.2.3. Traffic Ordering Constraints . . . . . . . . . . . . . 20 90 4.2.4. Accounting for IP and LDP Traffic . . . . . . . . . . 20 91 5. Existing Mechanisms . . . . . . . . . . . . . . . . . . . . . 21 92 5.1. Link Bundling . . . . . . . . . . . . . . . . . . . . . . 21 93 5.2. Classic Multipath . . . . . . . . . . . . . . . . . . . . 23 94 6. Mechanisms Proposed in Other Documents . . . . . . . . . . . . 23 95 6.1. Loss and Delay Measurement . . . . . . . . . . . . . . . . 23 96 6.2. Link Bundle Extensions . . . . . . . . . . . . . . . . . . 24 97 6.3. Fat PW and Entropy Labels . . . . . . . . . . . . . . . . 25 98 6.4. Multipath Extensions . . . . . . . . . . . . . . . . . . . 25 99 7. Required Protocol Extensions and Mechanisms . . . . . . . . . 26 100 7.1. Brief Review of Requirements . . . . . . . . . . . . . . . 26 101 7.2. Required Document Coverage . . . . . . . . . . . . . . . . 27 102 7.2.1. Component Link Grouping . . . . . . . . . . . . . . . 28 103 7.2.2. Component Group Metric . . . . . . . . . . . . . . . . 28 104 7.2.3. Delay and Jitter Extensions . . . . . . . . . . . . . 28 105 7.2.4. Path Selection and Admission Control . . . . . . . . . 28 106 7.2.5. Dynamic Multipath Balance . . . . . . . . . . . . . . 28 107 7.2.6. Frequency of Load Balance . . . . . . . . . . . . . . 29 108 7.2.7. Inter-Layer Communication . . . . . . . . . . . . . . 29 109 7.2.8. Packet Ordering Requirements . . . . . . . . . . . . . 29 110 7.2.9. Minimally Disruption Load Balance . . . . . . . . . . 29 111 7.2.10. Path Symmetry . . . . . . . . . . . . . . . . . . . . 29 112 7.2.11. Performance, Scalability, and Stability . . . . . . . 30 113 7.2.12. IP and LDP Traffic . . . . . . . . . . . . . . . . . . 30 114 7.2.13. LDP Extensions . . . . . . . . . . . . . . . . . . . . 30 115 7.2.14. Pseudowire Extensions . . . . . . . . . . . . . . . . 30 116 7.2.15. Multi-Domain Composite Link . . . . . . . . . . . . . 30 117 7.3. Open Issues Regarding Requirements . . . . . . . . . . . . 31 118 7.4. Framework Requirement Coverage by Protocol . . . . . . . . 31 119 7.4.1. OSPF-TE and ISIS-TE Protocol Extensions . . . . . . . 32 120 7.4.2. PW Protocol Extensions . . . . . . . . . . . . . . . . 32 121 7.4.3. LDP Protocol Extensions . . . . . . . . . . . . . . . 32 122 7.4.4. RSVP-TE Protocol Extensions . . . . . . . . . . . . . 32 123 7.4.5. RSVP-TE Path Selection Changes . . . . . . . . . . . . 32 124 7.4.6. RSVP-TE Admission Control and Preemption . . . . . . . 32 125 7.4.7. Flow Identification and Traffic Balance . . . . . . . 32 126 8. Security Considerations . . . . . . . . . . . . . . . . . . . 33 127 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 33 128 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 33 129 10.1. Normative References . . . . . . . . . . . . . . . . . . . 33 130 10.2. Informative References . . . . . . . . . . . . . . . . . . 34 131 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 37 133 1. Introduction 135 Composite Link functional requirements are specified in 136 [I-D.ietf-rtgwg-cl-requirement]. Composite Link use cases are 137 described in [I-D.symmvo-rtgwg-cl-use-cases]. This document 138 specifies a framework to meet these requirements. 140 Classic multipath, including Ethernet Link Aggregation has been 141 widely used in today's MPLS networks [RFC4385][RFC4928]. Classic 142 multipath using non-Ethernet links are often advertised using MPLS 143 Link bundling. A link bundle [RFC4201] bundles a group of 144 homogeneous links as a TE link to make IGP-TE information exchange 145 and RSVP-TE signaling more scalable. A composite link allows 146 bundling non-homogenous links together as a single logical link. The 147 motivations for using a composite link are descried in 148 [I-D.ietf-rtgwg-cl-requirement] and [I-D.symmvo-rtgwg-cl-use-cases]. 150 This document describes a composite link framework in the context of 151 MPLS networks using an IGP-TE and RSVP-TE MPLS control plane with 152 GMPLS extensions [RFC3209][RFC3630][RFC3945][RFC5305]. 154 A composite link is a single logical link in MPLS network that 155 contains multiple parallel component links between two MPLS LSR. 156 Unlike a link bundle [RFC4201], the component links in a composite 157 link can have different properties such as cost or capacity. 159 Specific protocol solutions are outside the scope of this document, 160 however a framework for the extension of existing protocols is 161 provided. Backwards compatibility is best achieved by extending 162 existing protocols where practical rather than inventing new 163 protocols. The focus is on examining where existing protocol 164 mechanisms fall short with respect to [I-D.ietf-rtgwg-cl-requirement] 165 and on extensions that will be required to accommodate functionality 166 that is called for in [I-D.ietf-rtgwg-cl-requirement]. 168 1.1. Architecture Summary 170 Networks aggregate information, both in the control plane and in the 171 data plane, as a means to achieve scalability. A tradeoff exists 172 between the needs of scalability and the needs to identify differing 173 path and link characteristics and differing requirements among flows 174 contained within further aggregated traffic flows. These tradeoffs 175 are discussed in detail in Section 3. 177 Some aspects of Composite Link requirements present challenges for 178 which multiple solutions may exist. In Section 4 various challenges 179 and potential approaches are discussed. 181 A subset of the functionality called for in 182 [I-D.ietf-rtgwg-cl-requirement] is available through MPLS Link 183 Bundling [RFC4201]. Link bundling and other existing standards 184 applicable to Composite Link are covered in Section 5. 186 The most straightforward means of supporting Composite Link 187 requirements is to extend MPLS protocols and protocol semantics and 188 in particular to extend link bundling. Extensions which have already 189 been proposed in other documents which are applicable to Composite 190 Link are discussed in Section 6. 192 Goals of most new protocol work within IETF is to reuse existing 193 protocol encapsulations and mechanisms where they meet requirements 194 and extend existing mechanisms such that additional complexity is 195 minimized while meeting requirements and such that backwards 196 compatibility is preserved to the extent it is practical to do so. 197 These goals are considered in proposing a framework for further 198 protocol extensions and mechanisms in Section 7. 200 1.2. Conventions used in this document 202 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 203 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 204 document are to be interpreted as described in RFC 2119 [RFC2119]. 206 1.2.1. Terminology 208 Terminology defined in [I-D.ietf-rtgwg-cl-requirement] is used in 209 this document. 211 The abbreviation IGP-TE is used as a shorthand indicating either 212 OSPF-TE [RFC3630] or ISIS-TE [RFC5305]. 214 2. Composite Link Key Characteristics 216 [I-D.ietf-rtgwg-cl-requirement] defines external behavior of 217 Composite Links. The overall framework approach involves extending 218 existing protocols in a backwards compatible manner and reusing 219 ongoing work elsewhere in IETF where applicable, defining new 220 protocols or semantics only where necessary. Given the requirements, 221 and this approach of extending MPLS, Composite Link key 222 characteristics can be described in greater detail than given 223 requirements alone. 225 2.1. Flow Identification 227 Traffic mapping to component links is a data plane operation. 228 Control over how the mapping is done may be directly dictated or 229 constrained by the control plane or by the management plane. When 230 unconstrained by the control plane or management plane, distribution 231 of traffic is entirely a local matter. Regardless of constraints or 232 lack or constraints, the traffic distribution is required to keep 233 packets belonging to individual flows in sequence and meet QoS 234 criteria specified per LSP by either signaling or management 235 [RFC2475][RFC3260]. A key objective of the traffic distribution is 236 to not overload any component link, and be able to perform local 237 recovery when one of component link fails. 239 Operator may have other objectives such as placing a bidirectional 240 flow or LSP on the same component link in both direction, load 241 balance over component links, composite link energy saving, and etc. 242 These new requirements are described in 243 [I-D.ietf-rtgwg-cl-requirement]. 245 Examples of means to identify a flow may in principle include: 247 1. an LSP identified by an MPLS label, 249 2. a sub-LSP [I-D.kompella-mpls-rsvp-ecmp] identified by an MPLS 250 label, 252 3. a pseudowire (PW) [RFC3985] identified by an MPLS label, 254 4. a flow or group of flows within a pseudowire (PW) [RFC6391] 255 identified by an MPLS label, 257 5. an entropy flow in an LSP [I-D.ietf-mpls-entropy-label] 258 identified by an MPLS label, 260 6. all traffic between a pair of IP hosts, identified by an IP 261 source and destination pair, 263 7. a specific connection between a pair of IP hosts, identified by 264 an IP source and destination pair, protocol, and protocol port 265 pair, 267 8. a layer-2 conversation within a pseudowire (PW), where the 268 identification is PW payload type specific, such as Ethernet MAC 269 addresses and VLAN tags within an Ethernet PW (RFC4448). 271 Although in principle a layer-2 conversation within a pseudowire 272 (PW), may be identified by PW payload type specific information, in 273 practice this is impractical at LSP midpoints when PW are carried. 274 The PW ingress may provide equivalent information in a PW flow label 275 [RFC6391]. Therefore, in practice, item #8 above is covered by 276 [RFC6391] and may be dropped from the list. 278 An LSR must at least be capable of identifying flows based on MPLS 279 labels. Most MPLS LSP do not require that traffic carried by the LSP 280 are carried in order. MPLS-TP is a recent exception. If it is 281 assumed that no LSP require strict packet ordering of the LSP itself 282 (only of flows within the LSP), then the entire label stack can be 283 used as flow identification. If some LSP may require strict packet 284 ordering but those LSP cannot be distinguished from others, then only 285 the top label can be used as a flow identifier. If only the top 286 label is used, then there may not be adequate flow granularity to 287 accomplish well balanced traffic distribution and it will not be 288 possible to carry LSP that are larger than any individual component 289 link. 291 The number of flows can be extremely large. This may be the case 292 when the entire label stack is used and is always the case when IP 293 addresses are used in provider networks carrying Internet traffic. 294 Current practice at the time of writing were documented in [RFC2991] 295 and [RFC2992]. These practices as described, make use of IP 296 addresses. The common practices were extended to include the MPLS 297 label stack and the common practice of looking at IP addresses within 298 the MPLS payload. These extended practices are described in 299 [RFC4385] and [RFC4928] due to their impact on pseudowires without a 300 PWE3 Control Word. 302 Using only the top label supports too course a traffic balance. 303 Using the full label stack or IP addresses as flow identification 304 provides a sufficiently fine traffic balance, but is capable of 305 identifying such a high number of distinct flows, that a technique of 306 grouping flows, such as hashing on the flow identification criteria, 307 becomes essential to reduce the stored state, and is an essential 308 scaling technique. Other means of grouping flows may be possible. 310 2.2. Composite Link in Control Plane 312 A composite Link is advertised as a single logical interface between 313 two connected routers, which forms forwarding adjacency (FA) between 314 the routers. The FA is advertised as a TE-link in a link state IGP, 315 using either OSPF-TE or ISIS-TE. The IGP-TE advertised interface 316 parameters for the composite link can be preconfigured by the network 317 operator or be derived from its component links. Composite link 318 advertisement requirements are specified in 319 [I-D.ietf-rtgwg-cl-requirement]. 321 In IGP-TE, a composite link is advertised as a single TE link between 322 two connected routers. This is similar to a link bundle [RFC4201]. 323 Link bundle applies to a set of homogenous component links. 324 Composite link allows homogenous and non-homogenous component links. 325 Due to the similarity, and for backwards compatibility, extending 326 link bundling is viewed as both simple and as the best approach. 328 In order for a route computation engine to calculate a proper path 329 for a LSP, it is necessary for composite link to advertise the 330 summarized available bandwidth as well as the maximum bandwidth that 331 can be made available for single flow (or single LSP where no finer 332 flow identification is available). If a composite link contains some 333 non-homogeneous component links, the composite link also should 334 advertise the summarized bandwidth and the maximum bandwidth for 335 single flow per each homogeneous component link group. 337 Both LDP [RFC5036] and RSVP-TE [RFC3209] can be used to signal a LSP 338 over a composite link. LDP cannot be extended to support traffic 339 engineering capabilities [RFC3468]. 341 When an LSP is signaled using RSVP-TE, the LSP MUST be placed on the 342 component link that meets the LSP criteria indicated in the signaling 343 message. 345 When an LSP is signaled using LDP, the LSP MUST be placed on the 346 component link that meets the LSP criteria, if such a component link 347 is available. LDP follows the IGP, therefore failure to forward on 348 the IGP path will often result in loss of connectivity if the IGP 349 adjacency is not withdrawn when an LDP FEC is refused. This is a 350 pathologic case that can occur if LDP is carried natively and there 351 is a high volume of LDP traffic. This situation can be avoided by 352 carrying LDP within RSVP-TE LSP. 354 LDP traffic engineering is specifically disallowed by [RFC3468]. It 355 may be possible to support multi-topology IGP extensions to 356 accommodate more than one set of criteria. If so, the additional IGP 357 could be bound to the forwarding criteria, and the LDP FEC bound to a 358 specific IGP instance, inheriting the forwarding criteria. {Note: WG 359 needs to discuss this.] 361 A composite link may contain non-homogeneous component links. The 362 route computing engine may select one group of component links for a 363 LSP. The routing protocol MUST make this grouping available in the 364 TE-LSDB. The route computation MUST be extended to include only the 365 capacity of groups within a composite link which meet LSP criteria. 366 The signaling protocol MUST be able to indicate either the criteria, 367 or which groups may be used. A composite link MUST place the LSP on 368 a component link or group which meets or exceeds the LSP criteria. 370 Composite link capacity is aggregated capacity and MAY be larger than 371 individual component link capacity. Any aggregated LSP can determine 372 a bounds on the largest microflow that could be carried and this 373 constraint can be handled as follows. 375 1. If no other information is available the largest microflow is 376 bound by one of the following: 378 A. the largest single LSP if most traffic is RSVP-TE signaled 379 and further aggregated, 381 B. the largest pseudowire if most traffic is carrying pseudowire 382 payloads that are aggregated within RSVP-TE LSP, 384 C. or the largest source and sink interface if a large amount of 385 IP or LDP traffic is contained within the aggregate. 387 If a very large amount of traffic being aggregated is IP or LDP, 388 then the largest microflow is bound by the largest component link 389 on which IP traffic can arrive. For example, if an LSR is acting 390 as an LER and IP and LDP traffic is arrving on 10 Gb/s edge 391 interfaces, then no microflow larger than 10 Gb/s will be present 392 on the RSVP-TE LSP that aggregate traffic across the core, even 393 if the core interfaces are 100 Gb/s interfaces. 395 2. The prior conditions provide a bound on the largest microflow 396 when no signaling extensions indicate a bounds. If an LSP is 397 aggregating smaller LSP for which the largest expected microflow 398 carried by the smaller LSP is signaled, then the largest 399 microflow expected in the containing LSP (the aggregate) is the 400 maximum of the largest expected microflow for any contained LSP. 401 For example, RSVP-TE LSP may be large but aggregate traffic for 402 which the source or sink are all 1 Gb/s or smaller interfaces 403 (such as in mobile applications in which cell sites backhauls are 404 no larger than 1 Gb/s). If this information is carried in the 405 LSP originated at the cell sites, then further aggregates across 406 a core may make use of this information. 408 3. The IGP must provide the bounds on the largest microflow that a 409 composite link can accommodate, which is the maximum capacity on 410 a component link that can be made available by moving other 411 traffic. This information is needed by the ingress LER for path 412 determination. 414 4. A means to signal an LSP whose capacity is larger than individual 415 component link capacity is needed [I-D.ietf-rtgwg-cl-requirement] 416 and also signal the largest microflow expected to be contained in 417 the LSP. If a bounds on the largest microflow is not signaled 418 there is no means to determine if an LSP which is larger than any 419 component link can be subdivided into flows and therefore should 420 be accepted by admission control. 422 When a bidirectional LSP request is signaled over a composite link, 423 if the request indicates that the LSP must be placed on the same 424 component link, the routers of the composite link MUST place the LSP 425 traffic in both directions on a same component link. This is 426 particularly challenging for aggregated capacity which makes use of 427 the label stack for traffic distribution. The two requirements are 428 mutually exclusive for any one LSP. No one LSP may be both larger 429 than any individual component link and require symmetrical paths for 430 every flow. Both requirements can be accommodated by the same 431 composite link for different LSP, with any one LSP requiring no more 432 than one of these two features. 434 Individual component link may fail independently. Upon component 435 link failure, a composite link MUST support a minimally disruptive 436 local repair, preempting any LSP which can no longer be supported. 437 Available capacity in other component links MUST be used to carry 438 impacted traffic. The available bandwidth after failure MUST be 439 advertised immediately to avoid looped crankback. 441 When a composite link is not able to transport all flows, it preempts 442 some flows based upon local management configuration and informs the 443 control plane on these preempted flows. The composite link MUST 444 support soft preemption [RFC5712]. This action ensures the remaining 445 traffic is transported properly. FR#10 requires that the traffic be 446 restored. FR#12 requires that any change be minimally disruptive. 447 These two requirements are interpreted to include preemption among 448 the types of changes that must be minimally disruptive. 450 2.3. Composite Link in Data Plane 452 The traffic over a composite link is distributed over individual 453 component links. Traffic distribution may be determined by or 454 constrained by control plane or management plane. Traffic 455 distribution may be changed due to component link status change, 456 subject to constraints imposed by either the management plane or 457 control plane. The distribution function is local to the routers in 458 which a composite link belongs to and is not specified here. 460 When performing traffic placement, a composite link does not 461 differentiate multicast traffic vs. unicast traffic. 463 3. Architecture Tradeoffs 465 Scalability and stability are critical considerations in protocol 466 design where protocols may be used in a large network. Composite 467 Link is applicable to large networks, and therefore scalability must 468 be a major consideration. Some of the requirements of Composite Link 469 require additional information to be carried in situations where 470 component links differ in some significant way. 472 3.1. Scalability Motivations 474 In the interest of scalability information is aggregated in 475 situations where information about a large amount of network capacity 476 or a large amount of network demand provides is adequate to meet 477 requirements. Routing information is aggregated to reduce the amount 478 of information exchange related to routing and to simplify route 479 computation (see Section 3.2). Reducing the amount of information 480 allows the exchange of information during a large routing change to 481 be accomplished more quickly, and simplifying route computation 482 improves convergence time after very significant network faults which 483 cannot be handled by preprovisioned or precomputed protection 484 mechanisms. Aggregating smaller LSP into larger LSP is a means to 485 reduce RSVP-TE signaling (see Section 3.3). 487 Neglecting scaling issues can result in performance issues, such as 488 slow convergence. Neglecting scaling in some cases can result in 489 networks which perform so poorly as to become unstable. 491 3.2. Reducing Routing Information and Exchange 493 Link bundling at the very least provides a means of aggregating 494 control plane information. Even where the all-ones component link 495 supported by link bundling is not used, the amount of control 496 information is reduced by the average number of component links in a 497 bundle. 499 Fully deaggregating link bundle information would negate this 500 benefit. If there is a need to deaggregate, such as to distinguish 501 between groups of links within specified ranges of delay, then no 502 more deaggregation than is necessary should be done. 504 For example, in supporting the requirement for heterogeneous 505 component links, it makes little sense to fully deaggregate link 506 bundles when adding support for groups of component links with common 507 attributes within a link bundle can maintain most of the benefit of 508 aggregation while adequately supporting the requirement to support 509 heterogeneous component links. 511 Routing information exchange is also reduced by making sensible 512 choices regarding the amount of change to link parameters that 513 require link readvertisement. For example, if delay measurements 514 include queuing delay, then a much more course granularity of delay 515 measurement would be called for than if the delay does not include 516 queuing and is dominated by geographic delay (speed of light delay). 518 3.3. Reducing Signaling Load 520 Aggregating traffic into very large hierarchical LSP in the core very 521 substantially reduces the number of LSP that need to be signaled and 522 the number of path computations any given LSR will be required to 523 perform when a major network fault occurs. 525 In the extreme, applying MPLS to a very large network without 526 hierarchy could exceed the 20 bit label space. For example, in a 527 network with 4,000 nodes, with 2,000 on either side of a cutset, 528 would have 4,000,000 LSP crossing the cutset. Even in a degree four 529 cutset, an uneven distribution of LSP across the cutset, or the loss 530 of one link would result in a need to exceed the size of the label 531 space. Among provider networks, 4,000 access nodes is not at all 532 large. 534 In less extreme cases, having each node terminate hundreds of LSP to 535 achieve a full mesh creates a very large computational load. The 536 time complexity of one CSPF computation is order(N log N), where L is 537 proportional to N, and N and L are the number of nodes and number of 538 links, respectively. If each node must perform order(N) computations 539 when a fault occurs, then the computational load increases as 540 order(N^2 log N) as the number of nodes increases. In practice at 541 the time of writing, this imposes a limit of a few hundred nodes in a 542 full mesh of MPLS LSP before the computational load is sufficient to 543 result in unacceptable convergence times. 545 Two solutions are applied to reduce the amount of RSVP-TE signaling. 546 Both involve subdividing the MPLS domain into a core and a set of 547 regions. 549 3.3.1. Reducing Signaling Load using LDP 551 LDP can be used for edge-to-edge LSP, using RSVP-TE to carry the LDP 552 intra-core traffic and also optionally also using RSVP-TE to carry 553 the LDP intra-region traffic within each region. LDP does not 554 support traffic engineering, but does support multipoint-to-point 555 (MPTP) LSP, which require less signaling than edge-to-edge RSVP-TE 556 point-to-point (PTP) LSP. A drawback of this approach is the 557 inability to use RSVP-TE protection (FRR or GMPLS protection) against 558 failure of the border LSR sitting at a core/region boundary. 560 3.3.2. Reducing Signaling Load using Hierarchy 562 When the number of nodes grows too large, the amount of RSVP-TE 563 signaling can be reduced using the MPLS PSC hierarchy [RFC4206]. A 564 core within the hierarchy can divide the topology into M regions of 565 on average N/M nodes. Within a region the computational load is 566 reduced by more than M^2. Within the core, the computational load 567 generally becomes quite small since M is usually a fairly small 568 number (a few tens of regions) and each region is generally attached 569 to the core in typically only two or three places on average. 571 Using hierarchy improves scaling but has two consequences. First, 572 hierarchy effectively forces the use of platform label space. When a 573 containing LSP is rerouted, the labels assigned to the contained LSP 574 cannot be changed but may arrive on a different interface. Second, 575 hierarchy results in much larger LSP. These LSP today are larger 576 than any single component link and therefore force the use of the 577 all-ones component in link bundles. 579 3.3.3. Using Both LDP and RSVP-TE Hierarchy 581 It is also possible to use both LDP and RSVP-TE hierarchy. MPLS 582 networks with a very large number of nodes may benefit from the use 583 of both LDP and RSVP-TE hierarchy. The two techniques are certainly 584 not mutually exclusive. 586 3.4. Reducing Forwarding State 588 Both LDP and MPLS hierarchy have the benefit of reducing the amount 589 of forwarding state. Using the example from Section 3.3, and using 590 MPLS hierarchy, the worst case generally occurs at borders with the 591 core. 593 For example, consider a network with approximately 1,000 nodes 594 divided into 10 regions. At the edges, each node requires 1,000 LSP 595 to other edge nodes. The edge nodes also require 100 intra-region 596 LSP. Within the core, if the core has only 3 attachments to each 597 region the core LSR have less than 100 intra-core LSP. At the border 598 cutset between the core and a given region, in this example there are 599 100 edge nodes with inter-region LSP crossing that cutset, destined 600 to 900 other edge nodes. That yields forwarding state for on the 601 order of 90,000 LSP at the border cutset. These same routers need 602 only reroute well under 200 LSP when a multiple fault occurs, as long 603 as only links are affected and a border LSR does not go down. 605 In the core, the forwarding state is greatly reduced. If inter- 606 region LSP have different characteristics, it makes sense to make use 607 of aggregates with different characteristics. Rather than exchange 608 information about every inter-region LSP within the intra-core LSP it 609 makes more sense to use multiple intra-core LSP between pairs of core 610 nodes, each aggregating sets of inter-region LSP with common 611 characteristics or common requirements. 613 3.5. Avoiding Route Oscillation 615 Networks can become unstable when a feedback loop exists such that 616 moving traffic to a link causes a metric such as delay to increase, 617 which then causes traffic to move elsewhere. For example, the 618 original ARPANET routing used a delay based cost metric and proved 619 prone to route oscillations [DBP]. 621 Delay may be used as a constraint in routing for high priority 622 traffic, where the movement of traffic cannot impact the delay. The 623 safest way to measure delay is to make measurements based on traffic 624 which is prioritized such that it is queued ahead of the traffic 625 which will be affected. This is a reasonable measure of delay for 626 high priority traffic for which constraints have been set which allow 627 this type of traffic to consume only a fraction of link capacities 628 with the remaining capacity available to lower priority traffic. 630 Any measurement of jitter (delay variation) that is used in route 631 decision is likely to cause oscillation. Jitter that is caused by 632 queuing effects and cannot be measured using a very high priority 633 measurement traffic flow. 635 It may be possible to find links with constrained queuing delay or 636 jitter using a theoretical maximum or a probability based bound on 637 queuing delay or jitter at a given priority based on the types and 638 amounts of traffic accepted and combining that theoretical limit with 639 a measured delay at very high priority. 641 Instability can occur due to poor performance and interaction with 642 protocol timers. In this way a computational scaling problem can 643 become a stability problem when a network becomes sufficiently large. 644 For this reason, [I-D.ietf-rtgwg-cl-requirement] has a number of 645 requirements focusing on minimally impacting scalability. 647 4. New Challenges 649 New technical challenges are posed by [I-D.ietf-rtgwg-cl-requirement] 650 in both the control plane and data plane. 652 Among the more difficult challenges are the following. 654 1. requirements related delay or jitter (see Section 4.1.1), 656 2. the combination of ingress control over LSP placement and 657 retaining an ability to move traffic as demands dictate can pose 658 challenges and such requirements can even be conflicting (see 659 target="sect.local-control" />), 661 3. path symmetry requires extensions and is particularly challenging 662 for very large LSP (see target="sect.path-symmetry" />), 664 4. accommodating a very wide range of requirements among contained 665 LSP can lead to inefficiency if the most stringent requirements 666 are reflected in aggregates, or reduce scalability if a large 667 number of aggregates are used to provide a too fine a reflection 668 of the requirements in the contained LSP (see 669 target="sect.contained-lsp" />), 671 5. backwards compatibility is somewhat limited due to the need to 672 accommodate legacy multipath interfaces which provide too little 673 information regarding their configured default behavior, and 674 legacy LSP which provide too little information regarding their 675 requirements (see target="sect.compat" />), 677 6. data plane challenges include those of accommodating very large 678 LSP, large microflows, traffic ordering constraints imposed by a 679 subsent of LSP, and accounting for IP and LDP traffic (see 680 target="sect.dp-challenge" />). 682 4.1. Control Plane Challenges 684 Some of the control plane requirements are particularly challenging. 685 Handling large flows which aggregate smaller flows must be 686 accomplished with minimal impact on scalability. Potentially 687 conflicting are requirements for jitter and requirements for 688 stability. Potentially conflicting are the requirements for ingress 689 control of a large number of parameters, and the requirements for 690 local control needed to achieve traffic balance across a composite 691 link. These challenges and potential solutions are discussed in the 692 following sections. 694 4.1.1. Delay and Jitter Sensitive Routing 696 Delay and jitter sensitive routing are called for in 697 [I-D.ietf-rtgwg-cl-requirement] in requirements FR#2, FR#7, FR#8, 698 FR#9, FR#15, FR#16, FR#17, FR#18. Requirement FR#17 is particularly 699 problematic, calling for constraints on jitter. 701 A tradeoff exists between scaling benefits of aggregating 702 information, and potential benefits of using a finer granularity in 703 delay reporting. To maintain the scaling benefit, measured link 704 delay for any given composite link SHOULD be aggregated into a small 705 number of delay ranges. IGP-TE extensions MUST be provided which 706 advertise the available capacities for each of the selected ranges. 708 For path selection of delay sensitive LSP, the ingress SHOULD bias 709 link metrics based on available capacity and select a low cost path 710 which meets LSP total path delay criteria. To communicate the 711 requirements of an LSP, the ERO MUST be extended to indicate the per 712 link constraints. To communicate the type of resource used, the RRO 713 SHOULD be extended to carry an identification of the group that is 714 used to carry the LSP at each link bundle hop. 716 4.1.2. Local Control of Traffic Distribution 718 Many requirements in [I-D.ietf-rtgwg-cl-requirement] suggest that a 719 node immediately adjacent to a component link should have a high 720 degree of control over how traffic is distributed, as long as network 721 performance objectives are met. Particularly relevant are FR#18 and 722 FR#19. 724 The requirements to allow local control are potentially in conflict 725 with requirement FR#21 which gives full control of component link 726 select to the LSP ingress. While supporting this capability is 727 mandatory, use of this feature is optional per LSP. 729 A given network deployment will have to consider this pair of 730 conflicting requirements and make appropriate use of local control of 731 traffic placement and ingress control of traffic placement to best 732 meet network requirements. 734 4.1.3. Path Symmetry Requirements 736 Requirement FR#21 in [I-D.ietf-rtgwg-cl-requirement] includes a 737 provision to bind both directions of a bidirectional LSP to the same 738 component. This is easily achieved if the LSP is directly signaled 739 across a composite link. This is not as easily achieved if a set of 740 LSP with this requirement are signaled over a large hierarchical LSP 741 which is in turn carried over a composite link. The basis for load 742 distribution in such as case is the label stack. The labels in 743 either direction are completely independent. 745 This could be accommodated if the ingress, egress, and all midpoints 746 of the hierarchical LSP make use of an entropy label in the 747 distribution, and use only that entropy label. A solution for this 748 problem may add complexity with very little benefit. There is little 749 or no true benefit of using symmetrical paths rather than component 750 links of identical characteristics. 752 Traffic symmetry and large LSP capacity are a second pair of 753 conflicting requirements. Any given LSP can meet one of these two 754 requirements but not both. A given network deployment will have to 755 make appropriate use of each of these features to best meet network 756 requirements. 758 4.1.4. Requirements for Contained LSP 760 [I-D.ietf-rtgwg-cl-requirement] calls for new LSP constraints. These 761 constraints include frequency of load balancing rearrangement, delay 762 and jitter, packet ordering constraints, and path symmetry. 764 When LSP are contained within hierarchical LSP, there is no signaling 765 available at midpoint LSR which identifies the contained LSP let 766 alone providing the set of requirements unique to each contained LSP. 767 Defining extensions to provide this information would severely impact 768 scalability and defeat the purpose of aggregating control information 769 and forwarding information into hierarchical LSP. For the same 770 scalability reasons, not aggregating at all is not a viable option 771 for large networks where scalability and stability problems may occur 772 as a result. 774 As pointed out in Section 4.1.3, the benefits of supporting symmetric 775 paths among LSP contained within hierarchical LSP may not be 776 sufficient to justify the complexity of supporting this capability. 778 A scalable solution which accommodates multiple sets of LSP between 779 given pairs of LSR is to provide multiple hierarchical LSP for each 780 given pair of LSR, each hierarchical LSP aggregating LSP with common 781 requirements and a common pair of endpoints. This is a network 782 design technique available to the network operator rather than a 783 protocol extension. This technique can accommodate multiple sets of 784 delay and jitter parameters, multiple sets of frequency of load 785 balancing parameters, multiple sets of packet ordering constraints, 786 etc. 788 4.1.5. Retaining Backwards Compatibility 790 Backwards compatibility and support for incremental deployment 791 requires considering the impact of legacy LSR in the role of LSP 792 ingress, and considering the impact of legacy LSR advertising 793 ordinary links, advertising Ethernet LAG as ordinary links, and 794 advertising link bundles. 796 Legacy LSR in the role of LSP ingress cannot signal requirements 797 which are not supported by their control plane software. The 798 additional capabilities supported by other LSR has no impact on these 799 LSR. These LSR however, being unaware of extensions, may try to make 800 use of scarce resources which support specific requirements such as 801 low delay. To a limited extent it may be possible for a network 802 operator to avoid this issue using existing mechanisms such as link 803 administrative attributes and attribute affinities [RFC3209]. 805 Legacy LSR advertising ordinary links will not advertise attributes 806 needed by some LSP. For example, there is no way to determine the 807 delay or jitter characteristics of such a link. Legacy LSR 808 advertising Ethernet LAG pose additional problems. There is no way 809 to determine that packet ordering constraints would be violated for 810 LSP with strict packet ordering constraints, or that frequency of 811 load balancing rearrangement constraints might be violated. 813 Legacy LSR advertising link bundles have no way to advertise the 814 configured default behavior of the link bundle. Some link bundles 815 may be configured to place each LSP on a single component link and 816 therefore may not be able to accommodate an LSP which requires 817 bandwidth in excess of the size of a component link. Some link 818 bundles may be configured to spread all LSP over the all-ones 819 component. For LSR using the all-ones component link, there is no 820 documented procedure for correctly setting the "Maximum LSP 821 Bandwidth". There is currently no way to indicate the largest 822 microflow that could be supported by a link bundle using the all-ones 823 component link. 825 Having received the RRO, it is possible for an ingress to look for 826 the all-ones component to identify such link bundles after having 827 signaled at least one LSP. Whether any LSR collects this information 828 on legacy LSR and makes use of it to set defaults, is an 829 implementation choice. 831 4.2. Data Plane Challenges 833 Regardless of implementation choices, there are tradeoffs regarding 834 the flow identification granularity. If flow identification 835 granularity is very course, such as using top lable only, LSP larger 836 than the size of a component link are not feasible. In practice 837 using the MPLS label stack alone has proven too course to acheive a 838 reasonably good load balance, due to bin-packing issues and 839 discrpencies between signaled bandwidth and actual traffic loads on 840 LSP. If a finer granualrity is based on IP addresses, then a table 841 approach is infeasible, due to the extremely large number of IP 842 address pairs found in Internet traffic. The preferred 843 implementation has been to use a hash over the IP address pairs to 844 provide a fine granuarity but with a feasible implementation. Where 845 hashing is used, the hash itself can be done at ingress and placed in 846 a fat-pw label or entropy label (see Section 6.3) to avoid performing 847 the deeper MPLS header parse in the network core and the hash in the 848 network core. 850 Other implementations are possible, but must still face the problem 851 that not using IP addresses provides a granularity which is too 852 course and using IP host pairs yields a table size which is so 853 impractical as to be considered an infeasible solution. This section 854 assumes that this problem is addressed, but makes no assumption as to 855 the implementation. Where a hash based solution is mentioned, such 856 mention should be considered a reference approach, not a requirement. 858 In order to maintain scalability, existing data plane forwarding 859 retains state associated with the top label only. Data plane 860 forwarding makes use of the top label to select a composite link, or 861 a group of components within a composite link or for the case where 862 an LSP is pinned (see [RFC4201]), a specific component link. For 863 those LSP for which the LSP selects only the composite link or a 864 group of components within a composite link, the load balancing may 865 make use of the entire label stack and in some cases may make use of 866 information in the payload, though no state on specific contained LSP 867 is retained. 869 Load balancing makes use of techniques which allow large sets of 870 flows to be moved to rearrange traffic. These large sets of flows 871 may be at a finer granularity than contained LSP. Requirements to 872 limit frequency of load balancing rearrangement can be adhered to by 873 constraining the frequency at which these large sets of flows are 874 moved. 876 4.2.1. Very Large LSP 878 Very large LSP may exceed the capacity of any single component of a 879 composite link. In some cases contained LSP may exceed the capacity 880 of any single component. These LSP require the use of the equivalent 881 of the all-ones component of a link bundle. 883 4.2.2. Very Large Microflows 885 Within a very large LSP there may be very large microflows, or very 886 large flows which cannot be further subdivided for other reasons. 887 Flows which cannot be subdivided must be no larger that the capacity 888 of any single component. 890 Current signaling provides no way to specify the largest microflow 891 that a can be supported on a given link bundle in routing 892 advertisements. Extensions which address this are discussed in 893 Section 6.4. Absent extensions of this type, traffic containing 894 microflows that are too large for a given composite link may be 895 present. There is no data plane solution for this problem that would 896 not require reordering traffic at the composite link egress. 898 Some techniques are susceptible to statistical collisions where an 899 algorithm to distribute traffic is unable to disambiguate traffic 900 among two or more very large microflow where their sum is in excess 901 of the capacity of any single component. Hash based algorithms which 902 use too small a hash space are particularly susceptible and require a 903 change in hash seed in the event that this were to occur. A change 904 in hash seed is highly disruptive, causing traffic reordering among 905 all traffic flows over which the hash function is applied. 907 4.2.3. Traffic Ordering Constraints 909 Some LSP have strict traffic ordering constraints. Most notable 910 among these are MPLS-TP LSP. In the absence of aggregation into 911 hierarchical LSP, those LSP with strict traffic ordering constraints 912 can be placed on individual component links if there is a means of 913 identifying which LSP have such a constraint. If LSP with strict 914 traffic ordering constraints are aggregated in hierarchical LSP, the 915 hierarchical LSP capacity may exceed the capacity of any single 916 component link. In such a case the load balancing for the containing 917 may be constrained to look only at the top label and the first 918 contained label. This and related issues are discussed further in 919 Section 6.4. 921 4.2.4. Accounting for IP and LDP Traffic 923 Networks which carry RSVP-TE signaled MPLS traffic generally carry 924 low volumes of native IP traffic, often only carrying control traffic 925 as native IP. There is no architectural guarantee of this, it is 926 just how network operators have made use of the protocols. 928 [I-D.ietf-rtgwg-cl-requirement] requires that native IP and native 929 LDP be accommodated. In some network, a subset of services may be 930 carried as native IP or carried as native LDP. Today this may be 931 accommodated by the network operator estimating the contribution of 932 IP and LDP and configuring a lower set of available bandwidth figures 933 on the RSVP-TE advertisements. 935 The only improvement that Composite Link can offer is that of 936 measuring the IP and LDP traffic levels and automatically reducing 937 the available bandwidth figures on the RSVP-TE advertisements. The 938 measurements would have to be significantly filtered. This is 939 similar to a feature in existing LSR, commonly known as 940 "autobandwidth" with a key difference. In the "autobandwidth" 941 feature, the bandwidth request of an RSVP-TE signaled LSP is adjusted 942 in response to traffic measurements. In this case the IP or LDP 943 traffic measurements are used to reduce the link bandwidth directly, 944 without first encapsulating in an RSVP-TE LSP. 946 This may be a subtle and perhaps even a meaningless distinction if 947 Composite Link is used to form a Sub-Path Maintenance Element (SPME). 948 A SPME is in practice essentially an unsignaled single hop LSP with 949 PHP enabled [RFC5921]. A Composite Link SPME looks very much like 950 classic multipath, where there is no signaling, only management plane 951 configuration creating the multipath entity (of which Ethernet Link 952 Aggregation is a subset). 954 IP does not offer traffic engineering. LDP cannot be extended to 955 offer traffic engineering [RFC3468]. Therefore there is no traffic 956 engineered fallback to an alternate path for IP and LDP traffic if 957 resources are not adequate for the IP and/or LDP traffic alone on a 958 given link in the primary path. The only option available to LDP is 959 to fail to forward a FEC if conditions are not met for that FEC. In 960 either LDP DOD mode or DU mode, another path will be used if 961 available in a manner similar to a link which is present in the IGP 962 but does not have LDP enabled. 964 5. Existing Mechanisms 966 In MPLS the one mechanisms which support explicit signaling of 967 multiple parallel links is Link Bundling [RFC4201]. The set of 968 techniques known as "classis multipath" support no explicit 969 signaling, except in two cases. In Ethernet Link Aggregation the 970 Link Aggregation Control Protocol (LACP) coordinates the addition or 971 removal of members from an Ethernet Link Aggregation Group (LAG). 972 The use of the "all-ones" component of a link bundle indicates use of 973 classis multipath, however the ability to determine if a link bundle 974 makes use of classis multipath is not yet supported. 976 5.1. Link Bundling 978 Link bundling supports advertisement of a set of homogenous links as 979 a single route advertisement. Link bundling supports placement of an 980 LSP on any single component link, or supports placement of an LSP on 981 the all-ones component link. Not all link bundling implementations 982 support the all-ones component link. There is no way for an ingress 983 LSR to tell which potential midpoint LSR support this feature and use 984 it by default and which do not. Based on [RFC4201] it is unclear how 985 to advertise a link bundle for which the all-ones component link is 986 available and used by default. Common practice is to violate the 987 specification and set the Maximum LSP Bandwidth to the Available 988 Bandwidth. There is no means to determine the largest microflow that 989 could be supported by a link bundle that is using the all-ones 990 component link. 992 [RFC6107] extends the procedures for hierarchical LSP but also 993 extends link bundles. An LSP can be explicitly signaled to indicate 994 that it is an LSP to be used as a component of a link bundle. Prior 995 to that the common practice was to simply not advertise the component 996 link LSP into the IGP, since only the ingress and egress of the link 997 bundle needed to be aware of their existence, which they would be 998 aware of due to the RSVP-TE signaling used in setting up the 999 component LSP. 1001 While link bundling can be the basis for composite links, a 1002 significant number of small extension needs to be added. 1004 1. To support link bundles of heterogeneous links, a means of 1005 advertising the capacity available within a group of homogeneous 1006 needs to be provided. 1008 2. Attributes need to be defined to support the following parameters 1009 for the link bundle or for a group of homogeneous links. 1011 A. delay range 1013 B. jitter (delay variation) range 1015 C. group metric 1017 D. all-ones component capable 1019 E. capable of dynamically balancing load 1021 F. largest supportable microflow 1023 G. abilities to support strict packet ordering requirements 1024 within contained LSP 1026 3. For each of the prior extended attributes, the constraint based 1027 routing path selection needs to be extended to reflect new 1028 constraints based on the extended attributes. 1030 4. For each of the prior extended attributes, LSP admission control 1031 needs to be extended to reflect new constraints based on the 1032 extended attributes. 1034 5. Dynamic load balance must be provided for flows within a given 1035 set of links with common attributes such that NPO are not 1036 violated including frequency of load balance adjustment for any 1037 given flow. 1039 5.2. Classic Multipath 1041 Classic multipath is defined in [I-D.symmvo-rtgwg-cl-use-cases]. 1043 Classic multipath refers to the most common current practice in 1044 implementation and deployment of multipath. The most common current 1045 practice makes use of a hash on the MPLS label stack and if IPv4 or 1046 IPv6 are indicated under the label stack, makes use of the IP source 1047 and destination addresses [RFC4385] [RFC4928]. 1049 Classic multipath provides a highly scalable means of load balancing. 1050 Adaptive multipath has proven value in assuring an even loading on 1051 component link and an ability to adapt to change in offerred load 1052 that occurs over periods of hundreds of milliseconds or more. 1053 Classic multipath scalability is due to the ability to effectively 1054 work with an extremely large number of flows (IP host pairs) using 1055 relatively little resources (a data structure accessed using a hash 1056 result as a key or using ranges of hash results). 1058 Classic multipath meets a small subset of Composite Link 1059 requirements. Due to scalability of the approach, classic multipath 1060 seems to be an excellent candidate for extension to meet the full set 1061 of Composite Link forwarding requirements. 1063 Additional detail can be found in [I-D.symmvo-rtgwg-cl-use-cases]. 1065 6. Mechanisms Proposed in Other Documents 1067 A number of documents which at the time of writing are works in 1068 progress address parts of the requirements of Composite Link, or 1069 assist in making some of the goals achievable. 1071 6.1. Loss and Delay Measurement 1073 Procedures for measuring loss and delay are provided in [RFC6374]. 1074 These are OAM based measurements. This work could be the basis of 1075 delay measurements and delay variation measurement used for metrics 1076 called for in [I-D.ietf-rtgwg-cl-requirement]. 1078 Currently there are two additional Internet-Drafts that address delay 1079 and delay variation metrics. 1081 [I-D.wang-ccamp-latency-te-metric] is designed specifically to 1082 meet this requirement. OSPF-TE and ISIS-TE extensions are 1083 defined to indicate link delay and delay variance. The RSVP-TE 1084 ERO is extended to include service level requirements. A latency 1085 accumulation object is defined to provide a means of verification 1086 of the service level requirements. This draft is intended to 1087 proceed in the CCAMP WG. It is currently and individual 1088 submission. The 03 version of this draft expired in September 1089 2012. 1091 draft-giacalone-ospf-te-express-path 1092 This document proposes to extend OSPF-TE only. Extensions 1093 support delay, delay variance, loss, residual bandwidth, and 1094 available bandwidth. No extensions to RSVP-TE are proposed. 1095 This draft is intended to proceed in the CCAMP WG. It is 1096 currently and individual submission. The 02 version will expire 1097 in March 2012. 1099 A possible course of action may be to combine these two drafts. The 1100 delay variance, loss, residual bandwidth, and available bandwidth 1101 extensions are particular prone to network instability. The question 1102 as to whether queuing delay and delay variation should be considered, 1103 and if so for which diffserv Per-Hop Service Class (PSC) is not 1104 addressed. 1106 Note to co-authors: The ccamp-latency-te-metric draft refers to 1107 [I-D.ietf-rtgwg-cl-requirement] and is well matched to those 1108 requirements, including stability. The ospf-te-express-path draft 1109 refers to the "Alto Protocol" (draft-ietf-alto-protocol) and 1110 therefore may not be intended for RSVP-TE use. The authors of the 1111 two drafts may be able to resolve this. It may be best to drop ospf- 1112 te-express-path from this framework document. 1114 6.2. Link Bundle Extensions 1116 A set of link bundling extensions are defined in 1117 [I-D.ietf-mpls-explicit-resource-control-bundle]. This document 1118 provides extensions to the ERO and RRO to explicitly control the 1119 labels and resources within a bundle used by an LSP. 1121 The extensions in this document could be further extended to support 1122 indicating a group of component links in the ERO or RRO, where the 1123 group is given an interface identification like the bundle itself. 1124 The extensions could also be further extended to support 1125 specification of the all-ones component link in the ERO or RRO. 1127 [I-D.ietf-mpls-explicit-resource-control-bundle] does not provide a 1128 means to advertise the link bundle components. It is not certain how 1129 the ingress LSR would determine the set of link bundle component 1130 links available for a given link bundle. 1132 [I-D.ospf-cc-stlv] provides a baseline draft for extending link 1133 bundling to advertise components. A new component TVL (C-TLV) is 1134 proposed, which must reference a Composite Link Link TLV. 1135 [I-D.ospf-cc-stlv] is intended for the OSPF WG and submitted for the 1136 "Experimental" track. The 00 version expired in February 2012. 1138 6.3. Fat PW and Entropy Labels 1140 Two documents provide a means to add entropy for the purpose of 1141 improving load balance. MPLS encapsulation can bury information that 1142 is needed to identify microflows. These two documents allow a 1143 pseudowire ingress and LSP ingress respectively to add a label solely 1144 for the purpose of providing a finer granularity of microflow groups. 1146 [RFC6391] allows pseudowires which carry a large volume of traffic, 1147 where microflows can be identified to be load balanced across 1148 multiple members of an Ethernet LAG or an MPLS link bundle. This is 1149 accomplished by adding a flow label below the pseudowire label in the 1150 MPLS label stack. For this to be effective the link bundle load 1151 balance must make use of the label stack up to and including this 1152 flow label. 1154 [I-D.ietf-mpls-entropy-label] provides a means for a LER to put an 1155 additional label known as an entropy label on the MPLS label stack. 1156 As defined, only the LER can add the entropy label and this label 1157 must be at the bottom of stack. 1159 If the bottom of stack restriction on entropy labels were to be 1160 relaxed, then core LSR could add entropy labels based on deep packet 1161 inspection and place the entropy label just below the label being 1162 acted on. This would be helpful in situations where the label stack 1163 depth to which load distribution can operate is limited by 1164 implementation or is limited for other reasons such as carrying both 1165 MPLS-TP and MPLS with entropy labels within the same hierarchical 1166 LSP. 1168 6.4. Multipath Extensions 1170 The multipath extensions drafts address one aspect of Composite Link. 1171 These drafts deal with the issue of accommodating LSP which have 1172 strict packet ordering constraints in a network containing multipath. 1173 MPLS-TP has become the one important instance of LSP with strict 1174 packet ordering constraints and has driven this work. 1176 [I-D.villamizar-mpls-tp-multipath] outlines requirements and gives a 1177 number of options for dealing with the apparent incompatibility of 1178 MPLS-TP and multipath. A preferred option is described. 1180 [I-D.villamizar-mpls-tp-multipath-te-extn] provides protocol 1181 extensions needed to implement the preferred option described in 1182 [I-D.villamizar-mpls-tp-multipath]. 1184 Other issues pertaining to multipath are also addressed. Means to 1185 advertise the largest microflow supportable are defined. Means to 1186 indicate the largest expected microflow within an LSP are defined. 1187 Issues related to hierarchy are addressed. 1189 7. Required Protocol Extensions and Mechanisms 1191 Prior sections have reviewed key characteristics, architecture 1192 tradeoffs, new challenges, existing mechanisms, and relevant 1193 mechanisms proposed in existing new documents. 1195 This section first summarizes and groups requirements. A set of 1196 documents coverage groupings are proposed with existing works-in- 1197 progress noted where applicable. The set of extensions are then 1198 grouped by protocol affected as a convenience to implementors. 1200 7.1. Brief Review of Requirements 1202 The following list provides a categorization of requirements 1203 specified in [I-D.ietf-rtgwg-cl-requirement] along with a short 1204 phrase indication what topic the requirement covers. 1206 routing information aggregation 1207 FR#1 (routing summarization), FR#20 (composite link may be a 1208 component of another composite link) 1210 restoration speed 1211 FR#2 (restoration speed meeting NPO), FR#12 (minimally disruptive 1212 load rebalance), DR#6 (fast convergence), DR#7 (fast worst case 1213 failure convergence) 1215 load distribution, stability, minimal disruption 1216 FR#3 (automatic load distribution), FR#5 (must not oscillate), 1217 FR#11 (dynamic placement of flows), FR#12 (minimally disruptive 1218 load rebalance), FR#13 (bounded rearrangement frequency), FR#18 1219 (flow placement must satisfy NPO), FR#19 (flow identification 1220 finer than per top level LSP), MR#6 (operator initiated flow 1221 rebalance) 1223 backward compatibility and migration 1224 FR#4 (smooth incremental deployment), FR#6 (management and 1225 diagnostics must continue to function), DR#1 (extend existing 1226 protocols), DR#2 (extend LDP, no LDP TE) 1228 delay and delay variation 1229 FR#7 (expose lower layer measured delay), FR#8 (precision of 1230 latency reporting), FR#9 (limit latency on per LSP basis), FR#15 1231 (minimum delay path), FR#16 (bounded delay path), FR#17 (bounded 1232 jitter path) 1234 admission control, preemption, traffic engineering 1235 FR#10 (admission control, preemption), FR#14 (packet ordering), 1236 FR#21 (ingress specification of path), FR#22 (path symmetry), 1237 DR#3 (IP and LDP traffic), MR#3 (management specification of 1238 path) 1240 single vs multiple domain 1241 DR#4 (IGP extensions allowed within single domain), DR#5 (IGP 1242 extensions disallowed in multiple domain case) 1244 general network management 1245 MR#1 (polling, configuration, and notification), MR#2 (activation 1246 and de-activation) 1248 path determination, connectivity verification 1249 MR#4 (path trace), MR#5 (connectivity verification) 1251 The above list is not intended as a substitute for 1252 [I-D.ietf-rtgwg-cl-requirement], but rather as a concise grouping and 1253 reminder or requirements to serve as a means of more easily 1254 determining requirements coverage of a set of protocol documents. 1256 7.2. Required Document Coverage 1258 The primary areas where additional protocol extensions and mechanisms 1259 are required include the topics described in the following 1260 subsections. 1262 There are candidate documents for a subset of the topics below. This 1263 grouping of topics does not require that each topic be addressed by a 1264 separate document. In some cases, a document may cover multiple 1265 topics, or a specific topic may be addressed as applicable in 1266 multiple documents. 1268 7.2.1. Component Link Grouping 1270 An extension to link bundling is needed to specify a group of 1271 components with common attributes. This can be a TLV defined within 1272 the link bundle that carries the same encapsulations as the link 1273 bundle. Two interface indices would be needed for each group. 1275 a. An index is needed that if included in an ERO would indicate the 1276 need to place the LSP on any one component within the group. 1278 b. A second index is needed that if included in an ERO would 1279 indicate the need to balance flows within the LSP across all 1280 components of the group. This is equivalent to the "all-ones" 1281 component for the entire bundle. 1283 [I-D.ospf-cc-stlv] can be extended to include multipath treatment 1284 capabilities. An ISIS solution is also needed. An extension of 1285 RSVP-TE signaling is needed to indicate multipath treatment 1286 preferences. 1288 7.2.2. Component Group Metric 1290 If a group is allowed to support all of the parameters of a link 1291 bundle, then a group TE metric would be accommodated. This can be 1292 supported with the component TLV (C-TLV) defined in 1293 [I-D.ospf-cc-stlv]. 1295 7.2.3. Delay and Jitter Extensions 1297 A extension is needed in the IGP-TE advertisement to support delay 1298 and delay variation for links, link bundles, and forwarding 1299 adjacencies. Whatever mechanism is described must take precautions 1300 that insure that route oscillations cannot occur. 1301 [I-D.wang-ccamp-latency-te-metric] may be a good starting point. 1303 7.2.4. Path Selection and Admission Control 1305 Path selection and admission control changes must be documented in 1306 each document that proposes a protocol extension that advertises a 1307 new capability or parameter that must be supported by changes in path 1308 selection and admission control. 1310 7.2.5. Dynamic Multipath Balance 1312 FR#11 explicitly calls for dynamic load balancing similar to existing 1313 adaptive multipath. In implementations where flow identification 1314 uses a course granularity, the adjustments would have to be equally 1315 course, in the worst case moving entire LSP. The impact of flow 1316 identification granularity and potential adaptive multipath 1317 approaches may need to be documented in greater detail than provided 1318 here. 1320 7.2.6. Frequency of Load Balance 1322 IGP-TE and RSVP-TE extensions are needed to support frequency of load 1323 balancing rearrangement called for in FR#13, and FR#15-FR#17. 1324 Constraints are not defined in RSVP-TE, but could be modeled after 1325 administrative attribute affinities in RFC3209 and elsewhere. 1327 7.2.7. Inter-Layer Communication 1329 Lower layer to upper layer communication called for in FR#7 and 1330 FR#20. This is addressed for a subset of parameters related to 1331 packet ordering in [I-D.villamizar-mpls-tp-multipath] where layers 1332 are MPLS. Remaining parameters, specifically delay and delay 1333 variation, need to be addressed. Passing information from a lower 1334 non-MPLS layer to an MPLS layer needs to be addressed, though this 1335 may largely be generic advice encouraging a coupling of MPLS to lower 1336 layer management plane or control plane interfaces. This topic can 1337 be addressed in each document proposing a protocol extension, where 1338 applicable. 1340 7.2.8. Packet Ordering Requirements 1342 A document is needed to define extensions supporting various packet 1343 ordering requirements, ranging from requirements to preservce 1344 microflow ordering only, to requirements to preservce full LSP 1345 ordering (as in MPLS-TP). This is covered by 1346 [I-D.villamizar-mpls-tp-multipath] and 1347 [I-D.villamizar-mpls-tp-multipath-te-extn]. 1349 7.2.9. Minimally Disruption Load Balance 1351 The behavior of hash methods used in classic multipath needs to be 1352 described in terms of FR#12 which calls for minimally disruptive load 1353 adjustments. For example, reseeding the hash violates FR#12. Using 1354 modulo operations is significantly disruptive if a link comes or goes 1355 down, as pointed out in [RFC2992]. In addition, backwards 1356 compatibility with older hardware needs to be accommodated. 1358 7.2.10. Path Symmetry 1360 Protocol extensions are needed to support dynamic load balance as 1361 called for to meet FR#22 (path symmetry) and to meet FR#11 (dynamic 1362 placement of flows). Currently path symmetry can only be supported 1363 in link bundling if the path is pinned. When a flow is moved both 1364 ingress and egress must make the move as close to simultaneously as 1365 possible to satisfy FR#22 and FR#12 (minimally disruptive load 1366 rebalance). If a group of flows are identified using a hash, then 1367 the hash must be identical on the pair of LSR at the endpoint, using 1368 the same hash seed and with one side swapping source and destination. 1369 If the label stack is used, then either the entire label stack must 1370 be a special case flow identification, since the set of labels in 1371 either direction are not correlated, or the two LSR must conspire to 1372 use the same flow identifier. For example, using a common entropy 1373 label value, and using only the entropy label in the flow 1374 identification would satisfy this requirement. 1376 7.2.11. Performance, Scalability, and Stability 1378 A separate document providing analysis of performance, scalability, 1379 and stability impacts of changes may be needed. The topic of traffic 1380 adjustment oscillation must also be covered. If sufficient coverage 1381 is provided in each document covering a protocol extension, a 1382 separate document would not be needed. 1384 7.2.12. IP and LDP Traffic 1386 A document is needed to define the use of measurements native IP and 1387 native LDP traffic levels to reduce link advertised bandwidth 1388 amounts. 1390 7.2.13. LDP Extensions 1392 Extending LDP is called for in DR#2. LDP can be extended to couple 1393 FEC admission control to local resource availability without 1394 providing LDP traffic engineering capability. Other LDP extensions 1395 such as signaling a bound on microflow size and LDP LSP requirements 1396 would provide useful information without providing LDP traffic 1397 engineering capability. 1399 7.2.14. Pseudowire Extensions 1401 PW extensions such as signaling a bound on microflow size and PW 1402 requirements would provide useful information. 1404 7.2.15. Multi-Domain Composite Link 1406 DR#5 calls for Composite Link to span multiple network topologies. 1407 Component LSP may already span multiple network topologies, though 1408 most often in practice these are LDP signaled. Component LSP which 1409 are RSVP-TE signaled may also span multiple network topologies using 1410 at least three existing methods (per domain [RFC5152], BRPC 1411 [RFC5441], PCE [RFC4655]). When such component links are combined in 1412 a Composite Link, the Composite Link spans multiple network 1413 topologies. It is not clear in which document this needs to be 1414 described or whether this description in the framework is sufficient. 1415 The authors and/or the WG may need to discuss this. DR#5 mandates 1416 that IGP-TE extension cannot be used. This would disallow the use of 1417 [RFC5316] or [RFC5392] in conjunction with [RFC5151]. 1419 7.3. Open Issues Regarding Requirements 1421 Note to co-authors: This section needs to be reduced to an empty 1422 section and then removed. 1424 The following topics in the requirements document are not addressed. 1425 Since they are explicitly mentioned in the requirements document some 1426 mention of how they are supported is needed, even if to say nother 1427 needed to be done. If we conclude any particular topic is 1428 irrelevant, maybe the topic should be removed from the requirement 1429 document. At that point we could add the management requirements 1430 that have come up and were missed. 1432 1. L3VPN RFC 4364, RFC 4797,L2VPN RFC 4664, VPWS, VPLS RFC 4761, RFC 1433 4762 and VPMS VPMS Framework 1434 (draft-ietf-l2vpn-vpms-frmwk-requirements). It is not clear what 1435 additional Composite Link requirements these references imply, if 1436 any. If no additional requirements are implied, then these 1437 references are considered to be informational only. 1439 2. Migration may not be adequately covered in Section 4.1.5. It 1440 might also be necessary to say more here on performance, 1441 scalability, and stability. Comments on this from co-authors or 1442 the WG? 1444 3. We may need a performance section in this document to 1445 specifically address #DR6 (fast convergence), and #DR7 (fast 1446 worst case failure convergence), though we do already have 1447 scalability discussion. The performance section would have to 1448 say "no worse than before, except were there was no alternative 1449 to make it very slightly worse" (in a bit more detail than that). 1450 It would also have to better define the nature of the performance 1451 criteria. 1453 7.4. Framework Requirement Coverage by Protocol 1455 As an aid to implementors, this section summarizes requirement 1456 coverage listed in Section 7.2 by protocol or LSR functionality 1457 affected. 1459 Some documentation may be purely informational, proposing no changes 1460 and proposing usage at most. This includes Section 7.2.4, 1461 Section 7.2.9, Section 7.2.11, and Section 7.2.15. 1463 Section 7.2.10 may require a new protocol. 1465 7.4.1. OSPF-TE and ISIS-TE Protocol Extensions 1467 Many of the changes listed in Section 7.2 require IGP-TE changes, 1468 though most are small extensions to provide additional information. 1469 This set includes Section 7.2.1, Section 7.2.2, Section 7.2.3, 1470 Section 7.2.6, Section 7.2.7, and Section 7.2.8. An adjustment to 1471 existing advertised parameters is suggested in Section 7.2.12. 1473 7.4.2. PW Protocol Extensions 1475 The only suggestion of pseudowire (PW) extensions is in 1476 Section 7.2.14. 1478 7.4.3. LDP Protocol Extensions 1480 Potential LDP extensions are described in Section 7.2.13. 1482 7.4.4. RSVP-TE Protocol Extensions 1484 RSVP-TE protocol extensions are called for in Section 7.2.1, 1485 Section 7.2.2, Section 7.2.6, Section 7.2.8, and Section 7.2.10. 1487 7.4.5. RSVP-TE Path Selection Changes 1489 Section 7.2.4 calls for path selection to be addressed in individual 1490 documents that require change. These changes would include those 1491 proposed in Section 7.2.1, Section 7.2.2, Section 7.2.3, 1492 Section 7.2.6, and Section 7.2.8. 1494 7.4.6. RSVP-TE Admission Control and Preemption 1496 When a change is needed to path selection, a corresponding change is 1497 needed in admission control. The same set of sections applies: 1498 Section 7.2.1, Section 7.2.2, Section 7.2.3, Section 7.2.6, and 1499 Section 7.2.8. Some resource changes such as a link delay change 1500 might trigger preemption. The rules of preemption remain unchanged, 1501 still based on holding priority. 1503 7.4.7. Flow Identification and Traffic Balance 1505 The following describe either the state of the art in flow 1506 identification and traffic balance or propose changes: Section 7.2.5, 1507 Section 7.2.6, Section 7.2.8, and Section 7.2.9. 1509 8. Security Considerations 1511 The security considerations for MPLS/GMPLS and for MPLS-TP are 1512 documented in [RFC5920] and [I-D.ietf-mpls-tp-security-framework]. 1514 The types protocol extensions proposed in this framework document 1515 provide additional information about links, forwarding adjacencies, 1516 and LSP requirements. The protocol semantics changes described in 1517 this framework document propose additional LSP constraints applied at 1518 path computation time and at LSP admission at midpoints LSR. The 1519 additional information and constraints provide no additional security 1520 considerations beyond the security considerations already documented 1521 in [RFC5920] and [I-D.ietf-mpls-tp-security-framework]. 1523 9. Acknowledgments 1525 Authors would like to thank Adrian Farrel, Fred Jounay, Yuji Kamite 1526 for his extensive comments and suggestions, Ron Bonica, Nabil Bitar, 1527 Eric Gray, Lou Berger, and Kireeti Kompella for their reviews and 1528 great suggestions. 1530 10. References 1532 10.1. Normative References 1534 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1535 Requirement Levels", BCP 14, RFC 2119, March 1997. 1537 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1538 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1539 Tunnels", RFC 3209, December 2001. 1541 [RFC3630] Katz, D., Kompella, K., and D. Yeung, "Traffic Engineering 1542 (TE) Extensions to OSPF Version 2", RFC 3630, 1543 September 2003. 1545 [RFC4201] Kompella, K., Rekhter, Y., and L. Berger, "Link Bundling 1546 in MPLS Traffic Engineering (TE)", RFC 4201, October 2005. 1548 [RFC4206] Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP) 1549 Hierarchy with Generalized Multi-Protocol Label Switching 1550 (GMPLS) Traffic Engineering (TE)", RFC 4206, October 2005. 1552 [RFC5036] Andersson, L., Minei, I., and B. Thomas, "LDP 1553 Specification", RFC 5036, October 2007. 1555 [RFC5305] Li, T. and H. Smit, "IS-IS Extensions for Traffic 1556 Engineering", RFC 5305, October 2008. 1558 [RFC5712] Meyer, M. and JP. Vasseur, "MPLS Traffic Engineering Soft 1559 Preemption", RFC 5712, January 2010. 1561 [RFC6107] Shiomoto, K. and A. Farrel, "Procedures for Dynamically 1562 Signaled Hierarchical Label Switched Paths", RFC 6107, 1563 February 2011. 1565 [RFC6374] Frost, D. and S. Bryant, "Packet Loss and Delay 1566 Measurement for MPLS Networks", RFC 6374, September 2011. 1568 [RFC6391] Bryant, S., Filsfils, C., Drafz, U., Kompella, V., Regan, 1569 J., and S. Amante, "Flow-Aware Transport of Pseudowires 1570 over an MPLS Packet Switched Network", RFC 6391, 1571 November 2011. 1573 10.2. Informative References 1575 [DBP] Bertsekas, D., "Dynamic Behavior of Shortest Path Routing 1576 Algorithms for Communication Networks", IEEE Trans. Auto. 1577 Control 1982. 1579 [I-D.ietf-mpls-entropy-label] 1580 Drake, J., Kompella, K., Yong, L., Amante, S., and W. 1581 Henderickx, "The Use of Entropy Labels in MPLS 1582 Forwarding", draft-ietf-mpls-entropy-label-01 (work in 1583 progress), October 2011. 1585 [I-D.ietf-mpls-explicit-resource-control-bundle] 1586 Zamfir, A., Ali, Z., and P. Dimitri, "Component Link 1587 Recording and Resource Control for TE Links", 1588 draft-ietf-mpls-explicit-resource-control-bundle-10 (work 1589 in progress), April 2011. 1591 [I-D.ietf-mpls-tp-security-framework] 1592 Niven-Jenkins, B., Fang, L., Graveman, R., and S. 1593 Mansfield, "MPLS-TP Security Framework", 1594 draft-ietf-mpls-tp-security-framework-02 (work in 1595 progress), October 2011. 1597 [I-D.ietf-rtgwg-cl-requirement] 1598 Malis, A., Villamizar, C., McDysan, D., Yong, L., and N. 1599 So, "Requirements for MPLS Over a Composite Link", 1600 draft-ietf-rtgwg-cl-requirement-05 (work in progress), 1601 January 2012. 1603 [I-D.kompella-mpls-rsvp-ecmp] 1604 Kompella, K., "Multi-path Label Switched Paths Signaled 1605 Using RSVP-TE", draft-kompella-mpls-rsvp-ecmp-01 (work in 1606 progress), October 2011. 1608 [I-D.ospf-cc-stlv] 1609 Osborne, E., "Component and Composite Link Membership in 1610 OSPF", draft-ospf-cc-stlv-00 (work in progress), 1611 August 2011. 1613 [I-D.symmvo-rtgwg-cl-use-cases] 1614 Malis, A., Villamizar, C., McDysan, D., Yong, L., and N. 1615 So, "Composite Link USe Cases and Design Considerations", 1616 draft-symmvo-rtgwg-cl-use-cases-00 (work in progress), 1617 February 2012. 1619 [I-D.villamizar-mpls-tp-multipath] 1620 Villamizar, C., "Use of Multipath with MPLS-TP and MPLS", 1621 draft-villamizar-mpls-tp-multipath-01 (work in progress), 1622 March 2011. 1624 [I-D.villamizar-mpls-tp-multipath-te-extn] 1625 Villamizar, C., "Multipath Extensions for MPLS Traffic 1626 Engineering", 1627 draft-villamizar-mpls-tp-multipath-te-extn-00 (work in 1628 progress), July 2011. 1630 [I-D.wang-ccamp-latency-te-metric] 1631 Fu, X., Betts, M., Wang, Q., McDysan, D., and A. Malis, 1632 "GMPLS extensions to communicate latency as a traffic 1633 engineering performance metric", 1634 draft-wang-ccamp-latency-te-metric-03 (work in progress), 1635 March 2011. 1637 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1638 and W. Weiss, "An Architecture for Differentiated 1639 Services", RFC 2475, December 1998. 1641 [RFC2991] Thaler, D. and C. Hopps, "Multipath Issues in Unicast and 1642 Multicast Next-Hop Selection", RFC 2991, November 2000. 1644 [RFC2992] Hopps, C., "Analysis of an Equal-Cost Multi-Path 1645 Algorithm", RFC 2992, November 2000. 1647 [RFC3260] Grossman, D., "New Terminology and Clarifications for 1648 Diffserv", RFC 3260, April 2002. 1650 [RFC3468] Andersson, L. and G. Swallow, "The Multiprotocol Label 1651 Switching (MPLS) Working Group decision on MPLS signaling 1652 protocols", RFC 3468, February 2003. 1654 [RFC3945] Mannie, E., "Generalized Multi-Protocol Label Switching 1655 (GMPLS) Architecture", RFC 3945, October 2004. 1657 [RFC3985] Bryant, S. and P. Pate, "Pseudo Wire Emulation Edge-to- 1658 Edge (PWE3) Architecture", RFC 3985, March 2005. 1660 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 1661 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 1662 Use over an MPLS PSN", RFC 4385, February 2006. 1664 [RFC4655] Farrel, A., Vasseur, J., and J. Ash, "A Path Computation 1665 Element (PCE)-Based Architecture", RFC 4655, August 2006. 1667 [RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal 1668 Cost Multipath Treatment in MPLS Networks", BCP 128, 1669 RFC 4928, June 2007. 1671 [RFC5151] Farrel, A., Ayyangar, A., and JP. Vasseur, "Inter-Domain 1672 MPLS and GMPLS Traffic Engineering -- Resource Reservation 1673 Protocol-Traffic Engineering (RSVP-TE) Extensions", 1674 RFC 5151, February 2008. 1676 [RFC5152] Vasseur, JP., Ayyangar, A., and R. Zhang, "A Per-Domain 1677 Path Computation Method for Establishing Inter-Domain 1678 Traffic Engineering (TE) Label Switched Paths (LSPs)", 1679 RFC 5152, February 2008. 1681 [RFC5316] Chen, M., Zhang, R., and X. Duan, "ISIS Extensions in 1682 Support of Inter-Autonomous System (AS) MPLS and GMPLS 1683 Traffic Engineering", RFC 5316, December 2008. 1685 [RFC5392] Chen, M., Zhang, R., and X. Duan, "OSPF Extensions in 1686 Support of Inter-Autonomous System (AS) MPLS and GMPLS 1687 Traffic Engineering", RFC 5392, January 2009. 1689 [RFC5441] Vasseur, JP., Zhang, R., Bitar, N., and JL. Le Roux, "A 1690 Backward-Recursive PCE-Based Computation (BRPC) Procedure 1691 to Compute Shortest Constrained Inter-Domain Traffic 1692 Engineering Label Switched Paths", RFC 5441, April 2009. 1694 [RFC5920] Fang, L., "Security Framework for MPLS and GMPLS 1695 Networks", RFC 5920, July 2010. 1697 [RFC5921] Bocci, M., Bryant, S., Frost, D., Levrau, L., and L. 1698 Berger, "A Framework for MPLS in Transport Networks", 1699 RFC 5921, July 2010. 1701 Authors' Addresses 1703 So Ning 1704 Verizon 1705 2400 N. Glenville Ave. 1706 Richardson, TX 75082 1708 Phone: +1 972-729-7905 1709 Email: ning.so@verizonbusiness.com 1711 Dave McDysan 1712 Verizon 1713 22001 Loudoun County PKWY 1714 Ashburn, VA 20147 1716 Email: dave.mcdysan@verizon.com 1718 Eric Osborne 1719 Cisco 1721 Email: eosborne@cisco.com 1723 Lucy Yong 1724 Huawei USA 1725 1700 Alma Dr. Suite 500 1726 Plano, TX 75075 1728 Phone: +1 469-229-5387 1729 Email: lucyyong@huawei.com 1731 Curtis Villamizar 1732 Outer Cape Cod Network Consulting 1734 Email: curtis@occnc.com