idnits 2.17.1 draft-yong-pwe3-enhance-ecmp-lfat-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** You're using the IETF Trust Provisions' Section 6.b License Notice from 12 Sep 2009 rather than the newer Notice from 28 Dec 2009. (See https://trustee.ietf.org/license-info/) Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 5, 2010) is 5159 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Obsolete normative reference: RFC 2460 (Obsoleted by RFC 8200) ** Obsolete normative reference: RFC 3697 (Obsoleted by RFC 6437) ** Downref: Normative reference to an Informational RFC: RFC 3985 == Outdated reference: A later version (-07) exists of draft-ietf-pwe3-fat-pw-03 == Outdated reference: A later version (-02) exists of draft-kompella-mpls-entropy-label-01 == Outdated reference: A later version (-03) exists of draft-carpenter-flow-ecmp-01 Summary: 4 errors (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group L. Yong Ed. 2 Internet Draft P. L. Yang 3 Intended status: Standards Track Huawei 4 Expires: Sept. 2010 March 5, 2010 6 Enhanced ECMP and Large Flow Aware Transport 7 draft-yong-pwe3-enhance-ecmp-lfat-01.txt 9 Status of this Memo 11 This Internet-Draft is submitted to IETF in full conformance with 12 the provisions of BCP 78 and BCP 79. 14 Internet-Drafts are working documents of the Internet Engineering 15 Task Force (IETF), its areas, and its working groups. Note that 16 other groups may also distribute working documents as Internet- 17 Drafts. 19 Internet-Drafts are draft documents valid for a maximum of six 20 months and may be updated, replaced, or obsoleted by other 21 documents at any time. It is inappropriate to use Internet- 22 Drafts as reference material or to cite them other than as "work 23 in progress." 25 The list of current Internet-Drafts can be accessed at 26 http://www.ietf.org/ietf/1id-abstracts.txt 28 The list of Internet-Draft Shadow Directories can be accessed at 29 http://www.ietf.org/shadow.html 31 This Internet-Draft will expire on September 4, 2010. 33 Copyright Notice 35 Copyright (c) 2010 IETF Trust and the persons identified as the 36 document authors. All rights reserved. 38 This document is subject to BCP 78 and the IETF Trust's Legal 39 Provisions Relating to IETF Documents 40 (http://trustee.ietf.org/license-info) in effect on the date of 41 publication of this document. Please review these documents 42 carefully, as they describe your rights and restrictions with 43 respect to this document. Code Components extracted from this 44 document must include Simplified BSD License text as described in 45 Section 4.e of the Trust Legal Provisions and are provided 46 without warranty as described in the BSD License. 48 Abstract 50 Internet Traffic has constantly shown the pattern that a very 51 small amount of the traffic flows generate a high traffic volume 52 while a significant amount of small flows contribute a small 53 amount of traffic volume. Differentiating such large flow and 54 small flow in the packet switched network enables an enhanced 55 transport method over Equal Cost Multi Paths (ECMP). This draft 56 describes the enhanced ECMP transport with the large flow 57 awareness. 59 Table of Contents 61 1. Introduction...................................................2 62 2. Conventions used in this document..............................4 63 2.1. Terminology...............................................4 64 3. Large Flow Recognition.........................................4 65 4. Enhanced ECMP Process..........................................5 66 4.1. Congestion Control........................................7 67 5. Large Flow Indication..........................................8 68 6. Backward Compatibility.........................................9 69 7. Applicability..................................................9 70 7.1. Link Aggregation Groups..................................10 71 7.2. The Single Large Flow Case...............................10 72 7.3. Flow Rate Difference.....................................10 73 7.4. Multi-Segment Pseudowires................................11 74 7.5. IP Flows.................................................11 75 7.6. LSP with Entropy Label...................................11 76 8. Security Considerations.......................................12 77 9. IANA Considerations...........................................12 78 10. References...................................................12 79 10.1. Normative References....................................12 80 10.2. Informative References..................................13 81 11. Acknowledgments..............................................13 82 Appendix A. Simulation Analysis..................................14 84 1. Introduction 86 [FAT-PW] introduces the flow label on the label stack for some 87 pseudowires (PW) to take the advantage of ECMP transport. The 88 method inserts a flow label on each packet at ingress PE. The 89 ECMP process in the packet switched network (PSN) hashes the 90 label stack that contains the flow label. As a result, individual 91 flows in a PW can be transported over different ECMP paths. Since 92 the packets that belong to the same flow have the same label 93 value, the method gets ECMP transport benefit as well as 94 preserves the ordering of each individual transported IP flow. 96 However, the traffic over Internet today includes Web browsing 97 data and audio as well as video/downloading and streaming. 98 Video/downloading and streaming generates the very high rate 99 flows compared to Web browsing data/audio. This causes Internet 100 traffic clearly mixed with huge amount of small flows and small 101 amount of very high rate flows. Internet traffic analysis [CAIDA] 102 indicates that, today, ~2% of the top rate ranked flows takes 103 about 30% of traffic volume while the rest of 98% flows 104 contribute 70% of traffic volume. As Web HDTV and 3D TV will be 105 on the Internet, the traffic volume ratio between large and small 106 flows may be further higher. Although the flow label can 107 improve the load balancing per the flow basis within a 108 pseudowire, under such traffic pattern, hash based distribution 109 is inadequate for satisfactory load balancing. 111 Hash based distribution ensures any flow to be mapped into only 112 one of ECMP paths (fixed one) so the flow ordering is preserved 113 in the transport. However, hash based distribution disperses all 114 the possible flow identifiers over ECMP paths no matter a flow 115 exists or not at the time and does not consider individual flow 116 rate, i.e. it has the nature of stateless distribution. Such 117 distribution method generates adequate load balancing if the 118 traffic contains huge amount of flows that have similar flow 119 rates. The simulation has shown that given Internet traffic 120 pattern, the hash method does not evenly distribute the flows 121 over ECMP paths. The load difference between two of ECMP paths 122 can be significant large; the more ECMP paths exist, the more 123 severe the un-balancing syndrome presents. This implies that hash 124 based distribution can cause some path congested and some being 125 partial filled only. This results that the packets are dropped at 126 congestion point and the network reroute impacted traffic. In 127 other words, this syndrome lowers the network performance and 128 brings operator desires to improve load balancing over ECMP. One 129 option to prevent such syndrome is to add more transport resource 130 in the network. But this will lower the network utilization and 131 increase the service cost. 133 This draft describes an enhanced ECMP method for such traffic 134 pattern and also introduces the large/small flow indication on 135 the flow label to facilitate enhanced ECMP transport in PSN. The 136 enhanced method uses a table for a small amount of large flow 137 distribution and hashing on all other flows. The method gets 138 evenly load balance by maintaining a small set of large flow 139 states. The draft states the process procedures on PE and P 140 routers. 142 The simulation result has shown that the enhanced ECMP gets much 143 better improvement on load-balancing compared to hash based ECMP 144 under Internet traffic pattern. The load difference among paths 145 is less than 1%. 147 2. Conventions used in this document 149 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL 150 NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and 151 "OPTIONAL" in this document are to be interpreted as described in 152 [RFC2119]. 154 2.1. Terminology 156 Large Flow: a group of packets that contain the same "identity" 157 in their header and come to the network at a high rate, i.e. the 158 packet volume per time is high. 160 Small Flow: a group of packets that contain the same "identity" 161 in their header and come to the network at a low rate, i.e. the 162 packet volume per time is very low. Single packet can be 163 considered as a special small flow in the context of this 164 document. 166 3. Large Flow Recognition 168 The high technology now enables router devices to inspect the 169 received the packets and identifies the large flows from huge 170 amount of packets that belong to many flows (both large and 171 small). Large flow recognition process may use protocol 172 inspection, flow volume measurement, or other methods to detect 173 the large flows. If a router can differentiate packets that 174 belong to the high rate flows from all the received packets, it 175 can perform differentiated transports for the large flows and 176 small flows in PSN as described in section 4. 178 It is possible for hosts to insert a large flow indication on the 179 packet header. However, there is a huge security concern for a 180 network to perform on the customer inserted indication. 182 Typically, a large flow has the context for an entire packet 183 switched network. It has obvious benefit that if ingress PE 184 performs the large flow recognition and inserts a large flow 185 indication on the packets, then all the P nodes within PSN can 186 distinguish the large flow packets by checking this indication. 187 This can largely reduce the implementation cost and the impact on 188 the performance. 190 The native service processing function (NSP) [RFC3985] in the 191 ingress PE can identify the flow or groups of flows in the 192 service, and insert the flow (group) identity of each packet 193 before it is passed to the pseudowire forwarder. When ingress PE 194 performs large flow recognition, the pseudowire forwarder 195 [RFC3985] can perform packet inspection and detect the large flow 196 packets. The design method for the large flow recognition is 197 outside the scope of this document. The pseudowire forwarder can 198 insert a large flow indication on all the packets that belong to 199 the large flow once it is recognized as a large flow. The large 200 flow indication encoding schema is described in section 5. Since 201 a large flow comes and disappears when it is transported 202 completely, the list of large flows could dynamically change. 204 Large Flow Recognition has the assumption that a large flow 205 sustains for certain time on the network. This assumption applies 206 video, streaming, and file download applications. Although 207 application rate may vary over the time, its lowest rate value is 208 still much high compared to the small flows. Operator can set the 209 large flow criteria. 211 It is worth to mention that a large flow recognition process may 212 or may not need a time to recognize a large flow. If it needs and 213 even the time is very short, during this period, some packets 214 belonging to a large flow may be treated as small flow packets, 215 which may cause the packets for a large flow traversing different 216 paths during the transition. Thus this may cause a bit packet 217 disordering at a destination. To prevent the packet disordering, 218 Large Flow Recognition Process can use some temporary caching 219 technology to hold the large flow packets for short time at the 220 time the flow is recognized as a new large flow. Another factor 221 to consider is that today packet based applications at the end 222 points normally have a buffer to deal with packet delay variance 223 and loss/disordering, thus the seldom disordering during 224 transport is no longer a BIG issue for Internet traffic. Some 225 large flow recognition may not need time to detect the large 226 flow; it does not generate the ordering issue. 228 4. Enhanced ECMP Process 230 Label switched routers can implement the enhanced ECMP for 231 distributing flows over ECMP paths. The enhanced ECMP process 232 separates the packets that belong to a large flow from the 233 packets that belong to a small flow and applies different 234 treatments on these two types of packets. The process uses 235 hashing to select the path from equal cost multi paths for all 236 the small flow packets and uses a large flow table to select the 237 path for all the large flow packets. Figure 1 illustrates the 238 enhanced ECMP processing diagram. 240 +-------------+ | 4 ECMP Paths 241 | Small-Flow | | 242 +--->| Forwarding |--->|========= 243 +------------+ | | Process | | 244 Packets| Packet | | +-------------+ |========= 245 ------>| Separation |---+ | 246 | Process |---+ |========= 247 +------------+ | +-------------+ | 248 | | Large-Flow | |========= 249 +--->| Forwarding |--->| 250 | Process | | 251 +-------------+ | 253 Figure 1 Enhanced ECMP Process Diagram 255 Figure 1 depicts three function elements. There are four equal 256 cost paths shown as an example. Small-Flow Forwarding Process is 257 used for forwarding all the small flow packets, which can be the 258 same as existing ECMP process. Packet Separation Process and 259 Larger-Flow Forwarding Process are the new elements in the 260 enhanced ECMP proposed in this document. The Packet Separation 261 Process receives all the transported packets and evaluates all 262 the income packets; it uses the first nibble to distinguish 263 labeled packets or IP packets. If a labeled packet is marked as a 264 large flow, it will be sent to Large-Flow Forwarding Process; if 265 not, it will be sent to Small-Flow Forwarding Process. As a 266 result, the small flow transport path will be determined by 267 hashing method; the large flow transport path will be determined 268 by Large-Flow Forwarding Process. Since this draft focuses on the 269 labeled packets, IP packet process is described in section 7.5. 271 Since the bottom label at the label stack (S bit = 1) can be PW 272 label, LSP label, or Flow Label, it is necessary for Packet 273 Separation Process to differ Flow Label from PW Label and LSP 274 Label. Since Flow Label is never on the top of the label stack 275 and is not processed by the forwarder, it has its unique 276 position. The draft suggests setting FL TTL to 0. Thus, when S 277 bit =1 and TTL = 0, the label is Flow Label. Otherwise, it is 278 either PW Label or LSP label. 280 Large-Flow Forwarding Process uses a flow table for packet 281 forwarding. The flow table has an entry for each "live" flow. 282 When the process receives a packet, it retrieves the flow ID from 283 the packet and performs the table lookup by using flow ID. It 284 forwards the packets to the path indicated in the table. If the 285 process does not find an entry that matches the flow ID on a 286 packet, it calls the path selection algorithm. The algorithm can 287 select a path for the flow, say A, based on current path load, 288 i.e. select the path that has least load at the time. Then the 289 process forwards the packet to the selected path and inserts a 290 new entry for the flow A in the table. The following packets of 291 flow A will be forwarded to the path indicated in the table. When 292 a flow is transported completely, the process no longer receives 293 the packets that belong to the flow; the age function in the 294 process can delete the flow entry from the table, which prevents 295 the table size from the unnecessary growth. The age process 296 frequency is configurable based on operation needs. If one of 297 ECMP paths is down the algorithm will map impacted large flows to 298 other ECMP paths. If a new ECMP path is added, the new flows can 299 be assigned to the new path; it is optional for the process to 300 perform the "live" large flow reassignment since the "live" flows 301 may disappear itself anyway. The design method of Large-Flow 302 Forwarding Process is outside the scope of this document. 304 Note: Large-Flow Forwarding Process can work with any hash-key 305 generation scheme. Large-Flow distribution method using few large 306 flows effectively compensates the uneven distribution caused by 307 hashing and traffic pattern. It is worth to mention the 308 differences between DiffDerv and large and small flow 309 differentiation. Although both relate to traffic classification, 310 they aim the different purposes. DiffDerv is for the network to 311 perform differentiated service treatments. Large and small flow 312 differentiation is to improve the network utilization in ECMP. 314 4.1. Congestion Control 316 The enhanced ECMP also brings an advance in congestion control. 317 The congestion happens when the traffic volume exceeds the path 318 capacity. Since the large flows take much more bandwidth, 319 dropping few large flows can efficiently rescue the congestion 320 condition and keep the rest of services running normally. As a 321 result, the congestion control only impacts few services. Large- 322 Flow Process can easily select the large flows and block them 323 during the congestion. Whether it is worth to cache these blocked 324 flows or not is for further study and outside the scope of this 325 document. 327 5. Large Flow Indication 329 This draft specifies the protocol to encode a large flow 330 indication on the flow label specified in [FAT-PW]. Figure 2 331 illustrates current flow label format [RFC3031] with the 332 amendment given in [RFC5462]. Label field is filled with the flow 333 identity. Since the flow label is never on the top of label 334 stack, TTL field is not used. However, to prevent any 335 provisioning error, TTL filed is recommended to set as 1. S bit 336 is used to indicate the bottom of stack and set to 1 for the flow 337 label. 3 Traffic Class bits are not used in current ECMP 338 processing now. The document suggests using the first bit in the 339 Traffic Class bits to indicate the large flow or small flow, and 340 suggests value 1 for the large flow and value 0 for the small 341 flow. The two other bits reserve for the future. Figure 3 shows 342 proposed format. 344 0 1 2 3 345 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 346 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 347 | Label | TC |S| TTL | 348 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 350 Label: Label Value, 20 bits 351 TC: Traffic Class, 3 bits; 352 S: Bottom of Stack, 1 bit 353 TTL: Time to Live, 8 bits 355 Figure 2 Current Flow Label Formant 357 0 1 2 3 358 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 359 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 360 | Label |F| RV|S| TTL | 361 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 363 Label: Label Value, 20 bits 364 F: Flow Characteristics Indication, 1 bit; 365 RV: Reserved Bit, set to 0 366 S: Bottom of Stack, 1 bit 367 TTL: Time to Live, 8 bits 369 Figure 3 Flow Label Format with Large-Flow Indication 371 When Flow Label is used on a PW, ingress PE can insert the flow 372 label and a large flow indication on each packet; egress PE will 373 trim off the flow label before sending the packets to the right 374 AC. The procedure for informing flow label presence and label 375 insertion procedure remains the same as [FAT-PW]. 377 TC bits in MPLS Label are typically used for DiffServ purpose. 378 [RFC3270] [RFC5129] Since Flow Label is never used in the top of 379 Label Stack, DiffServ function does not apply to the Flow Label. 380 Hence the proposed Flow Label TC bit usage in this document does 381 not impact DiffServ function. 383 6. Backward Compatibility 385 The enhanced ECMP fully support backward compatibility in PSN. If 386 ingress PE does not support Large Flow Recognition, it SHALL set 387 flow label F bit to 0. Then all the flows are treated as small 388 flows in PSN. P routers with existing ECMP or enhanced ECMP 389 capability use hashing to discriminate the flows and distribute 390 those flows over ECMP paths. If ingress PE supports Large Flow 391 Recognition, it will insert the indication on the flow label. The 392 P routers with existing ECMP capability will ignore the 393 indication and just perform hashing on all the flows. The P 394 routers with enhanced ECMP capability will separate the large and 395 small flows and perform different treatments as proposed in this 396 document. Although P router with existing ECMP capability gets 397 uneven load balancing over its ECMP paths, it maintains the same 398 performance as today's network. If ingress PE does not support 399 the flow label on PW, when enhanced ECMP applies, it will treat 400 PW packets or LSP packets as small flow packets. 402 7. Applicability 404 Carriers have desires to improve transport network capability via 405 certain service awareness in packet transport and not be 406 constrained in just "pipe" transport service.[FAT-PW]brings such 407 potential by introducing the flow label in the label stack, which 408 enables ECMP transport discriminates traffic at flow granularity. 409 The large flow aware transport further enables ECMP transport to 410 distinguish the large and small flows and perform different 411 treatments on two types of flows, which can improve the load 412 balancing when traffic pattern contains very small percentage of 413 large flows. 415 The method described in this document requires the new capability 416 from the PSN and applies to packet switched routers. It requires 417 ingress PE to perform the large flow recognition and inserts a 418 large flow indication on the flow label; and P or PE routers 419 perform the enhanced ECMP function. Since each router node 420 performs ECMP function independently, a packet switched network 421 can work well even when some nodes support the enhanced ECMP 422 capability and some do not. This allows operator to gradually 423 upgrade the network. 425 7.1. Link Aggregation Groups 427 A Link Aggregation Group (LAG) is used to bond together several 428 physical circuits between two adjacent nodes so they appear to 429 higher-layer protocols as a single, higher bandwidth "virtual" 430 pipe. These may co-exist in various parts of a given network. The 431 enhanced ECMP proposed in this document can assist in producing a 432 more uniform flow distribution and controlling the congestion in 433 LAG. 435 7.2. The Single Large Flow Case 437 [FAT-PW] has suggested several options for the single large flow 438 in a PW. With the enhanced ECMP capability, it has beneficial to 439 insert a flow label even for a single large flow. Then ingress PE 440 can insert a large flow indication. P routers in PSN can treat it 441 as a large flow. 443 7.3. Flow Rate Difference 445 The enhanced ECMP method uses the different treatments between 446 large flows and small flows. Neither of treatments considers the 447 flow rate in the distribution process. This is because that even 448 load balance is achieved by hashing on the small flows and 449 selecting the least used the path for a new "live" flow. The 450 latter distribution using few large flows effectively compensates 451 the uneven balance caused by the former and is unnecessary to 452 consider individual flow rate. This is nice that enhanced ECMP 453 keeps the nature of statistical balancing. Therefore, the 454 enhanced ECMP method works well even flow rates are broad. 456 7.4. Multi-Segment Pseudowires 458 The flow label mechanism described in this document works on 459 multi-segment PWs [MS-PW] without requiring modification to the 460 Switched PEs (S-PEs). This is because the flow label is 461 transparent to the label swap operation. There is no need to 462 perform Large Flow Recognition at Switched PEs. 464 7.5. IP Flows 466 Today's ECMP method applies to both IP flows and MPLS labeled 467 flows in PSN. Typically, Hash method uses IP source and 468 destination address pair plus other elements to discriminate IP 469 flows and distribute them over ECMP paths. If PE can insert a 470 large flow indication in the packets of IP flows, the proposed 471 method can apply to IP flows as well. IPv6 protocol [RFC2460] 472 already has the flow label field. 3-tuple: source, destination, 473 and flow label is used in ECMP.[RFC3697] The method proposed here 474 meets the restriction on the flow label.[FLOW-ECMP] IETF just 475 needs to specify one bit in TC field (8 bits) to indicate small 476 and large flow. By default, all TC bits are set as 0 [RFC2460], 477 which is compatible to this solution. Although IPv4 protocol does 478 not have such flow label, IETF can decide if it is necessary to 479 improve IPv4 protocol to have the large flow indication or just 480 wait the time for IPv6 to take over. The IP large flow 481 recognition and indication is outside the scope of this document. 483 The Packet Separation Process in the enhanced ECMP uses the first 484 nibble to differentiate IP flows and non IP flows before 485 evaluating the large flow indication. When PSN does not support 486 large and small IP flow distinction, the enhanced ECMP treats all 487 IP flows as small flows. 489 7.6. LSP with Entropy Label 491 Entropy Label [Entropy] is inserted in LSP traffic at ingress LSR 492 to gain better ECMP load balancing at transit LSRs. Entropy label 493 is very similar as PW flow label and is used to differentiate 494 "microflow" within a LSP so ECMP process can get better 495 dispersion granularity. Enhanced ECMP and Large Flow Aware 496 Transport can apply to LSP with entropy label. Traffic class 497 field in the Entropy can use the same encoding scheme described 498 in this document. If ingress LSR does not support large flow 499 recognition, then it SHOULD set Large Flow indication bit to 0. 501 The same approach applies to Application Label [RFC4928] as well. 503 8. Security Considerations 505 Since the number of large flows is very small compared to the 506 number of small flows; packet switched routers only need to 507 maintain a small size of table or flow states. Notes operator can 508 use the large flow criteria to control the large flow volume. The 509 method won't create the scalability and performance issue. 511 9. IANA Considerations 513 IANA is for the further study. 515 10. References 517 10.1. Normative References 519 [RFC2460] Deering, S., Hinden, R., "Internet Protocol, Version 6 520 (IPv6) Specification", RFC 2460, December 1995. 522 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 523 Requirement Levels", BCP 14, RFC 2119, March 1997. 525 [RFC3031] Rosen, E., Viswanathan, A., and Callon, R., "Multiple 526 protocol Label Switching Architecture", RFC3031, January 527 2001. 528 [RFC3270] Faucheur F. Le, etc, "Multi-Protocol Label Switching 529 (MPLS) Support of Differentiated Services", RFC3270, May 530 2002 532 [RFC3697] Rajahalme, J., Conta, A., Carpenter, B., and S. Deering, 533 "IPv6 Flow Label Specification", RFC 3697, March 2004. 535 [RFC3985] Bryant, S., Pate P., "Multiprotocol Label Switching 536 (MPLS) Label Stack Entry: "EXP" Field Renamed to 537 "Traffic Class" Field", RFC3985, March 2005 539 [RFC4928] Swallow, G., Bryant, S., and Andersson, L., "Avoiding 540 Equal Cost Multipath Treatment in MPLS Network", 541 RFC4928, June 2007. 542 [RFC5129] Davie, B., Briscoe, B., and Jay, J., "Explicit 543 Congestion Marking in MPLS", RFC5129, January 2008 545 [RFC5462] Andersson, L. etc, "Multiprotocol Label Switching 546 (MPLS) Label Stack Entry: EXP Field Renamed to Traffic 547 Class Field", October 2009 549 10.2. Informative References 551 [FAT-PW] Bryant, S., Drafz, U Kompella, V., etc, "Flow Aware 552 Transport of Pseudowires over an MPLS PSN", draft-ietf- 553 pwe3-fat-pw-03, (work in progress), Jan. 2010 555 [Entropy] Kompella K, Amante S., "The use Entropy Labels in MPLS 556 Forwarding", draft-kompella-mpls-entropy-label-01, 557 January 2009 559 [MS-PW] Bocci, M. Bryant, S., "An Architecture fro Multi-Segment 560 Pseudowire Emulation Edge-to-Edge", RFC5659 October 561 2009 562 [FLOW-ECMP] Carpenter B., "Using the IPv6 flow label for equal 563 cost multipath routing in tunnels", draft-carpenter- 564 flow-ecmp-01, February 18, 2010 566 [CAIDA] Caida Internet Traffic Analysis, 567 www.caida.org/data/monitor 569 11. Acknowledgments 571 Authors like to thank Stewart Bryan, Frederic Jounay, Simon 572 Delord, Raymond Key for the review and suggestions. 574 Appendix A. Simulation Analysis 576 We create Internet Traffic Generator based on observed Internet 577 Traffic pattern. The generator randomly generates 98% of small 578 traffic flows and 2% of large traffic flows up to 10G traffic. 579 The traffic volume for the large flows and small flows are 30% 580 and 70%. Simulator uses hash based distribution to disperse the 581 traffic over 4 paths and 10 paths, respectively; and also uses 582 enhanced ECMP method to disperse the traffic over 4 paths and 10 583 paths. The results show the performance between ECMP and enhanced 584 ECMP from 6 simulations. Enhanced ECMP gets <1% load differences 585 among paths while ECMP have up to 15% load differences. It shows 586 how the simple distribution on few large flows can effectively 587 compensate the uneven load balance caused by hashing and the 588 traffic pattern. 590 Authors' Addresses 592 Lucy Yong 593 Huawei Technologies Co., Ltd. 594 1700 Alma Dr. 595 Plano, TX 75075 596 US 598 Phone: +14692295387 599 Email: lucyyong@huawei.com 601 Peilin Yang 602 Huawei Technologies Co., Ltd. 603 No.91, Baixia Road, Nanjing 210001 604 P. R. China 606 Phone: +86-25-84565881 607 EMail: yangpeilin@huawei.com