idnits 2.17.1 draft-yong-pwe3-enhance-ecmp-lfat-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** You're using the IETF Trust Provisions' Section 6.b License Notice from 12 Sep 2009 rather than the newer Notice from 28 Dec 2009. (See https://trustee.ietf.org/license-info/) Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (February 17, 2010) is 5154 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Obsolete normative reference: RFC 2460 (Obsoleted by RFC 8200) ** Downref: Normative reference to an Informational RFC: RFC 3985 == Outdated reference: A later version (-07) exists of draft-ietf-pwe3-fat-pw-03 == Outdated reference: A later version (-02) exists of draft-kompella-mpls-entropy-label-01 Summary: 3 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group L. Yong Ed. 2 Internet Draft P. L. Yang 3 Intended status: Standards Track Huawei 4 Expires: Sept. 2010 February 17, 2010 6 Enhanced ECMP and Large Flow Aware Transport 7 draft-yong-pwe3-enhance-ecmp-lfat-00.txt 9 Status of this Memo 11 This Internet-Draft is submitted to IETF in full conformance with 12 the provisions of BCP 78 and BCP 79. 14 Internet-Drafts are working documents of the Internet Engineering 15 Task Force (IETF), its areas, and its working groups. Note that 16 other groups may also distribute working documents as Internet- 17 Drafts. 19 Internet-Drafts are draft documents valid for a maximum of six 20 months and may be updated, replaced, or obsoleted by other 21 documents at any time. It is inappropriate to use Internet- 22 Drafts as reference material or to cite them other than as "work 23 in progress." 25 The list of current Internet-Drafts can be accessed at 26 http://www.ietf.org/ietf/1id-abstracts.txt 28 The list of Internet-Draft Shadow Directories can be accessed at 29 http://www.ietf.org/shadow.html 31 This Internet-Draft will expire on August 17, 2010. 33 Copyright Notice 35 Copyright (c) 2010 IETF Trust and the persons identified as the 36 document authors. All rights reserved. 38 This document is subject to BCP 78 and the IETF Trust's Legal 39 Provisions Relating to IETF Documents 40 (http://trustee.ietf.org/license-info) in effect on the date of 41 publication of this document. Please review these documents 42 carefully, as they describe your rights and restrictions with 43 respect to this document. Code Components extracted from this 44 document must include Simplified BSD License text as described in 45 Section 4.e of the Trust Legal Provisions and are provided 46 without warranty as described in the BSD License. 48 Abstract 50 Internet Traffic has constantly shown the pattern that a very 51 small amount of the traffic flows generate a high traffic volume 52 while a significant amount of small flows contribute a small 53 amount of traffic volume. Differentiating such large flow and 54 small flow in the packet switched network enables an enhanced 55 transport method over Equal Cost Multi Paths (ECMP). This draft 56 describes the enhanced ECMP transport with the large flow 57 awareness. 59 Table of Contents 61 1. Introduction...................................................2 62 2. Conventions used in this document..............................4 63 2.1. Terminology...............................................4 64 3. Large Flow Recognition.........................................4 65 4. Enhanced ECMP Process..........................................5 66 4.1. Congestion Control........................................7 67 5. Large Flow Indication..........................................7 68 6. Backward Compatibility.........................................8 69 7. Applicability..................................................9 70 7.1. Link Aggregation Groups...................................9 71 7.2. The Single Large Flow Case...............................10 72 7.3. Flow Rate Difference.....................................10 73 7.4. Multi-Segment Pseudowires................................10 74 7.5. IP Flows.................................................10 75 7.6. Entropy Label............................................11 76 8. Security Considerations.......................................11 77 9. IANA Considerations...........................................11 78 10. References...................................................12 79 10.1. Normative References....................................12 80 10.2. Informative References..................................12 81 11. Acknowledgments..............................................13 82 Appendix A. Simulation Analysis..................................14 84 1. Introduction 86 [FAT-PW] introduces the flow label on the label stack for some 87 pseudowires (PW) to take the advantage of ECMP transport. The 88 method inserts a flow label on each packet at ingress PE. The 89 ECMP process in the packet switched network (PSN) hashes the 90 label stack that contains the flow label. As a result, individual 91 flows in a PW can be transported over different ECMP paths. Since 92 the packets that belong to the same flow have the same label 93 value, the method gets ECMP transport benefit as well as 94 preserves the ordering of each individual transported IP flow. 96 However, the traffic over Internet today includes Web browsing 97 data and audio as well as video/downloading and streaming. 98 Video/downloading and streaming generates the very high rate 99 flows compared to Web browsing data/audio. This causes Internet 100 traffic clearly mixed with huge amount of small flows and small 101 amount of very high rate flows. Internet traffic analysis [CAIDA] 102 indicates that, today, ~2% of the top rate ranked flows takes 103 about 30% of traffic volume while the rest of 98% flows 104 contribute 70% of traffic volume. As Web HDTV and 3D TV will be 105 on the Internet, the traffic volume ratio between large and small 106 flows may be further higher. Although the flow label can 107 improve the load balancing per the flow basis within a 108 pseudowire, under such traffic pattern, hash based distribution 109 is inadequate for satisfactory load balancing. 111 Hash based distribution ensures any flow to be mapped into only 112 one of ECMP paths (fixed one) so the flow ordering is preserved 113 in the transport. However, hash based distribution disperses all 114 the possible flow identifiers over ECMP paths no matter a flow 115 exists or not at the time and does not consider individual flow 116 rate, i.e. it has the nature of stateless distribution. Such 117 distribution method generates adequate load balancing if the 118 traffic contains huge amount of flows that have similar flow 119 rates. The simulation has shown that given Internet traffic 120 pattern, the hash method does not evenly distribute the flows 121 over ECMP paths. The load difference between two of ECMP paths 122 can be significant large; the more ECMP paths exist, the more 123 severe the un-balancing syndrome presents. This implies that hash 124 based distribution can cause some path congested and some being 125 partial filled only. This results that congestion impacted 126 traffic are rerouted dynamically while other equal cost paths are 127 under utilized. In other words, this syndrome lowers the network 128 performance and brings operator desires to improve load balancing 129 over ECMP. One option to prevent such syndrome is to add more 130 transport resource into the network. But this will lower the 131 network utilization and increase the service cost. 133 This draft describes an enhanced ECMP method for such traffic 134 pattern and also introduces the large/small flow indication on 135 the flow label to facilitate enhanced ECMP transport in PSN. The 136 enhanced method uses a table for a small amount of large flow 137 distribution and hashing on all other flows. The method gets 138 evenly load balance by maintaining a small set of large flow 139 states. The draft states the process procedures on PE and P 140 routers. 142 The simulation result has shown that the enhanced ECMP gets much 143 better improvement on load-balancing compared to hash based ECMP 144 under Internet traffic pattern. The load difference among paths 145 is less than 1%. 147 2. Conventions used in this document 149 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL 150 NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and 151 "OPTIONAL" in this document are to be interpreted as described in 152 [RFC2119]. 154 2.1. Terminology 156 Large Flow: a group of packets that contain the same "identity" 157 in their header and come to the network at a high rate, i.e. the 158 packet volume per time is high. 160 Small Flow: a group of packets that contain the same "identity" 161 in their header and come to the network at a low rate, i.e. the 162 packet volume per time is very low. Single packet can be 163 considered as a special small flow in the context of this 164 document. 166 3. Large Flow Recognition 168 The high technology now enables router devices to inspect the 169 received the packets and identifies the large flows from huge 170 amount of packets that belong to many flows (both large and 171 small). Large flow recognition process may use protocol 172 inspection, flow volume measurement, or other methods to detect 173 the large flows. If a router can differentiate packets that 174 belong to the high rate flows from all the received packets, it 175 can perform differentiated transports for the large flows and 176 small flows in PSN as described in section 4. 178 It is possible for hosts to insert a large flow indication on the 179 packet header. However, there is a huge security concern for a 180 network to perform on the customer inserted indication. 182 Typically, a large flow has the context for an entire packet 183 switched network. It has obvious benefit that if ingress PE 184 performs the large flow recognition and inserts a large flow 185 indication on the packets, then all the P nodes within PSN can 186 distinguish the large flow packets by checking this indication. 187 This can largely reduce the implementation cost and the impact on 188 the performance. 190 The native service processing function (NSP) [RFC3985] in the 191 ingress PE can identify the flow or groups of flows in the 192 service, and insert the flow (group) identity of each packet 193 before it is passed to the pseudowire forwarder. When ingress PE 194 performs large flow recognition, the pseudowire forwarder 195 [RFC3985] can perform packet inspection and detect the large flow 196 packets. The design method for the large flow recognition is 197 outside the scope of this document. The pseudowire forwarder can 198 insert a large flow indication on all the packets that belong to 199 the large flow once it is recognized as a large flow. The large 200 flow indication encoding schema is described in section 5. Since 201 a large flow comes and disappears when it is transported 202 completely, the list of large flows could dynamically change. 204 Large Flow Recognition has the assumption that a large flow 205 sustains for certain time on the network. This assumption applies 206 video, streaming, and file download applications. Although 207 application rate may vary over the time, its lowest rate value is 208 still much high compared to the small flows. Operator can set the 209 large flow criteria. 211 4. Enhanced ECMP Process 213 Label switched routers can implement the enhanced ECMP for 214 distributing flows over ECMP paths. The enhanced ECMP process 215 separates the packets that belong to a large flow from the 216 packets that belong to a small flow and applies different 217 treatments on these two types of packets. The process uses 218 hashing to select the path from equal cost multi paths for all 219 the small flow packets and uses a large flow table to select the 220 path for all the large flow packets. Figure 1 illustrates the 221 enhanced ECMP processing diagram. 223 +-------------+ | 4 ECMP Paths 224 | Small-Flow | | 225 +--->| Forwarding |--->|========= 226 +------------+ | | Process | | 227 Packets| Packet | | +-------------+ |========= 228 ------>| Separation |---+ | 229 | Process |---+ |========= 230 +------------+ | +-------------+ | 231 | | Large-Flow | |========= 232 +--->| Forwarding |--->| 233 | Process | | 234 +-------------+ | 236 Figure 1 Enhanced ECMP Process Diagram 238 Figure 1 depicts three function elements. There are four equal 239 cost paths shown as an example. Small-Flow Forwarding Process is 240 used for forwarding all the small flow packets, which can be the 241 same as existing ECMP process. Packet Separation Process and 242 Larger-Flow Forwarding Process are the new elements in the 243 enhanced ECMP proposed in this document. The Packet Separation 244 Process receives all the transported packets and evaluates all 245 the income packets; it uses the first nibble to distinguish 246 labeled packets or IP packets. If a labeled packet is marked as a 247 large flow, it will be sent to Large-Flow Forwarding Process; if 248 not, it will be sent to Small-Flow Forwarding Process. As a 249 result, the small flow transport path will be determined by 250 hashing method; the large flow transport path will be determined 251 by Large-Flow Forwarding Process. Since this draft focuses on the 252 labeled packets, IP packet process is described in section 7.5. 254 Large-Flow Forwarding Process uses a flow table for packet 255 forwarding. The flow table has an entry for each "live" flow. 256 When the process receives a packet, it retrieves the flow ID from 257 the packet and performs the table lookup by using flow ID. It 258 forwards the packets to the path indicated in the table. If the 259 process does not find an entry that matches the flow ID on a 260 packet, it calls the path selection algorithm. The algorithm can 261 select a path for the flow, say A, based on current path load, 262 i.e. select the path that has least load at the time. Then the 263 process forwards the packet to the selected path and inserts a 264 new entry for the flow A in the table. The following packets of 265 flow A will be forwarded to the path indicated in the table. When 266 a flow is transported completely, the process no longer receives 267 the packets that belong to the flow; the age function in the 268 process can delete the flow entry from the table, which prevents 269 the table size from the unnecessary growth. The age process 270 frequency is configurable based on operation needs. If one of 271 ECMP paths is down the algorithm will map impacted large flows to 272 other ECMP paths. If a new ECMP path is added, the new flows can 273 be assigned to the new path; it is optional for the process to 274 perform the "live" large flow reassignment since the "live" flows 275 may disappear itself anyway. The design method of Large-Flow 276 Forwarding Process is outside the scope of this document. 278 Note: Large-Flow Forwarding Process can work with any hash-key 279 generation scheme. Large-Flow distribution method using few large 280 flows effectively compensates the uneven distribution caused by 281 hashing and traffic pattern. 283 4.1. Congestion Control 285 The enhanced ECMP also brings an advance in congestion control. 286 The congestion happens when the traffic volume exceeds the path 287 capacity. Since the large flows take much more bandwidth, 288 dropping few large flows can efficiently rescue the congestion 289 condition and keep the rest of services running normally. As a 290 result, the congestion control only impacts few services. Large- 291 Flow Process can easily select the large flows and block them 292 during the congestion. Whether it is worth to cache these blocked 293 flows or not is for further study and outside the scope of this 294 document. 296 5. Large Flow Indication 298 This draft specifies the protocol to encode a large flow 299 indication on the flow label specified in [FAT-PW]. Figure 2 300 illustrates current flow label format [RFC3031] with the 301 amendment given in [RFC5462]. Label field is filled with the flow 302 identity. Since the flow label is never on the top of label 303 stack, TTL field is not used. However, to prevent any 304 provisioning error, TTL filed is recommended to set as 1. S bit 305 is used to indicate the bottom of stack and set to 1 for the flow 306 label. 3 Traffic Class bits are not used in current ECMP 307 processing now. The document suggests using the first bit in the 308 Traffic Class bits to indicate the large flow or small flow, and 309 suggests value 1 for the large flow and value 0 for the small 310 flow. The two other bits reserve for the future. Figure 3 shows 311 proposed format. 313 0 1 2 3 314 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 315 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 316 | Label | TC |S| TTL | 317 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 319 Label: Label Value, 20 bits 320 TC: Traffic Class, 3 bits; 321 S: Bottom of Stack, 1 bit 322 TTL: Time to Live, 8 bits 324 Figure 2 Current Flow Label Formant 326 0 1 2 3 327 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 328 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 329 | Label |F| RV|S| TTL | 330 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 332 Label: Label Value, 20 bits 333 F: Flow Characteristics Indication, 1 bit; 334 RV: Reserved Bit, set to 0 335 S: Bottom of Stack, 1 bit 336 TTL: Time to Live, 8 bits 338 Figure 3 Flow Label Format with Large-Flow Indication 340 When flow label presents on a PW, ingress PE can insert the flow 341 label and a large flow indication on each packet; egress PE will 342 trim off the flow label before sending the packets to the right 343 AC. The procedure for informing flow label presence and label 344 insertion procedure remains the same as [FAT-PW]. 346 6. Backward Compatibility 348 The enhanced ECMP fully support backward compatibility in PSN. If 349 ingress PE does not support Large Flow Recognition, it SHALL set 350 flow label F bit to 0. Then all the flows are treated as small 351 flows in PSN. P routers with existing ECMP or enhanced ECMP 352 capability use hashing to discriminate the flows and distribute 353 those flows over ECMP paths. If ingress PE supports Large Flow 354 Recognition, it will insert the indication on the flow label. The 355 P routers with existing ECMP capability will ignore the 356 indication and just perform hashing on all the flows. The P 357 routers with enhanced ECMP capability will separate the large and 358 small flows and perform different treatments as proposed in this 359 document. Although P router with existing ECMP capability gets 360 uneven load balancing over its ECMP paths, it maintains the same 361 performance as today's network. If ingress PE does not support 362 the flow label on PW, when ECMP applies, the PW label serves as 363 flow label purpose, operator can decide if the PW should be 364 treated as small flow or large flow within PSN, then ingress PE 365 can set F bit based on the operator decision. The default value 366 SHOULD set to 0, i.e. as a small flow. 368 7. Applicability 370 Carriers have desires to improve transport network capability via 371 certain service awareness in packet transport and not be 372 constrained in just "pipe" transport service.[FAT-PW]brings such 373 potential by introducing the flow label in the label stack, which 374 enables ECMP transport discriminates traffic at flow granularity. 375 The large flow aware transport further enables ECMP transport to 376 distinguish the large and small flows and perform different 377 treatments on two types of flows, which can improve the load 378 balancing when traffic pattern contains very small percentage of 379 large flows. 381 The method described in this document requires the new capability 382 from the PSN and applies to packet switched routers. It requires 383 ingress PE to perform the large flow recognition and inserts a 384 large flow indication on the flow label; and P or PE routers 385 perform the enhanced ECMP function. Since each router node 386 performs ECMP function independently, a packet switched network 387 can work well even when some nodes support the enhanced ECMP 388 capability and some do not. This allows operator to gradually 389 upgrade the network. 391 7.1. Link Aggregation Groups 393 A Link Aggregation Group (LAG) is used to bond together several 394 physical circuits between two adjacent nodes so they appear to 395 higher-layer protocols as a single, higher bandwidth "virtual" 396 pipe. These may co-exist in various parts of a given network. The 397 enhanced ECMP proposed in this document can assist in producing a 398 more uniform flow distribution and controlling the congestion in 399 LAG. 401 7.2. The Single Large Flow Case 403 [FAT-PW] has suggested several options for the single large flow 404 in a PW. With the enhanced ECMP capability, it has beneficial to 405 insert a flow label even for a single large flow. Then ingress PE 406 can insert a large flow indication. P routers in PSN can treat it 407 as a large flow. 409 7.3. Flow Rate Difference 411 The enhanced ECMP method uses the different treatments between 412 large flows and small flows. Neither of treatments considers the 413 flow rate in the distribution process. This is because that even 414 load balance is achieved by hashing on the small flows and 415 selecting the least used the path for a new "live" flow. The 416 latter distribution using few large flows effectively compensates 417 the uneven balance caused by the former and is unnecessary to 418 consider individual flow rate. This is nice that enhanced ECMP 419 keeps the nature of statistical balancing. Therefore, the 420 enhanced ECMP method works well even flow rates are broad. 422 7.4. Multi-Segment Pseudowires 424 The flow label mechanism described in this document works on 425 multi-segment PWs [MS-PW] without requiring modification to the 426 Switched PEs (S-PEs). This is because the flow label is 427 transparent to the label swap operation. There is no need to 428 perform Large Flow Recognition at Switched PEs. 430 7.5. IP Flows 432 Today's ECMP method applies to both IP flows and MPLS labeled 433 flows in PSN. Typically, Hash method uses IP source and 434 destination address pair plus other elements to discriminate IP 435 flows and distribute them over ECMP paths. If PE can insert a 436 large flow indication in the packets of IP flows, the proposed 437 method can apply to IP flows as well. IPv6 protocol [RFC2460] 438 already has the flow label field. Although IPv4 protocol does not 439 have such flow label, IETF can decide if it is necessary to 440 improve IPv4 protocol to have the large flow indication or just 441 wait the time for IPv6 to take over. The IP large flow 442 recognition and indication is outside the scope of this document. 444 The Packet Separation Process in the enhanced ECMP uses the first 445 nibble to differentiate IP flows and non IP flows before 446 evaluating the large flow indication. When PSN does not support 447 large and small IP flow distinction, the enhanced ECMP treats all 448 IP flows as small flows. 450 7.6. Entropy Label 452 Entropy Label [Entropy] is inserted in LSP traffic at ingress LSR 453 to gain better ECMP load balancing at transit LSRs. Entropy label 454 is very similar as PW flow label and is used to differentiate 455 "microflow" within a LSP so ECMP process can get better 456 dispersion granularity. Enhanced ECMP and Large Flow Aware 457 Transport can apply to LSP with entropy label. Traffic class 458 field in the Entropy can use the same encoding scheme described 459 in this document. If ingress LSR does not support large flow 460 recognition, then it SHOULD set Large Flow indication bit to 0. 462 The same approach applies to Application Label [RFC4928] as well. 464 8. Security Considerations 466 A large flow recognition process may or may not need a time to 467 recognize a large flow. If it needs and even the time is very 468 short, during this period, some packets belonging to a large flow 469 may be treated as small flow packets, which may cause the packets 470 for a large flow traversing different paths during the 471 transition. Thus this may cause a bit packet disordering at a 472 destination. If it is necessary, Large Flow Recognition Process 473 can use some temporary caching technology to hold the large flow 474 packets for short time at the time the flow is recognized as a 475 new large flow. Another factor to consider is that today packet 476 based applications at the end points normally have a buffer to 477 deal with packet delay variance and loss/mis-order, therefore the 478 seldom mis-ordering during transport is no longer an BIG issue 479 for Internet traffic. Some large flow recognition may not need 480 time to detect the large flow; it does not generate the mis- 481 ordering issue. 483 Since the number of large flows is very small compared to the 484 number of small flows; packet switched routers only need to 485 maintain a small size of table or flow states. Notes operator can 486 use the large flow criteria to control the large flow volume. The 487 method won't create the scalability and performance issue. 489 9. IANA Considerations 491 IANA is for the further study. 493 10. References 495 10.1. Normative References 497 [RFC2460] Deering, S., Hinden, R., "Internet Protocol, Version 6 498 (IPv6) Specification", RFC 2460, December 1995. 500 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 501 Requirement Levels", BCP 14, RFC 2119, March 1997. 503 [RFC3031] Rosen, E., Viswanathan, A., and Callon, R., "Multiple 504 protocol Label Switching Architecture", RFC3031, January 505 2001. 507 [RFC3985] Bryant, S., Pate P., "Multiprotocol Label Switching 508 (MPLS) Label Stack Entry: "EXP" Field Renamed to 509 "Traffic Class" Field", RFC3985, March 2005. 511 [RFC4928] Swallow, G., Bryant, S., Andersson, L "Avoiding Equal 512 Cost Multipath Treatment in MPLS Network", RFC4928, 513 June 2007. 515 [RFC5462] Andersson, L. Asati R.., "An Architecture fro Multi- 516 Segment Pseudowire Emulation Edge-to-Edge", October 517 2009 519 10.2. Informative References 521 [FAT-PW] Bryant, S., Drafz, U Kompella, V., etc, "Flow Aware 522 Transport of Pseudowires over an MPLS PSN", draft-ietf- 523 pwe3-fat-pw-03, (work in progress), Jan. 2010 525 [Entropy] Kompella K, Amante S., "The use Entropy Labels in MPLS 526 Forwarding", draft-kompella-mpls-entropy-label-01, 527 January 2009 529 [MS-PW] Bocci, M. Bryant, S., "An Architecture fro Multi-Segment 530 Pseudowire Emulation Edge-to-Edge", RFC5659 October 531 2009 533 [CAIDA] Caida Internet Traffic Analysis, 534 www.caida.org/data/monitor 536 11. Acknowledgments 538 Authors like to thank Stewart Bryan for the review and 539 suggestions. 541 Appendix A. Simulation Analysis 543 We create Internet Traffic Generator based on observed Internet 544 Traffic pattern. The generator randomly generates 98% of small 545 traffic flows and 2% of large traffic flows up to 10G traffic. 546 The traffic volume for the large flows and small flows are 30% 547 and 70%. Simulator uses hash based distribution to disperse the 548 traffic over 4 paths and 10 paths, respectively; and also uses 549 enhanced ECMP method to disperse the traffic over 4 paths and 10 550 paths. The results show the performance between ECMP and enhanced 551 ECMP from 6 simulations. Enhanced ECMP gets <1% load differences 552 among paths while ECMP have up to 15% load differences. It shows 553 how the simple distribution on few large flows can effectively 554 compensate the uneven load balance caused by hashing and the 555 traffic pattern. 557 Authors' Addresses 559 Lucy Yong 560 Huawei Technologies Co., Ltd. 561 1700 Alma Dr. 562 Plano, TX 75075 563 US 565 Phone: +14692295387 566 Email: lucyyong@huawei.com 568 Peilin Yang 569 Huawei Technologies Co., Ltd. 570 No.91, Baixia Road, Nanjing 210001 571 P. R. China 573 Phone: +86-25-84565881 574 EMail: yangpeilin@huawei.com