idnits 2.17.1 draft-hopps-ecmp-algo-analysis-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** Missing expiration date. The document expiration date should appear on the first and last page. ** The document seems to lack a 1id_guidelines paragraph about 6 months document validity -- however, there's a paragraph with a matching beginning. Boilerplate error? == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (13 April 1999) is 9145 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Possible downref: Non-RFC (?) normative reference: ref. '1' ** Obsolete normative reference: RFC 2362 (ref. '2') (Obsoleted by RFC 4601, RFC 5059) Summary: 7 errors (**), 0 flaws (~~), 2 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Internet Engineering Task Force Christian E. Hopps 2 INTERNET-DRAFT Merit Network 3 Expires September 1999 13 April 1999 5 Analysis of an Equal-Cost Multi-Path Algorithm 6 8 Status of this Memo 10 This document is an Internet-Draft and is in full conformance with 11 all provisions of Section 10 of RFC2026. 13 Internet-Drafts are working documents of the Internet Engineering 14 Task Force (IETF), its areas, and its working groups. Note that 15 other groups may also distribute working documents as Internet- 16 Drafts. 18 Internet-Drafts are draft documents valid for a maximum of six months 19 and may be updated, replaced, or obsoleted by other documents at any 20 time. It is inappropriate to use Internet- Drafts as reference 21 material or to cite them other than as "work in progress." 23 The list of current Internet-Drafts can be accessed at 24 http://www.ietf.org/ietf/1id-abstracts.txt 26 The list of Internet-Draft Shadow Directories can be accessed at 27 http://www.ietf.org/shadow.html. 29 Copyright Notice 31 Copyright (C) The Internet Society (1999). All Rights Reserved. 33 Abstract 35 Equal-cost multi-path (ECMP) is a routing technique for routing 36 packets along multiple paths of equal cost. The forwarding engine 37 identifies paths by next-hop. When forwarding a packet the router 38 must decide which next-hop (path) to use. This document gives an 39 analysis of one method for making that decision. The analysis 40 includes the performance of the algorithm and the disruption caused 41 by changes to the set of next-hops. 43 1. Hash-Threshold 45 One method for determining which next-hop to use when routing with 46 ECMP can be called hash-threshold. The router first selects a key by 47 performing a hash (e.g., modulo-K where K is large, or CRC16) over 48 the packet header fields that identify a flow. The N next-hops have 49 been assigned unique regions in the key space. The router uses the 50 key to determine which region and thus which next-hop to use. 52 As an example of hash-threshold, upon receiving a packet the router 53 performs a CRC16 on the packet's header fields that define the flow 54 (e.g., the source and destination fields of the packet), this is the 55 key. Say for this destination there are 4 next-hops to choose from. 56 Each next-hop is assigned a region in 16 bit space (the key space). 57 For equal usage the router may have chosen to divide it up evenly so 58 each region is 65536/4 or 16k large. The next-hop is chosen by 59 determining which region contains the key (i.e., the CRC result). 61 2. Analysis 63 There are a few concerns when choosing an algorithm for deciding 64 which next-hop to use. One is performance, the computational 65 requirements to run the algorithm. Another is disruption (i.e., the 66 changing of which path a flow uses). Balancing is a third concern; 67 however since the algorithm's balancing characteristics are directly 68 related to the chosen hash function this analysis does not treat this 69 concern in depth. 71 For this analysis we will assume regions of equal size. If the hash 72 function is uniformly distributed the distribution of flows amongst 73 paths will also be uniform. 75 2.1. Performance 77 The performance of the hash-threshold algorithm can be broken down 78 into three parts: selection of regions for the next-hops, obtaining 79 the key and comparing the key to the regions to decide which next-hop 80 to use. 82 Since regions are restricted to be of equal size the calculation of 83 region boundaries is trivial. The boundaries can be calculated as 84 follows: 86 i = 0; 87 regionssize = keyspace.size / #{ next-hops } 88 for n in { next-hops } 89 n.start = i * regionsize; 90 n.end = n.start + regionsize; 91 i = i + 1 92 done 94 This calculation is O(N). Furthermore the calculation can be done 95 when the next-hops are added to or removed from the destination 96 route. 98 The algorithm doesn't specify the hash function used to obtain the 99 key. Its performance in this area will be exactly the performance of 100 the hash function. It is presumed that if this calculation proves to 101 be a concern it can be done in hardware parallel to other operations 102 that need to complete before deciding which next-hop to use. 104 Finally the next-hop must be chosen. This is done by determining 105 which region contains the key. The time required to do this is 106 dependent on the way the next-hops are organized in memory. The 107 problem reduces to a search. For example if the next-hops are stored 108 as a linked list the time is O(N) as the router must traverse the 109 list comparing each next-hop's region boundaries against the key. If 110 the next-hops are stored as an ordered array a binary search can be 111 used yielding O(logN). 113 As [1] notes if the number of next-hops is limited to a fixed maximum 114 the comparison can be done in parallel in hardware, thus O(1). 116 2.2. Disruption 118 Protocols such as TCP perform better if the path they flow along does 119 not change while the stream is connected. Disruption is the 120 measurement of how many flows have their paths changed due to some 121 change in the router. We measure disruption as the fraction of total 122 flows whose path changes in response to some change in the router. 124 Some algorithms such as round-robin (i.e., upon receiving a packet 125 the least recently used next-hop is chosen) are disruptive regardless 126 of any change in the router. Clearly this is not the case with hash- 127 threshold. As long as the region boundaries remain unchanged the 128 same next-hop will be chosen for a given flow. 130 Because we have required regions to be equal in size the only reason 131 for a change in region boundaries is the addition or removal of a 132 next-hop. In this case the regions must all grow or shrink to fill 133 the key space. The analysis begins with some examples of this. 135 0123456701234567012345670123456701234567 136 +-------+-------+-------+-------+-------+ 137 | 1 | 2 | 3 | 4 | 5 | 138 +-------+-+-----+---+---+-----+-+-------+ 139 | 1 | 2 | 4 | 5 | 140 +---------+---------+---------+---------+ 141 0123456789012345678901234567890123456789 143 Figure 1. Before and after deletion of region 3 145 In figure 1. region 3 has been deleted. The remaining regions grow 146 equally and shift to compensate. In this case 1/4 of region 2 is now 147 in region 1, 1/2 (2/4) of region 3 is in region 2, 1/2 of region 3 is 148 in region 4 and 1/4 of region 4 is in region 5. Since each of the 149 original regions represent 1/5 of the flows, the total disruption is 150 1/5*(1/4 + 1/2 + 1/2 + 1/4) or 3/10. 152 Note that the disruption to flows when adding a region is equivalent 153 to that of removing a region. That is, we are considering the 154 fraction of total flows that changes regions when moving from N to 155 N-1 regions, and that same fraction of flows will change when moving 156 from N-1 to N regions. 158 0123456701234567012345670123456701234567 159 +-------+-------+-------+-------+-------+ 160 | 1 | 2 | 3 | 4 | 5 | 161 +-------+-+-----+---+---+-----+-+-------+ 162 | 1 | 2 | 3 | 5 | 163 +---------+---------+---------+---------+ 164 0123456789012345678901234567890123456789 166 Figure 2. Before and after deletion of region 4 168 In figure 2. region 4 has been deleted. Again the remaining regions 169 grow equally and shift to compensate. 1/4 of region 2 is now in 170 region 1, 1/2 of region 3 is in region 2, 3/4 of region 4 is in 171 region 3 and 1/4 of region 4 is in region 5. Since each of the 172 original regions represent 1/5 of the flows the, total disruption is 173 7/20. 175 To generalize, upon removing a region K the remaining N-1 regions 176 grow to fill the 1/N space. This growth is evenly divided between 177 the N-1 regions and so the change in size for each region is 178 1/N/(N-1) or 1/(N(N-1)). This change in size causes non-end regions 179 to move. The first region grows and so the second region is shifted 180 towards K by the change in size of the first region. 1/(N(N-1)) of 181 the flows from region 2 are subsumed by the change in region 1's 182 size. 2/(N(N-1)) of the flows in region 3 are subsumed by region 2. 183 This is because region 2 has shifted by 1/(N(N-1)) and grown by 184 1/(N(N-1)). This continues from both ends until you reach the 185 regions that bordered K. The calculation for the number of flows 186 subsumed from the Kth region into the bordering regions accounts for 187 the removal of the Kth region. Thus we have the following equation. 189 K-1 N 190 --- i --- (i-K) 191 disruption = \ --- + \ --- 192 / (N)(N-1) / (N)(N-1) 193 --- --- 194 i=1 i=K+1 196 We can factor 1/((N)(N-1)) out as it is constant. 198 / K-1 N \ 199 1 | --- --- | 200 = --- | \ i + \ (i-K) | 201 (N)(N-1) | / / | 202 \ --- --- / 203 1 i=K+1 205 We now use the the concrete formulas for the sum of integers. The 206 first summation is (K)(K-1)/2. For the second summation notice that 207 we are summing the integers from 1 to N-K, thus it is (N-K)(N-K+1)/2. 209 (K-1)(K) + (N-K)(N-K+1) 210 = ----------------------- 211 2(N)(N-1) 213 Considering the summations, one can see that the least disruption is 214 when K is as close to half way between 1 and N as possible. This can 215 be proven by finding the minimum of the concrete formula for K 216 holding N constant. First break apart the quantities and collect. 218 2K*K - 2K - 2NK + N*N + N 219 = ------------------------- 220 2(N)(N-1) 222 K*K - K - NK N + 1 223 = -------------- + ------- 224 (N)(N-1) 2(N-1) 226 Since we are minimizing for K the right side (N+1)/2(N-1) is constant 227 as is the denominator (N)(N-1) so we can drop them. To minimize we 228 take the derivative. 230 d 231 -- (K*K - (N+1)K) 232 dk 234 = 2K - (N+1) 236 Which is zero when K is (N+1)/2. 238 The last thing to consider is that K must be an integer. When N is 239 odd (N+1)/2 will yield an integer, however when N is even (N+1)/2 240 yields an integer + 1/2. In the case, because of symmetry, we get 241 the least disruption when K is N/2 or N/2 + 1. 243 Now since the formula is quadratic with a global minimum half way 244 between 1 and N the maximum possible disruption must occur when edge 245 regions (1 and N) are removed. If K is 1 or N the formula reduces to 246 1/2. 248 The minimum possible disruption is obtained by letting K=(N+1)/2. In 249 this case the formula reduces to 1/4 + 1/(4*N). So the range of 250 possible disruption is (1/4, 1/2]. 252 To minimize disruption we recommend adding new regions to the center 253 rather than the ends. 255 3. Comparison to other algorithms 257 Other algorithms exist to decide which next-hop to use. These 258 algorithms all have different performance and disruptive 259 characteristics. Of these algorithms we will only consider ones that 260 are not disruptive by design (i.e., if no change to the set of next- 261 hops occurs the path a flow takes remains the same). This will 262 exclude round-robin and random choice. We will look at modulo-N and 263 highest random weight. 265 Modulo-N is a simpler form of hash-threshold. Given N next-hops the 266 hash function includes a final modulo-N which directly maps the 267 result to one of the next-hops. This operation is the fastest of the 268 three we consider, however if a next-hop is added or removed the 269 disruption is (N-1)/N. 271 Highest random weight (HRW) is another comparative method similar to 272 hash-threshold. The router seeds a pseudo-random number generator 273 with the packet header fields which describe the flow and the next- 274 hop to obtain a weight. The next-hop which receives the highest 275 weight is selected. The advantage with using HRW is that it has 276 minimal disruption (i.e., disruption due to adding or removing a 277 next-hop is always 1/N.) The disadvantage with HRW is an only 278 slightly more complex function to choose the next-hop. A description 279 of HRW along with comparisons to other methods can be found in [1]. 280 Although not used for next-hop calculation an example usage of HRW 281 can be found in [2]. 283 If the complexity of HRW's next-hop selection processes is acceptable 284 we think it should be considered as an alternative to hash-threshold. 286 4. Security Considerations 288 This document is an analysis of an algorithm used to implement an 289 ECMP routing decision. This analysis does not directly effect the 290 security of the Internet Infrastructure. 292 5. References 294 [1] Thaler, D., and C.V. Ravishankar, "Using Name-Based Mappings to 295 Increase Hit Rates", IEEE/ACM Transactions on Networking, February 296 1998. 298 [2] Estrin, D., Farinacci, D., Helmy, A., Thaler, D., Deering, S., 299 Handley, M., Jacobson, V., Liu, C., Sharma, P., and L. Wei, 300 "Protocol Independent Multicast-Sparse Mode (PIM-SM): Protocol 301 Specification", RFC 2362, June 1998. 303 6. Author's Address 305 Christian E. Hopps 306 Merit Network 307 4251 Plymouth Road, Suite C. 308 Ann Arbor, MI 48105 309 Phone: +1 734 936 0291 310 EMail: chopps@merit.edu 312 7. Full Copyright Statement 314 Copyright (C) The Internet Society (1999). All Rights Reserved. 316 This document and translations of it may be copied and furnished to 317 others, and derivative works that comment on or otherwise explain it 318 or assist in its implementation may be prepared, copied, published 319 and distributed, in whole or in part, without restriction of any 320 kind, provided that the above copyright notice and this paragraph are 321 included on all such copies and derivative works. However, this 322 document itself may not be modified in any way, such as by removing 323 the copyright notice or references to the Internet Society or other 324 Internet organizations, except as needed for the purpose of 325 developing Internet standards in which case the procedures for 326 copyrights defined in the Internet languages other than English. 328 The limited permissions granted above are perpetual and will not be 329 revoked by the Internet Society or its successors or assigns.