idnits 2.17.1 draft-hopps-ecmp-algo-analysis-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Cannot find the required boilerplate sections (Copyright, IPR, etc.) in this document. Expected boilerplate is as follows today (2024-04-18) according to https://trustee.ietf.org/license-info : IETF Trust Legal Provisions of 28-dec-2009, Section 6.a: This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. IETF Trust Legal Provisions of 28-dec-2009, Section 6.b(i), paragraph 2: Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved. IETF Trust Legal Provisions of 28-dec-2009, Section 6.b(i), paragraph 3: This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** Missing expiration date. The document expiration date should appear on the first and last page. ** The document seems to lack a 1id_guidelines paragraph about Internet-Drafts being working documents. ** The document seems to lack a 1id_guidelines paragraph about 6 months document validity -- however, there's a paragraph with a matching beginning. Boilerplate error? ** The document seems to lack a 1id_guidelines paragraph about the list of current Internet-Drafts. ** The document seems to lack a 1id_guidelines paragraph about the list of Shadow Directories. == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. Miscellaneous warnings: ---------------------------------------------------------------------------- -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (26 January 1999) is 9214 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Possible downref: Non-RFC (?) normative reference: ref. '1' ** Obsolete normative reference: RFC 2362 (ref. '2') (Obsoleted by RFC 4601, RFC 5059) Summary: 10 errors (**), 0 flaws (~~), 1 warning (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Internet Engineering Task Force Christian E. Hopps 2 INTERNET-DRAFT Merit Network 3 Expires July 1999 26 January 1999 5 Analysis of an Equal-Cost Multi-Path Algorithm 6 8 Status of this Memo 10 This document is an Internet-Draft. Internet-Drafts are working 11 documents of the Internet Engineering Task Force (IETF), its areas, 12 and its working groups. Note that other groups may also distribute 13 working documents as Internet-Drafts. 15 Internet-Drafts are draft documents valid for a maximum of six months 16 and may be updated, replaced, or obsoleted by other documents at any 17 time. It is inappropriate to use Internet- Drafts as reference 18 material or to cite them other than as "work in progress." 20 To view the list Internet-Draft Shadow Directories, see 21 http://www.ietf.org/shadow.html. 23 Abstract 25 Equal-cost multi-path (ECMP) is a routing technique for routing 26 packets along multiple paths of equal cost. The forwarding engine 27 identifies paths by next-hop. When forwarding a packet the router 28 must decide which next-hop (path) to use. This document gives an 29 analysis of one method for making that decision. The analysis 30 includes the performance of the algorithm and the disruption caused 31 by changes to the set of next-hops. 33 Draft Analysis of an ECMP Algorithm January 1999 35 1. Hash-Threshold 37 One method for determining which next-hop to use when routing with 38 ECMP can be called hash-threshold. The router first selects a key by 39 performing a hash (e.g., modulo-K where K is large, or CRC16) over 40 the packet header fields that identify a flow. The N next-hops have 41 been assigned unique regions in the key space. The router uses the 42 key to determine which region and thus which next-hop to use. 44 As an example of hash-threshold, upon receiving a packet the router 45 performs a CRC16 on the packet's header fields that define the flow 46 (e.g., the source and destination fields of the packet), this is the 47 key. Say for this destination there are 4 next-hops to choose from. 48 Each next-hop is assigned a region in 16 bit space (the key space). 49 For equal usage the router may have chosen to divide it up evenly so 50 each region is 65536/4 or 16k large. The next-hop is chosen by 51 determining which region contains the key (i.e., the CRC result). 53 2. Analysis 55 There are a few concerns when choosing an algorithm for deciding 56 which next-hop to use. One is performance, the computational 57 requirements to run the algorithm. Another is disruption (i.e., the 58 changing of which path a flow uses). Balancing is a third concern; 59 however since the algorithm's balancing characteristics are directly 60 related to the chosen hash function this analysis does not treat this 61 concern in depth. 63 For this analysis we will assume regions of equal size. If the hash 64 function is uniformly distributed the distribution of flows amongst 65 paths will also be uniform. 67 2.1. Performance 69 The performance of the hash-threshold algorithm can be broken down 70 into three parts: selection of regions for the next-hops, obtaining 71 the key and comparing the key to the regions to decide which next-hop 72 to use. 74 Since regions are restricted to be of equal size the calculation of 75 region boundaries is trivial. The boundaries can be calculated as 76 follows: 78 Draft Analysis of an ECMP Algorithm January 1999 80 i = 0; 81 regionssize = keyspace.size / #{ next-hops } 82 for n in { next-hops } 83 n.start = i * regionsize; 84 n.end = n.start + regionsize; 85 i = i + 1 86 done 88 This calculation is O(N). Furthermore the calculation can be done 89 when the next-hops are added to or removed from the destination 90 route. 92 The algorithm doesn't specify the hash function used to obtain the 93 key. Its performance in this area will be exactly the performance of 94 the hash function. It is presumed that if this calculation proves to 95 be a concern it can be done in hardware parallel to other operations 96 that need to complete before deciding which next-hop to use. 98 Finally the next-hop must be chosen. This is done by determining 99 which region contains the key. The time required to do this is 100 dependent on the way the next-hops are organized in memory. The 101 problem reduces to a search. For example if the next-hops are stored 102 as a linked list the time is O(N) as the router must traverse the 103 list comparing each next-hop's region boundaries against the key. If 104 the next-hops are stored as an ordered array a binary search can be 105 used yielding O(logN). 107 As [1] notes if the number of next-hops is limited to a fixed maximum 108 the comparison can be done in parallel in hardware, thus O(1). 110 2.2. Disruption 112 Protocols such as TCP perform better if the path they flow along does 113 not change while the stream is connected. Disruption is the 114 measurement of how many flows have their paths changed due to some 115 change in the router. We measure disruption as the fraction of total 116 flows whose path changes in response to some change in the router. 118 Some algorithms such as round-robin (i.e., upon receiving a packet 119 the least recently used next-hop is chosen) are disruptive regardless 120 of any change in the router. Clearly this is not the case with hash- 121 threshold. As long as the region boundaries remain unchanged the 122 same next-hop will be chosen for a given flow. 124 Draft Analysis of an ECMP Algorithm January 1999 126 Because we have required regions to be equal in size the only reason 127 for a change in region boundaries is the addition or removal of a 128 next-hop. In this case the regions must all grow or shrink to fill 129 the key space. The analysis begins with some examples of this. 131 0123456701234567012345670123456701234567 132 +-------+-------+-------+-------+-------+ 133 | 1 | 2 | 3 | 4 | 5 | 134 +-------+-+-----+---+---+-----+-+-------+ 135 | 1 | 2 | 4 | 5 | 136 +---------+---------+---------+---------+ 137 0123456789012345678901234567890123456789 139 Figure 1. Before and after deletion of region 3 141 In figure 1. region 3 has been deleted. The remaining regions grow 142 equally and shift to compensate. In this case 1/4 of region 2 is now 143 in region 1, 1/2 (2/4) of region 3 is in region 2, 1/2 of region 3 is 144 in region 4 and 1/4 of region 4 is in region 5. Since each of the 145 original regions represent 1/5 of the flows, the total disruption is 146 1/5*(1/4 + 1/2 + 1/2 + 1/4) or 3/10. 148 Note that the disruption to flows when adding a region is equivalent 149 to that of removing a region. That is, we are considering the 150 fraction of total flows that changes regions when moving from N to 151 N-1 regions, and that same fraction of flows will change when moving 152 from N-1 to N regions. 154 0123456701234567012345670123456701234567 155 +-------+-------+-------+-------+-------+ 156 | 1 | 2 | 3 | 4 | 5 | 157 +-------+-+-----+---+---+-----+-+-------+ 158 | 1 | 2 | 3 | 5 | 159 +---------+---------+---------+---------+ 160 0123456789012345678901234567890123456789 162 Figure 2. Before and after deletion of region 4 164 In figure 2. region 4 has been deleted. Again the remaining regions 165 grow equally and shift to compensate. 1/4 of region 2 is now in 166 region 1, 1/2 of region 3 is in region 2, 3/4 of region 4 is in 168 Draft Analysis of an ECMP Algorithm January 1999 170 region 3 and 1/4 of region 4 is in region 5. Since each of the 171 original regions represent 1/5 of the flows the, total disruption is 172 7/20. 174 To generalize, upon removing a region K the remaining N-1 regions 175 grow to fill the 1/N space. This growth is evenly divided between 176 the N-1 regions and so the change in size for each region is 177 1/N/(N-1) or 1/(N(N-1)). This change in size causes non-end regions 178 to move. The first region grows and so the second region is shifted 179 towards K by the change in size of the first region. 1/(N(N-1)) of 180 the flows from region 2 are subsumed by the change in region 1's 181 size. 2/(N(N-1)) of the flows in region 3 are subsumed by region 2. 182 This is because region 2 has shifted by 1/(N(N-1)) and grown by 183 1/(N(N-1)). This continues from both ends until you reach the 184 regions that bordered K. The calculation for the number of flows 185 subsumed from the Kth region into the bordering regions accounts for 186 the removal of the Kth region. Thus we have the following equation. 188 K-1 N 189 --- i --- (i-K) 190 disruption = \ --- + \ --- 191 / (N)(N-1) / (N)(N-1) 192 --- --- 193 i=1 i=K+1 195 We can factor 1/((N)(N-1)) out as it is constant. 197 / K-1 N \ 198 1 | --- --- | 199 = --- | \ i + \ (i-K) | 200 (N)(N-1) | / / | 201 \ --- --- / 202 1 i=K+1 204 We now use the the concrete formulas for the sum of integers. The 205 first summation is (K)(K-1)/2. For the second summation notice that 206 we are summing the integers from 1 to N-K, thus it is (N-K)(N-K+1)/2. 208 Draft Analysis of an ECMP Algorithm January 1999 210 (K-1)(K) + (N-K)(N-K+1) 211 = ----------------------- 212 2(N)(N-1) 214 Considering the summations, one can see that the least disruption is 215 when K is as close to half way between 1 and N as possible. This can 216 be proven by finding the minimum of the concrete formula for K 217 holding N constant. First break apart the quantities and collect. 219 2K*K - 2K - 2NK + N*N + N 220 = ------------------------- 221 2(N)(N-1) 223 K*K - K - NK N + 1 224 = -------------- + ------- 225 (N)(N-1) 2(N-1) 227 Since we are minimizing for K the right side (N+1)/2(N-1) is constant 228 as is the denominator (N)(N-1) so we can drop them. To minimize we 229 take the derivative. 231 d 232 -- (K*K - (N+1)K) 233 dk 235 = 2K - (N+1) 237 Which is zero when K is (N+1)/2. 239 The last thing to consider is that K must be an integer. When N is 240 odd (N+1)/2 will yield an integer, however when N is even (N+1)/2 241 yields an integer + 1/2. In the case, because of symmetry, we get 242 the least disruption when K is N/2 or N/2 + 1. 244 Draft Analysis of an ECMP Algorithm January 1999 246 Now since the formula is quadratic with a global minimum half way 247 between 1 and N the maximum possible disruption must occur when edge 248 regions (1 and N) are removed. If K is 1 or N the formula reduces to 249 1/2. 251 Thus to minimize disruption we recommend adding new regions to the 252 center rather than the ends. 254 3. Comparison to other algorithms 256 Other algorithms exist to decide which next-hop to use. These 257 algorithms all have different performance and disruptive 258 characteristics. Of these algorithms we will only consider ones that 259 are not disruptive by design (i.e., if no change to the set of next- 260 hops occurs the path a flow takes remains the same). This will 261 exclude round-robin and random choice. We will look at modulo-N and 262 highest random weight. 264 Modulo-N is a simpler form of hash-threshold. Given N next-hops the 265 hash function includes a final modulo-N which directly maps the 266 result to one of the next-hops. This operation is the fastest of the 267 three we consider, however if a next-hop is added or removed the 268 disruption is (N-1)/N. 270 Highest random weight (HRW) is another comparative method similar to 271 hash-threshold. The router seeds a pseudo-random number generator 272 with the packet header fields which describe the flow and the next- 273 hop to obtain a weight. The next-hop which receives the highest 274 weight is selected. The advantage with using HRW is that it has 275 minimal disruption (i.e., disruption due to adding or removing a 276 next-hop is always 1/N.) The disadvantage with HRW is an only 277 slightly more complex function to choose the next-hop. A description 278 of HRW along with comparisons to other methods can be found in [1]. 279 Although not used for next-hop calculation an example usage of HRW 280 can be found in [2]. 282 If the complexity of HRW's next-hop selection processes is acceptable 283 we think it should be considered as an alternative to hash-threshold. 285 4. Security Considerations 287 This document is an analysis of an algorithm used to implement an 288 ECMP routing decision. This analysis does not directly effect the 290 Draft Analysis of an ECMP Algorithm January 1999 292 security of the Internet Infrastructure. 294 5. References 296 [1] Thaler, D., and C.V. Ravishankar, "Using Name-Based Mappings to 297 Increase Hit Rates", IEEE/ACM Transactions on Networking, February 298 1998. 300 [2] Estrin, D., Farinacci, D., Helmy, A., Thaler, D., Deering, S., 301 Handley, M., Jacobson, V., Liu, C., Sharma, P., and L. Wei, 302 "Protocol Independent Multicast-Sparse Mode (PIM-SM): Protocol 303 Specification", RFC 2362, June 1998. 305 6. Author's Address 307 Christian E. Hopps 308 Merit Network 309 4251 Plymouth Road, Suite C. 310 Ann Arbor, MI 48105 311 Phone: +1 734 936 0291 312 EMail: chopps@merit.edu