idnits 2.17.1 draft-mcpherson-anycast-arch-implications-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (January 6, 2009) is 5587 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RSSAC 29' is defined on line 401, but no explicit reference was found in the text -- Obsolete informational reference (is this intentional?): RFC 1305 (Obsoleted by RFC 5905) -- Obsolete informational reference (is this intentional?): RFC 1884 (Obsoleted by RFC 2373) -- Obsolete informational reference (is this intentional?): RFC 2030 (Obsoleted by RFC 4330) -- Obsolete informational reference (is this intentional?): RFC 2373 (Obsoleted by RFC 3513) -- Obsolete informational reference (is this intentional?): RFC 2893 (Obsoleted by RFC 4213) -- Obsolete informational reference (is this intentional?): RFC 3068 (Obsoleted by RFC 7526) -- Obsolete informational reference (is this intentional?): RFC 3513 (Obsoleted by RFC 4291) -- Obsolete informational reference (is this intentional?): RFC 3775 (Obsoleted by RFC 6275) -- Obsolete informational reference (is this intentional?): RFC 4330 (Obsoleted by RFC 5905) Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 10 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 INTERNET-DRAFT Danny McPherson 2 Arbor Networks 3 Dave Oran 4 Cisco Systems 5 Expires: July 2009 January 6, 2009 6 Intended Status: Informational 8 Architectural Considerations of IP Anycast 9 11 Status of this Memo 13 This Internet-Draft is submitted to IETF in full conformance with the 14 provisions of BCP 78 and BCP 79. 16 Internet-Drafts are working documents of the Internet Engineering 17 Task Force (IETF), its areas, and its working groups. Note that 18 other groups may also distribute working documents as Internet- 19 Drafts. 21 Internet-Drafts are draft documents valid for a maximum of six months 22 and may be updated, replaced, or obsoleted by other documents at any 23 time. It is inappropriate to use Internet-Drafts as reference 24 material or to cite them other than as "work in progress." 26 The list of current Internet-Drafts can be accessed at 27 http://www.ietf.org/ietf/1id-abstracts.txt 29 The list of Internet-Draft Shadow Directories can be accessed at 30 http://www.ietf.org/shadow.html 32 Copyright Notice 34 Copyright (c) 2009 IETF Trust and the persons identified as the 35 document authors. All rights reserved. 37 This document is subject to BCP 78 and the IETF Trust's Legal 38 Provisions Relating to IETF Documents 39 (http://trustee.ietf.org/license-info) in effect on the date of 40 publication of this document. Please review these documents 41 carefully, as they describe your rights and restrictions with respect 42 to this document. 44 Abstract 46 This memo discusses architectural implications of IP anycast. 48 Table of Contents 50 1. Specification of Requirements. . . . . . . . . . . . . . . . . 4 51 2. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 52 3. Anycast History. . . . . . . . . . . . . . . . . . . . . . . . 4 53 4. Use of Anycast in RFCs . . . . . . . . . . . . . . . . . . . . 5 54 5. Anycast in IPv6. . . . . . . . . . . . . . . . . . . . . . . . 6 55 6. DNS Anycast. . . . . . . . . . . . . . . . . . . . . . . . . . 6 56 7. BCP 126 Revisited. . . . . . . . . . . . . . . . . . . . . . . 7 57 8. Layering and Resiliency. . . . . . . . . . . . . . . . . . . . 7 58 9. Anycast Addresses as Destinations. . . . . . . . . . . . . . . 8 59 10. Anycast Addresses as Sources. . . . . . . . . . . . . . . . . 8 60 11. Regarding Widespread Anycast Use. . . . . . . . . . . . . . . 9 61 12. Service Discovery . . . . . . . . . . . . . . . . . . . . . . 9 62 13. Middleboxes and Anycast . . . . . . . . . . . . . . . . . . . 9 63 14. Transport Implications. . . . . . . . . . . . . . . . . . . . 10 64 15. Security Considerations . . . . . . . . . . . . . . . . . . . 10 65 16. Deployment Considerations . . . . . . . . . . . . . . . . . . 11 66 17. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11 67 18. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 12 68 19. References. . . . . . . . . . . . . . . . . . . . . . . . . . 13 69 19.1. Normative References . . . . . . . . . . . . . . . . . . . 13 70 19.2. Informative References . . . . . . . . . . . . . . . . . . 13 71 20. Authors' Addresses. . . . . . . . . . . . . . . . . . . . . . 15 73 1. Specification of Requirements 75 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 76 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 77 document are to be interpreted as described in [RFC 2119]. 79 2. Overview 81 IP anycast is used for at least one critical Internet service, that 82 of the Domain Name System [RFC 1035] root servers. As of early 2008, 83 at least 10 of the 13 root name servers were using IP anycast [RSSAC 84 29]. Use of IP anycast is growing for other applications as well. 85 It has been deployed for over a decade for DNS resolution services 86 and is currently used by several DNS Top Level Domain (TLD) 87 operators. IP anycast is also used for other services in operational 88 environments, to include Network Time Protocol (NTP) [RFC 1305]. 90 Anycast addresses are syntactically indistinguishable from unicast 91 addresses. Allocation of anycast addresses typically follow a model 92 similar to that of unicast allocation policies. Anycast addressing 93 is inherent to that of unicast in multiple locations, and leverages 94 unicast destination routing to deliver a packet to either zero or one 95 interface among the interfaces asserting the address. The 96 expectation of delivery is to the "closest" instance as determined by 97 unicast routing topology metric(s). There is also an expectation of 98 load- balancing that exists among equal cost routes. 100 Unlike IP unicast, it is not considered an error to assert the same 101 anycast address on multiple interfaces within the same or multiple 102 systems. 104 Some consider anycast a "deceptively simple idea". That is, many 105 pitfalls and subtleties exist with application and transport, as well 106 as for routing configuration and operation, when IP anycast is 107 employed. In this document, we aim to capture many of the 108 architectural implications of IP anycast. 110 3. Anycast History 112 As of this writing, the term "anycast" appears in 126 RFCs, and ~360 113 Internet-Drafts (since 2006). 115 The first formal specification of anycast was provided in "Host 116 Anycasting Service" [RFC 1546]. The authors of this document did a 117 good job of capturing most of the issues that exist with IP anycast 118 today. 120 One of the first documented uses of anycast was in 1994 for a "Video 121 Registry" experiment [IMR 9401]. In the experiment, a UDP query was 122 transmitted to an anycast address to locate a server, and TCP was 123 used by the client to query the server, and then multicast was used 124 to distribute the server database. There is also discussion that 125 ISPs began using anycast for DNS resolution services around the same 126 time, although no public references to support this are available. 128 The IAB clarified in [RFC 2101] that IPv4 anycast addresses were pure 129 "locators", and could never serve as an "identifier" (of a host, 130 interface, or anything else). 132 In 1998 the IAB conducted a routing workshop [RFC 2902]. 133 Interestingly, of the conclusions and ouput actions items from the 134 report, an Anycast section is contained in S 2.10.3. Specifically 135 called out in the conclusions section is the need to describe the 136 advantages and disadvantages of anycast, and the belief that local- 137 scoped well-known anycast addresses will be useful to some 138 applications. In the subsequent section, an action item was outlined 139 that suggested a BOF should be held to plan work on progress, and if 140 a working group forms, a paper on the advantages and the 141 disadvantages of anycast should be included as part of the charter. 143 4. Use of Anycast in RFCs 145 SNTPv4 [RFC 2030] defined how to use anycast for server discovery. 146 This was extended in [RFC 4330] as an NTP-specific "manycast" 147 service, in which anycast was used for the discovery part. 149 IPv6 defined some reserved subnet anycast addresses [RFC 2526] and 150 assigned one to "Mobile IPv6 Home-Agents" [RFC 3775]. 152 The original IPv6 transition mechanism [RFC 2893] made use of IPv4 153 anycast addresses as tunnel endpoints for 6-over-4 tunnels, but this 154 was removed in the revision [RFC 4213]. Huitema's Relay Router 155 scheme [RFC 3068] for 6to4 translation also used anycast in a similar 156 fashion. 158 DNS use of anycast was first specified in [RFC 3258]. Is is noteable 159 that it used the term "shared unicast address" rather than "anycast 160 address" for the service. 162 Anycast was used for routing to rendezvous points (RPs) for MSDP and 163 PIM as documented in [RFC 4610]. 165 [RFC 4786] deals with how the routing system interacts with anycast 166 services, and the operation of anycast services. 168 [RFC 4892] cites the use of anycast with DNS as a motivation to 169 identify individual nameserver instances' [RFC 5001] defines the NSID 170 option for doing so. 172 "Reflections on Internet Transparency" [RFC 4924] briefly mentions 173 how violating transparency can also damage global services that use 174 anycast. 176 5. Anycast in IPv6 178 The original IPv6 addressing architecture [RFC 1884], carried forward 179 in [RFC 2373] and [RFC 3513], severely restricted the use of anycast 180 addresses. In particular, they provided that anycast addresses MUST 181 NOT be used as a source address, and MUST NOT be assigned to an IPv6 182 host (i.e., only routers). these restrictions were finally lifted in 183 2006, with the publication of [RFC 4291]. 185 In fact, the recent "IPv6 Transition/Co-existence Security 186 Considerations" [RFC 4942] overview now recommends: 188 "To avoid exposing knowledge about the internal structure of 189 the network, it is recommended that anycast servers now take 190 advantage of the ability to return responses with the anycast 191 address as the source address if possible." 193 6. DNS Anycast 195 "Distributed Authoritative Name Servers via Shared Unicast Addresses" 196 [RFC 3258] described how to reach authoritative name servers using 197 anycast. It made some interesting points: 199 o asserted (as an advantage) that no routing changes were needed 201 o recommended stopping DNS processes, rather than withdrawing 202 routes, to deal with fail-over. 204 o argued that failure modes involving state were not serious, 205 because: 207 - the vast majority of DNS queries are UDP 208 - large routing metric disparity among authoritative server 209 instances would localize queries to a single instance for 210 most clients 211 - when the resolver tries TCP and it breaks, the resolver 212 will move to a different server instance (where presumably 213 it doesn't break) 215 7. BCP 126 Revisited 217 BCP 126 [RFC 4786] was a product of the IETF's GROW working group. 218 The primary design constraint considered was that routing "be stable" 219 for significantly longer than a "transaction time", where 220 "transaction time" is loosely defined as "a single interaction 221 between a single client and a single server". It takes no position 222 on what applications are suitable candidates for anycast usage. 224 Furthermore, it views anycast service disruptions as an operational 225 problem, "Operators should be aware that, especially for long running 226 flows, there are potential failure modes using anycast that are more 227 complex than a simple 'destination unreachable' failure using 228 unicast." 230 The document primary deals with global Internet-wide services 231 provided by anycast. Where internal topology issues are discussed 232 they're mostly regarding routing implications, not application design 233 implications. BCP 126 also views networks employing per-packet load 234 balancing on equal cost paths as "pathological". 236 8. Layering and Resiliency 238 Preserving the integrity of a modular layered design for IP protocols 239 on the Internet is critical to its continued success and flexibility. 240 One such consideration is that of whether an application should have 241 to adapt to changes in the routing system. 243 Higher layer protocols should make minimal assumptions about lower 244 layer protocols. E.g., applications should make minimal assumptions 245 about routing stability, just as they should make minimal assumptions 246 about congestion and packet loss. When designing applications, it 247 would perhaps be safe to assume that the routing system may deliver 248 each packet to a different service instance, in any pattern, with 249 termporal re-ordering being a not-so-rare phenomenon. 251 Stateful transport protocols (TCP, DCCP, SCTP), without modification, 252 do not understand the properties of anycast and hence will fail 253 probabilistically, but possibly catastrophically, in the presence of 254 "normal" routing dynamics. 256 9. Anycast Addresses as Destinations 258 Anycast addresses are "safe" to use as a destination addresses for an 259 application if: 261 o A request message or "one shot" message is self-contained in a 262 single transport packet 264 o A stateless transport (e.g., UDP) is used for the above 266 o Replies are always sent to a unicast address; these can be 267 multi-packet since the unicast destination is "stable" 269 * Note: this constrains the use of anycast as source addresses 270 as reply messages to that address may reach a device that was 271 not the source that initially triggered it. 273 o The server side of the application keeps no hard state across 274 requests 276 o Retries are idempotent; in addition to not assuming server state, 277 they do not encode any assumptions about loss of requests versus 278 loss of replies. 280 10. Anycast Addresses as Sources 282 Anycast addresses are "safe" to use as source addresses for an 283 application if: 285 o No reflexive (response) message is generated by the receiver 286 with the anycast source used as a destination 288 * unless the application has some private state synchronization 289 that allows for the reflexive message arriving at a different 290 instance 292 o The source anycast address is a bona fide interface address if 293 reverse path forwarding (RPF) checking is on, or a service 294 address explicitly provisioned to bypass RPF 296 11. Regarding Widespread Anycast Use 298 Widespread use of anycast for global Internet-wide services or inter- 299 domain services has some scaling challenges. Similar in ways to 300 multicast, each service generates at least one unique route in the 301 global BGP routing system. As a result, additional anycast instances 302 result in additional paths for a given prefix, which scales super- 303 linearly as a function of denseness of inter-domain interconnection 304 within the routing system (i.e., more paths result in more resources, 305 more network interconnections result in more paths).. 307 12. Service Discovery 309 Applications able to tolerate an extra round trip time (RTT) to learn 310 a unicast destination address for multi-packet exchanges can safely 311 use anycast destination addresses for service instance discovery. 313 o "Instance discovery" message sent to anycast destination address 315 o Reply sent from unicast source address of the interface that 316 received the discovery message 318 o Subsequent exchanges use the unicast address 320 13. Middleboxes and Anycast 322 Middleboxes (e.g., NATs, firewalls) may cause problems when used in 323 conjunction with anycast. In particular, a switch from anycast to 324 unicast requires may require a new binding, and this may not exist in 325 the middlebox. 327 14. Transport Implications 329 UDP is the "lingua franca" for anycast today. Stateful transports 330 could be enhanced to be more anycast friendly. It seems as though 331 this was anticipated in Host Anycasting Services [RFC 1546], 332 specifically: 334 "The solution to this problem is to only permit anycast addresses as 335 the remote address of a TCP SYN segment (without the ACK bit set). A 336 TCP can then initiate a connection to an anycast address. When the 337 SYN-ACK is sent back by the host that received the anycast segment, 338 the initiating TCP should replace the anycast address of its peer, 339 with the address of the host returning the SYN-ACK. (The initiating 340 TCP can recognize the connection for which the SYN-ACK is destined by 341 treating the anycast address as a wildcard address, which matches any 342 incoming SYN-ACK segment with the correct destination port and 343 address and source port, provided the SYN-ACK's full address, 344 including source address, does not match another connection and the 345 sequence numbers in the SYN-ACK are correct.) This approach ensures 346 that a TCP, after receiving the SYN-ACK is always communicating with 347 only one host." 349 Multi-address transports (e.g., SCTP) might be more amenable to such 350 extensions than TCP. 352 Some similarities exist between what is needed for anycast and what 353 is needed for address discovery when doing multi-homing in the 354 transport layer. **NEED TO EXPAND ON THIS*** 356 15. Security Considerations 358 Anycast is often employed to mitigate or at least localize the 359 effects of distributed denial of service (DDOS) attacks. For 360 example, with the Netgear NTP fiasco [RFC 4085] anycast was used in a 361 distributed sinkhole model to mitigate the effects of embedded 362 globally-routed Internet addresses in network elements. 364 "Internet Denial-of-Service Considerations" [RFC 4732] notes that "A 365 number of the root nameservers have since been replicated using 366 anycast to further improve their resistance to DoS". 368 [RFC 4786] cites DoS mitigation, constraining DoS to localized 369 regions, and identifying attack sources using spoofed addresses as 370 some motivations to deploy services using anycast. Multiple anycast 371 service instances such as those used by the root name servers also 372 add resiliency when network partitioning occurs (e.g., as the result 373 of transoceanic fiber cuts or natural disasters). 375 16. Deployment Considerations 377 This document covers issues associated with the architectural 378 implications of anycast. Operators should heed these considerations 379 when evaluating the use of anycast in their specific environments. 381 17. IANA Considerations 383 No IANA actions are required. 385 18. Acknowledgments 387 Many thanks for Dave Thaler and Kurtis Lindqvist for their early 388 review and feedback on this document. 390 Your name could be here.... 392 19. References 394 19.1. Normative References 396 19.2. Informative References 398 [IMR 9401] "INTERNET MONTHLY REPORT", January 1994, 399 http://mirror.facebook.com/rfc/museum/imr/imr9401.txt 401 [RSSAC 29] "RSSAC 29 Meeting Minutes", December 2, 2007, 402 http://www.rssac.org/meetings/04-08/rssac29.pdf 404 [RFC 1035] Mockapetris, P., "DOMAIN NAMES - IMPLEMENTATION 405 AND SPECIFICATION", RFC 1035, November 1987. 407 [RFC 1305] Mills, D., "Network Time Protocol (Version 3) 408 Specification, Implementation and Analysis", RFC 409 1305, March 1992. 411 [RFC 1546] Partridge, C., Mendez, T., Milliken, W., "Host 412 Anycasting Service", RFC 1546, November 1993. 414 [RFC 1884] Hinden, R., Deering, S., "IP Version 6 Addressing 415 Architecture", RFC 1884, December 1995. 417 [RFC 2030] Mills, D., "Simple Network Time Protocol (SNTP) 418 Version 4 for IPv4, IPv6 and OSI", RFC 2030, 419 October 1996. 421 [RFC 2101] Carpenter, B., Crowcroft, J., Rekhter, Y., "IPv4 422 Address Behaviour Today", RFC 2101, February 1997. 424 [RFC 2119] Bradner, S., "Key words for use in RFCs to Indicate 425 Requirement Levels", RFC 2119, March 1997. 427 [RFC 2373] Hinden, R., Deering, S., "IP Version 6 Addressing 428 Architecture", RFC 2373, July 1998. 430 [RFC 2526] Johnson, D., Deering, S., "Reserved IPv6 Subnet 431 Anycast Addresses", RFC 2526, March 1999. 433 [RFC 2893] Gilligan, R., Nordmark, E., "Transition Mechanisms 434 for IPv6 Hosts and Routers", RFC 2893, August 2000. 436 [RFC 2902] Deering, S., Hares, S., Perkins, C., Perlman, R., 437 "Overview of the 1998 IAB Routing Workshop", RFC 438 2902, August 2000. 440 [RFC 3068] Huitema, C., "An Anycast Prefix for 6to4 Relay 441 Routers", RFC 3068, June 2001. 443 [RFC 3258] Hardie, R., "Distributing Authoritative Name Servers 444 via Shared Unicast Addresses", RFC 3258, April 2002. 446 [RFC 3513] Hinden, R., Deering, S., "Internet Protocol Version 447 6 (IPv6) Addressing Architecture", RFC 3513, April 448 2003. 450 [RFC 3775] Johnson, D., Perkins, C., Arkko, J., "Mobility 451 Support in IPv6", RFC 3775, June 2004. 453 [RFC 4085] Plonka, D., "Embedding Globally-Routable Internet 454 Addresses Considered Harmful", RFC 4085, June 2005. 456 [RFC 4213] Normark, E., Gilligan, R., "Basic Transition 457 Mechanisms for IPv6 Hosts and Routers", RFC 4213, 458 October 2005. 460 [RFC 4291] Hinden, R., Deering, S., "IP Version 6 Addressing 461 Architecture", RFC 4291, February 2006. 463 [RFC 4330] Mills, D., "Simple Network Time Protocol (SNTP) 464 Version 4 for IPv4, IPv6 and OSI", RFC 4330, 465 January 2006. 467 [RFC 4610] Farinacci, D., Cai, Y., "Anycast-RP Using Protocol 468 Independent Multicast (PIM)", RFC 4610, August 2006. 470 [RFC 4732] Handley, M., Rescorla, E., IAB, "Internet Denial-of- 471 Service Considerations", RFC 4732, November 2006. 473 [RFC 4786] Abley, J., Lindqvist, K., "Operation of Anycast 474 Services", RFC 4786, December 2006. 476 [RFC 4892] Woolf, S., Conrad, D., "Requirements for a Mechanism 477 Identifying a Name Server Instance", RFC 4892, June 478 2007. 480 [RFC 4924] Aboba, B., Davies, E., " Reflections on Internet 481 Transparency", RFC 4924, July 2007. 483 [RFC 4942] Davies, E., Krishnan, S., Savola, P., "IPv6 484 Transition/Coexistence Security Considerations", 485 RFC 4942, September 2007. 487 [RFC 5001] Austein, R., "DNS Name Server Identifier (NSID) 488 Option", RFC 5001, August 2007. 490 20. Authors' Addresses 492 Danny McPherson 493 Arbor Networks, Inc. 494 Email: danny@arbor.net 496 Dave Oran 497 Cisco Systems 498 Email: oran@cisco.com 500 Acknowledgment 502 Funding for the RFC Editor function is provided by the IETF 503 Administrative Support Activity (IASA).