idnits 2.17.1 draft-iab-anycast-arch-implications-12.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 733: '...ddress Detection MUST NOT be performed...' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (November 04, 2013) is 3824 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- No information found for draft-ietf-ipv6-dns-discovery - is the name correct? -- Obsolete informational reference (is this intentional?): RFC 793 (Obsoleted by RFC 9293) -- Obsolete informational reference (is this intentional?): RFC 1884 (Obsoleted by RFC 2373) -- Obsolete informational reference (is this intentional?): RFC 2030 (Obsoleted by RFC 4330) -- Obsolete informational reference (is this intentional?): RFC 2373 (Obsoleted by RFC 3513) -- Obsolete informational reference (is this intentional?): RFC 2893 (Obsoleted by RFC 4213) -- Obsolete informational reference (is this intentional?): RFC 3068 (Obsoleted by RFC 7526) -- Obsolete informational reference (is this intentional?): RFC 3513 (Obsoleted by RFC 4291) -- Obsolete informational reference (is this intentional?): RFC 3775 (Obsoleted by RFC 6275) -- Obsolete informational reference (is this intentional?): RFC 4305 (Obsoleted by RFC 4835) -- Obsolete informational reference (is this intentional?): RFC 4330 (Obsoleted by RFC 5905) Summary: 1 error (**), 0 flaws (~~), 1 warning (==), 12 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force D. McPherson 3 Internet-Draft Verisign, Inc. 4 Intended status: Informational D. Oran 5 Expires: May 08, 2014 Cisco Systems 6 D. Thaler 7 Microsoft Corporation 8 E. Osterweil 9 Verisign, Inc. 10 November 04, 2013 12 Architectural Considerations of IP Anycast 13 draft-iab-anycast-arch-implications-12 15 Abstract 17 This memo discusses architectural implications of IP anycast, and 18 provides some historical analysis of anycast use by various IETF 19 protocols. 21 Status of This Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at http://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on May 08, 2014. 38 Copyright Notice 40 Copyright (c) 2013 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents 45 (http://trustee.ietf.org/license-info) in effect on the date of 46 publication of this document. Please review these documents 47 carefully, as they describe your rights and restrictions with respect 48 to this document. Code Components extracted from this document must 49 include Simplified BSD License text as described in Section 4.e of 50 the Trust Legal Provisions and are provided without warranty as 51 described in the Simplified BSD License. 53 Table of Contents 55 1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 2 56 2. Background . . . . . . . . . . . . . . . . . . . . . . . . . 3 57 2.1. Anycast History . . . . . . . . . . . . . . . . . . . . . 4 58 2.2. Anycast in IPv6 . . . . . . . . . . . . . . . . . . . . . 6 59 2.3. DNS Anycast . . . . . . . . . . . . . . . . . . . . . . . 6 60 2.4. BCP 126 on Operation of Anycast Services . . . . . . . . 8 61 3. Principles . . . . . . . . . . . . . . . . . . . . . . . . . 8 62 3.1. Layering and Resiliency . . . . . . . . . . . . . . . . . 8 63 3.2. Anycast Addresses as Destinations . . . . . . . . . . . . 9 64 3.3. Anycast Addresses as Sources . . . . . . . . . . . . . . 10 65 3.4. Service Discovery . . . . . . . . . . . . . . . . . . . . 10 66 4. Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 11 67 4.1. Regarding Widespread Anycast Use . . . . . . . . . . . . 11 68 4.2. Transport Implications . . . . . . . . . . . . . . . . . 12 69 4.3. Stateful Firewalls, Middleboxes and Anycast . . . . . . . 12 70 4.4. Security Considerations . . . . . . . . . . . . . . . . . 13 71 4.5. Deployment Considerations . . . . . . . . . . . . . . . . 15 72 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17 73 6. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 17 74 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 17 75 8. Informative References . . . . . . . . . . . . . . . . . . . 17 76 Appendix A. IAB Members at the Time of Approval . . . . . . . . 20 77 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 21 79 1. Overview 81 IP anycast is a technique with a long legacy, with interesting 82 engineering challenges associated with it. However, at its core it 83 is a relatively simple concept. As described in BCP 126 [RFC4786], 84 the general form of IP anycast is the practice of making a particular 85 Service Address available in multiple, discrete, autonomous 86 locations, such that datagrams sent are routed to one of several 87 available locations. 89 IP anycast is used for at least one critical Internet service, that 90 of the Domain Name System [RFC1035] root servers. By late 2007, at 91 least 10 of the 13 root name servers were already using IP anycast 92 [RSSAC29]. Use of IP anycast is growing for other applications as 93 well. It has been deployed for over a decade for DNS resolution 94 services and is currently used by several DNS Top Level Domain (TLD) 95 operators. IP anycast is also used for other services in operational 96 environments, including Network Time Protocol (NTP) [RFC5905] 97 services. 99 Anycast addresses are syntactically indistinguishable from unicast 100 addresses. Anycast addressing is equivalent to that of unicast in 101 multiple locations. Destination-based routing does best-effort 102 delivery of a packet to one interface among the set of interfaces 103 asserting the reachability for the address. The expectation of 104 delivery is to the "closest" instance as determined by unicast 105 routing topology metric(s), and there is also a possibility that 106 various load-balancing techniques (e.g., per-packet, per-microflow) 107 may be used among multiple equal cost routes to distribute load for 108 an anycasted prefix. 110 Unlike IP unicast, it is not considered an error to assert the same 111 anycast address on multiple interfaces within the same or multiple 112 systems. 114 When IP anycast is employed, many pitfalls and subtleties exist with 115 applications and transports, as well as for routing configuration and 116 operation. In this document, we aim to capture many of the 117 architectural implications of IP anycast. 119 BCP 126 [RFC4786] discusses several different deployment models with 120 IP anycast. Two additional distinctions beyond that document involve 121 "off-link anycast" and "on-link anycast". "Off-link anycast" takes 122 advantage of routing protocol preferences and IP's hop-by-hop 123 destination-based forwarding paradigm in order to direct packets to 124 the "closest" destination. This is the traditional method of anycast 125 largely considered in BCP 126 [RFC4786] and can be used for IPv4 and 126 IPv6. "On-link anycast" is the formal support of anycast in the 127 address resolution (duplicate address detection) protocol and is only 128 standardized for IPv6, with the introduction of designated anycast 129 addresses on the anycasted hosts, and the Override flag in Neighbor 130 Discovery (ND) Neighbor Advertisements (NAs) [RFC4861]. There is no 131 standardized mechanism for this in IPv4. 133 2. Background 135 As of this writing, the term "anycast" appears in 176 RFCs and 144 136 active Internet-Drafts. The following sections capture some of the 137 key appearances and discussion of anycasting within the IETF over the 138 years. 140 2.1. Anycast History 142 The first formal specification of anycast was provided in "Host 143 Anycasting Service" [RFC1546]. The authors of this document did a 144 good job of capturing most of the issues that exist with IP anycast 145 today. 147 One of the first documented uses of anycast was in 1994 for a "Video 148 Registry" experiment [IMR9401]. In the experiment a UDP query was 149 transmitted to an anycasted address to locate the topologically 150 closest "supposedly equivalent network resource": 152 "A video resource (for example, a catalog server that lists 153 available video clips) sends an anycast UDP datagram to locate the 154 nearest video registry. At most one registry responds with a 155 unicast UDP datagram containing the registry's IP address. Said 156 resource then opens a TCP connection to that [the received 157 registry address] address and sends a request to register itself. 158 Every 5 minutes or so, each registry multicasts to all other 159 registries all of the resources it knows from local registration 160 requests. It also immediately announces newly registered 161 resources. Remotely registered resources not heard about for 20 162 minutes are dropped." 164 There is also discussion that ISPs began using anycast for DNS 165 resolution services around the same time, although no public 166 references to support this are available. 168 In 1997 the IAB clarified that IPv4 anycast addresses were pure 169 "locators", and could never serve as an "identifier" of a host or an 170 interface [RFC2101]. 172 In 1998 the IAB conducted a routing workshop [RFC2902]. Of the 173 conclusions and output action items from the report, an Anycast 174 section is contained in Section 2.10.3. Specifically called out is 175 the need to describe the advantages and disadvantages of anycast, and 176 the belief that local-scoped well-known anycast addresses will be 177 useful to some applications. In the subsequent section, an action 178 item was outlined that suggested a BOF should be held to plan work on 179 anycast, and if a working group forms, a paper on the advantages and 180 the disadvantages of anycast should be included as part of the 181 charter. 183 As a result of the recommendation in [RFC2902], in November of 1999 184 an Anycast BOF [ANYCASTBOF] was held at IETF 46. A number of uses 185 for anycast were discussed. No firm conclusion was reached regarding 186 use of TCP with anycasted services, but it was observed that 187 anycasting was useful for DNS, although it did introduce some new 188 complexities. The use of global anycast was not expected to scale 189 (see Section 4.1 below for more discussion), and hence was expected 190 to be limited to a small number of key uses. 192 In 2001, the Multicast and Anycast Group Membership [MAGMA] WG was 193 chartered to address host-to-router signaling, including initial 194 authentication and access control issues for multicast and anycast 195 group membership, but other aspects of anycast, including 196 architecture and routing, were outside the group's scope. 198 SNTPv4 [RFC2030] defined how to use SNTP anycast for server 199 discovery. This was extended in [RFC4330] as an NTP-specific 200 "manycast" service, in which anycast was used for the discovery part. 202 IPv6 defined some reserved subnet anycast addresses [RFC2526] and 203 assigned one to "Mobile IPv6 Home-Agents" [RFC3775] (obsoleted by 204 [RFC6275]). 206 The original IPv6 transition mechanism [RFC2893] made use of IPv4 207 anycast addresses as tunnel endpoints for IPv6 encapsulated in IPv4, 208 but this was later removed [RFC4213]. The 6to4 tunneling protocol 209 [RFC3056] was augmented by a 6to4 relay anycast prefix [RFC3068] 210 aiming to simplify the configuration of 6to4 routers. Incidentally, 211 6to4 deployment has shown a fair number of operational and security 212 issues [RFC3964] that result from using anycast as a discovery 213 mechanism. Specifically, one inference is that operational 214 consideration is needed to ensure that anycast addresses get 215 advertised and/or filtered in a way that produces the intended scope 216 (e.g., only advertise a route for your 6to4 relay to ASes that 217 conform to your own acceptable usage policy), an attribute that can 218 easily become quite operationally expensive. 220 In 2002, DNS' use of anycast was first specified in "Distributing 221 Authoritative Name Servers via Shared Unicast Addresses" [RFC3258]. 222 It is notable that it used the term "shared unicast address" rather 223 than "anycast address" for the service. This distinction was made 224 due to the IPv6 distinction in the on-link model. "Shared unicast" 225 addresses are unicast (not multicast) in the IPv6 model and, 226 therefore, support the off-link anycast model (described earlier), 227 but not the on-link anycast model. At the same time, site-local- 228 scoped well-known addresses began being used for recursive resolvers 229 [I-D.ietf-ipv6-dns-discovery], but this use was never standardized 230 (see below in Section 3.4 for more discussion). 232 Anycast was used for routing to rendezvous points (RPs) for PIM 233 [RFC4610]. 235 "Operation of Anycast Services" BCP 126 [RFC4786] deals with how the 236 routing system interacts with anycast services, and the operation of 237 anycast services. 239 "Requirements for a Mechanism Identifying a Name Server Instance" 240 [RFC4892] cites the use of anycast with DNS as a motivation to 241 identify individual name server instances, and the Name Server ID 242 (NSID) option was defined for this purpose [RFC5001]. One could view 243 the addition of NSID as an incarnation of locator and identifier 244 separation (where the anycast address is a locator and the NSID is an 245 identifier). 247 The IAB's "Reflections on Internet Transparency" [RFC4924] briefly 248 mentions how violating transparency can also damage global services 249 that use anycast. 251 2.2. Anycast in IPv6 253 Originally, the IPv6 addressing architecture [RFC1884] [RFC2373] 254 [RFC3513] severely restricted the use of anycast addresses. In 255 particular, they provided that anycast addresses must not be used as 256 a source address, and must not be assigned to an IPv6 host (i.e., 257 only routers). These restrictions were later lifted in 2006 258 [RFC4291]. 260 In fact, the recent "IPv6 Transition/Co-existence Security 261 Considerations" [RFC4942] overview now recommends: 263 "To avoid exposing knowledge about the internal structure of the 264 network, it is recommended that anycast servers now take advantage 265 of the ability to return responses with the anycast address as the 266 source address if possible." 268 As discussed in the Overview, "on-link anycast" is employed expressly 269 in IPv6 via ND NAs; see Section 7.2.7 of [RFC4861] for additional 270 information. 272 2.3. DNS Anycast 274 "Distributed Authoritative Name Servers via Shared Unicast Addresses" 275 [RFC3258] described how to reach authoritative name servers using 276 multiple unicast addresses, each one configured on a different set of 277 servers. It stated in Section 2.3: 279 "This document presumes that the usual DNS failover methods are 280 the only ones used to ensure reachability of the data for clients. 281 It does not advise that the routes be withdrawn in the case of 282 failure; it advises instead that the DNS process shutdown so that 283 servers on other addresses are queried. This recommendation 284 reflects a choice between performance and operational complexity. 285 While it would be possible to have some process withdraw the route 286 for a specific server instance when it is not available, there is 287 considerable operational complexity involved in ensuring that this 288 occurs reliably. Given the existing DNS failover methods, the 289 marginal improvement in performance will not be sufficient to 290 justify the additional complexity for most uses." 292 In anycast more generally, most anycast benefits cannot be realized 293 without route withdrawals since traffic will continue to be directed 294 to the link with the failed server. When multiple unicast addresses 295 are used with different sets of servers, a client can still fail over 296 to using a different server address and hence a different set of 297 servers. There can still be reliability problems, however, when each 298 set contains a failed server. If all servers in the same set are on 299 the same subnet, such problems could be minimized where address 300 resolution within the subnet will cause traffic to go to an available 301 server. 303 Other assertions included: 305 o it asserted (as an advantage) that no routing changes were needed 307 o it recommended stopping DNS processes, rather than withdrawing 308 routes, to deal with failures, data synchronization issues, and 309 fail-over, as provided in the quoted text above. The spirit of 310 this advice was that DNS resolvers may (indeed) reach out and 311 query unavailable DNS name servers, but as their queries time out, 312 they will elect to pin themselves to other server addresses, and 313 hence different servers. 315 o it argued that failure modes involving state were not serious, 316 because: 318 * the vast majority of DNS queries are UDP 320 * large routing metric disparity among authoritative server 321 instances would localize queries to a single instance for most 322 clients 324 * when the resolver tries TCP and it breaks, the resolver will 325 try to move to a different server address. In order to ensure 326 that this is possible, it is important that the DNS zone be 327 configured with multiple server addresses for different sets of 328 name servers. The advice given in section 3.3 of 329 [I-D.ietf-ipv6-dns-discovery] describes, in more detail, why 330 using multiple addresses is important. 332 "Unique Per-Node Origin ASNs for Globally Anycasted Services" 333 [RFC6382] makes recommendations regarding the use of per-node unique 334 origin Autonomous System Numbers (ASNs) for globally anycasted 335 critical infrastructure services in order to provide routing system 336 discriminators for a given anycasted prefix. The object was to allow 337 network management and monitoring techniques, or other operational 338 mechanisms to employ this new origin AS as a discriminator in 339 whatever manner fits their operating environment, either for 340 detection or policy associated with a given anycasted node. 342 2.4. BCP 126 on Operation of Anycast Services 344 "Operation of Anycast Services" BCP 126 [RFC4786] was a product of 345 the IETF's GROW working group. The primary design constraint 346 considered was that routing "be stable" for significantly longer than 347 a "transaction time", where "transaction time" is loosely defined as 348 "a single interaction between a single client and a single server". 349 It takes no position on what applications are suitable candidates for 350 anycast usage. 352 Furthermore, it views anycast service disruptions as an operational 353 problem: "Operators should be aware that, especially for long running 354 flows, there are potential failure modes using anycast that are more 355 complex than a simple 'destination unreachable' failure using 356 unicast." 358 The document primary deals with global Internet-wide services 359 provided by anycast. Where internal topology issues are discussed 360 they're mostly regarding routing implications, rather than 361 application design implications. BCP 126 also views networks 362 employing per-packet load balancing on equal cost paths as 363 "pathological". This was also discussed in [RFC2991]. 365 3. Principles 367 3.1. Layering and Resiliency 369 Preserving the integrity of a modular layered design for IP protocols 370 on the Internet is critical to its continued success and flexibility. 371 One such consideration is that of whether an application should have 372 to adapt to changes in the routing system. 374 Applications should make minimal assumptions about routing stability, 375 just as they should make minimal assumptions about congestion and 376 packet loss. When designing applications, it would perhaps be safe 377 to assume that the routing system may deliver each anycast packet to 378 a different service instance, in any pattern, with temporal re- 379 ordering being a not-so-rare phenomenon. 381 Most stateful transport protocols (e.g., TCP), without modification, 382 do not understand the properties of anycast and hence will fail 383 probabilistically, but possibly catastrophically, when using anycast 384 addresses in the presence of "normal" routing dynamics. 385 Specifically, if datagrams associated with a given active transaction 386 are routed to a new anycasted end system and that end system lacks 387 state data associated with the active transaction, the session will 388 be reset and hence will need to be reinitiated. As another example, 389 different networks have different routing properties and therefore 390 will experience problems under different conditions. This can lead 391 to a protocol working fine in, say, a test lab but not in the global 392 Internet. 394 3.2. Anycast Addresses as Destinations 396 When an anycast address is used as a destination address, different 397 packets with the same destination IP address may reach different 398 destination hosts, even if the packets are generated by the same 399 source host. Anycast addresses are thus "safe" to use as destination 400 addresses for an application if the following design points are all 401 met: 403 o A request message or "one shot" message is self-contained in a 404 single transport packet 406 o A stateless transport (e.g., UDP) is used for the above 408 o Replies are always sent to a unicast address; these can be multi- 409 packet since the unicast destination is presumed to be associated 410 with a single "stable" end system and not an anycasted source 411 address. Note that this constrains the use of anycast as source 412 addresses in request messages, since reply messages sent back to 413 that address may reach a device that was not the source that 414 initially triggered it. 416 o The server side of the application keeps no hard state across 417 requests. 419 o Retries are idempotent; in addition to not assuming server state, 420 they do not encode any assumptions about loss of requests versus 421 loss of replies. 423 It is noteworthy, though, that even under the above circumstances 424 ICMP messages against packets with anycast source addresses may be 425 routed to servers other than those expected. In addition, Path 426 Maximum Transmission Unit Discovery (PMTUD) can encounter 427 complications when employed against anycast addresses, since 428 iterations in the PMTU discovery process may have packets routed to 429 different anycast service instances. 431 3.3. Anycast Addresses as Sources 433 When an anycast address is used as a source address, the source 434 address does not uniquely identify the source host, and hence replies 435 might be sent to a different host. As noted earlier, this concept is 436 sometimes referred to (e.g., in [RFC3258]) as a "shared unicast 437 address". Anycast addresses are "safe" to use as source addresses 438 for an application if all of the following design points are met: 440 o No response message is generated by the receiver with the anycast 441 source used as a destination unless the application has some 442 private state synchronization that allows for the response message 443 arriving at a different instance 445 o The source anycast address is reachable via the interface address 446 if unicast reverse path forwarding (RPF) [RFC4778] checking is on, 447 or the service address is explicitly provisioned to bypass RPF 448 checks. In addition to the application defined in [RFC4778], 449 Section 4.4.5 of BCP 126 [RFC4786] gives explicit consideration to 450 RPF checks in anycasting operations. 452 3.4. Service Discovery 454 Applications able to tolerate an extra round trip time (RTT) to learn 455 a unicast destination address for multi-packet exchanges might safely 456 use anycast destination addresses for service instance discovery. 457 For example, "instance discovery" messages are sent to an anycast 458 destination address, and a reply is subsequently sent from the unique 459 unicast source address of the interface that received the discovery 460 message, or a reply is sent from the anycast source address of the 461 interface that received the message, containing the unicast address 462 to be used to invoke the service. Only the latter of these will 463 avoid potential NAT binding and stateful firewall issues. 465 [I-D.ietf-ipv6-dns-discovery] discussed several options to address 466 the need to configure DNS servers, including the use of a "Well-known 467 Anycast Address" for recursive DNS service configuration in clients 468 to ease configuration and allow those systems to ship with these 469 well-known addresses configured "from the beginning, as, say, factory 470 default". The proposal was later dropped, but the analysis was used 471 in publishing [RFC4339]. 473 After the final round of revisions to [I-D.ietf-ipv6-dns-discovery] 474 were made, [RFC4339] was published with a very similar focus, and 475 overlapping content. The difference was that the writing in 476 [RFC4339] focused on analysis, while [I-D.ietf-ipv6-dns-discovery] 477 covered both the analysis and a specific proposal. The proposal 478 details were removed in what became [RFC4339] although section 3.3 of 479 that RFC still discusses the approach of using a well-known anycast 480 address in this scenario. During publication the IESG requested that 481 the following "IESG Note" be contained in the document: 483 "This document describes three different approaches for the 484 configuration of DNS name resolution server information in IPv6 485 hosts. 487 There is not an IETF consensus on which approach is preferred. 488 The analysis in this document was developed by the proponents for 489 each approach and does not represent an IETF consensus. 491 The 'RA option' and 'Well-known anycast' approaches described in 492 this document are not standardized. Consequently the analysis for 493 these approaches might not be completely applicable to any 494 specific proposal that might be proposed in the future." 496 4. Analysis 498 4.1. Regarding Widespread Anycast Use 500 Widespread use of anycast for global Internet-wide services or inter- 501 domain services has some scaling challenges. Similar in ways to 502 multicast, each service generates at least one unique route in the 503 global BGP routing system. As a result, additional anycast instances 504 result in additional paths for a given prefix, which scales super- 505 linearly as a function of denseness of inter-domain interconnection 506 within the routing system (i.e., more paths result in more resources, 507 more network interconnections result in more paths). 509 This is why the Anycast BOF concluded that "the use of global anycast 510 addresses was not expected to scale and hence was expected to be 511 limited to a small number of key uses." 512 However, one interesting note is that multiple anycast services can 513 share a route if they are all located in a single announced prefix, 514 and if all the servers of all the services are always collocated. If 515 the announced prefix is aggregated differently in different 516 locations, though, longest-match routing might result in some anycast 517 locations being unreachable. Hence extra precaution must be taken 518 when aggregating prefixes used by anycast services. 520 4.2. Transport Implications 522 UDP is the "lingua franca" for anycast today. Stateful transports 523 could be enhanced to be more anycast friendly. This was anticipated 524 in Host Anycasting Services [RFC1546], specifically: 526 "The solution to this problem is to only permit anycast addresses 527 as the remote address of a TCP SYN segment (without the ACK bit 528 set). A TCP can then initiate a connection to an anycast address. 529 When the SYN-ACK is sent back by the host that received the 530 anycast segment, the initiating TCP should replace the anycast 531 address of its peer, with the address of the host returning the 532 SYN-ACK. (The initiating TCP can recognize the connection for 533 which the SYN-ACK is destined by treating the anycast address as a 534 wildcard address, which matches any incoming SYN-ACK segment with 535 the correct destination port and address and source port, provided 536 the SYN-ACK's full address, including source address, does not 537 match another connection and the sequence numbers in the SYN-ACK 538 are correct.) This approach ensures that a TCP, after receiving 539 the SYN-ACK is always communicating with only one host." 541 The reason for such considerations can be illustrated through an 542 example: one operationally observed shortcoming of using the 543 Transmission Control Protocol (TCP) [RFC0793] and anycast nodes in 544 DNS is that even during the TCP connection establishment, IP control 545 packets from a DNS client may initially be routed to one anycast 546 instance, but subsequent IP packets may be delivered to a different 547 anycast instance if (for example) a route has changed. In such a 548 case, the TCP connection will likely elicit a connection reset, but 549 will certainly result in the disruption of the connection. 551 Multi-address transports (e.g., SCTP) might be more amenable to such 552 extensions than TCP. 554 The features needed for address discovery when doing multi-homing in 555 the transport layer are similar to those needed to support anycast. 557 4.3. Stateful Firewalls, Middleboxes and Anycast 558 Middleboxes (e.g., NATs) and stateful firewalls cause problems when 559 used in conjunction with some ways to use anycast. In particular, a 560 server-side transition from an anycast source IP address to a unique 561 unicast address may require new or additional session state, and this 562 may not exist in the middlebox, as discussed previously in 563 Section 3.4. 565 4.4. Security Considerations 567 Anycast is often deployed to mitigate or at least localize the 568 effects of distributed denial of service (DDOS) attacks. For 569 example, with the Netgear NTP fiasco [RFC4085] anycast was used in a 570 distributed sinkhole model [RFC3882] to mitigate the effects of 571 embedded globally-routed Internet addresses in network elements. 573 "Internet Denial-of-Service Considerations" [RFC4732] notes: that "A 574 number of the root nameservers have since been replicated using 575 anycast to further improve their resistance to DoS". 577 "Operation of Anycast Services" BCP 126 [RFC4786] cites DoS 578 mitigation, constraining DoS to localized regions, and identifying 579 attack sources using spoofed addresses as some motivations to deploy 580 services using anycast. Multiple anycast service instances such as 581 those used by the root name servers also add resiliency when network 582 partitioning occurs (e.g., as the result of transoceanic fiber cuts 583 or natural disasters). 585 When using anycast, care must be taken not to simply withdraw an 586 anycast route in the presence of a sustained DoS attack, since the 587 result would simply move the attack to another service instance, 588 potentially causing a cascaded failure. Anycast adds resiliency when 589 such an attack is instead constrained to a single service instance. 591 It should be noted that there is a significant man in the middle 592 (MITM) exposure in either variant of anycast discovery (see 593 Section 3.4) that in many applications may necessitate the need for 594 end to end security models [RFC4302] [RFC4303] [RFC4305], or even 595 [RFC4033] that enable end systems to authenticate one another, or the 596 data itself. 598 However, when considering the above suggestion of enabling end 599 systems to authenticate each other, a potential complication can 600 arise. If the service nodes of an anycast deployment are 601 administered by separate authorities (as in the case of the DNS root 602 servers), any server-side authentication credentials that are used 603 must (necessarily) be shared across the administrative boundaries in 604 the anycast deployment. This would likely also be the case with 605 Secure Neighbor Discovery, described in [RFC5909]. 607 Furthermore, as discussed earlier in this document, operational 608 consideration needs to be given to ensure that anycast addresses get 609 advertised and/or filtered in a way that produces intended scope (for 610 example, only advertise a route to your 6to4 relay to ASes that 611 conform to your own Acceptable Use Policy, AUP). This seems to be 612 operationally expensive, and is often vulnerable to errors outside of 613 the local routing domain, in particular when anycasted services are 614 deployed with the intent to scope associated announcements within 615 some local or regional boundary. 617 As previously discussed, [RFC6382] makes recommendations regarding 618 the use of per-node unique origin ASNs for globally anycasted 619 critical infrastructure services in order to provide routing system 620 discriminators for a given anycasted prefix. Network management and 621 monitoring techniques, or other operational mechanisms may then 622 employ this new discriminator in whatever manner fits their operating 623 environment, either for detection or policy associated with a given 624 anycasted node. 626 Moreover, the use of per-node unique origin ASNs has the additional 627 benefit of overcoming complications that might arise with the 628 potential deployment of the Resource Public Key Infrastructure (RPKI) 629 [RFC6480]. Without per-node unique origin ASNs, the cryptographic 630 certificates needed to attest to the Route Origin Authorizations 631 (ROAs) of a multi-administrative deployment of anycast would need to 632 be shared. However, if each service instance has a separate ASN, 633 then those ASNs can be managed separately in the RPKI. 635 Unlike multicast (but like unicast), anycast allows traffic stealing. 636 That is, with multicast, joining a multicast group doesn't prevent 637 anyone else who was receiving the traffic from continuing to receive 638 the traffic. With anycast, adding an anycasted node to the routing 639 system can prevent a previous recipient from continuing to receive 640 traffic because it may now be delivered to the new node instead. As 641 such, if an unauthorized anycast node can inject a route into the 642 network, or be resolved using ARP/Neighbor Discovery on a link with 643 an authorized anycast node, traffic can be diverted thereby 644 triggering DoS or other attacks. Section 6.3 of BCP 126 [RFC4786] 645 provides expanded discussion on "Service Hijacking" and "traffic 646 stealing", and [FanInfocom13] discusses measured instances of anycast 647 nodes and "benign masquerading or hostile hijacking of anycast 648 services", by unauthorized nodes. 650 Unlike unicast (but like multicast), the desire is to allow 651 applications to cause route injection. In multicast, one often 652 allows arbitrary applications on hosts to join multicast groups, 653 resulting in multicast routing state. Trying to apply that same 654 model to anycast would present new security concerns, which is why 655 [MAGMA] only got so far. The security concerns include: 657 1. Allowing route injection can cause DOS to a legitimate address 658 owner. 660 2. Allowing route injection consumes routing resources and can hence 661 cause DOS to the routing system and impact legitimate 662 communications as a result. 664 These are two of the core issues that were part of the discussion 665 during [RFC1884], the [ANYCASTBOF], and the MAGMA [MAGMA] chartering. 667 Additional security considerations are scattered throughout the list 668 of references provided herein. 670 4.5. Deployment Considerations 672 BCP 126 [RFC4786] provides some very solid guidance related to 673 operations of anycasted services, and in particular DNS. 675 This document covers issues associated with the architectural 676 implications of anycast. This document does not treat in any depth 677 the fact that there are deployed services with TCP transport using 678 anycast today. Evidence exists to suggest that such practice is not 679 "safe" in the traditional and architectural sense (as described in 680 Section 4.2). These sorts of issues are indeed relative, and we 681 recognize sometimes unpredictability in the routing system beyond the 682 local administrative domain can be manageable. That is, despite the 683 inherent architectural problems in the use of anycast with stateful 684 transport and connection-oriented protocols, there is expanding 685 deployment (e.g., for content distribution networks) and situations 686 exist where it may make sense (e.g., such as with service discovery, 687 short-lived transactions, or in cases where dynamically directing 688 traffic to topologically optimal service instances is required). In 689 general, operators should consider the content and references 690 provided herein, and evaluate the benefits and implications of 691 anycast in their specific environments and applications. 693 In addition, (as noted in Section 2.3) the issue of whether to 694 withdraw anycast routes when there is a service failure is only 695 briefly broached in [RFC3258]. The advice given is that routes 696 should not be withdrawn, in order to reduce operational complexity. 697 However, the issue of route advertisements and service outages 698 deserves greater attention. 700 There is an inherent tradeoff that exists between the operational 701 complexity of matching service outages with anycast route 702 withdrawals, and allowing anycast routes to persist for services that 703 are no longer available. [RFC3258] maintains that DNS' inherent 704 failure recovery mechanism is sufficient to overcome failed nodes, 705 but even this advice enshrines the notion that these decisions are 706 both application-specific and subject to the operational needs of 707 each deployment. For example, the routing system plays a larger role 708 in DNS when services are anycast. Therefore, operational 709 consideration must be given to the fact that relying on anycast for 710 DNS deployment optimizations means that there are operational 711 tradeoffs related to keeping route advertisements (and withdrawals) 712 symmetric with service availability. For example, in order to ensure 713 that the DNS resolvers in a failed anycast instance's catchment 714 [RFC4786] are able to fail over and reach a non-failed catchment, a 715 route withdrawal is almost certainly required. On the other hand, 716 instability of a DNS process that triggers frequent route 717 advertisement and withdrawal might result in suppression of 718 legitimate paths to available nodes, e.g., as a result of route flap 719 damping [RFC2439]. 721 Rather than prescribing advice that attempts to befit all situations, 722 it should simply be recognized that when using anycast with network 723 services that provide redundancy or resilience capabilities at other 724 layers of the protocol stack, operators should carefully consider the 725 optimal layer(s) at which to provide said functions. 727 As noted in Section 2.3, use of anycast within a subnet does not 728 necessarily suffer from the potential issues with route withdrawals. 729 As such, use of anycast to reach servers that reside in the same 730 subnet can be made more reliable than use of anycast to reach 731 topologically disparate server instances. Within a subnet, however, 732 care must be taken as stated in Section 5.4 of [RFC4862], "Duplicate 733 Address Detection MUST NOT be performed on anycast addresses", and 734 hence the servers must be configured appropriately. 736 5. IANA Considerations 738 No IANA actions are required. 740 6. Conclusions 742 In summary, operators and application vendors alike should consider 743 the benefits and implications of anycast in their specific 744 environments and applications, and also give forward consideration to 745 how new network protocols and application functions may take 746 advantage of anycast, or how they may be negatively impacted if 747 anycasting is employed. 749 7. Acknowledgements 751 Many thanks to Kurtis Lindqvist for his early review and feedback on 752 this document. Thanks to Brian Carpenter, Alfred Hoenes, and Joe 753 Abley for their usual careful review and feedback as well as Mark 754 Smith for his detailed review. 756 8. Informative References 758 [ANYCASTBOF] 759 Deering, S., "IAB Anycast BOF Announcement", October 1999, 760 . 763 [FanInfocom13] 764 Fan, X., Heidemann, J., and R. Govindan, "Evaluating 765 Anycast in the Domain Name System", Proceedings of the 766 IEEE Infocom 2013, April 2013. 768 [I-D.ietf-ipv6-dns-discovery] 769 Durand, A., Hagino, J., and D. Thaler, "Well known site 770 local unicast addresses for DNS resolver", September 2002. 772 [IMR9401] "INTERNET MONTHLY REPORT", January 1994, . 775 [MAGMA] "Multicast and Anycast Group Membership (MAGMA), 776 concluded", April 2006, 777 . 779 [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, RFC 780 793, September 1981. 782 [RFC1035] Mockapetris, P., "Domain names - implementation and 783 specification", STD 13, RFC 1035, November 1987. 785 [RFC1546] Partridge, C., Mendez, T., and W. Milliken, "Host 786 Anycasting Service", RFC 1546, November 1993. 788 [RFC1884] Hinden, R. and S. Deering, "IP Version 6 Addressing 789 Architecture", RFC 1884, December 1995. 791 [RFC2030] Mills, D., "Simple Network Time Protocol (SNTP) Version 4 792 for IPv4, IPv6 and OSI", RFC 2030, October 1996. 794 [RFC2101] Carpenter, B., Crowcroft, J., and Y. Rekhter, "IPv4 795 Address Behaviour Today", RFC 2101, February 1997. 797 [RFC2373] Hinden, R. and S. Deering, "IP Version 6 Addressing 798 Architecture", RFC 2373, July 1998. 800 [RFC2439] Villamizar, C., Chandra, R., and R. Govindan, "BGP Route 801 Flap Damping", RFC 2439, November 1998. 803 [RFC2526] Johnson, D. and S. Deering, "Reserved IPv6 Subnet Anycast 804 Addresses", RFC 2526, March 1999. 806 [RFC2893] Gilligan, R. and E. Nordmark, "Transition Mechanisms for 807 IPv6 Hosts and Routers", RFC 2893, August 2000. 809 [RFC2902] Deering, S., Hares, S., Perkins, C., and R. Perlman, 810 "Overview of the 1998 IAB Routing Workshop", RFC 2902, 811 August 2000. 813 [RFC2991] Thaler, D. and C. Hopps, "Multipath Issues in Unicast and 814 Multicast Next-Hop Selection", RFC 2991, November 2000. 816 [RFC3056] Carpenter, B. and K. Moore, "Connection of IPv6 Domains 817 via IPv4 Clouds", RFC 3056, February 2001. 819 [RFC3068] Huitema, C., "An Anycast Prefix for 6to4 Relay Routers", 820 RFC 3068, June 2001. 822 [RFC3258] Hardie, T., "Distributing Authoritative Name Servers via 823 Shared Unicast Addresses", RFC 3258, April 2002. 825 [RFC3513] Hinden, R. and S. Deering, "Internet Protocol Version 6 826 (IPv6) Addressing Architecture", RFC 3513, April 2003. 828 [RFC3775] Johnson, D., Perkins, C., and J. Arkko, "Mobility Support 829 in IPv6", RFC 3775, June 2004. 831 [RFC3882] Turk, D., "Configuring BGP to Block Denial-of-Service 832 Attacks", RFC 3882, September 2004. 834 [RFC3964] Savola, P. and C. Patel, "Security Considerations for 835 6to4", RFC 3964, December 2004. 837 [RFC4033] Arends, R., Austein, R., Larson, M., Massey, D., and S. 838 Rose, "DNS Security Introduction and Requirements", RFC 839 4033, March 2005. 841 [RFC4085] Plonka, D., "Embedding Globally-Routable Internet 842 Addresses Considered Harmful", BCP 105, RFC 4085, June 843 2005. 845 [RFC4213] Nordmark, E. and R. Gilligan, "Basic Transition Mechanisms 846 for IPv6 Hosts and Routers", RFC 4213, October 2005. 848 [RFC4291] Hinden, R. and S. Deering, "IP Version 6 Addressing 849 Architecture", RFC 4291, February 2006. 851 [RFC4302] Kent, S., "IP Authentication Header", RFC 4302, December 852 2005. 854 [RFC4303] Kent, S., "IP Encapsulating Security Payload (ESP)", RFC 855 4303, December 2005. 857 [RFC4305] Eastlake, D., "Cryptographic Algorithm Implementation 858 Requirements for Encapsulating Security Payload (ESP) and 859 Authentication Header (AH)", RFC 4305, December 2005. 861 [RFC4330] Mills, D., "Simple Network Time Protocol (SNTP) Version 4 862 for IPv4, IPv6 and OSI", RFC 4330, January 2006. 864 [RFC4339] Jeong, J., "IPv6 Host Configuration of DNS Server 865 Information Approaches", RFC 4339, February 2006. 867 [RFC4610] Farinacci, D. and Y. Cai, "Anycast-RP Using Protocol 868 Independent Multicast (PIM)", RFC 4610, August 2006. 870 [RFC4732] Handley, M., Rescorla, E., IAB, "Internet Denial-of- 871 Service Considerations", RFC 4732, December 2006. 873 [RFC4778] Kaeo, M., "Operational Security Current Practices in 874 Internet Service Provider Environments", RFC 4778, January 875 2007. 877 [RFC4786] Abley, J. and K. Lindqvist, "Operation of Anycast 878 Services", BCP 126, RFC 4786, December 2006. 880 [RFC4861] Narten, T., Nordmark, E., Simpson, W., and H. Soliman, 881 "Neighbor Discovery for IP version 6 (IPv6)", RFC 4861, 882 September 2007. 884 [RFC4862] Thomson, S., Narten, T., and T. Jinmei, "IPv6 Stateless 885 Address Autoconfiguration", RFC 4862, September 2007. 887 [RFC4892] Woolf, S. and D. Conrad, "Requirements for a Mechanism 888 Identifying a Name Server Instance", RFC 4892, June 2007. 890 [RFC4924] Aboba, B. and E. Davies, "Reflections on Internet 891 Transparency", RFC 4924, July 2007. 893 [RFC4942] Davies, E., Krishnan, S., and P. Savola, "IPv6 Transition/ 894 Co-existence Security Considerations", RFC 4942, September 895 2007. 897 [RFC5001] Austein, R., "DNS Name Server Identifier (NSID) Option", 898 RFC 5001, August 2007. 900 [RFC5905] Mills, D., Martin, J., Burbank, J., and W. Kasch, "Network 901 Time Protocol Version 4: Protocol and Algorithms 902 Specification", RFC 5905, June 2010. 904 [RFC5909] Combes, J-M., Krishnan, S., and G. Daley, "Securing 905 Neighbor Discovery Proxy: Problem Statement", RFC 5909, 906 July 2010. 908 [RFC6275] Perkins, C., Johnson, D., and J. Arkko, "Mobility Support 909 in IPv6", RFC 6275, July 2011. 911 [RFC6382] McPherson, D., Donnelly, R., and F. Scalzo, "Unique Origin 912 Autonomous System Numbers (ASNs) per Node for Globally 913 Anycasted Services", BCP 169, RFC 6382, October 2011. 915 [RFC6480] Lepinski, M. and S. Kent, "An Infrastructure to Support 916 Secure Internet Routing", RFC 6480, February 2012. 918 [RSSAC29] "RSSAC 29 Meeting Minutes", December 2007, . 921 Appendix A. IAB Members at the Time of Approval 923 Bernard Aboba 924 Jari Arkko 925 Marc Blanchet 926 Ross Callon 927 Alissa Cooper 928 Joel Halpern 929 Russ Housley 930 Eliot Lear 931 Xing Li 932 Erik Nordmark 933 Andrew Sullivan 934 Dave Thaler 935 Hannes Tschofenig 937 Authors' Addresses 939 Danny McPherson 940 Verisign, Inc. 941 12061 Bluemont Way 942 Reston, VA 943 USA 945 Email: dmcpherson@verisign.com 947 Dave Oran 948 Cisco Systems 949 USA 951 Email: oran@cisco.com 953 Dave Thaler 954 Microsoft Corporation 955 One Microsoft Way 956 Redmond, WA 957 USA 959 Email: dthaler@microsoft.com 961 Eric Osterweil 962 Verisign, Inc. 963 12061 Bluemont Way 964 Reston, VA 965 USA 967 Email: eosterweil@verisign.com