idnits 2.17.1 draft-mcmurry-dime-overload-reqs-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (June 18, 2012) is 4323 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-34) exists of draft-ietf-dime-rfc3588bis-33 Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group E. McMurry 3 Internet-Draft B. Campbell 4 Intended status: Standards Track Tekelec 5 Expires: December 20, 2012 June 18, 2012 7 Diameter Overload Control Requirements 8 draft-mcmurry-dime-overload-reqs-01 10 Abstract 12 When a Diameter server or agent becomes overloaded, it needs to be 13 able to gracefully reduce its load, typically by informing clients to 14 reduce sending traffic for some period of time. Otherwise, it must 15 continue to expend resources parsing and responding to Diameter 16 messages, possibly resulting in congestion collapse. The existing 17 mechanisms provided by Diameter are not sufficient for this purpose. 18 This document describes the limitations of the existing mechanisms, 19 and provides requirements for new overload management mechanisms. 21 Status of this Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at http://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on December 20, 2012. 38 Copyright Notice 40 Copyright (c) 2012 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents 45 (http://trustee.ietf.org/license-info) in effect on the date of 46 publication of this document. Please review these documents 47 carefully, as they describe your rights and restrictions with respect 48 to this document. Code Components extracted from this document must 49 include Simplified BSD License text as described in Section 4.e of 50 the Trust Legal Provisions and are provided without warranty as 51 described in the Simplified BSD License. 53 Table of Contents 55 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 56 1.1. Causes of Overload . . . . . . . . . . . . . . . . . . . . 3 57 1.2. Effects of Overload . . . . . . . . . . . . . . . . . . . 5 58 1.3. Overload vs. Network Congestion . . . . . . . . . . . . . 5 59 1.4. Diameter Applications in a Broader Network . . . . . . . . 5 60 1.5. Documentation Conventions . . . . . . . . . . . . . . . . 6 61 2. Overload Scenarios . . . . . . . . . . . . . . . . . . . . . . 6 62 2.1. Peer to Peer Scenarios . . . . . . . . . . . . . . . . . . 7 63 2.2. Agent Scenarios . . . . . . . . . . . . . . . . . . . . . 9 64 3. Existing Mechanisms . . . . . . . . . . . . . . . . . . . . . 12 65 4. Issues with the Current Mechanisms . . . . . . . . . . . . . . 13 66 4.1. Problems with Implicit Mechanism . . . . . . . . . . . . . 13 67 4.2. Problems with Explicit Mechanisms . . . . . . . . . . . . 14 68 5. Diameter Overload Case Studies . . . . . . . . . . . . . . . . 15 69 5.1. Overload in Mobile Data Networks . . . . . . . . . . . . . 15 70 5.2. 3GPP Study on Core Network Overload . . . . . . . . . . . 16 71 6. Solution Requirements . . . . . . . . . . . . . . . . . . . . 16 72 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 73 8. Security Considerations . . . . . . . . . . . . . . . . . . . 21 74 8.1. Access Control . . . . . . . . . . . . . . . . . . . . . . 22 75 8.2. Denial-of-Service Attacks . . . . . . . . . . . . . . . . 22 76 8.3. Replay Attacks . . . . . . . . . . . . . . . . . . . . . . 22 77 8.4. Man-in-the-Middle Attacks . . . . . . . . . . . . . . . . 22 78 8.5. Compromised Hosts . . . . . . . . . . . . . . . . . . . . 23 79 9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 23 80 9.1. Normative References . . . . . . . . . . . . . . . . . . . 23 81 9.2. Informative References . . . . . . . . . . . . . . . . . . 23 82 Appendix A. Contributors . . . . . . . . . . . . . . . . . . . . 24 83 Appendix B. Acknowledgements . . . . . . . . . . . . . . . . . . 24 84 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 24 86 1. Introduction 88 When a Diameter [I-D.ietf-dime-rfc3588bis] server or agent becomes 89 overloaded, it needs to be able to gracefully reduce its load, 90 typically by informing clients to reduce sending traffic for some 91 period of time. Otherwise, it must continue to expend resources 92 parsing and responding to Diameter messages, possibly resulting in 93 congestion collapse. The existing mechanisms provided by Diameter 94 are not sufficient for this purpose. This document describes the 95 limitations of the existing mechanisms, and provides requirements for 96 new overload management mechanisms. 98 This document draws on [RFC5390] and the work done on SIP overload 99 control as well as on overload practices in SS7 networks and studies 100 done by 3GPP. 102 Diameter is not typically an end-user protocol; rather it is 103 generally used as one component in support of some end-user activity. 104 For example, a WiFi access point might use Diameter to authenticate 105 and authorize user access via 802.11. Overload in a network that 106 uses Diameter applications will likely spill over into the end-user 107 application network. The impact of Diameter overload on the client 108 application (a client application may use the Diameter protocol and 109 other protocols to do its job) is beyond the scope of this document. 111 This document presents non-normative descriptions of causes of 112 overload along with related scenarios and studies. Finally, it 113 offers a set of normative requirements for an improved overload 114 indication mechanism. 116 1.1. Causes of Overload 118 Overload occurs when an element, such as a Diameter server or agent, 119 has insufficient resources to successfully process all of the traffic 120 it is receiving. Resources include all of the capabilities of the 121 element used to process a request, including CPU processing, memory, 122 I/O, and disk resources. It can also include external resources such 123 as a database or DNS server, in which case the CPU, processing, 124 memory, I/O, and disk resources of those elements are effectively 125 part of the logical element processing the request. 127 Overload can occur for many reasons, including: 129 Inadequate capacity: When designing Diameter networks, that is, 130 application layer multi-node Diameter deployments, it can be very 131 difficult to predict all scenarios that may cause elevated 132 traffic. It may also be more costly to implement support for some 133 scenarios than a network operator may deem worthwhile. This 134 results in the likelihood that a Diameter network will not have 135 adequate capacity to handle all situations. 137 Dependency failures: A Diameter node can become overloaded because a 138 resource on which it is dependent has failed or become overloaded, 139 greatly reducing the logical capacity of the node. In these 140 cases, even minimal traffic might cause the node to go into 141 overload. Examples of such dependency overloads include DNS 142 servers, databases, disks, and network interfaces. 144 Component failures: A Diameter node can become overloaded when it is 145 a member of a cluster of servers that each share the load of 146 traffic, and one or more of the other members in the cluster fail. 147 In this case, the remaining nodes take over the work of the failed 148 nodes. Normally, capacity planning takes such failures into 149 account, and servers are typically run with enough spare capacity 150 to handle failure of another node. However, unusual failure 151 conditions can cause many nodes to fail at once. This is often 152 the case with software failures, where a bad packet or bad 153 database entry hits the same bug in a set of nodes in a cluster. 155 Network Initiated Traffic Flood: Issues with the radio access 156 network in a mobile network such as radio overlays with frequent 157 handovers, and operational changes are examples of network events 158 that can precipitate a flood of Diameter signaling traffic, such 159 as an avalanche restart. Failure of a Diameter proxy may also 160 result in a large amount of signaling as connections and sessions 161 are reestablished. 163 Subscriber Initiated Traffic Flood: Large gatherings of subscribers 164 or events that result in many subscribers interacting with the 165 network in close time proximity can result in Diameter signaling 166 traffic floods. For example, the finale of a large fireworks show 167 could be immediately followed by many subscribers posting 168 messages, pictures, and videos concentrated on one portion of a 169 network. Subscriber devices, such as smartphones, may use 170 aggressive registration strategies that generate unusually high 171 Diameter traffic loads. 173 DoS attacks: An attacker, wishing to disrupt service in the network, 174 can cause a large amount of traffic to be launched at a target 175 element. This can be done from a central source of traffic or 176 through a distributed DoS attack. In all cases, the volume of 177 traffic well exceeds the capacity of the element, sending the 178 system into overload. 180 1.2. Effects of Overload 182 Modern Diameter networks, comprised of application layer multi-node 183 deployments of Diameter elements, may operate at very large 184 transaction volumes. If a Diameter node becomes overloaded, or even 185 worse, fails completely, a large number of messages may be lost very 186 quickly. Even with redundant servers, many messages can be lost in 187 the time it takes for failover to complete. While a Diameter client 188 or agent should be able to retry such requests, an overloaded peer 189 may cause a sudden large increase in the number of transaction 190 transactions needing to be retried, rapidly filling local queues or 191 otherwise contributing to local overload. Therefore Diameter devices 192 need to be able to shed load before critical failures can occur. 194 Diameter depends heavily on The "Authentication, Authorization, 195 and Accounting (AAA) Transport Profile" [RFC3539], which states 196 assumptions about the scale of AAA services which may be incorrect 197 for current uses of Diameter. In particular, the document 198 suggests that AAA services will typically be low volume and that 199 traffic will typically be application-driven. Section 2.1 of that 200 document uses an example of a 48 port NAS. However, Diameter is 201 commonly used in large-scale mobile data environments, where a 202 typical client could be a packet gateway that serves millions of 203 users, and generates Diameter messages at network-driven rates. 205 1.3. Overload vs. Network Congestion 207 This document uses the term "overload" to refer to application-layer 208 overload at Diameter nodes. This is distinct from "network 209 congestion", that is, congestion that occurs at the lower networking 210 layers that may impact the delivery of Diameter messages between 211 nodes. The authors recognize that element overload and network 212 congestion are interrelated, and that overload can contribute to 213 network congestion and vice versa. 215 Network congestion issues are better handled by the transport 216 protocols. Diameter uses TCP and SCTP, both of which include 217 congestion management features. Analysis of whether those features 218 are sufficient for transport level congestion between Diameter nodes, 219 and any work to further mitigate network congestion is out of scope 220 both for this document, and for the work proposed by this document. 222 1.4. Diameter Applications in a Broader Network 224 Most elements using Diameter applications do not use Diameter 225 exclusively. It is important to realize that overload of an element 226 can be caused by a number of factors that may be unrelated to the 227 processing of Diameter or Diameter applications. 229 A element communicating via protocols other than Diameter that is 230 also using a Diameter application needs to be able to signal to 231 Diameter peers that it is experiencing overload regardless of the 232 cause of the overload, since the overload will affect that element's 233 ability to process Diameter transactions. The element may also need 234 to signal this on other protocols depending on its function and the 235 architecture of the network and application it is providing services 236 for. Whether that is necessary can only be decided within the 237 context of that architecture and application. A mechanism for 238 signaling overload with Diameter, which this specification details 239 the requirements for, provides applications the ability to signal 240 their Diameter peers of overload, mitigating that part of the issue. 241 Applications may need to use this, as well as other mechanisms, to 242 solve their broader overload issues. Indicating overload on 243 protocols other than Diameter is out of scope for this document, and 244 for the work proposed by this document. 246 1.5. Documentation Conventions 248 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 249 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 250 document are to be interpreted as described in [RFC2119]. 252 The terms "client", "server", "agent", "node", "peer", "upstream", 253 and "downstream" are used as defined in [I-D.ietf-dime-rfc3588bis]. 255 2. Overload Scenarios 257 Several Diameter deployment scenarios exist that may impact overload 258 management. The following scenarios help motivate the requirements 259 for an overload management mechanism. 261 These scenarios are by no means exhaustive, and are in general 262 simplified for the sake of clarity. In particular, the authors 263 assume for the sake of clarity that the client sends Diameter 264 requests to the server, and the server sends responses to client, 265 even though Diameter supports bidirectional applications. Each 266 direction in such an application can be modeled separately. 268 In a large scale deployment, many of the nodes represented in these 269 scenarios would be deployed as clusters of servers. The authors 270 assume that such a cluster is responsible for managing its own 271 internal load balancing and overload management so that it appears as 272 a single Diameter node. That is, other Diameter nodes can treat it 273 as single, monolithic node for the purposes of overload management. 275 These scenarios do not illustrate the client application. As 276 mentioned in Section 1, Diameter is not typically an end-user 277 protocol; rather it is generally used in support of some other client 278 application. These scenarios do not consider the impact of Diameter 279 overload on the client application. 281 2.1. Peer to Peer Scenarios 283 This section describes Diameter peer-to-peer scenarios. That is, 284 scenarios where a Diameter client talks directly with a Diameter 285 server, without the use of a Diameter agent. 287 Figure 1 illustrates the simplest possible Diameter relationship. 288 The client and server share a one-to-one peer-to-peer relationship. 289 If the server becomes overloaded, either because the client exceeds 290 the server's capacity, or because the server's capacity is reduced 291 due to some resource dependency, the client needs to reduce the 292 amount of Diameter traffic it sends to the server. Since the client 293 cannot forward requests to another server, it must either queue 294 requests until the server recovers, or itself become overloaded in 295 the context of the client application and other protocols it may also 296 use. 298 +------------------+ 299 | | 300 | | 301 | Server | 302 | | 303 +--------+---------+ 304 | 305 | 306 +--------+---------+ 307 | | 308 | | 309 | Client | 310 | | 311 +------------------+ 313 Figure 1: Basic Peer to Peer Scenario 315 Figure 2 shows a similar scenario, except in this case the client has 316 multiple servers that can handle work for a specific realm and 317 application. If server 1 becomes overloaded, the client can forward 318 traffic to server 2. Assuming server 2 has sufficient reserve 319 capacity to handle the forwarded traffic, the client should be able 320 to continue serving client application protocol users. If server 1 321 is approaching overload, but can still handle some number of new 322 request, it needs to be able to instruct the client to forward a 323 subset of its traffic to server 2. 325 +------------------+ +------------------+ 326 | | | | 327 | | | | 328 | Server 1 | | Server 2 | 329 | | | | 330 +--------+-`.------+ +------.'+---------+ 331 `. .' 332 `. .' 333 `. .' 334 `. .' 335 +-------`.'--------+ 336 | | 337 | | 338 | Client | 339 | | 340 +------------------+ 342 Figure 2: Multiple Server Peer to Peer Scenario 344 Figure 3 illustrates a peer-to-peer scenario with multiple Diameter 345 realm and application combinations. In this example, server 2 can 346 handle work for both applications. Each application might have 347 different resource dependencies. For example, a server might need to 348 access one database for application A, and another for application B. 349 This creates a possibility that Server 2 could become overloaded for 350 application A but not for application B, in which case the client 351 would need to divert some part of its application A requests to 352 server 1, but should not divert any application B requests. This 353 requires server 2 to be able to distinguish between applications when 354 it indicates an overload condition to the client. 356 On the other hand, it's possible that the servers host many 357 applications. If server 2 becomes overloaded for all applications, 358 it would be undesirable for it to have to notify the client 359 separately for each application. Therefore it also needs a way to 360 indicate that it is overloaded for all possible applications. 362 +----------------------------------------------+ 363 | Application A +------------------------+----------------------+ 364 |+------------------+ | +------------------+ | +------------------+| 365 || | | | | | | || 366 || | | | | | | || 367 || Server 1 | | | Server 2 | | | Server 3 || 368 || | | | | | | || 369 |+--------+---------+ | +--------+---------+ | +-+----------------+| 370 | | | | | | | 371 +---------+-----------+-----------+------------+ | | 372 | | | | | 373 | | | | Application B | 374 | +-----------+-----------------+-----------------+ 375 ``-.._ | | 376 `-..__ | _.-'' 377 `--._ | _.-'' 378 ``-.__ | _.-'' 379 +------`-.-''------+ 380 | | 381 | | 382 | Client | 383 | | 384 +------------------+ 386 Figure 3: Multiple Application Peer to Peer Scenario 388 2.2. Agent Scenarios 390 This section describes scenarios that include a Diameter agent, 391 either in the form of a Diameter relay or Diameter proxy. These 392 scenarios do not consider Diameter redirect agents, since they are 393 more readily modeled as end-servers. 395 Figure 4 illustrates a simple Diameter agent scenario with a single 396 client, agent, and server. In this case, overload can occur at the 397 server, at the agent, or both. But in most cases, client behavior is 398 the same whether overload occurs at the server or at the agent. From 399 the client's perspective, server overload and agent overload is the 400 same thing. 402 +------------------+ 403 | | 404 | | 405 | Server | 406 | | 407 +--------+---------+ 408 | 409 | 410 +--------+---------+ 411 | | 412 | | 413 | Agent | 414 | | 415 +--------+---------+ 416 | 417 | 418 +--------+---------+ 419 | | 420 | | 421 | Client | 422 | | 423 +------------------+ 425 Figure 4: Basic Agent Scenario 427 Figure 5 shows an agent scenario with multiple servers. If server 1 428 becomes overloaded, but server 2 has sufficient reserve capacity, the 429 agent may be able to transparently divert some or all Diameter 430 requests originally bound for server 1 to server 2. 432 In most cases, the client does not have detailed knowledge of the 433 Diameter topology upstream of the agent. If the agent uses dynamic 434 discovery to find eligible servers, the set of eligible servers may 435 not be enumerable from the perspective of the client. Therefore, in 436 most cases the agent needs to deal with any upstream overload issues 437 in a way that is transparent to the client. If one server notifies 438 the agent that it has become overloaded, the notification should not 439 be passed back to the client in a way where the client could 440 mistakenly perceive the agent itself as being overloaded. If the set 441 of all possible destinations upstream of the agent no longer has 442 sufficient capacity for incoming load, the agent itself becomes 443 effectively overloaded. 445 On the other hand, there are cases where the client needs to be able 446 to select a particular server from behind an agent. For example, if 447 a Diameter request is part of a multiple-round-trip authentication, 448 or is otherwise part of a Diameter "session", it may have a 449 DestinationHost AVP that requires the request to be served by server 450 1. Therefore the agent may need to inform a client that a particular 451 upstream server is overloaded or otherwise unavailable. Note that 452 there can be many ways a server can be specified, which may have 453 different implications (e.g. by IP address, by host name, etc). 455 +------------------+ +------------------+ 456 | | | | 457 | | | | 458 | Server 1 | | Server 2 | 459 | | | | 460 +--------+-`.------+ +------.'+---------+ 461 `. .' 462 `. .' 463 `. .' 464 `. .' 465 +-------`.'--------+ 466 | | 467 | | 468 | Agent | 469 | | 470 +--------+---------+ 471 | 472 | 473 | 474 +--------+---------+ 475 | | 476 | | 477 | Client | 478 | | 479 +------------------+ 481 Figure 5: Multiple Server Agent Scenario 483 Figure 6 shows a scenario where an agent routes requests to a set of 484 servers for more than one Diameter realm and application. In this 485 scenario, if server 1 becomes overloaded or unavailable, the agent 486 may effectively operate at reduced capacity for application A, but at 487 full capacity for application B. Therefore, the agent needs to be 488 able to report that it is overloaded for one application, but not for 489 another. 491 +----------------------------------------------+ 492 | Application A +------------------------+----------------------+ 493 |+------------------+ | +------------------+ | +------------------+| 494 || | | | | | | || 495 || | | | | | | || 496 || Server 1 | | | Server 2 | | | Server 3 || 497 || | | | | | | || 498 |+---------+--------+ | +--------+---------+ | +--+---------------+| 499 | | | | | | | 500 +----------+----------+-----------+------------+ | | 501 | | | | | 502 | | | | Application B | 503 | +-----------+------------------+----------------+ 504 | | | 505 ``--.__ | _. 506 ``-.__ | __.--'' 507 `--.._ | _..--' 508 +-----``-+.-''-----+ 509 | | 510 | | 511 | Agent | 512 | | 513 +--------+---------+ 514 | 515 | 516 +--------+---------+ 517 | | 518 | | 519 | Client | 520 | | 521 +------------------+ 523 Figure 6: Multiple Application Agent Scenario 525 3. Existing Mechanisms 527 Diameter offers both implicit and explicit mechanisms for a Diameter 528 node to learn that a peer is overloaded or unreachable. The implicit 529 mechanism is simply the lack of responses to requests. If a client 530 fails to receive a response in a certain time period, it assumes the 531 upstream peer is unavailable, or overloaded to the point of effective 532 unavailability. The watchdog mechanism [RFC3539] ensures that a 533 certain rate of transaction responses occur even when there is 534 otherwise little or no other Diameter traffic. 536 The explicit mechanism involves specific protocol error responses, 537 where an agent or server can tell a downstream peer that it is either 538 too busy to handle a request (DIAMETER_TOO_BUSY) or unable to route a 539 request to an upstream destination (DIAMETER_UNABLE_TO_DELIVER), 540 perhaps because that destination itself is overloaded to the point of 541 unavailability. 543 Once a Diameter node learns that an upstream peer has become 544 overloaded via one of these mechanisms, it can then attempt to take 545 action to reduce the load. This usually means forwarding traffic to 546 an alternate destination, if available. If no alternate destination 547 is available, the node must either reduce the number of messages it 548 originates (in the case of a client) or inform the client to reduce 549 traffic (in the case of an agent.) 551 Diameter requires the use of a congestion-managed transport layer, 552 currently TCP or SCTP, to mitigate network congestion. It is 553 expected that these transports manage network congestion and that 554 issues with transport (e.g. congestion propagation and window 555 management) are managed at that level. But even with a congestion- 556 managed transport, a Diameter node can become overloaded at the 557 Diameter protocol or application layers due to the causes described 558 in Section 1.1 and congestion managed transports do not provide 559 facilities (and are at the wrong level) to handle server overload. 560 Transport level congestion management is also not sufficient to 561 address overload in cases of multi-hop and multi-destination 562 signaling. 564 4. Issues with the Current Mechanisms 566 The currently available Diameter mechanisms for indicating an 567 overload condition are not adequate to avoid service outages due to 568 overload. This may, in turn, contribute to broader congestion 569 collapse due to unresponsive Diameter nodes causing application or 570 transport layer retransmissions. In particular, they do not allow a 571 Diameter agent or server to shed load as it approaches overload. At 572 best, a node can only indicate that it needs to entirely stop 573 receiving requests, i.e. that it has effectively failed. Even that 574 is problematic due to the inability to indicate durational validity 575 on the transient errors available in the base Diameter protocol. 576 Diameter offers no mechanism to allow a node to indicate different 577 overload states for different categories of messages, for example, if 578 it is overloaded for one Diameter application but not another. 580 4.1. Problems with Implicit Mechanism 582 The implicit mechanism doesn't allow an agent or server to inform the 583 client of a problem until it is effectively too late to do anything 584 about it. The client does not know to take action until the upstream 585 node has effectively failed. A Diameter node has no opportunity to 586 shed load early to avoid collapse in the first place. 588 Additionally, the implicit mechanism cannot distinguish between 589 overload of a Diameter node and network congestion. Diameter treats 590 the failure to receive an answer as a transport failure. 592 4.2. Problems with Explicit Mechanisms 594 The Diameter specification is ambiguous on how a client should handle 595 receipt of a DIAMETER_TOO_BUSY response. The base specification 596 [I-D.ietf-dime-rfc3588bis] indicates that the sending client should 597 attempt to send the request to a different peer. It makes no 598 suggestion that a the receipt of a DIAMETER_TOO_BUSY response should 599 affect future Diameter messages in any way. 601 The Authentication, Authorization, and Accounting (AAA) Transport 602 Profile [RFC3539] recommends that a AAA node that receives a "Busy" 603 response failover all remaining requests to a different agent or 604 server. But while the Diameter base specification explicitly depends 605 on RFC3539 to define transport behavior, it does not refer to RFC3539 606 in the description of behavior on receipt of DIAMETER_TOO_BUSY. 607 There's a strong likelihood that at least some implementations will 608 continue to send Diameter requests to an upstream peer even after 609 receiving a DIAMETER_TOO_BUSY error. 611 BCP 41 [RFC2914] describes, among other things, how end-to-end 612 application behavior can help avoid congestion collapse. In 613 particular, an application should avoid sending messages that will 614 never be delivered or processed. The DIAMETER_TOO_BUSY behavior as 615 described in the Diameter base specification fails at this, since if 616 an upstream node becomes overloaded, a client attempts each request, 617 and does not discover the need to failover the request until the 618 initial attempt fails. 620 The situation is improved if implementations follow the [RFC3539] 621 recommendation and keep state about upstream peer overload. But even 622 then, the Diameter specification offers no guidance on how long a 623 client should wait before retrying the overloaded destination. If an 624 agent or server supports multiple realms and/or applications, 625 DIAMETER_TOO_BUSY only offers no way to indicate that it is 626 overloaded for one application but not another. A DIAMETER_TOO_BUSY 627 error can only indicate overload at a "whole server" scope. 629 Agent processing of a DIAMETER_TOO_BUSY response is also problematic 630 as described in the base specification. DIAMETER_TOO_BUSY is defined 631 as a protocol error. If an agent receives a protocol error, it may 632 either handle it locally or it may forward the response back towards 633 the downstream peer. (The Diameter specification is inconsistent 634 about whether a protocol error MAY or SHOULD be handled by an agent, 635 rather than forwarded downstream.) If a downstream peer receives the 636 DIAMETER_TOO_BUSY response, it may stop sending all requests to the 637 agent for some period of time, even though the agent may still be 638 able to deliver requests to other upstream peers. 640 DIAMETER_UNABLE_TO_DELIVER also has no mechanisms for specifying the 641 scope or cause of the failure, or the durational validity. 643 5. Diameter Overload Case Studies 645 5.1. Overload in Mobile Data Networks 647 As the number of Third Generation (3G) and Long Term Evolution (LTE) 648 enabled smartphone devices continue to expand in mobility networks, 649 there have been situations where high signaling traffic load led to 650 overload events at the Diameter-based Home Location Registries (HLR) 651 and/or Home Subscriber Servers (HSS). The root causes of the HLR 652 congestion events were manifold but included hardware failure and 653 procedural errors. The result was high signaling traffic load on the 654 HLR and HSS. 656 The 3GPP standards specification[need citation] for the end-to-end 657 signaling call flows in 3G and LTE, from the end user device 658 traversing through the radio and the core networks to the HLR/HSS, 659 did not have an equivalent load control mechanism which is provided 660 in the more traditional SS7 elements in GSM [need citation]. The 661 capabilities specified in the 3GPP standards do not adequately 662 address the abnormal condition where excessively high signaling 663 traffic load situations are experienced. 665 Smartphones contribute much more heavily to the continuation of a 666 registration surge due to their very aggressive registration 667 algorithms. The aggressive smartphone logic is designed to: 669 a. always have voice and data registration, and 671 b. constantly try to be on 3G data (and thus on 3G voice) for their 672 added benefits. 674 Non-smartphones typically have logic to wait for a time period after 675 registering successfully on voice and data. 677 The smartphone aggressive registration is problematic in two ways: 679 o first by generating excessive signaling load towards the HLR that 680 is ten times that from a non-smartphone, 682 o and second by causing continual registration attempts when a 683 network failure affects registrations through the 3G data network. 685 5.2. 3GPP Study on Core Network Overload 687 A study in 3GPP SA2 on core network overload has produced the 688 technical report [TR23.843]. This enumerates several causes of 689 overload in mobile core networks including portions that are signaled 690 using Diameter. This document is a work in progress and is not 691 complete. However, it is useful for pointing out scenarios and the 692 general need for an overload control mechanism for Diameter. 694 It is common for mobile networks to employ more than one radio 695 technology and to do so in an overlay fashion with multiple 696 technologies present in the same location (such as GSM, UMTS or CDMA 697 along with LTE). This presents opportunities for traffic storms when 698 issues occur on one overlay and not another as all devices that had 699 been on the overlay with issues switch. This causes a large amount 700 of Diameter traffic as locations and policies are updated. 702 Another scenario called out by this study is a flood of registration 703 and mobility management events caused by some element in the core 704 network failing. This flood of traffic from end nodes falls under 705 the network initiated traffic flood category. There is likely to 706 also be traffic resulting directly from the component failure in this 707 case. 709 Subscriber initiated traffic floods are also indicated in this study 710 as an overload mechanism where a large number of mobile devices 711 attempting to access services at the same time, such as in response 712 to an entertainment event or a catastrophic event. 714 While this study is concerned with the broader effects of these 715 scenarios on wireless networks and their elements, they have 716 implications specifically for Diameter signaling. One of the goals 717 of this document is to provide guidance for a core mechanism that can 718 be used to mitigate the scenarios called out by this study. 720 6. Solution Requirements 722 This section proposes requirements for an improved mechanism to 723 control Diameter overload, with the goals of improving the issues 724 described in Section 4 and supporting the scenarios described in 725 Section 2 726 REQ 1: The overload mechanism MUST provide a communication method 727 for Diameter nodes to exchange overload information. 729 REQ 2: The overload mechanism MUST be useable with any existing or 730 future Diameter application. It MUST NOT require 731 specification changes for existing Diameter applications. 733 REQ 3: The overload mechanism MUST limit the impact of overload on 734 the overall useful throughput of a Diameter server, even 735 when the incoming load on the network is far in excess of 736 its capacity. The overall useful throughput under load is 737 the ultimate measure of the value of an overload control 738 mechanism. 740 REQ 4: Diameter allows requests to be sent from either side of a 741 connection and either side of a connection may have need to 742 provide its overload status. The mechanism MUST allow each 743 side of a connection to independently inform the other of 744 its overload status. 746 REQ 5: Diameter allows nodes to determine their peers via dynamic 747 discovery or manual configuration. The mechanism MUST work 748 consistently without regard to how peers are determined. 750 REQ 6: The mechanism designers SHOULD seek to minimize the amount 751 of new configuration required in order to work. For 752 example, it is better to allow peers to advertise or 753 negotiate support for the mechanism, rather than to require 754 this knowledge to be configured at each node. 756 REQ 7: The overload mechanism MUST ensure that the system remains 757 stable. When the offered load drops from above the overall 758 capacity of the network to below the overall capacity, the 759 throughput MUST stabilize and become equal to the offered 760 load. 762 REQ 8: The mechanism MUST allow nodes to shed load without 763 introducing oscillations. Note that this requirement 764 implies a need for supporting nodes to be able to 765 distinguish current overload information from stale 766 information, and to make decisions using the most currently 767 available information. 769 REQ 9: The mechanism MUST function across fully loaded as well as 770 quiescent transport connections. This is partially derived 771 from the requirements for stability and hysteresis control 772 above. 774 REQ 10: Consumers of overload state indications MUST be able to 775 determine when the overload condition improves or ends. 777 REQ 11: The overload mechanism MUST be scalable. That is, it MUST 778 be able to operate in different sized networks. 780 REQ 12: When a single network node fails, goes into overload, or 781 suffers from reduced processing capacity, the mechanism MUST 782 make it possible to limit the impact of this on other nodes 783 in the network. This helps to prevent a small-scale failure 784 from becoming a widespread outage. 786 REQ 13: The mechanism MUST NOT introduce substantial additional work 787 for node in an overloaded state. For example, a requirement 788 for an overloaded node to send overload information every 789 time it received a new request would introduce substantial 790 work. Existing messaging is likely to have the 791 characteristic of increasing as an overload condition 792 approaches, allowing for the possibility of increased 793 feedback for information piggybacked on it. 795 REQ 14: Some scenarios that result in overload involve a rapid 796 increase of traffic with little time between normal levels 797 and overload inducing levels. The mechanism SHOULD provide 798 for increased feedback when traffic levels increase. The 799 mechanism MUST NOT do this in such a way that it increases 800 the number of messages while at high loads. 802 REQ 15: The mechanism MUST NOT interfere with the congestion control 803 mechanisms of underlying transport protocols. For example, 804 a mechanism that opened additional TCP connections when the 805 network is congested would reduce the effectiveness of the 806 underlying congestion control mechanisms. 808 REQ 16: The mechanism MUST operate without malfunction in an 809 environment with a mix of nodes that do, and nodes that do 810 not, support the mechanism. 812 REQ 17: In a mixed environment with nodes that support the overload 813 control mechanism and that do not, the mechanism MUST NOT 814 result in less useful throughput than would have resulted if 815 it were not present. It SHOULD result in less severe 816 congestion in this environment. 818 REQ 18: In a mixed environment of nodes that support the overload 819 control mechanism and that do not, users and operators of 820 nodes that do not support the mechanism MUST NOT benefit 821 from the mechanism more than users and operators of nodes 822 that support the mechanism. 824 REQ 19: It MUST be possible to use the mechanism between nodes in 825 different realms and in different administrative domains. 827 REQ 20: Any explicit overload indication MUST distinguish between 828 actual overload, as opposed to other, non-overload related 829 failures. 831 REQ 21: In cases where a network node fails, is so overloaded that 832 it cannot process messages, or cannot communicate due to a 833 network failure, it may not be able to provide explicit 834 indications of the nature of the failure or its levels of 835 congestion. The mechanism MUST properly function in these 836 cases. 838 REQ 22: The mechanism MUST provide a way for an node to throttle the 839 amount of traffic it receives from an peer node. This 840 throttling SHOULD be graded so that it can be applied 841 gradually as offered load increases. Overload is not a 842 binary state; there may be degrees of overload. 844 REQ 23: The mechanism MUST enable a supporting node to minimize the 845 chance that retries due to an overloaded or failed node 846 result in additional traffic to other overloaded nodes, or 847 cause additional nodes to become overloaded. Moreover, the 848 mechanism SHOULD provide unambiguous directions to clients 849 on when they should retry a request and when they should not 850 considering the various causes of overload such as avalanche 851 restart. 853 REQ 24: The mechanism MUST provide sufficient information to enable 854 a load balancing node to divert messages that are rejected 855 or otherwise throttled by an overloaded upstream node to 856 other upstream nodes that are the most likely to have 857 sufficient capacity to process them. 859 REQ 25: The mechanism MUST provide a mechanism for indicating load 860 levels even when not in an overloaded condition, to assist 861 nodes making decisions to prevent overload conditions from 862 occurring. 864 REQ 26: The specification for the overload mechanism SHOULD offer 865 guidance on which message types might be desirable to send 866 or process over others during times of overload, based on 867 Diameter-specific considerations. For example, it may be 868 more beneficial to process messages for existing sessions 869 ahead of new sessions. 871 REQ 27: The mechanism MUST NOT prevent a node from prioritizing 872 requests based on any local policy, so that certain requests 873 are given preferential treatment, given additional 874 retransmission, or processed ahead of others. 876 REQ 28: The overload mechanism MUST NOT provide new vulnerabilities 877 to malicious attack, or increase the severity of any 878 existing vulnerabilities. This includes vulnerabilities to 879 DoS and DDoS attacks as well as replay and man-in-the middle 880 attacks. 882 REQ 29: The mechanism MUST provide a means to match an overload 883 indication with the node that originated it. In particular, 884 the mechanism MUST allow a node to distinguish between 885 overload at a next-hop peer from overload at a node upstream 886 of the peer. For example, in Figure 5, the client must not 887 mistake overload at server 1 for overload at the agent, 888 whether or not the agent supports the mechanism.( see REQ 889 4). 891 REQ 30: The mechanism MUST NOT depend on being deployed in 892 environments where all Diameter nodes are completely 893 trusted. It SHOULD operate as effectively as possible in 894 environments where other nodes are malicious; this includes 895 preventing malicious nodes from obtaining more than a fair 896 share of service. Note that this does not imply any 897 responsibility on the mechanism to detect, or take 898 countermeasures against, malicious nodes. 900 REQ 31: It MUST be possible for a supporting node to make 901 authorization decisions about what information will be sent 902 to peer nodes based on the identity of those nodes. This 903 allows a domain administrator who considers the load of 904 their nodes to be sensitive information to restrict access 905 to that information. Of course, in such cases, there is no 906 expectation that the overload mechanism itself will help 907 prevent overload from that peer node. 909 REQ 32: The mechanism MUST NOT interfere with any Diameter compliant 910 method that a node may use to protect itself from overload 911 from non-supporting nodes, or from denial of service 912 attacks. 914 REQ 33: There are multiple situations where a Diameter node may be 915 overloaded for some purposes but not others. For example, 916 this can happen to an agent or server that supports multiple 917 applications, or when a server depends on multiple external 918 resources, some of which may become overloaded while others 919 are fully available. The mechanism MUST allow Diameter 920 nodes to indicate overload with sufficient granularity to 921 allow clients to take action based on the overloaded 922 resources without forcing available capacity to go unused. 923 The mechanism MUST support specification of overload 924 information with granularities of at least "Diameter node", 925 "realm", "Diameter application", and "Diameter session", and 926 SHOULD allow extensibility for others to be added in the 927 future. 929 REQ 34: The mechanism MUST provide a method for extending the 930 information communicated and the algorithms used for 931 overload control. 933 7. IANA Considerations 935 This document makes no requests of IANA. 937 8. Security Considerations 939 A Diameter overload control mechanism is primarily concerned with the 940 load and overload related behavior of nodes in a Diameter network, 941 and the information used to affect that behavior. Load and overload 942 information is shared between nodes and directly affects the behavior 943 and thus is potentially vulnerable to a number of methods of attack. 945 Load and overload information may also be sensitive from both 946 business and network protection viewpoints. Operators of Diameter 947 equipment want to control visibility to load and overload information 948 to keep it from being used for competitive intelligence or for 949 targeting attacks. It is also important that the Diameter overload 950 control mechanism not introduce any way in which any other 951 information carried by Diameter is sent inappropriately. 953 This document includes requirements intended to mitigate the effects 954 of attacks and to protect the information used by the mechanism. 956 8.1. Access Control 958 To control the visibility of load and overload information, sending 959 should be subject to some form of authentication and authorization of 960 the receiver. It is also important to the receivers that they are 961 confident the load and overload information they receive is from a 962 legitimate source. Note that this implies a certain amount of 963 configurability on the nodes supporting the Diameter overload control 964 mechanism. 966 8.2. Denial-of-Service Attacks 968 An overload control mechanism provides a very attractive target for 969 denial-of-service attacks. A small number of messages may affect a 970 large service disruption by falsely reporting overload conditions. 971 Alternately, attacking servers nearing, or in, overload may also be 972 facilitated by disrupting their overload indications, potentially 973 preventing them from mitigating their overload condition. 975 A design goal for the Diameter overload control mechanism is to 976 minimize or eliminate the possibility of using the mechanism for this 977 type of attack. 979 As the intent of some denial-of-service attacks is to induce overload 980 conditions, an effective overload control mechanism should help to 981 mitigate the effects of an such an attack. 983 8.3. Replay Attacks 985 An attacker that has managed to obtain some messages from the 986 overload control mechanism may attempt to affect the behavior of 987 nodes supporting the mechanism by sending those messages at 988 potentially inopportune times. In addition to time shifting, replay 989 attacks may send messages to other nodes as well (target shifting). 991 A design goal for the Diameter overload control mechanism is to 992 minimize or eliminate the possibility of causing disruption by using 993 a replay attack on the Diameter overload control mechanism. 995 8.4. Man-in-the-Middle Attacks 997 By inserting themselves in between two nodes supporting the Diameter 998 overload control mechanism, an attacker may potentially both access 999 and alter the information sent between those nodes. This can be used 1000 for information gathering for business intelligence and attack 1001 targeting, as well as direct attacks. 1003 A design goal for the Diameter overload control mechanism is to 1004 minimize or eliminate the possibility of causing disruption man-in- 1005 the-middle attacks on the Diameter overload control mechanism. A 1006 transport using TLS and/or IPSEC may be desirable for this. 1008 8.5. Compromised Hosts 1010 A compromised host that supports the Diameter overload control 1011 mechanism could be used for information gathering as well as for 1012 sending malicious information to any Diameter node that would 1013 normally accept information from it. While is is beyond the scope of 1014 the Diameter overload control mechanism to mitigate any operational 1015 interruption to the compromised host, a reasonable design goal is to 1016 minimize the impact that a compromised host can have on other nodes 1017 through the use of the Diameter overload control mechanism. Of 1018 course, a compromised host could be used to cause damage in a number 1019 of other ways. This is out of scope for a Diameter overload control 1020 mechanism. 1022 9. References 1024 9.1. Normative References 1026 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1027 Requirement Levels", BCP 14, RFC 2119, March 1997. 1029 [I-D.ietf-dime-rfc3588bis] 1030 Fajardo, V., Arkko, J., Loughney, J., and G. Zorn, 1031 "Diameter Base Protocol", draft-ietf-dime-rfc3588bis-33 1032 (work in progress), May 2012. 1034 [RFC2914] Floyd, S., "Congestion Control Principles", BCP 41, 1035 RFC 2914, September 2000. 1037 [RFC3539] Aboba, B. and J. Wood, "Authentication, Authorization and 1038 Accounting (AAA) Transport Profile", RFC 3539, June 2003. 1040 9.2. Informative References 1042 [RFC5390] Rosenberg, J., "Requirements for Management of Overload in 1043 the Session Initiation Protocol", RFC 5390, December 2008. 1045 [TR23.843] 1046 3GPP, "Study on Core Network Overload Solutions", 1047 TR 23.843 0.4.0, April 2011. 1049 Appendix A. Contributors 1051 Significant contributions to this document were made by Adam Roach 1052 and Eric Noel. 1054 Appendix B. Acknowledgements 1056 Review of, and contributions to, this specification by Martin Dolly, 1057 Carolyn Johnson, Jianrong Wang, Imtiaz Shaikh, and Robert Sparks were 1058 most appreciated. We would like to thank them for their time and 1059 expertise. 1061 Authors' Addresses 1063 Eric McMurry 1064 Tekelec 1065 17210 Campbell Rd. 1066 Suite 250 1067 Dallas, TX 75252 1068 US 1070 Email: emcmurry@estacado.net 1072 Ben Campbell 1073 Tekelec 1074 17210 Campbell Rd. 1075 Suite 250 1076 Dallas, TX 75252 1077 US 1079 Email: ben@nostrum.com