idnits 2.17.1 draft-ietf-dime-overload-reqs-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (December 17, 2012) is 4147 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) No issues found here. Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group E. McMurry 3 Internet-Draft B. Campbell 4 Intended status: Standards Track Tekelec 5 Expires: June 20, 2013 December 17, 2012 7 Diameter Overload Control Requirements 8 draft-ietf-dime-overload-reqs-02 10 Abstract 12 When a Diameter server or agent becomes overloaded, it needs to be 13 able to gracefully reduce its load, typically by informing clients to 14 reduce sending traffic for some period of time. Otherwise, it must 15 continue to expend resources parsing and responding to Diameter 16 messages, possibly resulting in congestion collapse. The existing 17 mechanisms provided by Diameter are not sufficient for this purpose. 18 This document describes the limitations of the existing mechanisms, 19 and provides requirements for new overload management mechanisms. 21 Status of this Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at http://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on June 20, 2013. 38 Copyright Notice 40 Copyright (c) 2012 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents 45 (http://trustee.ietf.org/license-info) in effect on the date of 46 publication of this document. Please review these documents 47 carefully, as they describe your rights and restrictions with respect 48 to this document. Code Components extracted from this document must 49 include Simplified BSD License text as described in Section 4.e of 50 the Trust Legal Provisions and are provided without warranty as 51 described in the Simplified BSD License. 53 Table of Contents 55 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 56 1.1. Causes of Overload . . . . . . . . . . . . . . . . . . . . 3 57 1.2. Effects of Overload . . . . . . . . . . . . . . . . . . . 5 58 1.3. Overload vs. Network Congestion . . . . . . . . . . . . . 5 59 1.4. Diameter Applications in a Broader Network . . . . . . . . 5 60 1.5. Documentation Conventions . . . . . . . . . . . . . . . . 6 61 2. Overload Scenarios . . . . . . . . . . . . . . . . . . . . . . 6 62 2.1. Peer to Peer Scenarios . . . . . . . . . . . . . . . . . . 7 63 2.2. Agent Scenarios . . . . . . . . . . . . . . . . . . . . . 9 64 2.3. Interconnect Scenario . . . . . . . . . . . . . . . . . . 12 65 3. Extensibility . . . . . . . . . . . . . . . . . . . . . . . . 13 66 4. Existing Mechanisms . . . . . . . . . . . . . . . . . . . . . 14 67 5. Issues with the Current Mechanisms . . . . . . . . . . . . . . 14 68 5.1. Problems with Implicit Mechanism . . . . . . . . . . . . . 15 69 5.2. Problems with Explicit Mechanisms . . . . . . . . . . . . 15 70 6. Diameter Overload Case Studies . . . . . . . . . . . . . . . . 16 71 6.1. Overload in Mobile Data Networks . . . . . . . . . . . . . 16 72 6.2. 3GPP Study on Core Network Overload . . . . . . . . . . . 17 73 7. Solution Requirements . . . . . . . . . . . . . . . . . . . . 18 74 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 75 9. Security Considerations . . . . . . . . . . . . . . . . . . . 23 76 9.1. Access Control . . . . . . . . . . . . . . . . . . . . . . 24 77 9.2. Denial-of-Service Attacks . . . . . . . . . . . . . . . . 24 78 9.3. Replay Attacks . . . . . . . . . . . . . . . . . . . . . . 24 79 9.4. Man-in-the-Middle Attacks . . . . . . . . . . . . . . . . 25 80 9.5. Compromised Hosts . . . . . . . . . . . . . . . . . . . . 25 81 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 25 82 10.1. Normative References . . . . . . . . . . . . . . . . . . . 25 83 10.2. Informative References . . . . . . . . . . . . . . . . . . 26 84 Appendix A. Contributors . . . . . . . . . . . . . . . . . . . . 26 85 Appendix B. Acknowledgements . . . . . . . . . . . . . . . . . . 26 86 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 27 88 1. Introduction 90 When a Diameter [RFC6733] server or agent becomes overloaded, it 91 needs to be able to gracefully reduce its load, typically by 92 informing clients to reduce sending traffic for some period of time. 93 Otherwise, it must continue to expend resources parsing and 94 responding to Diameter messages, possibly resulting in congestion 95 collapse. The existing mechanisms provided by Diameter are not 96 sufficient for this purpose. This document describes the limitations 97 of the existing mechanisms, and provides requirements for new 98 overload management mechanisms. 100 This document draws on [RFC5390] and the work done on SIP overload 101 control as well as on overload practices in SS7 networks and studies 102 done by 3GPP. 104 Diameter is not typically an end-user protocol; rather it is 105 generally used as one component in support of some end-user activity. 106 For example, a WiFi access point might use Diameter to authenticate 107 and authorize user access via 802.11. Overload in a network that 108 uses Diameter applications will likely spill over into the end-user 109 application network. The impact of Diameter overload on the client 110 application (a client application may use the Diameter protocol and 111 other protocols to do its job) is beyond the scope of this document. 113 This document presents non-normative descriptions of causes of 114 overload along with related scenarios and studies. Finally, it 115 offers a set of normative requirements for an improved overload 116 indication mechanism. 118 1.1. Causes of Overload 120 Overload occurs when an element, such as a Diameter server or agent, 121 has insufficient resources to successfully process all of the traffic 122 it is receiving. Resources include all of the capabilities of the 123 element used to process a request, including CPU processing, memory, 124 I/O, and disk resources. It can also include external resources such 125 as a database or DNS server, in which case the CPU, processing, 126 memory, I/O, and disk resources of those elements are effectively 127 part of the logical element processing the request. 129 Overload can occur for many reasons, including: 131 Inadequate capacity: When designing Diameter networks, that is, 132 application layer multi-node Diameter deployments, it can be very 133 difficult to predict all scenarios that may cause elevated 134 traffic. It may also be more costly to implement support for some 135 scenarios than a network operator may deem worthwhile. This 136 results in the likelihood that a Diameter network will not have 137 adequate capacity to handle all situations. 139 Dependency failures: A Diameter node can become overloaded because a 140 resource on which it is dependent has failed or become overloaded, 141 greatly reducing the logical capacity of the node. In these 142 cases, even minimal traffic might cause the node to go into 143 overload. Examples of such dependency overloads include DNS 144 servers, databases, disks, and network interfaces. 146 Component failures: A Diameter node can become overloaded when it is 147 a member of a cluster of servers that each share the load of 148 traffic, and one or more of the other members in the cluster fail. 149 In this case, the remaining nodes take over the work of the failed 150 nodes. Normally, capacity planning takes such failures into 151 account, and servers are typically run with enough spare capacity 152 to handle failure of another node. However, unusual failure 153 conditions can cause many nodes to fail at once. This is often 154 the case with software failures, where a bad packet or bad 155 database entry hits the same bug in a set of nodes in a cluster. 157 Network Initiated Traffic Flood: Issues with the radio access 158 network in a mobile network such as radio overlays with frequent 159 handovers, and operational changes are examples of network events 160 that can precipitate a flood of Diameter signaling traffic, such 161 as an avalanche restart. Failure of a Diameter proxy may also 162 result in a large amount of signaling as connections and sessions 163 are reestablished. 165 Subscriber Initiated Traffic Flood: Large gatherings of subscribers 166 or events that result in many subscribers interacting with the 167 network in close time proximity can result in Diameter signaling 168 traffic floods. For example, the finale of a large fireworks show 169 could be immediately followed by many subscribers posting 170 messages, pictures, and videos concentrated on one portion of a 171 network. Subscriber devices, such as smartphones, may use 172 aggressive registration strategies that generate unusually high 173 Diameter traffic loads. 175 DoS attacks: An attacker, wishing to disrupt service in the network, 176 can cause a large amount of traffic to be launched at a target 177 element. This can be done from a central source of traffic or 178 through a distributed DoS attack. In all cases, the volume of 179 traffic well exceeds the capacity of the element, sending the 180 system into overload. 182 1.2. Effects of Overload 184 Modern Diameter networks, comprised of application layer multi-node 185 deployments of Diameter elements, may operate at very large 186 transaction volumes. If a Diameter node becomes overloaded, or even 187 worse, fails completely, a large number of messages may be lost very 188 quickly. Even with redundant servers, many messages can be lost in 189 the time it takes for failover to complete. While a Diameter client 190 or agent should be able to retry such requests, an overloaded peer 191 may cause a sudden large increase in the number of transaction 192 transactions needing to be retried, rapidly filling local queues or 193 otherwise contributing to local overload. Therefore Diameter devices 194 need to be able to shed load before critical failures can occur. 196 Diameter depends heavily on The "Authentication, Authorization, 197 and Accounting (AAA) Transport Profile" [RFC3539], which states 198 assumptions about the scale of AAA services which may be incorrect 199 for current uses of Diameter. In particular, the document 200 suggests that AAA services will typically be low volume and that 201 traffic will typically be application-driven. Section 2.1 of that 202 document uses an example of a 48 port NAS. However, Diameter is 203 commonly used in large-scale mobile data environments, where a 204 typical client could be a packet gateway that serves millions of 205 users, and generates Diameter messages at network-driven rates. 207 1.3. Overload vs. Network Congestion 209 This document uses the term "overload" to refer to application-layer 210 overload at Diameter nodes. This is distinct from "network 211 congestion", that is, congestion that occurs at the lower networking 212 layers that may impact the delivery of Diameter messages between 213 nodes. The authors recognize that element overload and network 214 congestion are interrelated, and that overload can contribute to 215 network congestion and vice versa. 217 Network congestion issues are better handled by the transport 218 protocols. Diameter uses TCP and SCTP, both of which include 219 congestion management features. Analysis of whether those features 220 are sufficient for transport level congestion between Diameter nodes, 221 and any work to further mitigate network congestion is out of scope 222 both for this document, and for the work proposed by this document. 224 1.4. Diameter Applications in a Broader Network 226 Most elements using Diameter applications do not use Diameter 227 exclusively. It is important to realize that overload of an element 228 can be caused by a number of factors that may be unrelated to the 229 processing of Diameter or Diameter applications. 231 A element communicating via protocols other than Diameter that is 232 also using a Diameter application needs to be able to signal to 233 Diameter peers that it is experiencing overload regardless of the 234 cause of the overload, since the overload will affect that element's 235 ability to process Diameter transactions. The element may also need 236 to signal this on other protocols depending on its function and the 237 architecture of the network and application it is providing services 238 for. Whether that is necessary can only be decided within the 239 context of that architecture and application. A mechanism for 240 signaling overload with Diameter, which this specification details 241 the requirements for, provides applications the ability to signal 242 their Diameter peers of overload, mitigating that part of the issue. 243 Applications may need to use this, as well as other mechanisms, to 244 solve their broader overload issues. Indicating overload on 245 protocols other than Diameter is out of scope for this document, and 246 for the work proposed by this document. 248 1.5. Documentation Conventions 250 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 251 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 252 document are to be interpreted as described in [RFC2119]. 254 The terms "client", "server", "agent", "node", "peer", "upstream", 255 and "downstream" are used as defined in [RFC6733]. 257 2. Overload Scenarios 259 Several Diameter deployment scenarios exist that may impact overload 260 management. The following scenarios help motivate the requirements 261 for an overload management mechanism. 263 These scenarios are by no means exhaustive, and are in general 264 simplified for the sake of clarity. In particular, the authors 265 assume for the sake of clarity that the client sends Diameter 266 requests to the server, and the server sends responses to client, 267 even though Diameter supports bidirectional applications. Each 268 direction in such an application can be modeled separately. 270 In a large scale deployment, many of the nodes represented in these 271 scenarios would be deployed as clusters of servers. The authors 272 assume that such a cluster is responsible for managing its own 273 internal load balancing and overload management so that it appears as 274 a single Diameter node. That is, other Diameter nodes can treat it 275 as single, monolithic node for the purposes of overload management. 277 These scenarios do not illustrate the client application. As 278 mentioned in Section 1, Diameter is not typically an end-user 279 protocol; rather it is generally used in support of some other client 280 application. These scenarios do not consider the impact of Diameter 281 overload on the client application. 283 2.1. Peer to Peer Scenarios 285 This section describes Diameter peer-to-peer scenarios. That is, 286 scenarios where a Diameter client talks directly with a Diameter 287 server, without the use of a Diameter agent. 289 Figure 1 illustrates the simplest possible Diameter relationship. 290 The client and server share a one-to-one peer-to-peer relationship. 291 If the server becomes overloaded, either because the client exceeds 292 the server's capacity, or because the server's capacity is reduced 293 due to some resource dependency, the client needs to reduce the 294 amount of Diameter traffic it sends to the server. Since the client 295 cannot forward requests to another server, it must either queue 296 requests until the server recovers, or itself become overloaded in 297 the context of the client application and other protocols it may also 298 use. 300 +------------------+ 301 | | 302 | | 303 | Server | 304 | | 305 +--------+---------+ 306 | 307 | 308 +--------+---------+ 309 | | 310 | | 311 | Client | 312 | | 313 +------------------+ 315 Figure 1: Basic Peer to Peer Scenario 317 Figure 2 shows a similar scenario, except in this case the client has 318 multiple servers that can handle work for a specific realm and 319 application. If server 1 becomes overloaded, the client can forward 320 traffic to server 2. Assuming server 2 has sufficient reserve 321 capacity to handle the forwarded traffic, the client should be able 322 to continue serving client application protocol users. If server 1 323 is approaching overload, but can still handle some number of new 324 request, it needs to be able to instruct the client to forward a 325 subset of its traffic to server 2. 327 +------------------+ +------------------+ 328 | | | | 329 | | | | 330 | Server 1 | | Server 2 | 331 | | | | 332 +--------+-`.------+ +------.'+---------+ 333 `. .' 334 `. .' 335 `. .' 336 `. .' 337 +-------`.'--------+ 338 | | 339 | | 340 | Client | 341 | | 342 +------------------+ 344 Figure 2: Multiple Server Peer to Peer Scenario 346 Figure 3 illustrates a peer-to-peer scenario with multiple Diameter 347 realm and application combinations. In this example, server 2 can 348 handle work for both applications. Each application might have 349 different resource dependencies. For example, a server might need to 350 access one database for application A, and another for application B. 351 This creates a possibility that Server 2 could become overloaded for 352 application A but not for application B, in which case the client 353 would need to divert some part of its application A requests to 354 server 1, but should not divert any application B requests. This 355 requires server 2 to be able to distinguish between applications when 356 it indicates an overload condition to the client. 358 On the other hand, it's possible that the servers host many 359 applications. If server 2 becomes overloaded for all applications, 360 it would be undesirable for it to have to notify the client 361 separately for each application. Therefore it also needs a way to 362 indicate that it is overloaded for all possible applications. 364 +----------------------------------------------+ 365 | Application A +------------------------+----------------------+ 366 |+------------------+ | +------------------+ | +------------------+| 367 || | | | | | | || 368 || | | | | | | || 369 || Server 1 | | | Server 2 | | | Server 3 || 370 || | | | | | | || 371 |+--------+---------+ | +--------+---------+ | +-+----------------+| 372 | | | | | | | 373 +---------+-----------+-----------+------------+ | | 374 | | | | | 375 | | | | Application B | 376 | +-----------+-----------------+-----------------+ 377 ``-.._ | | 378 `-..__ | _.-'' 379 `--._ | _.-'' 380 ``-.__ | _.-'' 381 +------`-.-''------+ 382 | | 383 | | 384 | Client | 385 | | 386 +------------------+ 388 Figure 3: Multiple Application Peer to Peer Scenario 390 2.2. Agent Scenarios 392 This section describes scenarios that include a Diameter agent, 393 either in the form of a Diameter relay or Diameter proxy. These 394 scenarios do not consider Diameter redirect agents, since they are 395 more readily modeled as end-servers. 397 Figure 4 illustrates a simple Diameter agent scenario with a single 398 client, agent, and server. In this case, overload can occur at the 399 server, at the agent, or both. But in most cases, client behavior is 400 the same whether overload occurs at the server or at the agent. From 401 the client's perspective, server overload and agent overload is the 402 same thing. 404 +------------------+ 405 | | 406 | | 407 | Server | 408 | | 409 +--------+---------+ 410 | 411 | 412 +--------+---------+ 413 | | 414 | | 415 | Agent | 416 | | 417 +--------+---------+ 418 | 419 | 420 +--------+---------+ 421 | | 422 | | 423 | Client | 424 | | 425 +------------------+ 427 Figure 4: Basic Agent Scenario 429 Figure 5 shows an agent scenario with multiple servers. If server 1 430 becomes overloaded, but server 2 has sufficient reserve capacity, the 431 agent may be able to transparently divert some or all Diameter 432 requests originally bound for server 1 to server 2. 434 In most cases, the client does not have detailed knowledge of the 435 Diameter topology upstream of the agent. If the agent uses dynamic 436 discovery to find eligible servers, the set of eligible servers may 437 not be enumerable from the perspective of the client. Therefore, in 438 most cases the agent needs to deal with any upstream overload issues 439 in a way that is transparent to the client. If one server notifies 440 the agent that it has become overloaded, the notification should not 441 be passed back to the client in a way that the client could 442 mistakenly perceive the agent itself as being overloaded. If the set 443 of all possible destinations upstream of the agent no longer has 444 sufficient capacity for incoming load, the agent itself becomes 445 effectively overloaded. 447 On the other hand, there are cases where the client needs to be able 448 to select a particular server from behind an agent. For example, if 449 a Diameter request is part of a multiple-round-trip authentication, 450 or is otherwise part of a Diameter "session", it may have a 451 DestinationHost AVP that requires the request to be served by server 452 1. Therefore the agent may need to inform a client that a particular 453 upstream server is overloaded or otherwise unavailable. Note that 454 there can be many ways a server can be specified, which may have 455 different implications (e.g. by IP address, by host name, etc). 457 +------------------+ +------------------+ 458 | | | | 459 | | | | 460 | Server 1 | | Server 2 | 461 | | | | 462 +--------+-`.------+ +------.'+---------+ 463 `. .' 464 `. .' 465 `. .' 466 `. .' 467 +-------`.'--------+ 468 | | 469 | | 470 | Agent | 471 | | 472 +--------+---------+ 473 | 474 | 475 | 476 +--------+---------+ 477 | | 478 | | 479 | Client | 480 | | 481 +------------------+ 483 Figure 5: Multiple Server Agent Scenario 485 Figure 6 shows a scenario where an agent routes requests to a set of 486 servers for more than one Diameter realm and application. In this 487 scenario, if server 1 becomes overloaded or unavailable, the agent 488 may effectively operate at reduced capacity for application A, but at 489 full capacity for application B. Therefore, the agent needs to be 490 able to report that it is overloaded for one application, but not for 491 another. 493 +----------------------------------------------+ 494 | Application A +------------------------+----------------------+ 495 |+------------------+ | +------------------+ | +------------------+| 496 || | | | | | | || 497 || | | | | | | || 498 || Server 1 | | | Server 2 | | | Server 3 || 499 || | | | | | | || 500 |+---------+--------+ | +--------+---------+ | +--+---------------+| 501 | | | | | | | 502 +----------+----------+-----------+------------+ | | 503 | | | | | 504 | | | | Application B | 505 | +-----------+------------------+----------------+ 506 | | | 507 ``--.__ | _. 508 ``-.__ | __.--'' 509 `--.._ | _..--' 510 +-----``-+.-''-----+ 511 | | 512 | | 513 | Agent | 514 | | 515 +--------+---------+ 516 | 517 | 518 +--------+---------+ 519 | | 520 | | 521 | Client | 522 | | 523 +------------------+ 525 Figure 6: Multiple Application Agent Scenario 527 2.3. Interconnect Scenario 529 Another scenario to consider when looking at Diameter overload is 530 that of multiple network operators using Diameter components 531 connected through an interconnect service, e.g. using IPX. IPX (IP 532 eXchange) [IR.34] is an Inter-Operator IP Backbone that provides 533 roaming interconnection network between mobile operators and service 534 providers. The IPX is also used to transport Diameter signaling 535 between operators [IR.88]. Figure 7 shows two network operators with 536 an interconnect network in-between. There could be any number of 537 these networks between any two network operator's networks. 539 +-------------------------------------------+ 540 | Interconnect | 541 | | 542 | +--------------+ +--------------+ | 543 | | Server 3 |------| Server 4 | | 544 | +--------------+ +--------------+ | 545 | .' `. | 546 +------.-'--------------------------`.------+ 547 .' `. 548 .-' `. 549 ------------.'-----+ +----`.--------------- 550 +----------+ | | +----------+ 551 | Server 1 | | | | Server 2 | 552 +----------+ | | +----------+ 553 | | 554 Network Operator 1 | | Network Operator 2 555 -------------------+ +--------------------- 557 Figure 7: Two Network Interconnect Scenario 559 The characteristics of the information that an operator would want to 560 share over such a connection are different from the information 561 shared between components within a network operator's network. 562 Network operators may not want to convey topology or operational 563 information, which limits how much overload and loading information 564 can be sent. For the interconnect scenario shown, Server 2 may want 565 to signal overload to Server 1, to affect traffic coming from Network 566 Operator 1. 568 This case is distinct from those internal to a network operator's 569 network, where there may be many more elements in a more complicated 570 topology. Also, the elements in the interconnect network may not 571 support diameter overload control, and the network operators may not 572 want the interconnect network to use overload or loading information. 573 They may only want the information to pass through the interconnect 574 network without further processing or action by the interconnect 575 network even if the elements in the interconnect network do support 576 diameter overload control. 578 3. Extensibility 580 Given the variety of scenarios diameter elements can be deployed in, 581 and the variety of roles they can fulfill with diameter and other 582 technologies, a single algorithm for handling overload may not be 583 sufficient. This effort cannot anticipate all possible future 584 scenarios and roles. Extensibility, particularly of algorithms used 585 to deal with overload, will be important to cover these cases. 587 4. Existing Mechanisms 589 Diameter offers both implicit and explicit mechanisms for a Diameter 590 node to learn that a peer is overloaded or unreachable. The implicit 591 mechanism is simply the lack of responses to requests. If a client 592 fails to receive a response in a certain time period, it assumes the 593 upstream peer is unavailable, or overloaded to the point of effective 594 unavailability. The watchdog mechanism [RFC3539] ensures that a 595 certain rate of transaction responses occur even when there is 596 otherwise little or no other Diameter traffic. 598 The explicit mechanism can involve specific protocol error responses, 599 where an agent or server tells a downstream peer that it is either 600 too busy to handle a request (DIAMETER_TOO_BUSY) or unable to route a 601 request to an upstream destination (DIAMETER_UNABLE_TO_DELIVER), 602 perhaps because that destination itself is overloaded to the point of 603 unavailability. 605 Another explicit mechanism, a DPR (Disconnect-Peer-Request) message, 606 can be sent with a Disconnect-Cause of BUSY. This signals the 607 sender's intent to close the transport connection, and requests the 608 client not to reconnect. 610 Once a Diameter node learns that an upstream peer has become 611 overloaded via one of these mechanisms, it can then attempt to take 612 action to reduce the load. This usually means forwarding traffic to 613 an alternate destination, if available. If no alternate destination 614 is available, the node must either reduce the number of messages it 615 originates (in the case of a client) or inform the client to reduce 616 traffic (in the case of an agent.) 618 Diameter requires the use of a congestion-managed transport layer, 619 currently TCP or SCTP, to mitigate network congestion. It is 620 expected that these transports manage network congestion and that 621 issues with transport (e.g. congestion propagation and window 622 management) are managed at that level. But even with a congestion- 623 managed transport, a Diameter node can become overloaded at the 624 Diameter protocol or application layers due to the causes described 625 in Section 1.1 and congestion managed transports do not provide 626 facilities (and are at the wrong level) to handle server overload. 627 Transport level congestion management is also not sufficient to 628 address overload in cases of multi-hop and multi-destination 629 signaling. 631 5. Issues with the Current Mechanisms 633 The currently available Diameter mechanisms for indicating an 634 overload condition are not adequate to avoid service outages due to 635 overload. This inadequacy may, in turn, contribute to broader 636 congestion collapse due to unresponsive Diameter nodes causing 637 application or transport layer retransmissions. In particular, they 638 do not allow a Diameter agent or server to shed load as it approaches 639 overload. At best, a node can only indicate that it needs to 640 entirely stop receiving requests, i.e. that it has effectively 641 failed. Even that is problematic due to the inability to indicate 642 durational validity on the transient errors available in the base 643 Diameter protocol. Diameter offers no mechanism to allow a node to 644 indicate different overload states for different categories of 645 messages, for example, if it is overloaded for one Diameter 646 application but not another. 648 5.1. Problems with Implicit Mechanism 650 The implicit mechanism doesn't allow an agent or server to inform the 651 client of a problem until it is effectively too late to do anything 652 about it. The client does not know to take action until the upstream 653 node has effectively failed. A Diameter node has no opportunity to 654 shed load early to avoid collapse in the first place. 656 Additionally, the implicit mechanism cannot distinguish between 657 overload of a Diameter node and network congestion. Diameter treats 658 the failure to receive an answer as a transport failure. 660 5.2. Problems with Explicit Mechanisms 662 The Diameter specification is ambiguous on how a client should handle 663 receipt of a DIAMETER_TOO_BUSY response. The base specification 664 [RFC6733] indicates that the sending client should attempt to send 665 the request to a different peer. It makes no suggestion that a the 666 receipt of a DIAMETER_TOO_BUSY response should affect future Diameter 667 messages in any way. 669 The Authentication, Authorization, and Accounting (AAA) Transport 670 Profile [RFC3539] recommends that a AAA node that receives a "Busy" 671 response failover all remaining requests to a different agent or 672 server. But while the Diameter base specification explicitly depends 673 on RFC3539 to define transport behavior, it does not refer to RFC3539 674 in the description of behavior on receipt of DIAMETER_TOO_BUSY. 675 There's a strong likelihood that at least some implementations will 676 continue to send Diameter requests to an upstream peer even after 677 receiving a DIAMETER_TOO_BUSY error. 679 BCP 41 [RFC2914] describes, among other things, how end-to-end 680 application behavior can help avoid congestion collapse. In 681 particular, an application should avoid sending messages that will 682 never be delivered or processed. The DIAMETER_TOO_BUSY behavior as 683 described in the Diameter base specification fails at this, since if 684 an upstream node becomes overloaded, a client attempts each request, 685 and does not discover the need to failover the request until the 686 initial attempt fails. 688 The situation is improved if implementations follow the [RFC3539] 689 recommendation and keep state about upstream peer overload. But even 690 then, the Diameter specification offers no guidance on how long a 691 client should wait before retrying the overloaded destination. If an 692 agent or server supports multiple realms and/or applications, 693 DIAMETER_TOO_BUSY offers no way to indicate that it is overloaded for 694 one application but not another. A DIAMETER_TOO_BUSY error can only 695 indicate overload at a "whole server" scope. 697 Agent processing of a DIAMETER_TOO_BUSY response is also problematic 698 as described in the base specification. DIAMETER_TOO_BUSY is defined 699 as a protocol error. If an agent receives a protocol error, it may 700 either handle it locally or it may forward the response back towards 701 the downstream peer. (The Diameter specification is inconsistent 702 about whether a protocol error MAY or SHOULD be handled by an agent, 703 rather than forwarded downstream.) If a downstream peer receives the 704 DIAMETER_TOO_BUSY response, it may stop sending all requests to the 705 agent for some period of time, even though the agent may still be 706 able to deliver requests to other upstream peers. 708 DIAMETER_UNABLE_TO_DELIVER, or using DPR with cause code BUSY also 709 have no mechanisms for specifying the scope or cause of the failure, 710 or the durational validity. 712 The issues with error responses in [RFC6733] extend beyond the 713 particular issues for overload control and have been addressed in an 714 ad hoc fashion by various implementations. Addressing these in a 715 standard way would be a useful exercise, but it us beyond the scope 716 of this document. 718 6. Diameter Overload Case Studies 720 6.1. Overload in Mobile Data Networks 722 As the number of Third Generation (3G) and Long Term Evolution (LTE) 723 enabled smartphone devices continue to expand in mobility networks, 724 there have been situations where high signaling traffic load led to 725 overload events at the Diameter-based Home Location Registries (HLR) 726 and/or Home Subscriber Servers (HSS) [TR23.843]. The root causes of 727 the HLR congestion events were manifold but included hardware failure 728 and procedural errors. The result was high signaling traffic load on 729 the HLR and HSS. 731 The 3GPP architecture [TS23.002] makes extensive use of Diameter. It 732 is used for mobility management [TS29.272] (and others), IMS 733 [TS29.228] (and others), policy and charging control [TS29.212] (and 734 others) as well as other functions. The details of the architecture 735 are out of scope for this document, but it is worth noting that there 736 are quite a few Diameter applications, some with quite large amounts 737 of Diameter signaling in deployed networks. 739 The 3GPP specifications do not currently address overload for 740 Diameter applications or provide an equivalent load control mechanism 741 to those provided in the more traditional SS7 elements in GSM 742 [TS29.002]. The capabilities specified in the 3GPP standards do not 743 adequately address the abnormal condition where excessively high 744 signaling traffic load situations are experienced. 746 Smartphones contribute much more heavily, relative to non- 747 smartphones, to the continuation of a registration surge due to their 748 very aggressive registration algorithms. The aggressive smartphone 749 logic is designed to: 751 a. always have voice and data registration, and 753 b. constantly try to be on 3G or LTE data (and thus on 3G voice or 754 VoLTE) for their added benefits. 756 Non-smartphones typically have logic to wait for a time period after 757 registering successfully on voice and data. 759 The smartphone aggressive registration is problematic in two ways: 761 o first by generating excessive signaling load towards the HLR that 762 is ten times that from a non-smartphone, 764 o and second by causing continual registration attempts when a 765 network failure affects registrations through the 3G data network. 767 6.2. 3GPP Study on Core Network Overload 769 A study in 3GPP SA2 on core network overload has produced the 770 technical report [TR23.843]. This enumerates several causes of 771 overload in mobile core networks including portions that are signaled 772 using Diameter. This document is a work in progress and is not 773 complete. However, it is useful for pointing out scenarios and the 774 general need for an overload control mechanism for Diameter. 776 It is common for mobile networks to employ more than one radio 777 technology and to do so in an overlay fashion with multiple 778 technologies present in the same location (such as GSM, UMTS or CDMA 779 along with LTE). This presents opportunities for traffic storms when 780 issues occur on one overlay and not another as all devices that had 781 been on the overlay with issues switch. This causes a large amount 782 of Diameter traffic as locations and policies are updated. 784 Another scenario called out by this study is a flood of registration 785 and mobility management events caused by some element in the core 786 network failing. This flood of traffic from end nodes falls under 787 the network initiated traffic flood category. There is likely to 788 also be traffic resulting directly from the component failure in this 789 case. A similar flood can occur when elements or components recover 790 as well. 792 Subscriber initiated traffic floods are also indicated in this study 793 as an overload mechanism where a large number of mobile devices 794 attempting to access services at the same time, such as in response 795 to an entertainment event or a catastrophic event. 797 While this 3GPP study is concerned with the broader effects of these 798 scenarios on wireless networks and their elements, they have 799 implications specifically for Diameter signaling. One of the goals 800 of this document is to provide guidance for a core mechanism that can 801 be used to mitigate the scenarios called out by this study. 803 7. Solution Requirements 805 This section proposes requirements for an improved mechanism to 806 control Diameter overload, with the goals of improving the issues 807 described in Section 5 and supporting the scenarios described in 808 Section 2 810 REQ 1: The overload control mechanism MUST provide a communication 811 method for Diameter nodes to exchange load and overload 812 information. 814 REQ 2: [Open Issue: The following requirement has generated list 815 discussion that is unresolved at the time of this writing. 816 The discussion concerns whether this requirement is needed 817 at all, whether it should include the "MUST NOT require 818 specification changes" language vs saying that it should not 819 force changes large enough to require new application IDs, 820 and whether we should include additional language to forbid 821 assumptions about the behavior of specific implementations.] 822 The overload control mechanism MUST be useable with any 823 existing or future Diameter application. It MUST NOT 824 require specification changes for existing Diameter 825 applications. 827 REQ 3: The overload control mechanism MUST limit the impact of 828 overload on the overall useful throughput of a Diameter 829 server, even when the incoming load on the network is far in 830 excess of its capacity. The overall useful throughput under 831 load is the ultimate measure of the value of an overload 832 control mechanism. 834 REQ 4: Diameter allows requests to be sent from either side of a 835 connection and either side of a connection may have need to 836 provide its overload status. The mechanism MUST allow each 837 side of a connection to independently inform the other of 838 its overload status. 840 REQ 5: Diameter allows nodes to determine their peers via dynamic 841 discovery or manual configuration. The mechanism MUST work 842 consistently without regard to how peers are determined. 844 REQ 6: The mechanism designers SHOULD seek to minimize the amount 845 of new configuration required in order to work. For 846 example, it is better to allow peers to advertise or 847 negotiate support for the mechanism, rather than to require 848 this knowledge to be configured at each node. 850 REQ 7: The overload control mechanism and any associated default 851 algorithm(s) MUST ensure that the system remains stable. 852 When the offered load drops from above the overall capacity 853 of the network to below the overall capacity, the throughput 854 MUST stabilize and become equal to the offered load. Note 855 that this also requires that the mechanism MUST allow nodes 856 to shed load without introducing oscillations. 858 REQ 8: Supporting nodes MUST be able to distinguish current 859 overload information from stale information, and SHOULD make 860 decisions using the most currently available information. 862 REQ 9: The mechanism MUST function across fully loaded as well as 863 quiescent transport connections. This is partially derived 864 from the requirements for stability and hysteresis control 865 above. 867 REQ 10: Consumers of overload state indications MUST be able to 868 determine when the overload condition improves or ends. 870 REQ 11: The overload control mechanism MUST be scalable. That is, 871 it MUST be able to operate in different sized networks. 873 REQ 12: When a single network node fails, goes into overload, or 874 suffers from reduced processing capacity, the mechanism MUST 875 make it possible to limit the impact of this on other nodes 876 in the network. This helps to prevent a small-scale failure 877 from becoming a widespread outage. 879 REQ 13: The mechanism MUST NOT introduce substantial additional work 880 for node in an overloaded state. For example, a requirement 881 for an overloaded node to send overload information every 882 time it received a new request would introduce substantial 883 work. Existing messaging is likely to have the 884 characteristic of increasing as an overload condition 885 approaches, allowing for the possibility of increased 886 feedback for information piggybacked on it. 888 REQ 14: Some scenarios that result in overload involve a rapid 889 increase of traffic with little time between normal levels 890 and overload inducing levels. The mechanism SHOULD provide 891 for rapid feedback when traffic levels increase. 893 REQ 15: The mechanism MUST NOT interfere with the congestion control 894 mechanisms of underlying transport protocols. For example, 895 a mechanism that opened additional TCP connections when the 896 network is congested would reduce the effectiveness of the 897 underlying congestion control mechanisms. 899 REQ 16: The mechanism MUST operate without malfunction in an 900 environment with a mix of nodes that do, and nodes that do 901 not, support the mechanism. 903 REQ 17: In a mixed environment with nodes that support the overload 904 control mechanism and that do not, the mechanism MUST result 905 in at least as much useful throughput as would have resulted 906 if the mechanism were not present. It SHOULD result in less 907 severe congestion in this environment. 909 REQ 18: In a mixed environment of nodes that support the overload 910 control mechanism and that do not, users and operators of 911 nodes that do not support the mechanism MUST NOT unfairly 912 benefit from the mechanism. 914 REQ 19: It MUST be possible to use the mechanism between nodes in 915 different realms and in different administrative domains. 917 REQ 20: Any explicit overload indication MUST distinguish between 918 actual overload, as opposed to other, non-overload related 919 failures. 921 REQ 21: In cases where a network node fails, is so overloaded that 922 it cannot process messages, or cannot communicate due to a 923 network failure, it may not be able to provide explicit 924 indications of the nature of the failure or its levels of 925 congestion. The mechanism MUST properly function in these 926 cases. 928 REQ 22: The mechanism MUST provide a way for an node to throttle the 929 amount of traffic it receives from an peer node. This 930 throttling SHOULD be graded so that it can be applied 931 gradually as offered load increases. Overload is not a 932 binary state; there may be degrees of overload. 934 REQ 23: The mechanism MUST enable a supporting node to minimize the 935 chance that retries due to an overloaded or failed node 936 result in additional traffic to other overloaded nodes, or 937 cause additional nodes to become overloaded. Moreover, the 938 mechanism SHOULD provide unambiguous directions to clients 939 on when they should retry a request and when they should not 940 considering the various causes of overload such as avalanche 941 restart. 943 REQ 24: The mechanism MUST provide sufficient information to enable 944 a load balancing node to divert messages that are rejected 945 or otherwise throttled by an overloaded upstream node to 946 other upstream nodes that are the most likely to have 947 sufficient capacity to process them. 949 REQ 25: The mechanism MUST provide a mechanism for indicating load 950 levels even when not in an overloaded condition, to assist 951 nodes making decisions to prevent overload conditions from 952 occurring. 954 REQ 26: The base specification for the overload control mechanism 955 SHOULD offer general guidance on which message types might 956 be desirable to send or process over others during times of 957 overload, based on application-specific considerations. For 958 example, it may be more beneficial to process messages for 959 existing sessions ahead of new sessions, or to give priority 960 to requests associated with emergency sessions or with high 961 priority users. Any normative or otherwise detailed 962 definition of the relative priorities of message types 963 during an overload condition will be the responsibility of 964 the application specification. 966 REQ 27: The mechanism MUST NOT prevent a node from prioritizing 967 requests based on any local policy, so that certain requests 968 are given preferential treatment, given additional 969 retransmission, not throttled, or processed ahead of others. 971 REQ 28: The overload control mechanism MUST NOT provide new 972 vulnerabilities to malicious attack, or increase the 973 severity of any existing vulnerabilities. This includes 974 vulnerabilities to DoS and DDoS attacks as well as replay 975 and man-in-the middle attacks. Note that the Diameter base 976 specification [RFC6733] lacks end to end security and this 977 must be considered. 979 REQ 29: The mechanism MUST provide a means to match an overload 980 indication with the node that originated it. In particular, 981 the mechanism MUST allow a node to distinguish between 982 overload at a next-hop peer from overload at a node upstream 983 of the peer. For example, in Figure 5, the client must not 984 mistake overload at server 1 for overload at the agent, 985 whether or not the agent supports the mechanism.( see REQ 986 4). 988 REQ 30: The mechanism MUST NOT depend on being deployed in 989 environments where all Diameter nodes are completely 990 trusted. It SHOULD operate as effectively as possible in 991 environments where other nodes are malicious; this includes 992 preventing malicious nodes from obtaining more than a fair 993 share of service. Note that this does not imply any 994 responsibility on the mechanism to detect, or take 995 countermeasures against, malicious nodes. 997 REQ 31: It MUST be possible for a supporting node to make 998 authorization decisions about what information will be sent 999 to peer nodes based on the identity of those nodes. This 1000 allows a domain administrator who considers the load of 1001 their nodes to be sensitive information to restrict access 1002 to that information. Of course, in such cases, there is no 1003 expectation that the overload control mechanism itself will 1004 help prevent overload from that peer node. 1006 REQ 32: The mechanism MUST NOT interfere with any Diameter compliant 1007 method that a node may use to protect itself from overload 1008 from non-supporting nodes, or from denial of service 1009 attacks. 1011 REQ 33: There are multiple situations where a Diameter node may be 1012 overloaded for some purposes but not others. For example, 1013 this can happen to an agent or server that supports multiple 1014 applications, or when a server depends on multiple external 1015 resources, some of which may become overloaded while others 1016 are fully available. The mechanism MUST allow Diameter 1017 nodes to indicate overload with sufficient granularity to 1018 allow clients to take action based on the overloaded 1019 resources without forcing available capacity to go unused. 1020 The mechanism MUST support specification of overload 1021 information with granularities of at least "Diameter node", 1022 "realm", "Diameter application", and "Diameter session", and 1023 SHOULD allow extensibility for others to be added in the 1024 future. 1026 REQ 34: The mechanism MUST provide a method for extending the 1027 information communicated and the algorithms used for 1028 overload control. 1030 REQ 35: The mechanism SHOULD provide a method for exchanging 1031 overload and load information between elements that are 1032 connected by intermediaries that do not support the 1033 mechanism. 1035 8. IANA Considerations 1037 This document makes no requests of IANA. 1039 9. Security Considerations 1041 A Diameter overload control mechanism is primarily concerned with the 1042 load and overload related behavior of nodes in a Diameter network, 1043 and the information used to affect that behavior. Load and overload 1044 information is shared between nodes and directly affects the behavior 1045 and thus is potentially vulnerable to a number of methods of attack. 1047 Load and overload information may also be sensitive from both 1048 business and network protection viewpoints. Operators of Diameter 1049 equipment want to control visibility to load and overload information 1050 to keep it from being used for competitive intelligence or for 1051 targeting attacks. It is also important that the Diameter overload 1052 control mechanism not introduce any way in which any other 1053 information carried by Diameter is sent inappropriately. 1055 Note that the Diameter base specification [RFC6733] lacks end to end 1056 security, making verifying the authenticity and ownership of load and 1057 overload information difficult for non-adjacent nodes. 1058 Authentication of load and overload information helps to alleviate 1059 several of the security issues listed in this section. 1061 This document includes requirements intended to mitigate the effects 1062 of attacks and to protect the information used by the mechanism. 1064 9.1. Access Control 1066 To control the visibility of load and overload information, sending 1067 should be subject to some form of authentication and authorization of 1068 the receiver. It is also important to the receivers that they are 1069 confident the load and overload information they receive is from a 1070 legitimate source. Note that this implies a certain amount of 1071 configurability on the nodes supporting the Diameter overload control 1072 mechanism. 1074 9.2. Denial-of-Service Attacks 1076 An overload control mechanism provides a very attractive target for 1077 denial-of-service attacks. A small number of messages may affect a 1078 large service disruption by falsely reporting overload conditions. 1079 Alternately, attacking servers nearing, or in, overload may also be 1080 facilitated by disrupting their overload indications, potentially 1081 preventing them from mitigating their overload condition. 1083 A design goal for the Diameter overload control mechanism is to 1084 minimize or eliminate the possibility of using the mechanism for this 1085 type of attack. 1087 As the intent of some denial-of-service attacks is to induce overload 1088 conditions, an effective overload control mechanism should help to 1089 mitigate the effects of an such an attack. 1091 9.3. Replay Attacks 1093 An attacker that has managed to obtain some messages from the 1094 overload control mechanism may attempt to affect the behavior of 1095 nodes supporting the mechanism by sending those messages at 1096 potentially inopportune times. In addition to time shifting, replay 1097 attacks may send messages to other nodes as well (target shifting). 1099 A design goal for the Diameter overload control mechanism is to 1100 minimize or eliminate the possibility of causing disruption by using 1101 a replay attack on the Diameter overload control mechanism. 1103 9.4. Man-in-the-Middle Attacks 1105 By inserting themselves in between two nodes supporting the Diameter 1106 overload control mechanism, an attacker may potentially both access 1107 and alter the information sent between those nodes. This can be used 1108 for information gathering for business intelligence and attack 1109 targeting, as well as direct attacks. 1111 A design goal for the Diameter overload control mechanism is to 1112 minimize or eliminate the possibility of causing disruption man-in- 1113 the-middle attacks on the Diameter overload control mechanism. A 1114 transport using TLS and/or IPSEC may be desirable for this. 1116 9.5. Compromised Hosts 1118 A compromised host that supports the Diameter overload control 1119 mechanism could be used for information gathering as well as for 1120 sending malicious information to any Diameter node that would 1121 normally accept information from it. While is is beyond the scope of 1122 the Diameter overload control mechanism to mitigate any operational 1123 interruption to the compromised host, a reasonable design goal is to 1124 minimize the impact that a compromised host can have on other nodes 1125 through the use of the Diameter overload control mechanism. Of 1126 course, a compromised host could be used to cause damage in a number 1127 of other ways. This is out of scope for a Diameter overload control 1128 mechanism. 1130 10. References 1132 10.1. Normative References 1134 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1135 Requirement Levels", BCP 14, RFC 2119, March 1997. 1137 [RFC6733] Fajardo, V., Arkko, J., Loughney, J., and G. Zorn, 1138 "Diameter Base Protocol", RFC 6733, October 2012. 1140 [RFC2914] Floyd, S., "Congestion Control Principles", BCP 41, 1141 RFC 2914, September 2000. 1143 [RFC3539] Aboba, B. and J. Wood, "Authentication, Authorization and 1144 Accounting (AAA) Transport Profile", RFC 3539, June 2003. 1146 10.2. Informative References 1148 [RFC5390] Rosenberg, J., "Requirements for Management of Overload in 1149 the Session Initiation Protocol", RFC 5390, December 2008. 1151 [TR23.843] 1152 3GPP, "Study on Core Network Overload Solutions", 1153 TR 23.843 0.6.0, October 2012. 1155 [IR.34] GSMA, "Inter-Service Provider IP Backbone Guidelines", 1156 IR 34 7.0, January 2012. 1158 [IR.88] GSMA, "LTE Roaming Guidelines", IR 88 7.0, January 2012. 1160 [TS23.002] 1161 3GPP, "Network Architecture", TS 23.002 12.0.0, 1162 September 2012. 1164 [TS29.272] 1165 3GPP, "Evolved Packet System (EPS); Mobility Management 1166 Entity (MME) and Serving GPRS Support Node (SGSN) related 1167 interfaces based on Diameter protocol", TS 29.272 11.4.0, 1168 September 2012. 1170 [TS29.212] 1171 3GPP, "Policy and Charging Control (PCC) over Gx/Sd 1172 reference point", TS 29.212 11.6.0, September 2012. 1174 [TS29.228] 1175 3GPP, "IP Multimedia (IM) Subsystem Cx and Dx interfaces; 1176 Signalling flows and message contents", TS 29.228 11.5.0, 1177 September 2012. 1179 [TS29.002] 1180 3GPP, "Mobile Application Part (MAP) specification", 1181 TS 29.002 11.4.0, September 2012. 1183 Appendix A. Contributors 1185 Significant contributions to this document were made by Adam Roach 1186 and Eric Noel. 1188 Appendix B. Acknowledgements 1190 Review of, and contributions to, this specification by Martin Dolly, 1191 Carolyn Johnson, Jianrong Wang, Imtiaz Shaikh, Jouni Korhonen, Robert 1192 Sparks, Dieter Jacobsohn, Janet Gunn, Jean-Jacques Trottin, Laurent 1193 Thiebaut, and Lionel Morand were most appreciated. We would like to 1194 thank them for their time and expertise. 1196 Authors' Addresses 1198 Eric McMurry 1199 Tekelec 1200 17210 Campbell Rd. 1201 Suite 250 1202 Dallas, TX 75252 1203 US 1205 Email: emcmurry@computer.org 1207 Ben Campbell 1208 Tekelec 1209 17210 Campbell Rd. 1210 Suite 250 1211 Dallas, TX 75252 1212 US 1214 Email: ben@nostrum.com