idnits 2.17.1 draft-ietf-tsvwg-circuit-breaker-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (September 10, 2015) is 3150 days in the past. Is this intentional? Checking references for intended status: Best Current Practice ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Obsolete normative reference: RFC 5405 (Obsoleted by RFC 8085) Summary: 1 error (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TSVWG Working Group G. Fairhurst 3 Internet-Draft University of Aberdeen 4 Intended status: Best Current Practice September 10, 2015 5 Expires: March 13, 2016 7 Network Transport Circuit Breakers 8 draft-ietf-tsvwg-circuit-breaker-03 10 Abstract 12 This document explains what is meant by the term "network transport 13 Circuit Breaker" (CB). It describes the need for circuit breakers 14 when using network tunnels, and other non-congestion controlled 15 applications. It also defines requirements for building a circuit 16 breaker and the expected outcomes of using a circuit breaker within 17 the Internet. 19 Status of This Memo 21 This Internet-Draft is submitted in full conformance with the 22 provisions of BCP 78 and BCP 79. 24 Internet-Drafts are working documents of the Internet Engineering 25 Task Force (IETF). Note that other groups may also distribute 26 working documents as Internet-Drafts. The list of current Internet- 27 Drafts is at http://datatracker.ietf.org/drafts/current/. 29 Internet-Drafts are draft documents valid for a maximum of six months 30 and may be updated, replaced, or obsoleted by other documents at any 31 time. It is inappropriate to use Internet-Drafts as reference 32 material or to cite them other than as "work in progress." 34 This Internet-Draft will expire on March 13, 2016. 36 Copyright Notice 38 Copyright (c) 2015 IETF Trust and the persons identified as the 39 document authors. All rights reserved. 41 This document is subject to BCP 78 and the IETF Trust's Legal 42 Provisions Relating to IETF Documents 43 (http://trustee.ietf.org/license-info) in effect on the date of 44 publication of this document. Please review these documents 45 carefully, as they describe your rights and restrictions with respect 46 to this document. Code Components extracted from this document must 47 include Simplified BSD License text as described in Section 4.e of 48 the Trust Legal Provisions and are provided without warranty as 49 described in the Simplified BSD License. 51 Table of Contents 53 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 54 1.1. Types of Circuit-Breaker . . . . . . . . . . . . . . . . 4 55 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 56 3. Design of a Circuit-Breaker (What makes a good circuit 57 breaker?) . . . . . . . . . . . . . . . . . . . . . . . . . . 5 58 3.1. Functional Components . . . . . . . . . . . . . . . . . . 5 59 4. Requirements for a Network Transport Circuit Breaker . . . . 8 60 4.1. Unidirectional Circuit Breakers over Controlled Paths . . 10 61 4.1.1. Use with a multicast control/routing protocol . . . . 10 62 4.1.2. Use with control potocols supporting pre-prosvisioned 63 capacity . . . . . . . . . . . . . . . . . . . . . . 12 64 5. Examples of Circuit Breakers . . . . . . . . . . . . . . . . 12 65 5.1. A Fast-Trip Circuit Breaker . . . . . . . . . . . . . . . 12 66 5.1.1. A Fast-Trip Circuit Breaker for RTP . . . . . . . . . 13 67 5.2. A Slow-trip Circuit Breaker . . . . . . . . . . . . . . . 13 68 5.3. A Managed Circuit Breaker . . . . . . . . . . . . . . . . 14 69 5.3.1. A Managed Circuit Breaker for SAToP Pseudo-Wires . . 14 70 6. Examples where circuit breakers may not be needed. . . . . . 15 71 6.1. CBs over pre-provisioned Capacity . . . . . . . . . . . . 15 72 6.2. CBs with tunnels carrying Congestion-Controlled Traffic . 15 73 6.3. CBs with Uni-directional Traffic and no Control Path . . 16 74 7. Security Considerations . . . . . . . . . . . . . . . . . . . 16 75 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17 76 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 17 77 10. Revision Notes . . . . . . . . . . . . . . . . . . . . . . . 17 78 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 18 79 11.1. Normative References . . . . . . . . . . . . . . . . . . 18 80 11.2. Informative References . . . . . . . . . . . . . . . . . 19 81 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 19 83 1. Introduction 85 A network transport Circuit Breaker (CB) is an automatic mechanism 86 that is used to estimate congestion caused by a flow, and to 87 terminate (or significantly reduce the rate of) the flow when 88 persistent congestion is detected. This is a safety measure to 89 prevent congestion collapse (starvation of resources available to 90 other flows), essential for an Internet that is heterogeneous and for 91 traffic that is hard to predict in advance. 93 The term "Circuit Breaker" originates in electricity supply, and has 94 nothing to do with network circuits or virtual circuits. In 95 electricity supply, a Circuit Breaker is intended as a protection 96 mechanism of last resort. Under normal circumstances, a Circuit 97 Breaker ought not to be triggered; It is designed to protect the 98 supply network and attached equipment when there is overload. Just 99 as people do not expect the electrical circuit-breaker (or fuse) in 100 their home to be triggered, except when there is a wiring fault or a 101 problem with an electrical appliance. 103 In networking, the Circuit Breaker principle can be used as a 104 protection mechanism of last resort to avoid persistent congestion. 105 Persistent congestion (also known as "congestion collapse") was a 106 feature of the early Internet of the 1980s. This resulted in excess 107 traffic starving other connection from access to the Internet. It 108 was countered by the requirement to use congestion control (CC) by 109 the Transmission Control Protocol (TCP) [Jacobsen88] [RFC1112]. 110 These mechanisms operate in Internet hosts to cause TCP connections 111 to "back off" during congestion. The introduction of a Congestion 112 Controller in TCP (currently documented in [RFC5681] ensured the 113 stability of the Internet, because it was able to detect congestion 114 and promptly react. This worked well while TCP was by far the 115 dominant traffic in the Internet, and most TCP flows were long-lived 116 (ensuring that they could detect and respond to congestion before the 117 flows terminated). This is no longer the case, and non-congestion 118 controlled traffic, including many applications of the User Datagram 119 Protocol (UDP) can form a significant proportion of the total traffic 120 traversing a link. The current Internet therefore requires that non- 121 congestion controlled traffic needs to be considered to avoid 122 congestion collapse. 124 There are important differences between a transport circuit-breaker 125 and a congestion-control method. Specifically, congestion control 126 (as implemented in TCP, SCTP, and DCCP) operates on the timescale on 127 the order of a packet round-trip-time (RTT), the time from sender to 128 destination and return. Congestion control methods are able to react 129 to a single packet loss/marking and reduce the transmission rate for 130 each loss or congestion event. The goal is usually to limit the 131 maximum transmission rate to a rate that reflects the available 132 capacity across a network path. These methods typically operate on 133 individual traffic flows (e.g., a 5-tuple). 135 In contrast, Circuit Breakers are recommended for non-congestion- 136 controlled Internet flows and for traffic aggregates, e.g., traffic 137 sent using a network tunnel. Later sections provide examples of 138 cases where circuit-breakers may or may not be desirable. 140 A Circuit Breaker needs to measure (meter) the traffic to determine 141 if the network is experiencing congestion and needs to be designed to 142 trigger robustly when there is persistent congestion. This means the 143 trigger needs to operate on a timescale much longer than the path 144 round trip time (e.g., seconds to possibly many tens of seconds). 145 This longer period is needed to provide sufficient time for 146 transports (or applications) to adjust their rate following 147 congestion, and for the network load to stabilise after any 148 adjustment. 150 A Circuit Breaker trigger will often utilise a series of successive 151 sample measurements metered at an ingress point and an egress point 152 (either of which could be a transport endpoint). These measurements 153 need taken over a reasonably long period of time. This is to ensure 154 that a Circuit Breaker does not accidentally trigger following a 155 single (or even successive) congestion events (congestion events are 156 what triggers congestion control, and are to be regarded as normal on 157 a network link operating near its capacity). Once triggered, a 158 control function needs to remove traffic from the network, either by 159 disabling the flow or by significantly reducing the level of traffic. 160 This reaction provides the required protection to prevent persistent 161 congestion being experienced by other flows that share the congested 162 part of the network path. 164 Section 4 defines requirements for building a Circuit Breaker. 166 1.1. Types of Circuit-Breaker 168 There are various forms of network transport circuit breaker. These 169 are differentiated mainly on the timescale over which they are 170 triggered, but also in the intended protection they offer: 172 o Fast-Trip Circuit Breakers: The relatively short timescale used by 173 this form of circuit breaker is intended to protect a flow or 174 related group of flows. 176 o Slow-Trip Circuit Breakers: This circuit breaker utilises a longer 177 timescale and is designed to protect traffic aggregates. 179 o Managed Circuit Breakers: Utilise the operations and management 180 functions that might be present in a managed service to implement 181 a circuit breaker. 183 Examples of each type of circuit breaker are provided in section 4. 185 2. Terminology 187 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 188 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 189 document are to be interpreted as described in [RFC2119]. 191 3. Design of a Circuit-Breaker (What makes a good circuit breaker?) 193 Although circuit breakers have been talked about in the IETF for many 194 years, there has not yet been guidance on the cases where circuit 195 breakers are needed or upon the design of circuit breaker mechanisms. 196 This document seeks to offer advise on these two topics. 198 Circuit Breakers are RECOMMENDED for IETF protocols and tunnels that 199 carry non-congestion-controlled Internet flows and for traffic 200 aggregates, e.g., traffic sent using a network tunnel. Designers of 201 other protocols and tunnel encapsulations also ought to consider the 202 use of these techniques to provide last resort protection to the 203 network paths that these are used. 205 This document defines the requirements for design of a Circuit 206 Breaker and provides examples of how a Circuit Breaker can be 207 constructed. The specifications of idividual protocols and tunnels 208 encapsulations need to detail the protocol mechanisms needed to 209 implement a Circuit Breaker. 211 Section 3.1 describes the functional components of a circuit breaker 212 and section 3.2 defines requirements for implementing a Circuit 213 Breaker. 215 3.1. Functional Components 217 The basic design of a transport circuit breaker involves 218 communication between an ingress point (a sender) and an egress point 219 (a receiver) of a network flow. A simple picture of Circuit Breaker 220 operation is provided in figure 1. This shows a set of routers (each 221 labelled R) connecting a set of endpoints. A Circuit Breaker is used 222 to control traffic passing through a subset of these routers, acting 223 between the ingress and a egress point network devices. The path 224 between the ingress and egress could be provided by a tunnel or other 225 network-layer technique. One expected use would be at the ingress 226 and egress of a service. 228 +--------+ +--------+ 229 |Endpoint| |Endpoint| 230 +--+-----+ >>> circuit breaker tarffic >>> +--+-----+ 231 | | 232 | +-+ +-+ +---------+ +-+ +-+ +-+ +--------+ +-+ +-+ | 233 +-+R+--+R+->+ Ingress +--+R+--+R+--+R+--+ Egress |--+R+--+R+-+ 234 +++ +-+ +------+--+ +-+ +-+ +-+ +-----+--+ +++ +-+ 235 | ^ | | | 236 | | +--+------+ +------+--+ | 237 | | | Ingress | | Egress | | 238 | | | Meter | | Meter | | 239 | | +----+----+ +----+----+ | 240 | | | | | 241 +-+ | | +----+----+ | | +-+ 242 |R+--+ | | Measure +<----------------+ +--+R| 243 +++ | +----+----+ Reported +++ 244 | | | Egress | 245 | | +----+----+ Measurement | 246 +--+-----+ | | Trigger + +--+-----+ 247 |Endpoint| | +----+----+ |Endpoint| 248 +--------+ | | +--------+ 249 +---<---+ 250 Reaction 252 Figure 1: A CB controlling the part of the end-to-end path between an 253 ingress point and an egress point. (Note: In some cases, the trigger 254 and measure functions could alternatively be located at other 255 locations (e.g., at a network operations centre.) 257 In the context of a Circuit Breaker, the ingress and egress functions 258 could be located in one or both network endpoints (see figure 2), for 259 example, implemented as components within a transport protocol. 261 +----------+ +----------+ 262 | Ingress | +-+ +-+ +-+ | Egress | 263 | Endpoint +->+R+--+R+--+R+--+ Endpoint | 264 +--+----+--+ +-+ +-+ +-+ +----+-----+ 265 ^ | | 266 | +--+------+ +----+----+ 267 | | Ingress | | Egress | 268 | | Meter | | Meter | 269 | +----+----+ +----+----+ 270 | | | 271 | +--- +----+ | 272 | | Measure +<-----------------+ 273 | +----+----+ Reported 274 | | Egress 275 | +----+----+ Measurement 276 | | Trigger | 277 | +----+----+ 278 | | 279 +---<--+ 280 Reaction 282 Figure 2: An endpoint CB implemented at the sender (ingress) and 283 receiver (egress). 285 The set of components needed to implement a Circuit Breaker are: 287 1. An ingress meter (at the sender or tunnel ingress) records the 288 number of packets/bytes sent in each measurement interval. This 289 measures the offered network load. For example, the measurement 290 interval could be every few seconds. 292 2. An egress meter (at the receiver or tunnel egress) records the 293 number/bytes received in each measurement interval. This 294 measures the supported load and could utilise other signals to 295 detect the effect of congestion (e.g., loss/marking experienced 296 over the path). 298 3. The measured values at the ingress and egress are communicated to 299 the Circuit Breaker Measurement function. This could use several 300 methods including: Sending return measurement packets from a 301 receiver to a trigger function at the sender; An implementation 302 using Operations, Administration and Management (OAM); or be 303 sending another in-band signalling datagram to the trigger 304 function. This could also be implemented purely as a control 305 plane function, e.g., using a software-defined network 306 controller. 308 4. The measurement function combines the ingress and egress 309 measurements to assess the present level of network congestion. 310 (For example, the loss rate for each measurement interval could 311 be deduced from calculating the difference between ingress and 312 egress counter values. Note the method does not require high 313 accuracy for the period of the measurement interval (or therefore 314 the measured value, since isolated and/or infrequent loss events 315 need to be disregarded.) 317 5. A trigger function determines if the measurements indicate 318 persistent congestion. This function defines an appropriate 319 threshold for determining there is persistent congestion between 320 the ingress and egress. This preferably consider rate or ratio, 321 rather than an absolute value (e.g., more than 10% loss, but 322 other methods could also be based on the rate of transmission as 323 well as the loss rate). The transport Circuit Breaker is 324 triggered when the threshold is exceeded in multiple measurement 325 intervals (e.g., 3 successive measurements). Designs need to be 326 robust so that single or spurious events do not trigger a 327 reaction. 329 6. A reaction that is applied that the Ingress when the Circuit 330 Breaker is triggered. This seeks to automatically remove the 331 traffic causing persistent congestion. 333 7. A feedback mechanism that triggers when either the receive or 334 ingress and egress measurements are not available, since this 335 also could indicate a loss of control packets (also a symptom of 336 heavy congestion or inability to control the load). 338 4. Requirements for a Network Transport Circuit Breaker 340 The requirements for implementing a Circuit Breaker are: 342 o There MUST be a control path from the ingress meter and the egress 343 meter to the point of measurement. The Circuit Breaker MUST 344 trigger if this control path fails. That is, the feedback 345 indicating a congested period needs to be designed so that the 346 Circuit Breaker is triggered when it fails to receive measurement 347 reports that indicate an absence of congestion, rather than 348 relying on the successful transmission of a "congested" signal 349 back to the sender. (The feedback signal could itself be lost 350 under congestion). 352 o A Circuit Breaker MUST define a measurement period over which the 353 receiver measures the level of congestion or loss. This method 354 does not have to detect individual packet loss, but MUST have a 355 way to know that packets have been lost/marked from the traffic 356 flow. If Explicit Congestion Notification (ECN) is enabled 357 [RFC3168], an egress meter MAY also count the number of ECN 358 congestion marks/event per measurement interval, but even if ECN 359 is used, loss MUST still be measured, since this better reflects 360 the impact of persistent congestion. In this context, loss 361 represents a reliable indication of congestion, as opposed to the 362 finer-grain marking of incipient congestion that can be provided 363 via ECN. The type of Circuit Breaker will determine how long this 364 measurement period needs to be. 366 o The measurement period MUST be longer than the time that current 367 Congestion Control algorithms need to reduce their rate following 368 detection of congestion. This is important because end-to-end 369 Congestion Control algorithms require at least one RTT to notify 370 and adjust to experienced congestion, and congestion bottlenecks 371 can share traffic with a diverse range of RTTs and Circuit 372 Breakers hence need to perform measurements over a sufficiently 373 long period to avoid additionally penalising flows with a long 374 path RTT (e.g., many path RTTs). In some implementations, this 375 may require a measurement to combine multiple meter samples to 376 achieve a sufficiently long measurement period. In most cases, 377 the measurement period is expected to be significantly longer than 378 the RTT experience by the Circuit Breaker itself. 380 o A Circuit Breaker is REQUIRED to define a threshold to determine 381 whether the measured congestion is considered excessive. 383 o A Circuit Breaker is REQUIRED to define the triggering interval, 384 defining the period over which the trigger uses the collected 385 measurements. 387 o A Circuit Breaker MUST be robust to multiple congestion events. 388 This usually will define a number of measured persistent 389 congestion events per triggering period. For example, a Circuit 390 Breaker MAY combine the results of several measurement periods to 391 determine if the Circuit Breaker is triggered. (e.g., triggered 392 when persistent congestion is detected in 3 of the measurements 393 within the triggering interval). 395 o A Circuit Breaker SHOULD be constructed so that it does not 396 trigger under light or intermittent congestion, with a default 397 response to a trigger that disables all traffic that contributed 398 to congestion. 400 o Once triggered, the Circuit Breaker MUST react decisively by 401 disabling or significantly reducing traffic at the source (e.g., 402 ingress). A reaction that results in a reduction SHOULD result in 403 reducing the traffic by at least a factor of ten, each time the 404 Circuit Breaker is triggered. 406 o Some circuit breaker designs use a reaction that reduces, rather 407 that disables, the flows it controls. This response MUST be much 408 more severe than that of a Congestion Controller algorithm, 409 because the Circuit Breaker reacts to more persistent congestion 410 and operates over longer timescales (i.e., the overload condition 411 will have persisted for a longer time before the Circuit Breaker 412 is triggered). A Circuit Breaker that reduces the rate of a flow, 413 MUST continue to monitor the level congestion and MUST further 414 reduce the rate if the Circuit Breaker is again triggered. 416 o The reaction to a triggered Circuit Breaker MUST continue for a 417 period that is at least the triggering interval. Manual operator 418 intervention will usually be required to restore a flow. If an 419 automated response is needed to reset the trigger, then this MUST 420 NOT be immediate. The design of an automated reset mechanism 421 needs to be sufficiently conservative that it does not adversely 422 interact with other mechanisms (including other Circuit Breaker 423 algorithms that control traffic over a common path). It SHOULD 424 NOT perform an automated reset when there is evidence of continued 425 congestion. 427 o When a Circuit Breaker is triggered, it SHOULD be regarded as an 428 abnormal network event. As such, this event SHOULD be logged. 429 The measurements that lead to triggering of the Circuit Breaker 430 SHOULD also be logged. 432 4.1. Unidirectional Circuit Breakers over Controlled Paths 434 A Circuit Breaker can be used to control uni-directional UDP traffic, 435 providing that there is a control path to connect the functional 436 components at the Ingress and Egress. This control path can exist in 437 networks for which the traffic flow is purely unidirectional. For 438 example, a multicast stream that sends packets across an Internet 439 path and can use multicast routing to prune flows to shed network 440 load. Some other types of subnetwork also utilise control protocols 441 that can be used to control traffic flows. 443 4.1.1. Use with a multicast control/routing protocol 444 +----------+ +--------+ +----------+ 445 | Ingress | +-+ +-+ +-+ | Egress | | Egress | 446 | Endpoint +->+R+--+R+--+R+--+ Router |--+ Endpoint +->+ 447 +----+-----+ +-+ +-+ +-+ +---+--+-+ +----+-----+ | 448 ^ ^ ^ ^ | ^ | | 449 | | | | | | | | 450 +----+----+ + - - - < - - - - + | +----+----+ | Reported 451 | Ingress | multicast Prune | | Egress | | Ingress 452 | Meter | | | Meter | | Measurement 453 +---------+ | +----+----+ | 454 | | | 455 | +----+----+ | 456 | | Measure +<--+ 457 | +----+----+ 458 | | 459 | +----+----+ 460 multicast | | Trigger | 461 Leave | +----+----+ 462 Message | | 463 +----<----+ 465 Figure 3: An example of a multicast CB controlling the end-to-end 466 path between an ingress endpoint and an egress endpoint. 468 Figure 3 shows one example of how a multicast circuit breaker could 469 be implemented at a pair of multicast endpoints (e.g. to implement a 470 Section 5.1). The ingress endpoint (the sender that sources the 471 multicast traffic) meters the ingress load, generating an ingress 472 measurement (e.g., recording timestamped packet counts), and sends 473 this measurement to the multicast group together with the traffic it 474 has measured. 476 Routers along a multicast path forward the multicast traffic 477 (including the ingress measurement) to all active endpoint receivers. 478 Each last hop (egress) router forwards the traffic to one or more 479 egress endpoint(s). 481 In this figure, each endpoint includes a meter that performs a local 482 egress load measurement. An endpoint also extracts the received 483 ingress measurement from the traffic, and compares the ingress and 484 egress measurements to determine if the Circuit Breaker ought to be 485 triggered. This measurement has to be robust to loss (see previous 486 section). If the Circuit Breaker is triggered, it generates a 487 multicast leave message for the egress (e.g., an IGMP or MLD message 488 sent to the last hop router), which causes the upstream router to 489 cease forwarding traffic to the egress endpoint. 491 Any multicast router that has no active receivers for a particular 492 multicast group will prune traffic for that group, sending a prune 493 message to its upstream router. This starts the process of releasing 494 the capacity used by the traffic and is a standard multicast routing 495 function (e.g., using the PIM-SM routing protocol). Each egress 496 operates autonomously, and the circuit breaker "reaction" is executed 497 by the multicast control plane (e.g., PIM l), requiring no explicit 498 signalling by the circuit breaker along the control path. Note: 499 there is no direct communication with the Ingress, and hence a 500 triggered Circuit Breaker only controls traffic downstream of the 501 first hop router. It does not stop traffic flowing from the sender 502 to the first hop router; this is however the common practice for 503 multicast deployment. 505 The method could also be used with a multicast tunnel or subnetwork 506 (e.g., Section 5.2, Section 5.3), where a meter at the ingress 507 generates additional control messages to carry the measurement data 508 towards the egress where the egress metering is implemented. 510 4.1.2. Use with control potocols supporting pre-prosvisioned capacity 512 Some paths are provisioned using a control protocol, e.g., flows 513 provisioned using the Multi-Protocol Label Switching (MPLS) services, 514 path provisioned using the Resource reservation protocol (RSVP), 515 networks utilizing Software Defining Network (SDN) functions, or 516 admission-controlled Differentiated Services. 518 Figure 1 shows one expected use case, where in this usage a separate 519 device could be used to perform the measurement and trigger 520 functions. The reaction generated by the trigger could take the form 521 of a network control message sent to the ingress and/or other network 522 elements causing these elements to react to the Circuit Breaker. 523 Examples of this type of use are provided in section Section 5.3. 525 5. Examples of Circuit Breakers 527 There are multiple types of Circuit Breaker that could be defined for 528 use in different deployment cases. This section provides examples of 529 different types of circuit breaker: 531 5.1. A Fast-Trip Circuit Breaker 533 A fast-trip circuit breaker is the most responsive form of Circuit 534 Breaker. It has a response time that is only slightly larger than 535 that of the traffic that it controls. It is suited to traffic with 536 well-understood characteristics (and could include one or more 537 trigger functions specifically tailored the type of traffic for which 538 it is designed). It is not be suited to arbitrary network traffic, 539 since it could prematurely trigger (e.g., when multiple congestion- 540 controlled flows lead to short-term overload). 542 5.1.1. A Fast-Trip Circuit Breaker for RTP 544 A set of fast-trip Circuit Breaker methods have been specified for 545 use together by a Real-time Transport Protocol (RTP) flow using the 546 RTP/AVP Profile [RTP-CB]. It is expected that, in the absence of 547 severe congestion, all RTP applications running on best-effort IP 548 networks will be able to run without triggering these circuit 549 breakers. A fast-trip RTP Circuit Breaker is therefore implemented 550 as a fail-safe. 552 The sender monitors reception of RTCP reception report blocks, as 553 contained in SR or RR packets, that convey reception quality feedback 554 information. This is used to measure (congestion) loss, possibly in 555 combination with ECN [RFC6679]. 557 The Circuit Breaker action (shutdown of the flow) is triggered when 558 any of the following trigger conditions are true: 560 1. An RTP Circuit Breaker triggers on reported lack of progress. 562 2. An RTP Circuit Breaker triggers when no receiver reports messages 563 are received. 565 3. An RTP Circuit Breaker uses a TFRC-style check and sets a hard 566 upper limit to the long-term RTP throughput (over many RTTs). 568 4. An RTP Circuit Breaker includes the notion of Media Usability. 569 This circuit breaker is triggered when the quality of the 570 transported media falls below some required minimum acceptable 571 quality. 573 5.2. A Slow-trip Circuit Breaker 575 A slow-trip Circuit Breaker could be implemented in an endpoint or 576 network device. This type of Circuit Breaker is much slower at 577 responding to congestion than a fast-trip Circuit Breaker and is 578 expected to be more common. 580 One example where a slow-trip Circuit Breaker is needed is where 581 flows or traffic-aggregates use a tunnel or encapsulation and the 582 flows within the tunnel do not all support TCP-style congestion 583 control (e.g., TCP, SCTP, TFRC), see [RFC5405] section 3.1.3. A use 584 case is where tunnels are deployed in the general Internet (rather 585 than "controlled environments" within an ISP or Enterprise), 586 especially when the tunnel could need to cross a customer access 587 router. 589 5.3. A Managed Circuit Breaker 591 A managed Circuit Breaker is implemented in the signalling protocol 592 or management plane that relates to the traffic aggregate being 593 controlled. This type of circuit breaker is typically applicable 594 when the deployment is within a "controlled environment". 596 A Circuit Breaker requires more than the ability to determine that a 597 network path is forwarding data, or to measure the rate of a path - 598 which are often normal network operational functions. There is an 599 additional need to determine a metric for congestion on the path and 600 to trigger a reaction when a threshold is crossed that indicates 601 persistent congestion. 603 5.3.1. A Managed Circuit Breaker for SAToP Pseudo-Wires 605 [RFC4553], SAToP Pseudo-Wires (PWE3), section 8 describes an example 606 of a managed circuit breaker for isochronous flows. 608 If such flows were to run over a pre-provisioned (e.g., MPLS) 609 infrastructure, then it could be expected that the Pseudo-Wire (PW) 610 would not experience congestion, because a flow is not expected to 611 either increase (or decrease) their rate. If instead Pseudo-Wire 612 traffic is multiplexed with other traffic over the general Internet, 613 it could experience congestion. [RFC4553] states: "If SAToP PWs run 614 over a PSN providing best-effort service, they SHOULD monitor packet 615 loss in order to detect "severe congestion". The currently 616 recommended measurement period is 1 second, and the trigger operates 617 when there are more than three measured Severely Errored Seconds 618 (SES) within a period. 620 If such a condition is detected, a SAToP PW ought to shut down 621 bidirectionally for some period of time...". The concept was that 622 when the packet loss ratio (congestion) level increased above a 623 threshold, the PW was by default disabled. This use case considered 624 fixed-rate transmission, where the PW had no reasonable way to shed 625 load. 627 The trigger needs to be set at the rate that the PW was likely to 628 experience a serious problem, possibly making the service non- 629 compliant. At this point, triggering the Circuit Breaker would 630 remove the traffic preventing undue impact on congestion-responsive 631 traffic (e.g., TCP). Part of the rationale, was that high loss 632 ratios typically indicated that something was "broken" and ought to 633 have already resulted in operator intervention, and therefore need to 634 trigger this intervention. 636 An operator-based response provides opportunity for other action to 637 restore the service quality, e.g., by shedding other loads or 638 assigning additional capacity, or to consciously avoid reacting to 639 the trigger while engineering a solution to the problem. This could 640 require the trigger to be sent to a third location (e.g., a network 641 operations centre, NOC) responsible for operation of the tunnel 642 ingress, rather than the tunnel ingress itself. 644 6. Examples where circuit breakers may not be needed. 646 A Circuit Breaker is not required for a single Congestion Controller- 647 controlled flow using TCP, SCTP, TFRC, etc. In these cases, the 648 Congestion Control methods are already designed to prevent congestion 649 collapse. 651 6.1. CBs over pre-provisioned Capacity 653 One common question is whether a Circuit Breaker is needed when a 654 tunnel is deployed in a private network with pre-provisioned 655 capacity? 657 In this case, compliant traffic that does not exceed the provisioned 658 capacity ought not to result in congestion collapse. A Circuit 659 Breaker will hence only be triggered when there is non-compliant 660 traffic. It could be argued that this event ought never to happen - 661 but it could also be argued that the Circuit Breaker equally ought 662 never to be triggered. If a Circuit Breaker were to be implemented, 663 it will provide an appropriate response if persistent congestion 664 occurs in an operational network. 666 Implementing a Circuit Breaker will not reduce the performance of the 667 flows, but offers protection in the event that persistent congestion 668 occurs. This also could be used to protect from a failure that 669 causes traffic to be routed over a non-pre-provisioned path. 671 6.2. CBs with tunnels carrying Congestion-Controlled Traffic 673 IP-based traffic is generally assumed to be congestion-controlled, 674 i.e., it is assumed that the transport protocols generating IP-based 675 traffic at the sender already employ mechanisms that are sufficient 676 to address congestion on the path [RFC5405]. A question therefore 677 arises when people deploy a tunnel that is thought to only carry an 678 aggregate of TCP (or some other Congestion Controller-controlled) 679 traffic: Is there advantage in this case in using a Circuit Breaker? 680 For sure, traffic in a such a tunnel will respond to congestion. 681 However, the answer to the question is not always obvious, because 682 the overall traffic formed by an aggregate of flows that implement a 683 Congestion Controller mechanism does not necessarily prevent 684 congestion collapse. For instance, most Congestion Controller 685 mechanisms require long-lived flows to react to reduce the rate of a 686 flow, an aggregate of many short flows could result in many 687 terminating before they experience congestion. It is also often 688 impossible for a tunnel service provider to know that the tunnel only 689 contains CC-controlled traffic (e.g., Inspecting packet headers could 690 not be possible). The important thing to note is that if the 691 aggregate of the traffic does not result in persistent congestion 692 (impacting other flows), then the Circuit Breaker will not trigger. 693 This is the expected case in this context - so implementing a Circuit 694 Breaker will not reduce performance of the tunnel, but offers 695 protection in the event that persistent congestion occur. 697 6.3. CBs with Uni-directional Traffic and no Control Path 699 A one-way forwarding path could have no associated control path, and 700 therefore cannot be controlled using an automated process. This 701 service could be provided using a path that has dedicated capacity 702 and does not share this capacity with other elastic Internet flows 703 (i.e., flows that vary their rate). 705 A way to mitigate the impact on other flows when capacity could be 706 shared is to manage the traffic envelope by using ingress policing. 708 Supporting this type of traffic in the general Internet requires 709 operator monitoring to detect and respond to persistent congestion. 711 7. Security Considerations 713 All Circuit Breaker mechanisms rely upon coordination between the 714 ingress and egress meters and communication with the trigger 715 function. This is usually achieved by passing network control 716 information (or protocol messages) across the network. Timely 717 operation of a circuit breaker depends on the choice of measurement 718 period. If the receiver has an interval that is overly long, then 719 the responsiveness of the circuit breaker decreases. This impacts 720 the ability of the circuit breaker to detect and react to congestion. 722 Mechanisms need to be implemented to prevent attacks on the network 723 control information that would result in Denial of Service (DoS). 724 The source and integrity of control information (measurements and 725 triggers) MUST be protected from off-path attacks. Without 726 protection, it could be trivial for an attacker to inject packets 727 with values that could prematurely trigger a circuit breaker 728 resulting in DoS. Simple protection can be provided by using a 729 randomised source port, or equivalent field in the packet header 730 (such as the RTP SSRC value and the RTP sequence number) expected not 731 to be known to an off-path attacker. Stronger protection can be 732 achieved using a secure authentication protocol. 734 Transmission of network control information consumes network 735 capacity. This control traffic needs to be considered in the design 736 of a circuit breaker and could potentially add to network congestion. 737 If this traffic is sent over a shared path, it is RECOMMENDED that 738 this control traffic is prioritized to reduce the probability of loss 739 under congestion. Control traffic also needs to be considered when 740 provisioning a network that uses a circuit breaker. 742 The circuit breaker MUST be designed to be robust to packet loss that 743 can also be experienced during congestion/overload. Loss of control 744 traffic could be a side-effect of a congested network, but also could 745 arise from other causes. 747 Each design of a Circuit Breaker MUST evaluate whether the particular 748 circuit breaker mechanism has new security implications. 750 8. IANA Considerations 752 This document makes no request from IANA. 754 9. Acknowledgments 756 There are many people who have discussed and described the issues 757 that have motivated this draft. Contributions and comments included: 758 Lars Eggert, Colin Perkins, David Black, Matt Mathis and Andrew 759 McGregor. This work was part-funded by the European Community under 760 its Seventh Framework Programme through the Reducing Internet 761 Transport Latency (RITE) project (ICT-317700). 763 10. Revision Notes 765 XXX RFC-Editor: Please remove this section prior to publication XXX 767 Draft 00 769 This was the first revision. Help and comments are greatly 770 appreciated. 772 Draft 01 773 Contained clarifications and changes in response to received 774 comments, plus addition of diagram and definitions. Comments are 775 welcome. 777 WG Draft 00 779 Approved as a WG work item on 28th Aug 2014. 781 WG Draft 01 783 Incorporates feedback after Dallas IETF TSVWG meeting. This version 784 is thought ready for WGLC comments. 786 WG Draft 02 788 Minor fixes for typos. Rewritten security considerations section. 790 WG Draft 03 792 Updates following WGLC comments (see TSV mailing list). Comments 793 from C Perkins; D Black and off-list feedback. 795 A clear recommendation of intended scope. 797 Changes include: Improvment of language on timescales and minimum 798 mesurement period; clearer articulation of endpoint and multicast 799 examples - with new diagrams; separation of the controlled network 800 case; updated text on position of trigger function; corrections to 801 RTP-CB text; clarification of loss v ECN metrics; checks against 802 submission checklist 9use of keywords, added meters to diagrams). 804 11. References 806 11.1. Normative References 808 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 809 Requirement Levels", BCP 14, RFC 2119, 810 DOI 10.17487/RFC2119, March 1997, 811 . 813 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 814 of Explicit Congestion Notification (ECN) to IP", 815 RFC 3168, DOI 10.17487/RFC3168, September 2001, 816 . 818 [RFC5405] Eggert, L. and G. Fairhurst, "Unicast UDP Usage Guidelines 819 for Application Designers", BCP 145, RFC 5405, 820 DOI 10.17487/RFC5405, November 2008, 821 . 823 11.2. Informative References 825 [Jacobsen88] 826 European Telecommunication Standards, Institute (ETSI), 827 "Congestion Avoidance and Control", SIGCOMM Symposium 828 proceedings on Communications architectures and 829 protocols", August 1998. 831 [RFC1112] Deering, S., "Host extensions for IP multicasting", STD 5, 832 RFC 1112, DOI 10.17487/RFC1112, August 1989, 833 . 835 [RFC4553] Vainshtein, A., Ed. and YJ. Stein, Ed., "Structure- 836 Agnostic Time Division Multiplexing (TDM) over Packet 837 (SAToP)", RFC 4553, DOI 10.17487/RFC4553, June 2006, 838 . 840 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 841 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 842 . 844 [RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P., 845 and K. Carlberg, "Explicit Congestion Notification (ECN) 846 for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August 847 2012, . 849 [RTP-CB] Perkins, and Singh, "Multimedia Congestion Control: 850 Circuit Breakers for Unicast RTP Sessions", February 2014. 852 Author's Address 854 Godred Fairhurst 855 University of Aberdeen 856 School of Engineering 857 Fraser Noble Building 858 Aberdeen, Scotland AB24 3UE 859 UK 861 Email: gorry@erg.abdn.ac.uk 862 URI: http://www.erg.abdn.ac.uk