idnits 2.17.1 draft-ietf-tsvwg-circuit-breaker-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (October 19, 2015) is 3083 days in the past. Is this intentional? Checking references for intended status: Best Current Practice ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFC3828' is mentioned on line 614, but not defined -- Obsolete informational reference (is this intentional?): RFC 2309 (Obsoleted by RFC 7567) Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TSVWG Working Group G. Fairhurst 3 Internet-Draft University of Aberdeen 4 Intended status: BCP October 19, 2015 5 Expires: April 21, 2016 7 Network Transport Circuit Breakers 8 draft-ietf-tsvwg-circuit-breaker-07 10 Abstract 12 This document explains what is meant by the term "network transport 13 Circuit Breaker" (CB). It describes the need for circuit breakers 14 when using network tunnels, and other non-congestion controlled 15 applications, and explains where circuit breakers are, and are not, 16 needed. It also defines requirements for building a circuit breaker 17 and the expected outcomes of using a circuit breaker within the 18 Internet. 20 Status of this Memo 22 This Internet-Draft is submitted in full conformance with the 23 provisions of BCP 78 and BCP 79. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF). Note that other groups may also distribute 27 working documents as Internet-Drafts. The list of current Internet- 28 Drafts is at http://datatracker.ietf.org/drafts/current/. 30 Internet-Drafts are draft documents valid for a maximum of six months 31 and may be updated, replaced, or obsoleted by other documents at any 32 time. It is inappropriate to use Internet-Drafts as reference 33 material or to cite them other than as "work in progress." 35 This Internet-Draft will expire on April 21, 2016. 37 Copyright Notice 39 Copyright (c) 2015 IETF Trust and the persons identified as the 40 document authors. All rights reserved. 42 This document is subject to BCP 78 and the IETF Trust's Legal 43 Provisions Relating to IETF Documents 44 (http://trustee.ietf.org/license-info) in effect on the date of 45 publication of this document. Please review these documents 46 carefully, as they describe your rights and restrictions with respect 47 to this document. Code Components extracted from this document must 48 include Simplified BSD License text as described in Section 4.e of 49 the Trust Legal Provisions and are provided without warranty as 50 described in the Simplified BSD License. 52 Table of Contents 54 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 55 1.1. Types of Circuit-Breaker . . . . . . . . . . . . . . . . . 4 56 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 57 3. Design of a Circuit-Breaker (What makes a good circuit 58 breaker?) . . . . . . . . . . . . . . . . . . . . . . . . . . 5 59 3.1. Functional Components . . . . . . . . . . . . . . . . . . 5 60 4. Requirements for a Network Transport Circuit Breaker . . . . . 8 61 5. Other network topologies . . . . . . . . . . . . . . . . . . . 11 62 5.1. Use with a multicast control/routing protocol . . . . . . 11 63 5.2. Use with control protocols supporting pre-provisioned 64 capacity . . . . . . . . . . . . . . . . . . . . . . . . . 13 65 5.3. Unidirectional Circuit Breakers over Controlled Paths . . 13 66 6. Examples of Circuit Breakers . . . . . . . . . . . . . . . . . 14 67 6.1. A Fast-Trip Circuit Breaker . . . . . . . . . . . . . . . 14 68 6.1.1. A Fast-Trip Circuit Breaker for RTP . . . . . . . . . 14 69 6.2. A Slow-trip Circuit Breaker . . . . . . . . . . . . . . . 15 70 6.3. A Managed Circuit Breaker . . . . . . . . . . . . . . . . 15 71 6.3.1. A Managed Circuit Breaker for SAToP Pseudo-Wires . . . 16 72 6.3.2. A Managed Circuit Breaker for Pseudowires (PWs) . . . 17 73 7. Examples where circuit breakers may not be needed. . . . . . . 17 74 7.1. CBs over pre-provisioned Capacity . . . . . . . . . . . . 17 75 7.2. CBs with tunnels carrying Congestion-Controlled Traffic . 18 76 7.3. CBs with Uni-directional Traffic and no Control Path . . . 18 77 8. Security Considerations . . . . . . . . . . . . . . . . . . . 19 78 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 79 10. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 20 80 11. Revision Notes . . . . . . . . . . . . . . . . . . . . . . . . 20 81 12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21 82 12.1. Normative References . . . . . . . . . . . . . . . . . . . 21 83 12.2. Informative References . . . . . . . . . . . . . . . . . . 22 84 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 23 86 1. Introduction 88 A network transport Circuit Breaker (CB) is an automatic mechanism 89 that is used to estimate congestion caused by a flow, and to 90 terminate (or significantly reduce the rate of) the flow when 91 persistent congestion is detected. This is a safety measure to 92 prevent starvation of network resources denying other flows from 93 access to the Internet, such measures are essential for an Internet 94 that is heterogeneous and for traffic that is hard to predict in 95 advance. Avoiding persistent prevention is important to reduce the 96 potential for "Congestion Collapse" [RFC2914]. 98 The term "Circuit Breaker" originates in electricity supply, and has 99 nothing to do with network circuits or virtual circuits. In 100 electricity supply, a Circuit Breaker is intended as a protection 101 mechanism of last resort. Under normal circumstances, a Circuit 102 Breaker ought not to be triggered; it is designed to protect the 103 supply network and attached equipment when there is overload. Just 104 as people do not expect the electrical circuit-breaker (or fuse) in 105 their home to be triggered, except when there is a wiring fault or a 106 problem with an electrical appliance. 108 In networking, the Circuit Breaker principle can be used as a 109 protection mechanism of last resort to avoid persistent congestion 110 impacting other flows that share network capacity. Persistent 111 congestion was a feature of the early Internet of the 1980s. This 112 resulted in excess traffic starving other connection from access to 113 the Internet. It was countered by the requirement to use congestion 114 control (CC) by the Transmission Control Protocol (TCP) [Jacobsen88] 115 [RFC1112]. These mechanisms operate in Internet hosts to cause TCP 116 connections to "back off" during congestion. The introduction of a 117 Congestion Controller in TCP (currently documented in [RFC5681] 118 ensured the stability of the Internet, because it was able to detect 119 congestion and promptly react. This worked well while TCP was by far 120 the dominant traffic in the Internet, and most TCP flows were long- 121 lived (ensuring that they could detect and respond to congestion 122 before the flows terminated). This is no longer the case, and non- 123 congestion controlled traffic, including many applications of the 124 User Datagram Protocol (UDP) can form a significant proportion of the 125 total traffic traversing a link. The current Internet therefore 126 requires that non-congestion controlled traffic needs to be 127 considered to avoid persistent congestion. 129 There are important differences between a transport circuit-breaker 130 and a congestion-control method. Specifically, congestion control 131 (as implemented in TCP, SCTP, and DCCP) operates on the timescale on 132 the order of a packet round-trip-time (RTT), the time from sender to 133 destination and return. Congestion control methods are able to react 134 to a single packet loss/marking and reduce the transmission rate for 135 each loss or congestion event. The goal is usually to limit the 136 maximum transmission rate to a rate that reflects the available 137 capacity across a network path. These methods typically operate on 138 individual traffic flows (e.g., a 5-tuple). 140 In contrast, Circuit Breakers are recommended for non-congestion- 141 controlled Internet flows and for traffic aggregates, e.g., traffic 142 sent using a network tunnel. People have been implementing what this 143 draft characterizes as circuit breakers on an ad hoc basis to protect 144 Internet traffic, this draft therefore provides guidance on how to 145 deploy and use these mechanisms. Later sections provide examples of 146 cases where circuit-breakers may or may not be desirable. 148 A Circuit Breaker needs to measure (meter) the traffic to determine 149 if the network is experiencing congestion and needs to be designed to 150 trigger robustly when there is persistent congestion. This means the 151 trigger needs to operate on a timescale much longer than the path 152 round trip time (e.g., seconds to possibly many tens of seconds). 153 This longer period is needed to provide sufficient time for 154 transports (or applications) to adjust their rate following 155 congestion, and for the network load to stabilize after any 156 adjustment. 158 A Circuit Breaker trigger will often utilize a series of successive 159 sample measurements metered at an ingress point and an egress point 160 (either of which could be a transport endpoint). These measurements 161 need to be taken over a reasonably long period of time. This is to 162 ensure that a Circuit Breaker does not accidentally trigger following 163 a single (or even successive) congestion events (congestion events 164 are what triggers congestion control, and are to be regarded as 165 normal on a network link operating near its capacity). Once 166 triggered, a control function needs to remove traffic from the 167 network, either by disabling the flow or by significantly reducing 168 the level of traffic. This reaction provides the required protection 169 to prevent persistent congestion being experienced by other flows 170 that share the congested part of the network path. 172 Section 4 defines requirements for building a Circuit Breaker. 174 1.1. Types of Circuit-Breaker 176 There are various forms of network transport circuit breaker. These 177 are differentiated mainly on the timescale over which they are 178 triggered, but also in the intended protection they offer: 180 o Fast-Trip Circuit Breakers: The relatively short timescale used by 181 this form of circuit breaker is intended to provide protection for 182 network traffic from a single flow or related group of flows. 184 o Slow-Trip Circuit Breakers: This circuit breaker utilizes a longer 185 timescale and is designed to protect network traffic from 186 congestion by traffic aggregates. 188 o Managed Circuit Breakers: Utilize the operations and management 189 functions that might be present in a managed service to implement 190 a circuit breaker. 192 Examples of each type of circuit breaker are provided in section 4. 194 2. Terminology 196 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 197 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 198 document are to be interpreted as described in [RFC2119]. 200 3. Design of a Circuit-Breaker (What makes a good circuit breaker?) 202 Although circuit breakers have been talked about in the IETF for many 203 years, there has not yet been guidance on the cases where circuit 204 breakers are needed or upon the design of circuit breaker mechanisms. 205 This document seeks to offer advice on these two topics. 207 Circuit Breakers are RECOMMENDED for IETF protocols and tunnels that 208 carry non-congestion-controlled Internet flows and for traffic 209 aggregates. This includes traffic sent using a network tunnel. 210 Designers of other protocols and tunnel encapsulations also ought to 211 consider the use of these techniques to provide last resort to 212 protect traffic that shares the network path being used. 214 This document defines the requirements for design of a Circuit 215 Breaker and provides examples of how a Circuit Breaker can be 216 constructed. The specifications of individual protocols and tunnel 217 encapsulations need to detail the protocol mechanisms needed to 218 implement a Circuit Breaker. 220 Section 3.1 describes the functional components of a circuit breaker 221 and section 3.2 defines requirements for implementing a Circuit 222 Breaker. 224 3.1. Functional Components 226 The basic design of a transport circuit breaker involves 227 communication between an ingress point (a sender) and an egress point 228 (a receiver) of a network flow or set of flows. A simple picture of 229 Circuit Breaker operation is provided in figure 1. This shows a set 230 of routers (each labelled R) connecting a set of endpoints. 232 A Circuit Breaker is used to control traffic passing through a subset 233 of these routers, acting between the ingress and a egress point 234 network devices. The path between the ingress and egress could be 235 provided by a tunnel or other network-layer technique. One expected 236 use would be at the ingress and egress of a service, where all 237 traffic being considered terminates beyond the egress point, and 238 hence the ingress and egress carry the same set of flows. 240 +--------+ +--------+ 241 |Endpoint| |Endpoint| 242 +--+-----+ >>> circuit breaker traffic >>> +--+-----+ 243 | | 244 | +-+ +-+ +---------+ +-+ +-+ +-+ +--------+ +-+ +-+ | 245 +-+R+--+R+->+ Ingress +--+R+--+R+--+R+--+ Egress |--+R+--+R+-+ 246 +++ +-+ +------+--+ +-+ +-+ +-+ +-----+--+ +++ +-+ 247 | ^ | | | 248 | | +--+------+ +------+--+ | 249 | | | Ingress | | Egress | | 250 | | | Meter | | Meter | | 251 | | +----+----+ +----+----+ | 252 | | | | | 253 +-+ | | +----+----+ | | +-+ 254 |R+--+ | | Measure +<----------------+ +--+R| 255 +++ | +----+----+ Reported +++ 256 | | | Egress | 257 | | +----+----+ Measurement | 258 +--+-----+ | | Trigger + +--+-----+ 259 |Endpoint| | +----+----+ |Endpoint| 260 +--------+ | | +--------+ 261 +---<---+ 262 Reaction 264 Figure 1: A CB controlling the part of the end-to-end path between an 265 ingress point and an egress point. (Note: In some cases, the trigger 266 and measure functions could alternatively be located at other 267 locations (e.g., at a network operations centre.) 269 In the context of a Circuit Breaker, the ingress and egress functions 270 could be implemented in different places. For example, they could be 271 located in network devices at a tunnel ingress and at the tunnel 272 egress. In some cases, they could be located at one or both network 273 endpoints (see figure 2), implemented as components within a 274 transport protocol. 276 +----------+ +----------+ 277 | Ingress | +-+ +-+ +-+ | Egress | 278 | Endpoint +->+R+--+R+--+R+--+ Endpoint | 279 +--+----+--+ +-+ +-+ +-+ +----+-----+ 280 ^ | | 281 | +--+------+ +----+----+ 282 | | Ingress | | Egress | 283 | | Meter | | Meter | 284 | +----+----+ +----+----+ 285 | | | 286 | +--- +----+ | 287 | | Measure +<-----------------+ 288 | +----+----+ Reported 289 | | Egress 290 | +----+----+ Measurement 291 | | Trigger | 292 | +----+----+ 293 | | 294 +---<--+ 295 Reaction 297 Figure 2: An endpoint CB implemented at the sender (ingress) and 298 receiver (egress). 300 The set of components needed to implement a Circuit Breaker are: 302 1. An ingress meter (at the sender or tunnel ingress) records the 303 number of packets/bytes sent in each measurement interval. This 304 measures the offered network load for a flow or set of flows. 305 For example, the measurement interval could be many seconds (or 306 every few tens of seconds or a series of successive shorter 307 measurements that are combined by the Circuit Breaker Measurement 308 function). 310 2. An egress meter (at the receiver or tunnel egress) records the 311 number/bytes received in each measurement interval. This 312 measures the supported load for the flow or set of flows, and 313 could utilize other signals to detect the effect of congestion 314 (e.g., loss/marking experienced over the path). The measurements 315 at the egress could be synchronised (including an offset for the 316 time of flight of the data, or referencing the measurements to a 317 particular packet) to ensure any counters refer to the same span 318 of packets. 320 3. The measured values at the ingress and egress are communicated to 321 the Circuit Breaker Measurement function. This could use several 322 methods including: Sending return measurement packets from a 323 receiver to a trigger function at the sender; An implementation 324 using Operations, Administration and Management (OAM); or be 325 sending another in-band signalling datagram to the trigger 326 function. This could also be implemented purely as a control 327 plane function, e.g., using a software-defined network 328 controller. 330 4. The measurement function combines the ingress and egress 331 measurements to assess the present level of network congestion. 332 (For example, the loss rate for each measurement interval could 333 be deduced from calculating the difference between ingress and 334 egress counter values.) Note the method does not require high 335 accuracy for the period of the measurement interval (or therefore 336 the measured value, since isolated and/or infrequent loss events 337 need to be disregarded.) 339 5. A trigger function determines if the measurements indicate 340 persistent congestion. This function defines an appropriate 341 threshold for determining there is persistent congestion between 342 the ingress and egress. This preferably considers a rate or 343 ratio, rather than an absolute value (e.g., more than 10% loss, 344 but other methods could also be based on the rate of transmission 345 as well as the loss rate). The transport Circuit Breaker is 346 triggered when the threshold is exceeded in multiple measurement 347 intervals (e.g., 3 successive measurements). Designs need to be 348 robust so that single or spurious events do not trigger a 349 reaction. 351 6. A reaction that is applied that the Ingress when the Circuit 352 Breaker is triggered. This seeks to automatically remove the 353 traffic causing persistent congestion. 355 7. A feedback mechanism that triggers when either the receive or 356 ingress and egress measurements are not available, since this 357 also could indicate a loss of control packets (also a symptom of 358 heavy congestion or inability to control the load). 360 4. Requirements for a Network Transport Circuit Breaker 362 The requirements for implementing a Circuit Breaker are: 364 o There MUST be a communication path used for control messages from 365 the ingress meter and the egress meter to the point of 366 measurement. The Circuit Breaker MUST trigger if there is a 367 failure of the communication path used for the control messages. 368 That is, the feedback indicating a congested period needs to be 369 designed so that the Circuit Breaker is triggered when it fails to 370 receive measurement reports that indicate an absence of 371 congestion, rather than relying on the successful transmission of 372 a "congested" signal back to the sender. (The feedback signal 373 could itself be lost under congestion). 375 o A Circuit Breaker MUST define a measurement period over which the 376 Circuit Breaker Measurement function measures the level of 377 congestion or loss. This method does not have to detect 378 individual packet loss, but MUST have a way to know that packets 379 have been lost/marked from the traffic flow. If Explicit 380 Congestion Notification (ECN) is enabled [RFC3168], an egress 381 meter MAY also count the number of ECN congestion marks/event per 382 measurement interval, but even if ECN is used, loss MUST still be 383 measured, since this better reflects the impact of persistent 384 congestion. In this context, loss represents a reliable 385 indication of congestion, as opposed to the finer-grain marking of 386 incipient congestion that can be provided via ECN. The type of 387 Circuit Breaker will determine how long this measurement period 388 needs to be. 390 o The measurement period used by a Circuit Breaker Measurement 391 function MUST be longer than the time that current Congestion 392 Control algorithms need to reduce their rate following detection 393 of congestion. This is important because end-to-end Congestion 394 Control algorithms require at least one RTT to notify and adjust 395 the traffic to experienced congestion, and congestion bottlenecks 396 can share traffic with a diverse range of RTTs. The measurement 397 period is therefore expected to be significantly longer than the 398 RTT experienced by the Circuit Breaker itself. 400 o If necessary, MAY combine successive individual meter samples from 401 the ingress and egress to ensure observation of an average over a 402 sufficiently long interval. (Note when meter samples need to be 403 combined, the combination needs to reflect the sum of the 404 individual sample counts divided by the total time/volume over 405 which the samples were measured. Individual samples over 406 different intervals can not be directly combined to generate an 407 average value.) 409 o A Circuit Breaker is REQUIRED to define a threshold to determine 410 whether the measured congestion is considered excessive. 412 o A Circuit Breaker is REQUIRED to define the triggering interval, 413 defining the period over which the trigger uses the collected 414 measurements. Circuit Breakers need to trigger over a 415 sufficiently long period to avoid additionally penalizing flows 416 with a long path RTT (e.g., many path RTTs). 418 o A Circuit Breaker MUST be robust to multiple congestion events. 419 This usually will define a number of measured persistent 420 congestion events per triggering period. For example, a Circuit 421 Breaker MAY combine the results of several measurement periods to 422 determine if the Circuit Breaker is triggered. (e.g., triggered 423 when persistent congestion is detected in 3 of the measurements 424 within the triggering interval). 426 o A Circuit Breaker SHOULD be constructed so that it does not 427 trigger under light or intermittent congestion. 429 o The default response to a trigger SHOULD disable all traffic that 430 contributed to congestion. 432 o Once triggered, the Circuit Breaker MUST react decisively by 433 disabling or significantly reducing traffic at the source (e.g., 434 ingress). A reaction that results in a reduction SHOULD result in 435 reducing the traffic by at least an order of magnitude, each time 436 the Circuit Breaker is triggered. This response needs to be much 437 more severe than that of a Congestion Controller algorithm (such 438 as TCP's congestion control [RFC5681] or TCP-Friendly Rate 439 Control, TFRC [RFC5348]), because the Circuit Breaker reacts to 440 more persistent congestion and operates over longer timescales 441 (i.e., the overload condition will have persisted for a longer 442 time before the Circuit Breaker is triggered). 444 o A Circuit Breaker that reduces the rate of a flow, MUST continue 445 to monitor the level of congestion and MUST further reduce the 446 rate if the Circuit Breaker is again triggered. 448 o The reaction to a triggered Circuit Breaker MUST continue for a 449 period that is at least the triggering interval. Operator 450 intervention will usually be required to restore a flow. If an 451 automated response is needed to reset the trigger, then this needs 452 to not be immediate. The design of an automated reset mechanism 453 needs to be sufficiently conservative that it does not adversely 454 interact with other mechanisms (including other Circuit Breaker 455 algorithms that control traffic over a common path). It SHOULD 456 NOT perform an automated reset when there is evidence of continued 457 congestion. 459 o When a Circuit Breaker is triggered, it SHOULD be regarded as an 460 abnormal network event. As such, this event SHOULD be logged. 461 The measurements that lead to triggering of the Circuit Breaker 462 SHOULD also be logged. 464 o A Circuit Breaker requires control communication between endpoints 465 and/or network devices. The source and integrity of control 466 information (measurements and triggers) MUST be protected from 467 off-path attacks (Section 8 ). The circuit breaker MUST be 468 designed to be robust to packet loss that can also be experienced 469 during congestion/overload. This does not imply that it is 470 desirable to provide reliable delivery (e.g., over TCP), since 471 this can incur additional delay in responding to congestion. 472 Appropriate mechanisms could be to duplicate control messages to 473 provide increased robustness to loss, or/and to regard a lack of 474 control traffic as an indication that excessive congestion may be 475 being experienced [ID-ietf-tsvwg-RFC5405.bis]. 477 o The control communication may be in-band or out-of-band. In-band 478 communication is RECOMMENDED when either design would be possible. 479 If this traffic is sent over a shared path, it is RECOMMENDED that 480 this control traffic is prioritized to reduce the probability of 481 loss under congestion. Control traffic also needs to be 482 considered when provisioning a network that uses a circuit 483 breaker. 485 in-Band: An in-band control method SHOULD assume that loss of 486 control messages is an indication of potential congestion on 487 the path, and repeated loss ought to cause the Circuit Breaker 488 to be triggered. This design has the advantage that it 489 provides fate-sharing of the traffic flow(s) and the control 490 communications. 492 Out-of-Band: An out-of-band control method SHOULD NOT trigger 493 Circuit Breaker reaction when there is loss of control messages 494 (e.g., a loss of measurements). This avoids failure 495 amplification/propagation when the measurement and data paths 496 fail independently. A failure of an out-of-band communication 497 path SHOULD be regarded as abnormal network event and be 498 handled as appropriate for the network, e.g., this event SHOULD 499 be logged, and additional network operator action might be 500 appropriate, depending on the network and the traffic involved. 502 5. Other network topologies 504 A Circuit Breaker can be deployed in networks with topologies 505 different to that presented in figure 2. This section describes 506 examples of such usage, and possible places where functions may be 507 implemented. 509 5.1. Use with a multicast control/routing protocol 510 +----------+ +--------+ +----------+ 511 | Ingress | +-+ +-+ +-+ | Egress | | Egress | 512 | Endpoint +->+R+--+R+--+R+--+ Router |--+ Endpoint +->+ 513 +----+-----+ +-+ +-+ +-+ +---+--+-+ +----+-----+ | 514 ^ ^ ^ ^ | ^ | | 515 | | | | | | | | 516 +----+----+ + - - - < - - - - + | +----+----+ | Reported 517 | Ingress | multicast Prune | | Egress | | Ingress 518 | Meter | | | Meter | | Measurement 519 +---------+ | +----+----+ | 520 | | | 521 | +----+----+ | 522 | | Measure +<--+ 523 | +----+----+ 524 | | 525 | +----+----+ 526 multicast | | Trigger | 527 Leave | +----+----+ 528 Message | | 529 +----<----+ 531 Figure 3: An example of a multicast CB controlling the end-to-end 532 path between an ingress endpoint and an egress endpoint. 534 Figure 3 shows one example of how a multicast circuit breaker could 535 be implemented at a pair of multicast endpoints (e.g., to implement a 536 Fast-Trip Circuit Breaker, Section 6.1). The ingress endpoint (the 537 sender that sources the multicast traffic) meters the ingress load, 538 generating an ingress measurement (e.g., recording timestamped packet 539 counts), and sends this measurement to the multicast group together 540 with the traffic it has measured. 542 Routers along a multicast path forward the multicast traffic 543 (including the ingress measurement) to all active endpoint receivers. 544 Each last hop (egress) router forwards the traffic to one or more 545 egress endpoint(s). 547 In this figure, each endpoint includes a meter that performs a local 548 egress load measurement. An endpoint also extracts the received 549 ingress measurement from the traffic, and compares the ingress and 550 egress measurements to determine if the Circuit Breaker ought to be 551 triggered. This measurement has to be robust to loss (see previous 552 section). If the Circuit Breaker is triggered, it generates a 553 multicast leave message for the egress (e.g., an IGMP or MLD message 554 sent to the last hop router), which causes the upstream router to 555 cease forwarding traffic to the egress endpoint. 557 Any multicast router that has no active receivers for a particular 558 multicast group will prune traffic for that group, sending a prune 559 message to its upstream router. This starts the process of releasing 560 the capacity used by the traffic and is a standard multicast routing 561 function (e.g., using the PIM-SM routing protocol). Each egress 562 operates autonomously, and the circuit breaker "reaction" is executed 563 by the multicast control plane (e.g., by the PIM multicast routing 564 protocol), requiring no explicit signalling by the circuit breaker 565 along the communication path used for the control messages. Note: 566 there is no direct communication with the Ingress, and hence a 567 triggered Circuit Breaker only controls traffic downstream of the 568 first hop router. It does not stop traffic flowing from the sender 569 to the first hop router; this is however the common practice for 570 multicast deployment. 572 The method could also be used with a multicast tunnel or subnetwork 573 (e.g., Section 6.2, Section 6.3), where a meter at the ingress 574 generates additional control messages to carry the measurement data 575 towards the egress where the egress metering is implemented. 577 5.2. Use with control protocols supporting pre-provisioned capacity 579 Some paths are provisioned using a control protocol, e.g., flows 580 provisioned using the Multi-Protocol Label Switching (MPLS) services, 581 path provisioned using the Resource reservation protocol (RSVP), 582 networks utilizing Software Defined Network (SDN) functions, or 583 admission-controlled Differentiated Services. 585 Figure 1 shows one expected use case, where in this usage a separate 586 device could be used to perform the measurement and trigger 587 functions. The reaction generated by the trigger could take the form 588 of a network control message sent to the ingress and/or other network 589 elements causing these elements to react to the Circuit Breaker. 590 Examples of this type of use are provided in section Section 6.3. 592 5.3. Unidirectional Circuit Breakers over Controlled Paths 594 A Circuit Breaker can be used to control uni-directional UDP traffic, 595 providing that there is a communication path that can be used for 596 control messages to connect the functional components at the Ingress 597 and Egress. This communication path for the control messages can 598 exist in networks for which the traffic flow is purely 599 unidirectional. For example, a multicast stream that sends packets 600 across an Internet path and can use multicast routing to prune flows 601 to shed network load. Some other types of subnetwork also utilize 602 control protocols that can be used to control traffic flows. 604 6. Examples of Circuit Breakers 606 There are multiple types of Circuit Breaker that could be defined for 607 use in different deployment cases. This section provides examples of 608 different types of circuit breaker: 610 6.1. A Fast-Trip Circuit Breaker 612 Applications ought to use a full-featured transport (TCP, SCTP, 613 DCCP), and if not, application (e.g. those using UDP and its UDP-Lite 614 variant [RFC3828])they need to provide appropriate congestion 615 avoidance. [RFC2309] discusses the dangers of congestion- 616 unresponsive flows and states that "all UDP-based streaming 617 applications should incorporate effective congestion avoidance 618 mechanisms". Guidance for applications that do not use congestion- 619 controlled transports is provided in [ID-ietf-tsvwg-RFC5405.bis]. 620 Such mechanisms can be designed to react on much shorter timescales 621 than a circuit breaker, that only observes a traffic envelope. These 622 methods can also interact with an application to more effectively 623 control its sending rate. 625 A fast-trip circuit breaker is the most responsive form of Circuit 626 Breaker. It has a response time that is only slightly larger than 627 that of the traffic that it controls. It is suited to traffic with 628 well-understood characteristics (and could include one or more 629 trigger functions specifically tailored the type of traffic for which 630 it is designed). It is not suited to arbitrary network traffic and 631 may be unsuitable fro traffic aggregates, since it could prematurely 632 trigger (e.g., when multiple congestion-controlled flows lead to 633 short-term overload). 635 These mechanisms are suitable for implementation in endpoints, where 636 they can also compliment end-to-end congestion control methods. A 637 shorter response time enables these mechanisms to triggers before 638 other forms of circuit breaker (e.g., circuit breakers operating on 639 traffic aggregates at a point along the network path). 641 6.1.1. A Fast-Trip Circuit Breaker for RTP 643 A set of fast-trip Circuit Breaker methods have been specified for 644 use together by a Real-time Transport Protocol (RTP) flow using the 645 RTP/AVP Profile [RTP-CB]. It is expected that, in the absence of 646 severe congestion, all RTP applications running on best-effort IP 647 networks will be able to run without triggering these circuit 648 breakers. A fast-trip RTP Circuit Breaker is therefore implemented 649 as a fail-safe that when triggered will terminate RTP traffic. 651 The sender monitors reception of in-band Real Time Control Protocol 652 (RTCP) reception report blocks, as contained in SR or RR packets, 653 that convey reception quality feedback information. This is used to 654 measure (congestion) loss, possibly in combination with ECN 655 [RFC6679]. 657 The Circuit Breaker action (shutdown of the flow) is triggered when 658 any of the following trigger conditions are true: 660 1. An RTP Circuit Breaker triggers on reported lack of progress. 662 2. An RTP Circuit Breaker triggers when no receiver reports messages 663 are received. 665 3. An RTP Circuit Breaker uses a TFRC-style check and sets a hard 666 upper limit to the long-term RTP throughput (over many RTTs). 668 4. An RTP Circuit Breaker includes the notion of Media Usability. 669 This circuit breaker is triggered when the quality of the 670 transported media falls below some required minimum acceptable 671 quality. 673 6.2. A Slow-trip Circuit Breaker 675 A slow-trip Circuit Breaker could be implemented in an endpoint or 676 network device. This type of Circuit Breaker is much slower at 677 responding to congestion than a fast-trip Circuit Breaker and is 678 expected to be more common. 680 One example where a slow-trip Circuit Breaker is needed is where 681 flows or traffic-aggregates use a tunnel or encapsulation and the 682 flows within the tunnel do not all support TCP-style congestion 683 control (e.g., TCP, SCTP, TFRC), see [ID-ietf-tsvwg-RFC5405.bis] 684 section 3.1.3. A use case is where tunnels are deployed in the 685 general Internet (rather than "controlled environments" within an 686 Internet service provider or enterprise network), especially when the 687 tunnel could need to cross a customer access router. 689 6.3. A Managed Circuit Breaker 691 A managed Circuit Breaker is implemented in the signalling protocol 692 or management plane that relates to the traffic aggregate being 693 controlled. This type of circuit breaker is typically applicable 694 when the deployment is within a "controlled environment". 696 A Circuit Breaker requires more than the ability to determine that a 697 network path is forwarding data, or to measure the rate of a path - 698 which are often normal network operational functions. There is an 699 additional need to determine a metric for congestion on the path and 700 to trigger a reaction when a threshold is crossed that indicates 701 persistent congestion. 703 The control messages can use either in-band or out-of-band 704 communications. 706 6.3.1. A Managed Circuit Breaker for SAToP Pseudo-Wires 708 [RFC4553], SAToP Pseudo-Wires (PWE3), section 8 describes an example 709 of a managed circuit breaker for isochronous flows. 711 If such flows were to run over a pre-provisioned (e.g., Multi- 712 Protocol Label Switching, MPLS) infrastructure, then it could be 713 expected that the Pseudowire (PW) would not experience congestion, 714 because a flow is not expected to either increase (or decrease) their 715 rate. If instead Pseudo-Wire traffic is multiplexed with other 716 traffic over the general Internet, it could experience congestion. 717 [RFC4553] states: "If SAToP PWs run over a PSN providing best-effort 718 service, they SHOULD monitor packet loss in order to detect "severe 719 congestion". The currently recommended measurement period is 1 720 second, and the trigger operates when there are more than three 721 measured Severely Errored Seconds (SES) within a period. If such a 722 condition is detected, a SAToP PW ought to shut down bidirectionally 723 for some period of time...". 725 The concept was that when the packet loss ratio (congestion) level 726 increased above a threshold, the PW was by default disabled. This 727 use case considered fixed-rate transmission, where the PW had no 728 reasonable way to shed load. 730 The trigger needs to be set at the rate that the PW was likely to 731 experience a serious problem, possibly making the service non- 732 compliant. At this point, triggering the Circuit Breaker would 733 remove the traffic preventing undue impact on congestion-responsive 734 traffic (e.g., TCP). Part of the rationale, was that high loss 735 ratios typically indicated that something was "broken" and ought to 736 have already resulted in operator intervention, and therefore need to 737 trigger this intervention. 739 An operator-based response provides opportunity for other action to 740 restore the service quality, e.g., by shedding other loads or 741 assigning additional capacity, or to consciously avoid reacting to 742 the trigger while engineering a solution to the problem. This could 743 require the trigger to be sent to a third location (e.g., a network 744 operations centre, NOC) responsible for operation of the tunnel 745 ingress, rather than the tunnel ingress itself. 747 6.3.2. A Managed Circuit Breaker for Pseudowires (PWs) 749 Pseudowires (PWs) [RFC3985] have become a common mechanism for 750 tunneling traffic, and may compete for network resources both with 751 other PWs and with non-PW traffic, such as TCP/IP flows. 753 [ID-ietf-pals-congcons] discusses congestion conditions that can 754 arise when PWs compete with elastic (i.e., congestion responsive) 755 network traffic (e.g, TCP traffic). Elastic PWs carrying IP traffic 756 (see [RFC4488]) do not raise major concerns because all of the 757 traffic involved responds, reducing the transmission rate when 758 network congestion is detected. 760 In contrast, inelastic PWs (e.g., a fixed bandwidth Time Division 761 Multiplex, TDM) [RFC4553] [RFC5086] [RFC5087]) have the potential to 762 harm congestion responsive traffic or to contribute to excessive 763 congestion because inelastic PWs do not adjust their transmission 764 rate in response to congestion. [ID-ietf-pals-congcons] analyses TDM 765 PWs, with an initial conclusion that a TDM PW operating with a degree 766 of loss that may result in congestion-related problems is also 767 operating with a degree of loss that results in an unacceptable TDM 768 service. For that reason, the draft suggests that a managed circuit 769 breaker that shuts down a PW when it persistently fails to deliver 770 acceptable TDM service is a useful means for addressing these 771 congestion concerns. 773 7. Examples where circuit breakers may not be needed. 775 A Circuit Breaker is not required for a single Congestion Controller- 776 controlled flow using TCP, SCTP, TFRC, etc. In these cases, the 777 Congestion Control methods are already designed to prevent persistent 778 congestion. 780 7.1. CBs over pre-provisioned Capacity 782 One common question is whether a Circuit Breaker is needed when a 783 tunnel is deployed in a private network with pre-provisioned 784 capacity. 786 In this case, compliant traffic that does not exceed the provisioned 787 capacity ought not to result in persistent congestion. A Circuit 788 Breaker will hence only be triggered when there is non-compliant 789 traffic. It could be argued that this event ought never to happen - 790 but it could also be argued that the Circuit Breaker equally ought 791 never to be triggered. If a Circuit Breaker were to be implemented, 792 it will provide an appropriate response if persistent congestion 793 occurs in an operational network. 795 Implementing a Circuit Breaker will not reduce the performance of the 796 flows, but in the event that persistent congestion occurs it protects 797 network traffic that shares network capacity with these flows. A 798 Circuit Breaker also could be used to protect other sharing network 799 traffic from a failure that causes the Circuit Breaker traffic to be 800 routed over a non-pre-provisioned path. 802 7.2. CBs with tunnels carrying Congestion-Controlled Traffic 804 IP-based traffic is generally assumed to be congestion-controlled, 805 i.e., it is assumed that the transport protocols generating IP-based 806 traffic at the sender already employ mechanisms that are sufficient 807 to address congestion on the path [ID-ietf-tsvwg-RFC5405.bis]. A 808 question therefore arises when people deploy a tunnel that is thought 809 to only carry an aggregate of TCP (or some other Congestion 810 Controller-controlled) traffic: Is there advantage in this case in 811 using a Circuit Breaker? 813 For sure, traffic in a such a tunnel will respond to congestion. 814 However, the answer to the question is not always obvious, because 815 the overall traffic formed by an aggregate of flows that implement a 816 Congestion Controller mechanism does not necessarily prevent 817 persistent congestion. For instance, most Congestion Controller 818 mechanisms require long-lived flows to react to reduce the rate of a 819 flow, an aggregate of many short flows could result in many 820 terminating before they experience congestion. It is also often 821 impossible for a tunnel service provider to know that the tunnel only 822 contains CC-controlled traffic (e.g., Inspecting packet headers could 823 not be possible). The important thing to note is that if the 824 aggregate of the traffic does not result in persistent congestion 825 (impacting other flows), then the Circuit Breaker will not trigger. 826 This is the expected case in this context - so implementing a Circuit 827 Breaker will not reduce performance of the tunnel, but in the event 828 that persistent congestion occurs this protects other network traffic 829 that shares capacity with the tunnel traffic. 831 7.3. CBs with Uni-directional Traffic and no Control Path 833 A one-way forwarding path could have no associated communication path 834 for sending control messages, and therefore cannot be controlled 835 using an automated process. This service could be provided using a 836 path that has dedicated capacity and does not share this capacity 837 with other elastic Internet flows (i.e., flows that vary their rate). 839 A way to mitigate the impact on other flows when capacity could be 840 shared is to manage the traffic envelope by using ingress policing. 842 Supporting this type of traffic in the general Internet requires 843 operator monitoring to detect and respond to persistent congestion. 845 8. Security Considerations 847 All Circuit Breaker mechanisms rely upon coordination between the 848 ingress and egress meters and communication with the trigger 849 function. This is usually achieved by passing network control 850 information (or protocol messages) across the network. Timely 851 operation of a circuit breaker depends on the choice of measurement 852 period. If the receiver has an interval that is overly long, then 853 the responsiveness of the circuit breaker decreases. This impacts 854 the ability of the circuit breaker to detect and react to congestion. 856 Mechanisms need to be implemented to prevent attacks on the network 857 control information that would result in Denial of Service (DoS). 858 The source and integrity of control information (measurements and 859 triggers) MUST be protected from off-path attacks. Without 860 protection, it could be trivial for an attacker to inject packets 861 with values that could prematurely trigger a circuit breaker 862 resulting in DoS. Simple protection can be provided by using a 863 randomized source port, or equivalent field in the packet header 864 (such as the RTP SSRC value and the RTP sequence number) expected not 865 to be known to an off-path attacker. Stronger protection can be 866 achieved using a secure authentication protocol. 868 Transmission of network control information consumes network 869 capacity. This control traffic needs to be considered in the design 870 of a Circuit Breaker and could potentially add to network congestion. 871 If this traffic is sent over a shared path, it is RECOMMENDED that 872 this control traffic is prioritized to reduce the probability of loss 873 under congestion. Control traffic also needs to be considered when 874 provisioning a network that uses a circuit breaker. 876 The circuit breaker MUST be designed to be robust to packet loss that 877 can also be experienced during congestion/overload. Loss of control 878 messages could be a side-effect of a congested network, but also 879 could arise from other causes Section 4. 881 The security implications depend on the design of the mechanisms, the 882 type of traffic being controlled and the intended deployment 883 scenario. Each design of a Circuit Breaker MUST therefore evaluate 884 whether the particular circuit breaker mechanism has new security 885 implications. 887 9. IANA Considerations 889 This document makes no request from IANA. 891 10. Acknowledgments 893 There are many people who have discussed and described the issues 894 that have motivated this draft. Contributions and comments included: 895 Lars Eggert, Colin Perkins, David Black, Matt Mathis, Andrew 896 McGregor, and Bob Briscoe. This work was part-funded by the European 897 Community under its Seventh Framework Programme through the Reducing 898 Internet Transport Latency (RITE) project (ICT-317700). 900 11. Revision Notes 902 XXX RFC-Editor: Please remove this section prior to publication XXX 904 Draft 00 906 This was the first revision. Help and comments are greatly 907 appreciated. 909 Draft 01 911 Contained clarifications and changes in response to received 912 comments, plus addition of diagram and definitions. Comments are 913 welcome. 915 WG Draft 00 917 Approved as a WG work item on 28th Aug 2014. 919 WG Draft 01 921 Incorporates feedback after Dallas IETF TSVWG meeting. This version 922 is thought ready for WGLC comments. Definitions of abbreviations. 924 WG Draft 02 926 Minor fixes for typos. Rewritten security considerations section. 928 WG Draft 03 930 Updates following WGLC comments (see TSV mailing list). Comments 931 from C Perkins; D Black and off-list feedback. 933 A clear recommendation of intended scope. 935 Changes include: Improvement of language on timescales and minimum 936 measurement period; clearer articulation of endpoint and multicast 937 examples - with new diagrams; separation of the controlled network 938 case; updated text on position of trigger function; corrections to 939 RTP-CB text; clarification of loss v ECN metrics; checks against 940 submission checklist 9use of keywords, added meters to diagrams). 942 WG Draft 04 944 Added section on PW CB for TDM - a newly adopted draft (D. Black). 946 WG Draft 05 948 Added clarifications requested during AD review. 950 WG Draft 06 952 Fixed some remaining typos. 954 Update following detailed review by Bob Briscoe, and comments by D. 955 Black. 957 WG Draft 07 959 Updated text on the response to lack of meter measurements with 960 managed circuit breakers. 962 12. References 964 12.1. Normative References 966 [ID-ietf-tsvwg-RFC5405.bis] 967 Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage 968 Guidelines (Work-in-Progress)", 2015. 970 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 971 Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/ 972 RFC2119, March 1997, 973 . 975 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 976 of Explicit Congestion Notification (ECN) to IP", 977 RFC 3168, DOI 10.17487/RFC3168, September 2001, 978 . 980 12.2. Informative References 982 [ID-ietf-pals-congcons] 983 Stein, YJ., Black, D., and B. Briscoe, "Pseudowire 984 Congestion Considerations (Work-in-Progress)", 2015. 986 [Jacobsen88] 987 European Telecommunication Standards, Institute (ETSI), 988 "Congestion Avoidance and Control", SIGCOMM Symposium 989 proceedings on Communications architectures and 990 protocols", August 1998. 992 [RFC1112] Deering, S., "Host extensions for IP multicasting", STD 5, 993 RFC 1112, DOI 10.17487/RFC1112, August 1989, 994 . 996 [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, 997 S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., 998 Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, 999 S., Wroclawski, J., and L. Zhang, "Recommendations on 1000 Queue Management and Congestion Avoidance in the 1001 Internet", RFC 2309, DOI 10.17487/RFC2309, April 1998, 1002 . 1004 [RFC2914] Floyd, S., "Congestion Control Principles", BCP 41, 1005 RFC 2914, DOI 10.17487/RFC2914, September 2000, 1006 . 1008 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 1009 Edge-to-Edge (PWE3) Architecture", RFC 3985, DOI 10.17487/ 1010 RFC3985, March 2005, 1011 . 1013 [RFC4488] Levin, O., "Suppression of Session Initiation Protocol 1014 (SIP) REFER Method Implicit Subscription", RFC 4488, 1015 DOI 10.17487/RFC4488, May 2006, 1016 . 1018 [RFC4553] Vainshtein, A., Ed. and YJ. Stein, Ed., "Structure- 1019 Agnostic Time Division Multiplexing (TDM) over Packet 1020 (SAToP)", RFC 4553, DOI 10.17487/RFC4553, June 2006, 1021 . 1023 [RFC5086] Vainshtein, A., Ed., Sasson, I., Metz, E., Frost, T., and 1024 P. Pate, "Structure-Aware Time Division Multiplexed (TDM) 1025 Circuit Emulation Service over Packet Switched Network 1026 (CESoPSN)", RFC 5086, DOI 10.17487/RFC5086, December 2007, 1027 . 1029 [RFC5087] Stein, Y(J)., Shashoua, R., Insler, R., and M. Anavi, 1030 "Time Division Multiplexing over IP (TDMoIP)", RFC 5087, 1031 DOI 10.17487/RFC5087, December 2007, 1032 . 1034 [RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP 1035 Friendly Rate Control (TFRC): Protocol Specification", 1036 RFC 5348, DOI 10.17487/RFC5348, September 2008, 1037 . 1039 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 1040 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 1041 . 1043 [RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P., 1044 and K. Carlberg, "Explicit Congestion Notification (ECN) 1045 for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, 1046 August 2012, . 1048 [RTP-CB] Perkins and Singh, "Multimedia Congestion Control: Circuit 1049 Breakers for Unicast RTP Sessions", February 2014. 1051 Author's Address 1053 Godred Fairhurst 1054 University of Aberdeen 1055 School of Engineering 1056 Fraser Noble Building 1057 Aberdeen, Scotland AB24 3UE 1058 UK 1060 Email: gorry@erg.abdn.ac.uk 1061 URI: http://www.erg.abdn.ac.uk