idnits 2.17.1 draft-ietf-tsvwg-circuit-breaker-15.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (April 04, 2016) is 2944 days in the past. Is this intentional? Checking references for intended status: Best Current Practice ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Obsolete informational reference (is this intentional?): RFC 2309 (Obsoleted by RFC 7567) -- Obsolete informational reference (is this intentional?): RFC 4601 (Obsoleted by RFC 7761) Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TSVWG Working Group G. Fairhurst 3 Internet-Draft University of Aberdeen 4 Intended status: Best Current Practice April 04, 2016 5 Expires: October 6, 2016 7 Network Transport Circuit Breakers 8 draft-ietf-tsvwg-circuit-breaker-15 10 Abstract 12 This document explains what is meant by the term "network transport 13 Circuit Breaker" (CB). It describes the need for circuit breakers 14 for network tunnels and applications when using non-congestion- 15 controlled traffic, and explains where circuit breakers are, and are 16 not, needed. It also defines requirements for building a circuit 17 breaker and the expected outcomes of using a circuit breaker within 18 the Internet. 20 Status of This Memo 22 This Internet-Draft is submitted in full conformance with the 23 provisions of BCP 78 and BCP 79. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF). Note that other groups may also distribute 27 working documents as Internet-Drafts. The list of current Internet- 28 Drafts is at http://datatracker.ietf.org/drafts/current/. 30 Internet-Drafts are draft documents valid for a maximum of six months 31 and may be updated, replaced, or obsoleted by other documents at any 32 time. It is inappropriate to use Internet-Drafts as reference 33 material or to cite them other than as "work in progress." 35 This Internet-Draft will expire on October 6, 2016. 37 Copyright Notice 39 Copyright (c) 2016 IETF Trust and the persons identified as the 40 document authors. All rights reserved. 42 This document is subject to BCP 78 and the IETF Trust's Legal 43 Provisions Relating to IETF Documents 44 (http://trustee.ietf.org/license-info) in effect on the date of 45 publication of this document. Please review these documents 46 carefully, as they describe your rights and restrictions with respect 47 to this document. Code Components extracted from this document must 48 include Simplified BSD License text as described in Section 4.e of 49 the Trust Legal Provisions and are provided without warranty as 50 described in the Simplified BSD License. 52 Table of Contents 54 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 55 1.1. Types of Circuit Breaker . . . . . . . . . . . . . . . . 6 56 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6 57 3. Design of a Circuit-Breaker (What makes a good circuit 58 breaker?) . . . . . . . . . . . . . . . . . . . . . . . . . . 6 59 3.1. Functional Components . . . . . . . . . . . . . . . . . . 7 60 3.2. Other network topologies . . . . . . . . . . . . . . . . 10 61 3.2.1. Use with a multicast control/routing protocol . . . . 10 62 3.2.2. Use with control protocols supporting pre-provisioned 63 capacity . . . . . . . . . . . . . . . . . . . . . . 11 64 3.2.3. Unidirectional Circuit Breakers over Controlled Paths 12 65 4. Requirements for a Network Transport Circuit Breaker . . . . 12 66 5. Examples of Circuit Breakers . . . . . . . . . . . . . . . . 15 67 5.1. A Fast-Trip Circuit Breaker . . . . . . . . . . . . . . . 15 68 5.1.1. A Fast-Trip Circuit Breaker for RTP . . . . . . . . . 16 69 5.2. A Slow-trip Circuit Breaker . . . . . . . . . . . . . . . 17 70 5.3. A Managed Circuit Breaker . . . . . . . . . . . . . . . . 17 71 5.3.1. A Managed Circuit Breaker for SAToP Pseudo-Wires . . 17 72 5.3.2. A Managed Circuit Breaker for Pseudowires (PWs) . . . 18 73 6. Examples where circuit breakers may not be needed. . . . . . 19 74 6.1. CBs over pre-provisioned Capacity . . . . . . . . . . . . 19 75 6.2. CBs with tunnels carrying Congestion-Controlled Traffic . 19 76 6.3. CBs with Uni-directional Traffic and no Control Path . . 20 77 7. Security Considerations . . . . . . . . . . . . . . . . . . . 20 78 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 79 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 22 80 10. Revision Notes . . . . . . . . . . . . . . . . . . . . . . . 22 81 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 25 82 11.1. Normative References . . . . . . . . . . . . . . . . . . 25 83 11.2. Informative References . . . . . . . . . . . . . . . . . 25 84 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 27 86 1. Introduction 88 The term "Circuit Breaker" originates in electricity supply, and has 89 nothing to do with network circuits or virtual circuits. In 90 electricity supply, a Circuit Breaker is intended as a protection 91 mechanism of last resort. Under normal circumstances, a Circuit 92 Breaker ought not to be triggered; it is designed to protect the 93 supply network and attached equipment when there is overload. People 94 do not expect an electrical circuit-breaker (or fuse) in their home 95 to be triggered, except when there is a wiring fault or a problem 96 with an electrical appliance. 98 In networking, the Circuit Breaker (CB) principle can be used as a 99 protection mechanism of last resort to avoid persistent excessive 100 congestion impacting other flows that share network capacity. 101 Persistent congestion was a feature of the early Internet of the 102 1980s. This resulted in excess traffic starving other connections 103 from access to the Internet. It was countered by the requirement to 104 use congestion control (CC) in the Transmission Control Protocol 105 (TCP) [Jacobsen88]. These mechanisms operate in Internet hosts to 106 cause TCP connections to "back off" during congestion. The addition 107 of a congestion control to TCP (currently documented in [RFC5681] 108 ensured the stability of the Internet, because it was able to detect 109 congestion and promptly react. This was effective in an Internet 110 where most TCP flows were long-lived (ensuring that they could detect 111 and respond to congestion before the flows terminated). Although TCP 112 was by far the dominant traffic, this is no longer the always the 113 case, and non-congestion-controlled traffic, including many 114 applications using the User Datagram Protocol (UDP), can form a 115 significant proportion of the total traffic traversing a link. The 116 current Internet therefore requires that non-congestion-controlled 117 traffic is considered to avoid persistent excessive congestion. 119 A network transport Circuit Breaker is an automatic mechanism that is 120 used to continuously monitor a flow or aggregate set of flows. The 121 mechanism seeks to detect when the flow(s) experience persistent 122 excessive congestion. When this is detected, a Circuit Breaker 123 terminates (or significantly reduce the rate of) the flow(s). This 124 is a safety measure to prevent starvation of network resources 125 denying other flows from access to the Internet. Such measures are 126 essential for an Internet that is heterogeneous and for traffic that 127 is hard to predict in advance. Avoiding persistent excessive 128 congestion is important to reduce the potential for "Congestion 129 Collapse" [RFC2914]. 131 There are important differences between a transport Circuit Breaker 132 and a congestion control method. Congestion control (as implemented 133 in TCP, SCTP, and DCCP) operates on a timescale on the order of a 134 packet round-trip-time (RTT), the time from sender to destination and 135 return. Congestion at a network link can also be detected using 136 Explicit Congestion Notification (ECN) [RFC3168], which allows the 137 network to signal congestion by marking ECN-capable packets with a 138 Congestion Experienced (CE) mark. Both loss and reception of CE- 139 marked packets are treated as congestion events. Congestion control 140 methods are able to react to a congestion event by continuously 141 adapting to reduce their transmission rate. The goal is usually to 142 limit the transmission rate to a maximum rate that reflects a fair 143 use of the available capacity across a network path. These methods 144 typically operate on individual traffic flows (e.g., a 5-tuple that 145 includes the IP addresses, protocol, and ports). 147 In contrast, Circuit Breakers are recommended for non-congestion- 148 controlled Internet flows and for traffic aggregates, e.g., traffic 149 sent using a network tunnel. They operate on timescales much longer 150 than the packet RTT, and trigger under situations of abnormal 151 (excessive) congestion. People have been implementing what this 152 document characterizes as circuit breakers on an ad hoc basis to 153 protect Internet traffic. This document therefore provides guidance 154 on how to deploy and use these mechanisms. Later sections provide 155 examples of cases where circuit-breakers may or may not be desirable. 157 A Circuit Breaker needs to measure (meter) some portion of the 158 traffic to determine if the network is experiencing congestion and 159 needs to be designed to trigger robustly when there is persistent 160 excessive congestion. 162 A Circuit Breaker trigger will often utilize a series of successive 163 sample measurements metered at an ingress point and an egress point 164 (either of which could be a transport endpoint). The trigger needs 165 to operate on a timescale much longer than the path round trip time 166 (e.g., seconds to possibly many tens of seconds). This longer period 167 is needed to provide sufficient time for transport congestion control 168 (or applications) to adjust their rate following congestion, and for 169 the network load to stabilize after any adjustment. Congestion 170 events can be common when a congestion-controlled transport is used 171 over a network link operating near capacity. Each event results in 172 reduction in the rate of the transport flow experiencing congestion. 173 The longer period seeks to ensure that a Circuit Breaker does not 174 accidentally trigger following a single (or even successive) 175 congestion events. 177 Once triggered, the Circuit Breaker needs to provide a control 178 function (called the "reaction"). This removes traffic from the 179 network, either by disabling the flow or by significantly reducing 180 the level of traffic. This reaction provides the required protection 181 to prevent persistent excessive congestion being experienced by other 182 flows that share the congested part of the network path. 184 Section 4 defines requirements for building a Circuit Breaker. 186 The operational conditions that cause a Circuit Breaker to trigger 187 ought to be regarded as abnormal. Examples of situations that could 188 trigger a Circuit Breaker include: 190 o anomalous traffic that exceeds the provisioned capacity (or whose 191 traffic characteristics exceed the threshold configured for the 192 Circuit Breaker); 194 o traffic generated by an application at a time when the provisioned 195 network capacity is being utilised for other purposes; 197 o routing changes that cause additional traffic to start using the 198 path monitored by the Circuit Breaker; 200 o misconfiguration of a service/network device where the capacity 201 available is insufficient to support the current traffic 202 aggregate; 204 o misconfiguration of an admission controller or traffic policer 205 that allows more traffic than expected across the path monitored 206 by the Circuit Breaker. 208 Other mechanisms could also be available to network operators to 209 detect excessive congestion (e.g., an observation of excessive 210 utilisation for a port on a network device). Utilising such 211 information, operational mechanisms could react to reduce network 212 load over a shorter timescale than those of a network transport 213 Circuit Breaker. The role of the Circuit Breaker over such paths 214 remains as a method of last resort. Because it acts over a longer 215 timescale, the Circuit Breaker ought to trigger only when other 216 reactions did not succeed in reducing persistent excessive 217 congestion. 219 In many cases, the reason for triggering a Circuit Breaker will not 220 be evident to the source of the traffic (user, application, endpoint, 221 etc). A Circuit Breaker can be used to limit traffic from 222 applications that are unable, or choose not, to use congestion 223 control, or where the congestion control properties of the traffic 224 cannot be relied upon (e.g., traffic carried over a network tunnel). 225 In such circumstances, it is all but impossible for the Circuit 226 Breaker to signal back to the impacted applications. In some cases 227 applications could therefore have difficulty in determining that a 228 Circuit Breaker has triggered, and where in the network this 229 happened. 231 Application developers are therefore advised, where possible, to 232 deploy appropriate congestion control mechanisms. An application 233 that uses congestion control will be aware of congestion events in 234 the network. This allows it to regulate the network load under 235 congestion, and is expected to avoid triggering a network Circuit 236 Breaker. For applications that can generate elastic traffic, this 237 will often be a preferred solution. 239 1.1. Types of Circuit Breaker 241 There are various forms of network transport circuit breaker. These 242 are differentiated mainly on the timescale over which they are 243 triggered, but also in the intended protection they offer: 245 o Fast-Trip Circuit Breakers: The relatively short timescale used by 246 this form of circuit breaker is intended to provide protection for 247 network traffic from a single flow or related group of flows. 249 o Slow-Trip Circuit Breakers: This circuit breaker utilizes a longer 250 timescale and is designed to protect network traffic from 251 congestion by traffic aggregates. 253 o Managed Circuit Breakers: Utilize the operations and management 254 functions that might be present in a managed service to implement 255 a circuit breaker. 257 Examples of each type of circuit breaker are provided in section 4. 259 2. Terminology 261 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 262 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 263 document are to be interpreted as described in [RFC2119]. 265 3. Design of a Circuit-Breaker (What makes a good circuit breaker?) 267 Although circuit breakers have been talked about in the IETF for many 268 years, there has not yet been guidance on the cases where circuit 269 breakers are needed or upon the design of circuit breaker mechanisms. 270 This document seeks to offer advice on these two topics. 272 Circuit Breakers are RECOMMENDED for IETF protocols and tunnels that 273 carry non-congestion-controlled Internet flows and for traffic 274 aggregates. This includes traffic sent using a network tunnel. 275 Designers of other protocols and tunnel encapsulations also ought to 276 consider the use of these techniques as a last resort to protect 277 traffic that shares the network path being used. 279 This document defines the requirements for design of a Circuit 280 Breaker and provides examples of how a Circuit Breaker can be 281 constructed. The specifications of individual protocols and tunnel 282 encapsulations need to detail the protocol mechanisms needed to 283 implement a Circuit Breaker. 285 Section 3.1 describes the functional components of a circuit breaker 286 and section 3.2 defines requirements for implementing a Circuit 287 Breaker. 289 3.1. Functional Components 291 The basic design of a Circuit Breaker involves communication between 292 an ingress point (a sender) and an egress point (a receiver) of a 293 network flow or set of flows. A simple picture of operation is 294 provided in figure 1. This shows a set of routers (each labelled R) 295 connecting a set of endpoints. 297 A Circuit Breaker is used to control traffic passing through a subset 298 of these routers, acting between the ingress and a egress point 299 network devices. The path between the ingress and egress could be 300 provided by a tunnel or other network-layer technique. One expected 301 use would be at the ingress and egress of a service, where all 302 traffic being considered terminates beyond the egress point, and 303 hence the ingress and egress carry the same set of flows. 305 +--------+ +--------+ 306 |Endpoint| |Endpoint| 307 +--+-----+ >>> circuit breaker traffic >>> +--+-----+ 308 | | 309 | +-+ +-+ +---------+ +-+ +-+ +-+ +--------+ +-+ +-+ | 310 +-+R+--+R+->+ Ingress +--+R+--+R+--+R+--+ Egress |--+R+--+R+-+ 311 +++ +-+ +------+--+ +-+ +-+ +-+ +-----+--+ +++ +-+ 312 | ^ | | | 313 | | +--+------+ +------+--+ | 314 | | | Ingress | | Egress | | 315 | | | Meter | | Meter | | 316 | | +----+----+ +----+----+ | 317 | | | | | 318 +-+ | | +----+----+ | | +-+ 319 |R+--+ | | Measure +<----------------+ +--+R| 320 +++ | +----+----+ Reported +++ 321 | | | Egress | 322 | | +----+----+ Measurement | 323 +--+-----+ | | Trigger + +--+-----+ 324 |Endpoint| | +----+----+ |Endpoint| 325 +--------+ | | +--------+ 326 +---<---+ 327 Reaction 329 Figure 1: A CB controlling the part of the end-to-end path between an 330 ingress point and an egress point. (Note: In some cases, the trigger 331 and measurement functions could alternatively be located at other 332 locations (e.g., at a network operations centre.) 334 In the context of a Circuit Breaker, the ingress and egress functions 335 could be implemented in different places. For example, they could be 336 located in network devices at a tunnel ingress and at the tunnel 337 egress. In some cases, they could be located at one or both network 338 endpoints (see figure 2), implemented as components within a 339 transport protocol. 341 +----------+ +----------+ 342 | Ingress | +-+ +-+ +-+ | Egress | 343 | Endpoint +->+R+--+R+--+R+--+ Endpoint | 344 +--+----+--+ +-+ +-+ +-+ +----+-----+ 345 ^ | | 346 | +--+------+ +----+----+ 347 | | Ingress | | Egress | 348 | | Meter | | Meter | 349 | +----+----+ +----+----+ 350 | | | 351 | +--- +----+ | 352 | | Measure +<-----------------+ 353 | +----+----+ Reported 354 | | Egress 355 | +----+----+ Measurement 356 | | Trigger | 357 | +----+----+ 358 | | 359 +---<--+ 360 Reaction 362 Figure 2: An endpoint CB implemented at the sender (ingress) and 363 receiver (egress). 365 The set of components needed to implement a Circuit Breaker are: 367 1. An ingress meter (at the sender or tunnel ingress) that records 368 the number of packets/bytes sent in each measurement interval. 369 This measures the offered network load for a flow or set of 370 flows. For example, the measurement interval could be many 371 seconds (or every few tens of seconds or a series of successive 372 shorter measurements that are combined by the Circuit Breaker 373 Measurement function). 375 2. An egress meter (at the receiver or tunnel egress) that records 376 the number/bytes received in each measurement interval. This 377 measures the supported load for the flow or set of flows, and 378 could utilize other signals to detect the effect of congestion 379 (e.g., loss/congestion marking [RFC3168] experienced over the 380 path). The measurements at the egress could be synchronised 381 (including an offset for the time of flight of the data, or 382 referencing the measurements to a particular packet) to ensure 383 any counters refer to the same span of packets. 385 3. A method that communicates the measured values at the ingress and 386 egress to the Circuit Breaker Measurement function. This could 387 use several methods including: Sending return measurement packets 388 (or control messages) from a receiver to a trigger function at 389 the sender; an implementation using Operations, Administration 390 and Management (OAM); or sending an in-band signalling datagram 391 to the trigger function. This could also be implemented purely 392 as a control plane function, e.g., using a software-defined 393 network controller. 395 4. A measurement function that combines the ingress and egress 396 measurements to assess the present level of network congestion. 397 (For example, the loss rate for each measurement interval could 398 be deduced from calculating the difference between ingress and 399 egress counter values.) Note the method does not require high 400 accuracy for the period of the measurement interval (or therefore 401 the measured value, since isolated and/or infrequent loss events 402 need to be disregarded.) 404 5. A trigger function that determines whether the measurements 405 indicate persistent excessive congestion. This function defines 406 an appropriate threshold for determining that there is persistent 407 excessive congestion between the ingress and egress. This 408 preferably considers a rate or ratio, rather than an absolute 409 value (e.g., more than 10% loss, but other methods could also be 410 based on the rate of transmission as well as the loss rate). The 411 Circuit Breaker is triggered when the threshold is exceeded in 412 multiple measurement intervals (e.g., 3 successive measurements). 413 Designs need to be robust so that single or spurious events do 414 not trigger a reaction. 416 6. A reaction that is applied at the Ingress when the Circuit 417 Breaker is triggered. This seeks to automatically remove the 418 traffic causing persistent excessive congestion. 420 7. A feedback control mechanism that triggers when either the 421 receive or ingress and egress measurements are not available, 422 since this also could indicate a loss of control packets (also a 423 symptom of heavy congestion or inability to control the load). 425 3.2. Other network topologies 427 A Circuit Breaker can be deployed in networks with topologies 428 different to that presented in figures 1 and 2. This section 429 describes examples of such usage, and possible places where functions 430 can be implemented. 432 3.2.1. Use with a multicast control/routing protocol 434 +----------+ +--------+ +----------+ 435 | Ingress | +-+ +-+ +-+ | Egress | | Egress | 436 | Endpoint +->+R+--+R+--+R+--+ Router |--+ Endpoint +->+ 437 +----+-----+ +-+ +-+ +-+ +---+--+-+ +----+-----+ | 438 ^ ^ ^ ^ | ^ | | 439 | | | | | | | | 440 +----+----+ + - - - < - - - - + | +----+----+ | Reported 441 | Ingress | multicast Prune | | Egress | | Ingress 442 | Meter | | | Meter | | Measurement 443 +---------+ | +----+----+ | 444 | | | 445 | +----+----+ | 446 | | Measure +<--+ 447 | +----+----+ 448 | | 449 | +----+----+ 450 multicast | | Trigger | 451 Leave | +----+----+ 452 Message | | 453 +----<----+ 455 Figure 3: An example of a multicast CB controlling the end-to-end 456 path between an ingress endpoint and an egress endpoint. 458 Figure 3 shows one example of how a multicast Circuit Breaker could 459 be implemented at a pair of multicast endpoints (e.g., to implement a 460 Fast-Trip Circuit Breaker, Section 5.1). The ingress endpoint (the 461 sender that sources the multicast traffic) meters the ingress load, 462 generating an ingress measurement (e.g., recording timestamped packet 463 counts), and sends this measurement to the multicast group together 464 with the traffic it has measured. 466 Routers along a multicast path forward the multicast traffic 467 (including the ingress measurement) to all active endpoint receivers. 468 Each last hop (egress) router forwards the traffic to one or more 469 egress endpoint(s). 471 In this figure, each endpoint includes a meter that performs a local 472 egress load measurement. An endpoint also extracts the received 473 ingress measurement from the traffic, and compares the ingress and 474 egress measurements to determine if the Circuit Breaker ought to be 475 triggered. This measurement has to be robust to loss (see previous 476 section). If the Circuit Breaker is triggered, it generates a 477 multicast leave message for the egress (e.g., an IGMP or MLD message 478 sent to the last hop router), which causes the upstream router to 479 cease forwarding traffic to the egress endpoint [RFC1112]. 481 Any multicast router that has no active receivers for a particular 482 multicast group will prune traffic for that group, sending a prune 483 message to its upstream router. This starts the process of releasing 484 the capacity used by the traffic and is a standard multicast routing 485 function (e.g., using Protocol Independent Multicast Sparse Mode 486 (PIM-SM) routing protocol [RFC4601]). Each egress operates 487 autonomously, and the Circuit Breaker "reaction" is executed by the 488 multicast control plane (e.g., by PIM) requiring no explicit 489 signalling by the Circuit Breaker along the communication path used 490 for the control messages. Note: there is no direct communication 491 with the Ingress, and hence a triggered Circuit Breaker only controls 492 traffic downstream of the first hop multicast router. It does not 493 stop traffic flowing from the sender to the first hop router; this is 494 common practice for multicast deployment. 496 The method could also be used with a multicast tunnel or subnetwork 497 (e.g., Section 5.2, Section 5.3), where a meter at the ingress 498 generates additional control messages to carry the measurement data 499 towards the egress where the egress metering is implemented. 501 3.2.2. Use with control protocols supporting pre-provisioned capacity 503 Some paths are provisioned using a control protocol, e.g., flows 504 provisioned using the Multi-Protocol Label Switching (MPLS) services, 505 paths provisioned using the resource reservation protocol (RSVP), 506 networks utilizing Software Defined Network (SDN) functions, or 507 admission-controlled Differentiated Services. Figure 1 shows one 508 expected use case, where in this usage a separate device could be 509 used to perform the measurement and trigger functions. The reaction 510 generated by the trigger could take the form of a network control 511 message sent to the ingress and/or other network elements causing 512 these elements to react to the Circuit Breaker. Examples of this 513 type of use are provided in section Section 5.3. 515 3.2.3. Unidirectional Circuit Breakers over Controlled Paths 517 A Circuit Breaker can be used to control uni-directional UDP traffic, 518 providing that there is a communication path that can be used for 519 control messages to connect the functional components at the Ingress 520 and Egress. This communication path for the control messages can 521 exist in networks for which the traffic flow is purely 522 unidirectional. For example, a multicast stream that sends packets 523 across an Internet path and can use multicast routing to prune flows 524 to shed network load. Some other types of subnetwork also utilize 525 control protocols that can be used to control traffic flows. 527 4. Requirements for a Network Transport Circuit Breaker 529 The requirements for implementing a Circuit Breaker are: 531 1. There needs to be a communication path for control messages to 532 carry measurement data from the ingress meter and from the 533 egress meter to the point of measurement. (Requirements 16-18 534 relate to the transmission of control messages.) 536 2. A CB is REQUIRED to define a measurement period over which the 537 CB Measurement function measures the level of congestion or 538 loss. This method does not have to detect individual packet 539 loss, but MUST have a way to know that packets have been lost/ 540 marked from the traffic flow. 542 3. An egress meter can also count ECN [RFC3168] congestion marks as 543 a part of measurement of congestion, but in this case, loss MUST 544 also be measured to provide a complete view of the level of 545 congestion. For tunnels, 546 [ID-ietf-tsvwg-tunnel-congestion-feedback] describes a way to 547 measure both loss and ECN-marking; these measurements could be 548 used on a relatively short timescale to drive a congestion 549 control response and/or aggregated over a longer timescale with 550 a higher trigger threshold to drive a CB. Subsequent bullet 551 items in this section discuss the necessity of using a longer 552 timescale and a higher trigger threshold. 554 4. The measurement period used by a CB Measurement function MUST be 555 longer than the time that current Congestion Control algorithms 556 need to reduce their rate following detection of congestion. 557 This is important because end-to-end Congestion Control 558 algorithms require at least one RTT to notify and adjust the 559 traffic when congestion is experienced, and congestion 560 bottlenecks can share traffic with a diverse range of RTTs. The 561 measurement period is therefore expected to be significantly 562 longer than the RTT experienced by the CB itself. 564 5. If necessary, a CB MAY combine successive individual meter 565 samples from the ingress and egress to ensure observation of an 566 average measurement over a sufficiently long interval. (Note 567 when meter samples need to be combined, the combination needs to 568 reflect the sum of the individual sample counts divided by the 569 total time/volume over which the samples were measured. 570 Individual samples over different intervals can not be directly 571 combined to generate an average value.) 573 6. A CB MUST be constructed so that it does not trigger under light 574 or intermittent congestion (see requirements 7-9). 576 7. A CB is REQUIRED to define a threshold to determine whether the 577 measured congestion is considered excessive. 579 8. A CB is REQUIRED to define the triggering interval, defining the 580 period over which the trigger uses the collected measurements. 581 CBs need to trigger over a sufficiently long period to avoid 582 additionally penalizing flows with a long path RTT (e.g., many 583 path RTTs). 585 9. A CB MUST be robust to multiple congestion events. This usually 586 will define a number of measured persistent congestion events 587 per triggering period. For example, a CB MAY combine the 588 results of several measurement periods to determine if the CB is 589 triggered (e.g., it is triggered when persistent excessive 590 congestion is detected in 3 of the measurements within the 591 triggering interval). 593 10. The normal reaction to a trigger SHOULD disable all traffic that 594 contributed to congestion (otherwise, see requirements 11,12). 596 11. The reaction MUST be much more severe than that of a Congestion 597 Control algorithm (such as TCP's congestion control [RFC5681] or 598 TCP-Friendly Rate Control, TFRC [RFC5348]), because the CB 599 reacts to more persistent congestion and operates over longer 600 timescales (i.e., the overload condition will have persisted for 601 a longer time before the CB is triggered). 603 12. A reaction that results in a reduction SHOULD result in reducing 604 the traffic by at least an order of magnitude. A response that 605 achieves the reduction by terminating flows, rather than 606 randomly dropping packets, will often be more desirable to users 607 of the service. A CB that reduces the rate of a flow, MUST 608 continue to monitor the level of congestion and MUST further 609 react to reduce the rate if the CB is again triggered. 611 13. The reaction to a triggered CB MUST continue for a period that 612 is at least the triggering interval. Operator intervention will 613 usually be required to restore a flow. If an automated response 614 is needed to reset the trigger, then this needs to not be 615 immediate. The design of an automated reset mechanism needs to 616 be sufficiently conservative that it does not adversely interact 617 with other mechanisms (including other CB algorithms that 618 control traffic over a common path). It SHOULD NOT perform an 619 automated reset when there is evidence of continued congestion. 621 14. A CB trigger SHOULD be regarded as an abnormal network event. 622 As such, this event SHOULD be logged. The measurements that 623 lead to triggering of the CB SHOULD also be logged. 625 15. The control communication needs to carry measurements 626 (requirement 1) and, in some uses, also needs to transmit 627 trigger messages to the ingress. This control communication may 628 be in-band or out-of-band. The use of in-band communication is 629 RECOMMENDED when either design would be possible. The preferred 630 CB design is one that triggers when it fails to receive 631 measurement reports that indicate an absence of congestion, in 632 contrast to relying on the successful transmission of a 633 "congested" signal back to the sender. (The feedback signal 634 could itself be lost under congestion). 636 in-Band: An in-band control method SHOULD assume that loss of 637 control messages is an indication of potential congestion on 638 the path, and repeated loss ought to cause the CB to be 639 triggered. This design has the advantage that it provides 640 fate-sharing of the traffic flow(s) and the control 641 communications. This fate-sharing property is weaker when 642 some or all of the measured traffic is sent using a path that 643 differs from the path taken by the control traffic (e.g., 644 where traffic and control messages follow a different path 645 due to use of equal-cost multipath routing, traffic 646 engineering, or tunnels for specific types of traffic). 648 Out-of-Band: An out-of-band control method SHOULD NOT trigger 649 CB reaction when there is loss of control messages (e.g., a 650 loss of measurements). This avoids failure amplification/ 651 propagation when the measurement and data paths fail 652 independently. A failure of an out-of-band communication 653 path SHOULD be regarded as abnormal network event and be 654 handled as appropriate for the network, e.g., this event 655 SHOULD be logged, and additional network operator action 656 might be appropriate, depending on the network and the 657 traffic involved. 659 16. The control communication MUST be designed to be robust to 660 packet loss. A control message can be lost if there is a 661 failure of the communication path used for the control messages, 662 loss is likely to also be experienced during congestion/ 663 overload. This does not imply that it is desirable to provide 664 reliable delivery (e.g., over TCP), since this can incur 665 additional delay in responding to congestion. Appropriate 666 mechanisms could be to duplicate control messages to provide 667 increased robustness to loss, or/and to regard a lack of control 668 traffic as an indication that excessive congestion could be 669 being experienced [ID-ietf-tsvwg-RFC5405.bis]. If control 670 messages traffic are sent over a shared path, it is RECOMMENDED 671 that this control traffic is prioritized to reduce the 672 probability of loss under congestion. Control traffic also 673 needs to be considered when provisioning a network that uses a 674 Circuit Breaker. 676 17. There are security requirements for the control communication 677 between endpoints and/or network devices (Section 7). The 678 authenticity of the source and integrity of the control messages 679 (measurements and triggers) MUST be protected from off-path 680 attacks. When there is a risk of on-path attack, a 681 cryptographic authentication mechanism for all control/ 682 measurement messages is RECOMMENDED. 684 5. Examples of Circuit Breakers 686 There are multiple types of Circuit Breaker that could be defined for 687 use in different deployment cases. There could be cases where a flow 688 become controlled by multiple Circuit Breakers (e.g., when the 689 traffic of an end-to-end flow is carried in a tunnel within the 690 network). This section provides examples of different types of 691 Circuit Breaker: 693 5.1. A Fast-Trip Circuit Breaker 695 [RFC2309] discusses the dangers of congestion-unresponsive flows and 696 states that "all UDP-based streaming applications should incorporate 697 effective congestion avoidance mechanisms". Some applications do not 698 use a full-featured transport (TCP, SCTP, DCCP). These applications 699 (e.g., using UDP and its UDP-Lite variant) need to provide 700 appropriate congestion avoidance. Guidance for applications that do 701 not use congestion-controlled transports is provided in 702 [ID-ietf-tsvwg-RFC5405.bis]. Such mechanisms can be designed to 703 react on much shorter timescales than a Circuit Breaker, that only 704 observes a traffic envelope. Congestion control methods can also 705 interact with an application to more effectively control its sending 706 rate. 708 A fast-trip Circuit Breaker is the most responsive form of Circuit 709 Breaker. It has a response time that is only slightly larger than 710 that of the traffic that it controls. It is suited to traffic with 711 well-understood characteristics (and could include one or more 712 trigger functions specifically tailored the type of traffic for which 713 it is designed). It is not suited to arbitrary network traffic and 714 could be unsuitable for traffic aggregates, since it could 715 prematurely trigger (e.g., when the combined traffic from multiple 716 congestion-controlled flows leads to short-term overload). 718 Although the mechanisms can be implemented in RTP-aware network 719 devices, these mechanisms are also suitable for implementation in 720 endpoints (e.g., as a part of the transport system) where they can 721 also compliment end-to-end congestion control methods. A shorter 722 response time enables these mechanisms to triggers before other forms 723 of Circuit Breaker (e.g., Circuit Breakers operating on traffic 724 aggregates at a point along the network path). 726 5.1.1. A Fast-Trip Circuit Breaker for RTP 728 A set of fast-trip Circuit Breaker methods have been specified for 729 use together by a Real-time Transport Protocol (RTP) flow using the 730 RTP/AVP Profile [RTP-CB]. It is expected that, in the absence of 731 severe congestion, all RTP applications running on best-effort IP 732 networks will be able to run without triggering these Circuit 733 Breakers. A fast-trip RTP Circuit Breaker is therefore implemented 734 as a fail-safe that when triggered will terminate RTP traffic. 736 The sending endpoint monitors reception of in-band RTP Control 737 Protocol (RTCP) reception report blocks, as contained in SR or RR 738 packets, that convey reception quality feedback information. This is 739 used to measure (congestion) loss, possibly in combination with ECN 740 [RFC6679]. 742 The Circuit Breaker action (shutdown of the flow) is triggered when 743 any of the following trigger conditions are true: 745 1. An RTP Circuit Breaker triggers on reported lack of progress. 747 2. An RTP Circuit Breaker triggers when no receiver reports messages 748 are received. 750 3. An RTP Circuit Breaker triggers when the long-term RTP throughput 751 (over many RTTs) exceeds a hard upper limit determined by a 752 method that resembles TCP-Friendly Rate Control (TFRC). 754 4. An RTP Circuit Breaker includes the notion of Media Usability. 755 This Circuit Breaker is triggered when the quality of the 756 transported media falls below some required minimum acceptable 757 quality. 759 5.2. A Slow-trip Circuit Breaker 761 A slow-trip Circuit Breaker could be implemented in an endpoint or 762 network device. This type of Circuit Breaker is much slower at 763 responding to congestion than a fast-trip Circuit Breaker. This is 764 expected to be more common. 766 One example where a slow-trip Circuit Breaker is needed is where 767 flows or traffic-aggregates use a tunnel or encapsulation and the 768 flows within the tunnel do not all support TCP-style congestion 769 control (e.g., TCP, SCTP, TFRC), see [ID-ietf-tsvwg-RFC5405.bis] 770 section 3.1.3. A use case is where tunnels are deployed in the 771 general Internet (rather than "controlled environments" within an 772 Internet service provider or enterprise network), especially when the 773 tunnel could need to cross a customer access router. 775 5.3. A Managed Circuit Breaker 777 A managed Circuit Breaker is implemented in the signalling protocol 778 or management plane that relates to the traffic aggregate being 779 controlled. This type of Circuit Breaker is typically applicable 780 when the deployment is within a "controlled environment". 782 A Circuit Breaker requires more than the ability to determine that a 783 network path is forwarding data, or to measure the rate of a path - 784 which are often normal network operational functions. There is an 785 additional need to determine a metric for congestion on the path and 786 to trigger a reaction when a threshold is crossed that indicates 787 persistent excessive congestion. 789 The control messages can use either in-band or out-of-band 790 communications. 792 5.3.1. A Managed Circuit Breaker for SAToP Pseudo-Wires 794 [RFC4553], SAToP Pseudo-Wires (PWE3), section 8 describes an example 795 of a managed Circuit Breaker for isochronous flows. 797 If such flows were to run over a pre-provisioned (e.g., Multi- 798 Protocol Label Switching, MPLS) infrastructure, then it could be 799 expected that the Pseudowire (PW) would not experience congestion, 800 because a flow is not expected to either increase (or decrease) their 801 rate. If, instead, PW traffic is multiplexed with other traffic over 802 the general Internet, it could experience congestion. [RFC4553] 803 states: "If SAToP PWs run over a PSN providing best-effort service, 804 they SHOULD monitor packet loss in order to detect "severe 805 congestion". The currently recommended measurement period is 1 806 second, and the trigger operates when there are more than three 807 measured Severely Errored Seconds (SES) within a period. If such a 808 condition is detected, a SAToP PW ought to shut down bidirectionally 809 for some period of time...". 811 The concept was that when the packet loss ratio (congestion) level 812 increased above a threshold, the PW was by default disabled. This 813 use case considered fixed-rate transmission, where the PW had no 814 reasonable way to shed load. 816 The trigger needs to be set at the rate that the PW was likely to 817 experience a serious problem, possibly making the service non- 818 compliant. At this point, triggering the Circuit Breaker would 819 remove the traffic preventing undue impact on congestion-responsive 820 traffic (e.g., TCP). Part of the rationale, was that high loss 821 ratios typically indicated that something was "broken" and ought to 822 have already resulted in operator intervention, and therefore need to 823 trigger this intervention. 825 An operator-based response to triggering of a Circuit Breaker 826 provides an opportunity for other action to restore the service 827 quality, e.g., by shedding other loads or assigning additional 828 capacity, or to consciously avoid reacting to the trigger while 829 engineering a solution to the problem. This could require the 830 trigger function to send a control message to a third location (e.g., 831 a network operations centre, NOC) that is responsible for operation 832 of the tunnel ingress, rather than the tunnel ingress itself. 834 5.3.2. A Managed Circuit Breaker for Pseudowires (PWs) 836 Pseudowires (PWs) [RFC3985] have become a common mechanism for 837 tunneling traffic, and could compete for network resources both with 838 other PWs and with non-PW traffic, such as TCP/IP flows. 840 [ID-ietf-pals-congcons] discusses congestion conditions that can 841 arise when PWs compete with elastic (i.e., congestion responsive) 842 network traffic (e.g, TCP traffic). Elastic PWs carrying IP traffic 843 (see [RFC4488]) do not raise major concerns because all of the 844 traffic involved responds, reducing the transmission rate when 845 network congestion is detected. 847 In contrast, inelastic PWs (e.g., a fixed bandwidth Time Division 848 Multiplex, TDM) [RFC4553] [RFC5086] [RFC5087]) have the potential to 849 harm congestion responsive traffic or to contribute to excessive 850 congestion because inelastic PWs do not adjust their transmission 851 rate in response to congestion. [ID-ietf-pals-congcons] analyses TDM 852 PWs, with an initial conclusion that a TDM PW operating with a degree 853 of loss that could result in congestion-related problems is also 854 operating with a degree of loss that results in an unacceptable TDM 855 service. For that reason, the document suggests that a managed 856 Circuit Breaker that shuts down a PW when it persistently fails to 857 deliver acceptable TDM service is a useful means for addressing these 858 congestion concerns. (See Appendix A of [ID-ietf-pals-congcons] for 859 further discussion.) 861 6. Examples where circuit breakers may not be needed. 863 A Circuit Breaker is not required for a single congestion-controlled 864 flow using TCP, SCTP, TFRC, etc. In these cases, the congestion 865 control methods are already designed to prevent persistent excessive 866 congestion. 868 6.1. CBs over pre-provisioned Capacity 870 One common question is whether a Circuit Breaker is needed when a 871 tunnel is deployed in a private network with pre-provisioned 872 capacity. 874 In this case, compliant traffic that does not exceed the provisioned 875 capacity ought not to result in persistent congestion. A Circuit 876 Breaker will hence only be triggered when there is non-compliant 877 traffic. It could be argued that this event ought never to happen - 878 but it could also be argued that the Circuit Breaker equally ought 879 never to be triggered. If a Circuit Breaker were to be implemented, 880 it will provide an appropriate response if persistent congestion 881 occurs in an operational network. 883 Implementing a Circuit Breaker will not reduce the performance of the 884 flows, but in the event that persistent excessive congestion occurs 885 it protects network traffic that shares network capacity with these 886 flows. It also protects network traffic from a failure when Circuit 887 Breaker traffic is (re)routed to cause additional network load on a 888 non-pre-provisioned path. 890 6.2. CBs with tunnels carrying Congestion-Controlled Traffic 892 IP-based traffic is generally assumed to be congestion-controlled, 893 i.e., it is assumed that the transport protocols generating IP-based 894 traffic at the sender already employ mechanisms that are sufficient 895 to address congestion on the path. A question therefore arises when 896 people deploy a tunnel that is thought to only carry an aggregate of 897 TCP traffic (or traffic using some other congestion control method): 898 Is there advantage in this case in using a Circuit Breaker? 899 TCP (and SCTP) traffic in a tunnel is expected to reduce the 900 transmission rate when network congestion is detected. Other 901 transports (e.g, using UDP) can employ mechanisms that are sufficient 902 to address congestion on the path [ID-ietf-tsvwg-RFC5405.bis]. 903 However, even if the individual flows sharing a tunnel each implement 904 a congestion control mechanism, and individually reduce their 905 transmission rate when network congestion is detected, the overall 906 traffic resulting from the aggregate of the flows does not 907 necessarily avoid persistent congestion. For instance, most 908 congestion control mechanisms require long-lived flows to react to 909 reduce the rate of a flow. An aggregate of many short flows could 910 result in many flows terminating before they experience congestion. 911 It is also often impossible for a tunnel service provider to know 912 that the tunnel only contains congestion-controlled traffic (e.g., 913 Inspecting packet headers might not be possible). Some IP-based 914 applications might not implement adequate mechanisms to address 915 congestion. The important thing to note is that if the aggregate of 916 the traffic does not result in persistent excessive congestion 917 (impacting other flows), then the Circuit Breaker will not trigger. 918 This is the expected case in this context - so implementing a Circuit 919 Breaker ought not to reduce performance of the tunnel, but in the 920 event that persistent excessive congestion occurs the Circuit Breaker 921 protects other network traffic that shares capacity with the tunnel 922 traffic. 924 6.3. CBs with Uni-directional Traffic and no Control Path 926 A one-way forwarding path could have no associated communication path 927 for sending control messages, and therefore cannot be controlled 928 using a Circuit Breaker (compare with Section 3.2.3). 930 A one-way service could be provided using a path with dedicated pre- 931 provisioned capacity that is not shared with other elastic Internet 932 flows (i.e., flows that vary their rate). A forwarding path could 933 also be shared with other flows. One way to mitigate the impact of 934 traffic on the other flows is to manage the traffic envelope by using 935 ingress policing. Supporting this type of traffic in the general 936 Internet requires operator monitoring to detect and respond to 937 persistent excessive congestion. 939 7. Security Considerations 941 All Circuit Breaker mechanisms rely upon coordination between the 942 ingress and egress meters and communication with the trigger 943 function. This is usually achieved by passing network control 944 information (or protocol messages) across the network. Timely 945 operation of a Circuit Breaker depends on the choice of measurement 946 period. If the receiver has an interval that is overly long, then 947 the responsiveness of the Circuit Breaker decreases. This impacts 948 the ability of the Circuit Breaker to detect and react to congestion. 949 If the interval is too short the Circuit Breaker could trigger 950 prematurely resulting in insufficient time for other mechanisms to 951 act, potentially resulting in unnecessary disruption to the service. 953 A Circuit Breaker could potentially be exploited by an attacker to 954 mount a Denial of Service (DoS) attack against the traffic being 955 controlled by the Circuit Breaker. Mechanisms therefore need to be 956 implemented to prevent attacks on the network control information 957 that would result in DoS. 959 The authenticity of the source and integrity of the control messages 960 (measurements and triggers) MUST be protected from off-path attacks. 961 Without protection, it could be trivial for an attacker to inject 962 fake or modified control/measurement messages (e.g., indicating high 963 packet loss rates) causing a Circuit Breaker to trigger and to 964 therefore mount a DoS attack that disrupts a flow. 966 Simple protection can be provided by using a randomized source port, 967 or equivalent field in the packet header (such as the RTP SSRC value 968 and the RTP sequence number) expected not to be known to an off-path 969 attacker. Stronger protection can be achieved using a secure 970 authentication protocol to mitigate this concern. 972 An attack on the control messages is relatively easy for an attacker 973 on the control path when the messages are neither encrypted nor 974 authenticated. Use of a cryptographic authentication mechanism for 975 all control/measurement messages is RECOMMENDED to mitigate this 976 concern, and would also provide protection from off-path attacks. 977 There is a design trade-off between the cost of introducing 978 cryptographic security for control messages and the desire to protect 979 control communication. For some deployment scenarios the value of 980 additional protection from DoS attack will therefore lead to a 981 requirement to authenticate all control messages. 983 Transmission of network control messages consumes network capacity. 984 This control traffic needs to be considered in the design of a 985 Circuit Breaker and could potentially add to network congestion. If 986 this traffic is sent over a shared path, it is RECOMMENDED that this 987 control traffic is prioritized to reduce the probability of loss 988 under congestion. Control traffic also needs to be considered when 989 provisioning a network that uses a Circuit Breaker. 991 The Circuit Breaker MUST be designed to be robust to packet loss that 992 can also be experienced during congestion/overload. Loss of control 993 messages could be a side-effect of a congested network, but also 994 could arise from other causes Section 4. 996 The security implications depend on the design of the mechanisms, the 997 type of traffic being controlled and the intended deployment 998 scenario. Each design of a Circuit Breaker MUST therefore evaluate 999 whether the particular Circuit Breaker mechanism has new security 1000 implications. 1002 8. IANA Considerations 1004 This document makes no request from IANA. 1006 9. Acknowledgments 1008 There are many people who have discussed and described the issues 1009 that have motivated this document. Contributions and comments 1010 included: Lars Eggert, Colin Perkins, David Black, Matt Mathis, 1011 Andrew McGregor, Bob Briscoe and Eliot Lear. This work was part- 1012 funded by the European Community under its Seventh Framework 1013 Programme through the Reducing Internet Transport Latency (RITE) 1014 project (ICT-317700). 1016 10. Revision Notes 1018 XXX RFC-Editor: Please remove this section prior to publication XXX 1020 Draft 00 1022 This was the first revision. Help and comments are greatly 1023 appreciated. 1025 Draft 01 1027 Contained clarifications and changes in response to received 1028 comments, plus addition of diagram and definitions. Comments are 1029 welcome. 1031 WG Draft 00 1033 Approved as a WG work item on 28th Aug 2014. 1035 WG Draft 01 1037 Incorporates feedback after Dallas IETF TSVWG meeting. This version 1038 is thought ready for WGLC comments. Definitions of abbreviations. 1040 WG Draft 02 1042 Minor fixes for typos. Rewritten security considerations section. 1044 WG Draft 03 1046 Updates following WGLC comments (see TSV mailing list). Comments 1047 from C Perkins; D Black and off-list feedback. 1049 A clear recommendation of intended scope. 1051 Changes include: Improvement of language on timescales and minimum 1052 measurement period; clearer articulation of endpoint and multicast 1053 examples - with new diagrams; separation of the controlled network 1054 case; updated text on position of trigger function; corrections to 1055 RTP-CB text; clarification of loss v ECN metrics; checks against 1056 submission checklist 9use of keywords, added meters to diagrams). 1058 WG Draft 04 1060 Added section on PW CB for TDM - a newly adopted draft (D. Black). 1062 WG Draft 05 1064 Added clarifications requested during AD review. 1066 WG Draft 06 1068 Fixed some remaining typos. 1070 Update following detailed review by Bob Briscoe, and comments by D. 1071 Black. 1073 WG Draft 07 1075 Additional update following review by Bob Briscoe. 1077 WG Draft 08 1079 Updated text on the response to lack of meter measurements with 1080 managed circuit breakers. Additional comments from Eliot Lear (APPs 1081 area). 1083 WG Draft 09 1085 Updated text on applications from Eliot Lear. Additional feedback 1086 from Bob Briscoe. 1088 WG Draft 10 1089 Updated text following comments by D Black including a rewritten ECN 1090 requirements bullet with of a reference to a tunnel measurement 1091 method in [ID-ietf-tsvwg-tunnel-congestion-feedback]. 1093 WG Draft 11 1095 Minor corrections after second WGLC. 1097 WG Draft 12 1099 Update following Gen-ART, RTG, and OPS review comments. 1101 WG Draft 13 1103 Fixed a typo. 1105 WG Draft 14 1107 Update after IESG discussion, including: 1109 Reworded introduction. Added definition of ECN. 1111 Requirement 1113 Addressed inconsistency between requirements for control messages. - 1114 Removed a "MUST" - following WG feedback on a anearlier version of 1115 the draft that "SHOULD" is more appropriate. 1117 Addressed comment about grouping requirements to help show they were 1118 inter-related. This reordered some requirements. 1120 Reworded the security considerations. 1122 Corrections to wording to improve clarity. 1124 WG Draft 15 (incorporating pending corrections) 1126 Corrected /applications might be implement/applications might not 1127 implement/ 1129 Corrected /Inspecting packet headers could/Inspecting packet headers 1130 might/ 1132 Removed Requirement 9, now duplicated (and renumbered remaining 1133 items). 1135 Added "(See Appendix A of [ID-ietf-pals-congcons] for further 1136 discussion.)" to end of 5.3.2 - missed comment. 1138 Simplified a sentence in section 6.1, without intended change of 1139 meaning. 1141 Added a linking sentence to the second para of Section 6.3. 1143 11. References 1145 11.1. Normative References 1147 [ID-ietf-tsvwg-RFC5405.bis] 1148 Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage 1149 Guidelines (Work-in-Progress)", 2015. 1151 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1152 Requirement Levels", BCP 14, RFC 2119, 1153 DOI 10.17487/RFC2119, March 1997, 1154 . 1156 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1157 of Explicit Congestion Notification (ECN) to IP", 1158 RFC 3168, DOI 10.17487/RFC3168, September 2001, 1159 . 1161 11.2. Informative References 1163 [ID-ietf-pals-congcons] 1164 Stein, YJ., Black, D., and B. Briscoe, "Pseudowire 1165 Congestion Considerations (Work-in-Progress)", 2015. 1167 [ID-ietf-tsvwg-tunnel-congestion-feedback] 1168 Wei, X., Zhu, L., and L. Dend, "Tunnel Congestion Feedback 1169 (Work-in-Progress)", 2015. 1171 [Jacobsen88] 1172 European Telecommunication Standards, Institute (ETSI), 1173 "Congestion Avoidance and Control", SIGCOMM Symposium 1174 proceedings on Communications architectures and 1175 protocols", August 1998. 1177 [RFC1112] Deering, S., "Host extensions for IP multicasting", STD 5, 1178 RFC 1112, DOI 10.17487/RFC1112, August 1989, 1179 . 1181 [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, 1182 S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., 1183 Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, 1184 S., Wroclawski, J., and L. Zhang, "Recommendations on 1185 Queue Management and Congestion Avoidance in the 1186 Internet", RFC 2309, DOI 10.17487/RFC2309, April 1998, 1187 . 1189 [RFC2914] Floyd, S., "Congestion Control Principles", BCP 41, 1190 RFC 2914, DOI 10.17487/RFC2914, September 2000, 1191 . 1193 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 1194 Edge-to-Edge (PWE3) Architecture", RFC 3985, 1195 DOI 10.17487/RFC3985, March 2005, 1196 . 1198 [RFC4488] Levin, O., "Suppression of Session Initiation Protocol 1199 (SIP) REFER Method Implicit Subscription", RFC 4488, 1200 DOI 10.17487/RFC4488, May 2006, 1201 . 1203 [RFC4553] Vainshtein, A., Ed. and YJ. Stein, Ed., "Structure- 1204 Agnostic Time Division Multiplexing (TDM) over Packet 1205 (SAToP)", RFC 4553, DOI 10.17487/RFC4553, June 2006, 1206 . 1208 [RFC4601] Fenner, B., Handley, M., Holbrook, H., and I. Kouvelas, 1209 "Protocol Independent Multicast - Sparse Mode (PIM-SM): 1210 Protocol Specification (Revised)", RFC 4601, 1211 DOI 10.17487/RFC4601, August 2006, 1212 . 1214 [RFC5086] Vainshtein, A., Ed., Sasson, I., Metz, E., Frost, T., and 1215 P. Pate, "Structure-Aware Time Division Multiplexed (TDM) 1216 Circuit Emulation Service over Packet Switched Network 1217 (CESoPSN)", RFC 5086, DOI 10.17487/RFC5086, December 2007, 1218 . 1220 [RFC5087] Stein, Y(J)., Shashoua, R., Insler, R., and M. Anavi, 1221 "Time Division Multiplexing over IP (TDMoIP)", RFC 5087, 1222 DOI 10.17487/RFC5087, December 2007, 1223 . 1225 [RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP 1226 Friendly Rate Control (TFRC): Protocol Specification", 1227 RFC 5348, DOI 10.17487/RFC5348, September 2008, 1228 . 1230 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 1231 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 1232 . 1234 [RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P., 1235 and K. Carlberg, "Explicit Congestion Notification (ECN) 1236 for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August 1237 2012, . 1239 [RTP-CB] Perkins, C. and V. Singh, "Multimedia Congestion Control: 1240 Circuit Breakers for Unicast RTP Sessions (draft-ietf- 1241 avtcore-rtp-circuit-breakers)", February 2014. 1243 Author's Address 1245 Godred Fairhurst 1246 University of Aberdeen 1247 School of Engineering 1248 Fraser Noble Building 1249 Aberdeen, Scotland AB24 3UE 1250 UK 1252 Email: gorry@erg.abdn.ac.uk 1253 URI: http://www.erg.abdn.ac.uk