idnits 2.17.1 draft-ietf-pwe3-congestion-frmwk-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** The document seems to lack a License Notice according IETF Trust Provisions of 28 Dec 2009, Section 6.b.i or Provisions of 12 Sep 2009 Section 6.b -- however, there's a paragraph with a matching beginning. Boilerplate error? (You're using the IETF Trust Provisions' Section 6.b License Notice from 12 Feb 2009 rather than one of the newer Notices. See https://trustee.ietf.org/license-info/.) Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (June 17, 2009) is 5426 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Obsolete informational reference (is this intentional?): RFC 2001 (Obsoleted by RFC 2581) -- Obsolete informational reference (is this intentional?): RFC 2581 (Obsoleted by RFC 5681) -- Obsolete informational reference (is this intentional?): RFC 3448 (Obsoleted by RFC 5348) -- Obsolete informational reference (is this intentional?): RFC 4447 (Obsoleted by RFC 8077) -- Obsolete informational reference (is this intentional?): RFC 4960 (Obsoleted by RFC 9260) Summary: 1 error (**), 0 flaws (~~), 1 warning (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group S. Bryant 3 Internet-Draft B. Davie 4 Intended status: Informational L. Martini 5 Expires: December 19, 2009 E. Rosen 6 Cisco Systems, Inc. 7 June 17, 2009 9 Pseudowire Congestion Control Framework 10 draft-ietf-pwe3-congestion-frmwk-02.txt 12 Status of this Memo 14 This Internet-Draft is submitted to IETF in full conformance with the 15 provisions of BCP 78 and BCP 79. 17 Internet-Drafts are working documents of the Internet Engineering 18 Task Force (IETF), its areas, and its working groups. Note that 19 other groups may also distribute working documents as Internet- 20 Drafts. 22 Internet-Drafts are draft documents valid for a maximum of six months 23 and may be updated, replaced, or obsoleted by other documents at any 24 time. It is inappropriate to use Internet-Drafts as reference 25 material or to cite them other than as "work in progress." 27 The list of current Internet-Drafts can be accessed at 28 http://www.ietf.org/ietf/1id-abstracts.txt. 30 The list of Internet-Draft Shadow Directories can be accessed at 31 http://www.ietf.org/shadow.html. 33 This Internet-Draft will expire on December 19, 2009. 35 Copyright Notice 37 Copyright (c) 2009 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents in effect on the date of 42 publication of this document (http://trustee.ietf.org/license-info). 43 Please review these documents carefully, as they describe your rights 44 and restrictions with respect to this document. 46 Abstract 48 Pseudowires are sometimes used to carry non-TCP data flows. In these 49 circumstances the service payload will not react to network 50 congestion by reducing its offered load. Pseudowires should 51 therefore reduces their network bandwidth demands in the face of 52 significant packet loss, including if necessary completely ceasing 53 transmission. Since it is difficult to determine a priori the number 54 of equivalent TCP flow that a pseudowire represents, a suitably 55 "fair" rate of back-off cannot be pre-determined. This document 56 describes pseudowire congestion problem and provides guidance on the 57 development suitable solutions. 59 Requirements Language 61 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 62 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 63 document are to be interpreted as described in RFC 2119 [RFC2119]. 65 Table of Contents 67 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 68 2. PW Congestion Context . . . . . . . . . . . . . . . . . . . . 4 69 2.1. Congestion in IP Networks . . . . . . . . . . . . . . . . 4 70 2.2. A Short Tutorial on TCP Congestion Control . . . . . . . . 5 71 2.3. Alternative Approaches to Congestion Control . . . . . . . 6 72 2.4. Pseudowires and TCP . . . . . . . . . . . . . . . . . . . 7 73 2.5. Arguments Against PW Congestion as a Practical Problem . . 8 74 2.6. Goals and non-goals of this draft . . . . . . . . . . . . 9 75 3. Challenges for PW Congestion Management . . . . . . . . . . . 10 76 3.1. Scale . . . . . . . . . . . . . . . . . . . . . . . . . . 10 77 3.2. Interaction among control loops . . . . . . . . . . . . . 10 78 3.3. Determining the Appropriate Rate . . . . . . . . . . . . . 11 79 3.4. Constant Bit Rate PWs . . . . . . . . . . . . . . . . . . 11 80 3.5. Valuable and Vulnerable PWs . . . . . . . . . . . . . . . 12 81 4. Congestion Control Mechanisms . . . . . . . . . . . . . . . . 12 82 5. Detecting Congestion . . . . . . . . . . . . . . . . . . . . . 13 83 5.1. Using Sequence Numbers to Detect Congestion . . . . . . . 14 84 5.2. Using VCCV to Detect Congestion . . . . . . . . . . . . . 15 85 5.3. Explicit Congestion Notification . . . . . . . . . . . . . 16 86 6. Feedback from Receiver to Transmitter . . . . . . . . . . . . 16 87 6.1. Control Plane Feedback . . . . . . . . . . . . . . . . . . 17 88 6.2. Using Reverse Data Packets for Feedback . . . . . . . . . 18 89 6.3. Reverse VCCV Traffic . . . . . . . . . . . . . . . . . . . 18 90 7. Responding to Congestion . . . . . . . . . . . . . . . . . . . 18 91 7.1. Interaction with TCP . . . . . . . . . . . . . . . . . . . 19 92 8. Rate Control per Tunnel vs. per PW . . . . . . . . . . . . . . 20 93 9. Constant Bit Rate Services . . . . . . . . . . . . . . . . . . 21 94 10. Managed vs Unmanaged Deployment . . . . . . . . . . . . . . . 21 95 11. Related Work: Pre-Congestion Notification . . . . . . . . . . 22 96 12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 97 13. Security Considerations . . . . . . . . . . . . . . . . . . . 23 98 14. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 23 99 15. Informative References . . . . . . . . . . . . . . . . . . . . 23 100 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 25 102 1. Introduction 104 Pseudowires are sometimes used to emulate services network that carry 105 non-TCP data flows. In these circumstances the service payload will 106 not react to network congestion by reducing its offered load. In 107 order to protect the newtork, pseudowires (PW) should therefore 108 reduces their network bandwidth demands in the face of significant 109 packet loss, including if necessary completely ceasing transmission. 111 Pseudowires are commonly deployed as part of the managed 112 infrastructure deployed by a service provider. It is likely in these 113 cases that the PW will be used to carry many TCP equivalent flows. 114 In these environments the operator needs to make a judgement call on 115 a fair estimate of the number of equivalent flows that the PW 116 represents. The counterpoint to a well managed network is a plug and 117 play network, of the type typically found in a consumer environment. 118 Whilst PWs have been designed to provide an infrastructure tool to 119 the service provider, we cannot be sure that they will not find some 120 consumer or similar plug and play application. In this environment a 121 conservative is needed, with PW congestion avoidance enabled by 122 default, and the throughput set by default to the equivalent of a 123 single TCP flow. 125 This framework first considers the context in which PWs operate, and 126 hence the context in which PW congestion control needs to operate. 127 It then explores the issues of detection, feedback and load control 128 in PW context, and the special issues that apply to constant bit rate 129 services. Finally it considers the applicability of congestion 130 control in various network environments. 132 This document defines the framework to be used in designing a 133 congestion control mechanism for PWs. It does not prescribe a 134 solution for an particular PW type, and it is possible that more than 135 one mechanism may be required, depending on the PW type and the 136 environment in which it is to be deployed. 138 2. PW Congestion Context 140 2.1. Congestion in IP Networks 142 Congestion in an IP or MPLS enabled IP network occurs when the amount 143 of traffic that needs to use a particular network resource exceeds 144 the capacity of that resource. This results first in long queues 145 within the network, and then in packet loss. If the amount of 146 traffic is not then reduced, the packet loss rate will climb, 147 potentially until it reaches 100%. If the situation is left unabated 148 the network enters a state where it is no longer able to sustain any 149 productive information flows and the result is "congestive collapse" 151 To prevent this sort of "congestive collapse", there must be 152 congestion control: a feedback loop by which the presence of 153 congestion anywhere on the path from the transmitter to the receiver 154 forces the transmitters to reduce the amount of traffic being sent. 155 As a connectionless protocol, IP has no way to push back directly on 156 the originator of the traffic. Procedures for (a) detecting 157 congestion, (b) providing the necessary feedback to the transmitters, 158 and (c) adjusting the transmission rates, are thus left the to higher 159 protocol layers such as TCP. 161 2.2. A Short Tutorial on TCP Congestion Control 163 TCP includes an elaborate congestion control mechanism that causes 164 the end systems to reduce their transmission rates when congestion 165 occurs. For those readers not intimately familiar with the details 166 of TCP congestion control, we give below a brief summary, greatly 167 simplified and not entirely accurate, of TCP's very complicated 168 feedback mechanism. The details of TCP congestion control can be 169 found in [RFC2581]. [RFC2001] is an earlier but more accessible 170 discussion. [RFC2914] articulates a number of general principles 171 governing congestion control in the Internet. 173 In TCP congestion control, a lost packet is considered to be an 174 indication of congestion. Roughly, TCP considers a given packet to 175 be lost if that packet is not acknowledged within a specified time, 176 or if three subsequent packets arrive at the receiver before the 177 given packet. The latter condition manifests itself at the 178 transmitter as the arrival of three duplicate acknowledgements in a 179 row. The algorithm by which TCP detects congestion is thus highly 180 dependent on the mechanisms used by TCP to ensure reliable and 181 sequential delivery. 183 Once a TCP transmitter becomes aware of congestion, it halves its 184 transmission rate. If congestion still occurs at the new rate, the 185 rate is halved again. When a rate is found at which congestion no 186 longer occurs, the rate is increased by one MSS ("Maximum Segment 187 Size") per RTT ("Round Trip Time"). The rate is increased each RTT 188 until congestion is encountered again, or until something else limits 189 it (e.g., the flow control window reached, or the application is 190 transmitting at its max desired rate, or at line rate). 192 This sort of mechanism is known as an "Additive Increase, 193 Multiplicative Decrease" (AIMD) mechanism. Congestion causes 194 relatively rapid decreases in the transmission rate, while the 195 absence of congestion causes relatively slow increases in the allowed 196 transmission rate. 198 2.3. Alternative Approaches to Congestion Control 200 [RFC2914] defines the notion of a "TCP-compatible flow": 202 "A TCP-compatible flow is responsive to congestion notification, and 203 in steady state uses no more bandwidth than a conformant TCP running 204 under comparable conditions (drop rate, RTT [round trip time], MTU 205 [maximum transmission unit], etc.)" 207 TCP-compatible flows respond to congestion in much the way TCP does, 208 so that they do not starve the TCP flows or otherwise obtain an 209 unfair advantage. [RFC2914] further points out: 211 "any form of congestion control that successfully avoids a high 212 sending rate in the presence of a high packet drop rate should be 213 sufficient to avoid congestion collapse from undelivered packets." 215 "This does not mean, however, that concerns about congestion collapse 216 and fairness with TCP necessitate that all best-effort traffic deploy 217 congestion control based on TCP's Additive-Increase Multiplicative- 218 Decrease (AIMD) algorithm of reducing the sending rate in half in 219 response to each packet drop." 221 "However, the list of TCP-compatible congestion control procedures is 222 not limited to AIMD with the same increase/ decrease parameters as 223 TCP. Other TCP-compatible congestion control procedures include 224 rate-based variants of AIMD; AIMD with different sets of increase/ 225 decrease parameters that give the same steady-state behavior; 226 equation-based congestion control where the sender adjusts its 227 sending rate in response to information about the long-term packet 228 drop rate ... and possibly other forms that we have not yet begun to 229 consider." 231 The AIMD procedures are not mandated for non-TCP traffic, and might 232 not be optimal for non-TCP PW traffic. Choosing a proper set of 233 procedures which are TCP-compatible while being optimized for a 234 particular type of traffic is no simple task. 236 [RFC3448], "TCP Friendly Rate Control (TFRC)" provides an 237 alternative: TFRC is designed to be reasonably fair when competing 238 for bandwidth with TCP flows, where a flow is "reasonably fair" if 239 its sending rate is generally within a factor of two of the sending 240 rate of a TCP flow under the same conditions. However, TFRC has a 241 much lower variation of throughput over time compared with TCP, which 242 makes it more suitable for applications such as telephony or 243 streaming media where a relatively smooth sending rate is of 244 importance." 245 "For its congestion control mechanism, TFRC directly uses a 246 throughput equation for the allowed sending rate as a function of the 247 loss event rate and round-trip time. In order to compete fairly with 248 TCP, TFRC uses the TCP throughput equation, which roughly describes 249 TCP's sending rate as a function of the loss event rate, round-trip 250 time, and packet size." 252 "Generally speaking, TFRC's congestion control mechanism works as 253 follows: 255 o The receiver measures the loss event rate and feeds this 256 information back to the sender. 258 o The sender also uses these feedback messages to measure the round- 259 trip time (RTT). 261 o The loss event rate and RTT are then fed into TFRC's throughput 262 equation, giving the acceptable transmit rate. 264 o The sender then adjusts its transmit rate to match the calculated 265 rate." 267 Note that the TFRC procedures require the transmitter to calculate a 268 throughput equation. For these procedures to be feasible as a means 269 of PW congestion control, they must be computationally efficient. 270 Section 8 of [RFC3448] describes an implementation technique that 271 appears to make it efficient to calculate the equation. 273 In the context of pseudowires, we note that TFRC aims to achieve 274 comparable throughput in the long term to a single TCP connection 275 experiencing similar loss and round trip time. As noted above, this 276 may not be an appropriate rate for a PW that is carrying many flows. 278 2.4. Pseudowires and TCP 280 Currently, traffic in IP and MPLS networks is predominantly TCP 281 traffic. Even the layer 2 tunnelled traffic (e.g., PPP frames 282 tunnelled through L2TP) is predominantly TCP traffic from the end- 283 users. If pseudowires (PWs) [RFC3985] were to be used only for 284 carrying TCP flows, there would be no need for any PW-specific 285 congestion mechanisms. The existing TCP congestion control 286 mechanisms would be all that is needed, since any loss of packets on 287 the PW would be detected as loss of packets on a TCP connection, and 288 the TCP flow control mechanisms would ensure a reduction of 289 transmission rate. 291 If a PW is carrying non-TCP traffic, then there is no feedback 292 mechanism to cause the end-systems to reduce their transmission rates 293 in response to congestion. When congestion occurs, any TCP traffic 294 that is sharing the congested resource with the non-TCP traffic will 295 be throttled, and the non-TCP traffic may "starve" the TCP traffic. 296 If there is enough non-TCP traffic to congest the network all by 297 itself, there is nothing to prevent congestive collapse. 299 The non-TCP traffic in a PW can belong to any higher layer 300 whatsoever, and there is no way to ensure that a TCP-like congestion 301 control mechanisms will be present to regulate the traffic rate. 302 Hence it appears that there is a need for an edge-to-edge (i.e., PE- 303 to-PE) feedback mechanism which forces a transmitting PE to reduce 304 its transmission rate in the face of network congestion. 306 As TCP uses window-based flow control, controlling the rate is really 307 a matter of limiting the amount of traffic which can be "in flight" 308 (i.e., transmitted but not yet acknowledged) at any one time. Where 309 a non-windowed protocol is used for transmitting data on a PW a 310 different technique is obviously required to control the transmission 311 rate. 313 2.5. Arguments Against PW Congestion as a Practical Problem 315 One may argue that congestion due to non-TCP PW traffic is only a 316 theoretical problem and that no congestion control mechanism is 317 needed. For example the following cases have been put forward: 319 o "99.9% of all the traffic in PWs is really IP traffic" 321 If this is the case, then the traffic is either TCP traffic, which 322 is already congestion-controlled, or "other" IP traffic. While 323 the congestion control issue may exist for the "other" IP traffic, 324 this is a general issue and hence is outside the scope of the 325 pseudowire design. 327 Unfortunately, we cannot be sure that this is the case. It may 328 well be the case for the PW offerings of certain providers, but 329 perhaps not for others. It does appear that many providers want 330 to be able to use PWs for transporting "legacy traffic" of various 331 non-IP protocols. Constant bit-rate services are an example of 332 this, and raise particular issues for congestion control 333 (discussed below). 335 o "PW traffic usually stays within one SP's network, and an SP 336 always engineers its network carefully enough so that congestion 337 is an impossibility" 339 Perhaps this will be true of "most" PWs, but inter-provider PWs 340 are certainly expected to have a significant presence. 342 Even within a single provider's network, the provider might 343 consider whether he is so confident of his network engineering 344 that he does not need a feedback loop reducing the transmission 345 rate in response to congestion. 347 There is also the issue of keeping the network running (i.e., out 348 of congestive collapse) after an unexpected reduction of capacity. 350 Finally the PW may be deployed on the Internet rather than in an 351 SP environment. 353 o "If one provider accepts PW traffic from another, policing will be 354 done at the entry point to the second provider's network, so that 355 the second provider is sure that the first provider is not sending 356 too much traffic. This policing, together with the second 357 provider's careful network engineering, makes congestion an 358 impossibility" 360 This could be the case given carefully controlled bilateral 361 peering arrangements. Note though that if the second provider is 362 merely providing transit services for a PW whose endpoints are in 363 other providers, it may be difficult for the transit provider to 364 tell which traffic is the PW traffic and which is "ordinary" IP 365 traffic. 367 o "The only time we really need a general congestion control 368 mechanism is when traffic goes through the public Internet. 369 Obviously this will never be the case for PW traffic." 371 It is not at all difficult to imagine someone using an IPsec 372 tunnel across the public Internet to transport a PW from one 373 private IP network to another. 375 Nor is it difficult to imagine some enterprise implementing a PW 376 and transporting it across some SP's backbone, e.g., if that SP is 377 providing VPN service to that enterprise. 379 The arguments that non-TCP traffic in PWs will never make any 380 significant contribution to congestion thus do not seem to be totally 381 compelling. 383 2.6. Goals and non-goals of this draft 385 As a framework, this draft aims to explore the issues surrounding PW 386 congestion and to lay out some of the design trade offs that will 387 need to be made if a solution is to be developed. It does not intend 388 to propose a particular solution, but it does point out some of the 389 problems that will arise with certain solution approaches. 391 The over-riding technical goal of this work is to avoid scenarios in 392 which the deployment of PW technology leads to congestive collapse of 393 the PSN over which the PWs are tunnelled. It is a non-goal of this 394 work to ensure that PWs receive a particular Quality of Service (QoS) 395 particular level in order to protect them from congestion( 396 (Section 3.5)). While such an outcome may be desirable, it is beyond 397 the scope of this draft. 399 3. Challenges for PW Congestion Management 401 3.1. Scale 403 It might appear at first glance that an easy solution to PW 404 congestion control would be to run the PWs through a TCP connection. 405 This would provide congestion control automatically. However, the 406 overhead is prohibitive for the PW application. The PWE3 data plane 407 may be implemented in a micro-coded or hardware engine which needs to 408 support thousands of PWs, and needs to do as little as possible for 409 each data packet; running a TCP state machine, and implementing TCP's 410 flow control procedures, would impose too high a processing and 411 memory cost in this environment. Nor do we want to add the large 412 overhead of TCP to the PWs -- the large headers, the plethora of 413 small acknowledgements in the reverse direction, etc., etc. In fact, 414 we need to avoid acknowledgements altogether. These same 415 considerations lead us away from using e.g., DCCP [RFC4340], or SCTP 416 [RFC4960], even in its partially reliable mode [RFC3758]. Therefore 417 we will investigate some PW-specific solutions for congestion 418 control. 420 The PW designed also needs to minimise the amount of interaction 421 between the data processing path (which is likely to be distributed 422 among a set of line cards) and the control path; be especially 423 careful of interactions which might require atomic read/modify/write 424 operations from the control path, or which might require atomic read/ 425 modify/write operations between different processors in a 426 multiprocessing implementation, as such interactions can cause 427 scaling problems. 429 Thus, feasible solutions for PW-specific congestion will require 430 scalable means to detect congestion and to reduce the amount of 431 traffic sent into the network when congestion is detected. These 432 topics are discussed in more detail in subsequent sections. 434 3.2. Interaction among control loops 436 As noted above, much of the traffic that is carried on PWs is likely 437 to be TCP traffic, and will therefore be subject the congestion 438 control mechanisms of TCP. It will typically be difficult for a PW 439 endpoint to tell whether or not this is the case. Thus there is the 440 risk that the PE-PE congestion control mechanisms applied over the PW 441 may interact in undesirable ways with the end-to-end congestion 442 control mechanisms of TCP. The PW-specific congestion control 443 mechanisms should be designed to minimise the negative impact of such 444 interaction. 446 3.3. Determining the Appropriate Rate 448 TCP tends to share the bandwidth of a bottleneck among flows somewhat 449 evenly, although flows with shorter round-trip-times and larger MSS 450 values will tend to get more throughput in the steady state than 451 those with longer RTTs or smaller MSS. TFRC simply tries to deliver 452 the same rate to a flow that a TCP flow would obtain in the steady 453 state under similar conditions (loss rate, RTT and MSS). The 454 challenge fin the PW environment is to determine what constitutes a 455 "flow". While it is tempting to consider a single PW to be 456 equivalent to a "flow" for the purposes of fair rate estimation, it 457 is likely in many cases that one PW will be carrying a large number 458 of flows, in which case it would seem quite unfair to throttle that 459 PW down to the same rate that a single TCP flow would obtain. 461 The issue of what constitutes fairness and the perils of using the 462 TCP flow as the basic unit of fairness have been explored at some 463 length in [I-D.briscoe-tsvarea-fair]. 465 In the PW environment some estimate (measured or configured) of the 466 number flows in the PW needs to be applied to the estimate of fair 467 throughput as part of the process of determining appropriate 468 congestion behavior. 470 3.4. Constant Bit Rate PWs 472 Some types of PW, for example SAToP (Structure Agnostic TDM over 473 Packet) [RFC4553], CESoPSN (Circuit Emulation over Packet Switched 474 Networks) [RFC5086], TDM over IP [RFC5087][RFC4842], SONET/SDH and 475 Constant Bit Rate ATM PWs represent an inelastic constant bit-rate 476 (CBR) flow. Such PWs cannot respond to congestion in a TCP-friendly 477 manner prescribed by [RFC2914]. However this inability to respond 478 well to congestion by reducing the amount of total bandwidth consumed 479 if offset by the fact that such a PW maintains a constant bandwidth 480 demand rather than being greedy (in the sense of trying to increase 481 their share of a link, as TCP does). AIMD or even more gradual TFRC 482 techniques are clearly not applicable to such services; it is not 483 feasible to reduce the rate of a CBR service without violating the 484 service definition. Such services are also frequently more sensitive 485 to packet loss than connectionless packet PWs. Given that CBR 486 services are not greedy, there may be a case for allowing them 487 greater latitude during congestion peaks. If some CBR PWs are not 488 able to endure any significant packet loss or reduction in rate 489 without compromising the transported service, such PWs must be 490 shutdown when the level of congestion becomes excessive, but at 491 suitably low levels of congestion they should be allowed to continue 492 to offer traffic to the network. 494 Some CBR services may be carried over connectionless packet PWs. An 495 example of such a case would be a CBR MPEG-2 video stream carried 496 over an Ethernet PW. One could argue that such a service - provided 497 the rate was policed at the ingress PE - should be offered the same 498 latitude as a PW that explicitly provided a CBR service. Likewise, 499 there may not be much value in trying to throttle such a service 500 rather than cutting it off completely during severe congestion. 501 However, this clearly raises the issue of how to know that a PW is 502 indeed carrying a CBR service. 504 3.5. Valuable and Vulnerable PWs 506 Some PWs are a premium service for which the user has paid a premium 507 to the provider for higher availability, lower packet loss rate etc. 508 Some PWs, for example the CBR services described in Section 3.4are 509 particularly vulnerable to packet loss. In both of these cases it 510 may be tempting to relax the congestion considerations so that these 511 services can press on as best they can in the event of congestion. 512 However a more appropriate strategy is to engineer the network paths, 513 QoS parameters, queue sizes and scheduling parameters so that these 514 services do not suffer congestion discard. If despite this network 515 engineering these services still experience congestion, that is an 516 indication that the network is having difficulty servicing their 517 needs, and the services, like any other service should reduce their 518 network load. 520 4. Congestion Control Mechanisms 522 There are three components of the congestion control mechanism that 523 we need to consider: 525 1. Congestion state detection 527 2. Feedback from the receiver to the transmitter 529 3. Method used by the transmitter to respond to congestion state 531 Reference to congestion state apply to both the detection of the 532 onset of the congestion state, and detection of the return of the 533 network to a state in which greater or normal bandwidth is available. 535 We discuss the design framework for each of these components in the 536 sections below. 538 5. Detecting Congestion 540 In TCP, congestion is detected by the transmitter; the receipt of 541 three successive duplicate TCP acknowledgements are taken to be 542 indicative of congestion. What this actually means is that the 543 several packets in a row were received at the remote end, such that 544 none of those packets had the next expected sequence number. This is 545 interpreted as meaning that the packet with the next expected 546 sequence number was lost in the network, and the loss of a single 547 packet in the network is taken as a sign of congestion. (Naturally, 548 the presence of congestion is also inferred if TCP has to retransmit 549 a packet.) Note that it is possible for misordered packets to be 550 misinterpreted as lost packets, if they do not arrive "soon enough". 552 In TCP, a time-out while awaiting an acknowledgement is also 553 interpreted as a sign of congestion. 555 Since there are normally no acknowledgements on a PW (the only PW 556 design that has so far attempted a sophisticated flow control is 557 [I-D.ietf-pwe3-fc-flow]), the PW-specific congestion control 558 mechanism cannot normally be based on either the presence of or the 559 absence of acknowledgements. Some types of pseudowire (the CBR PWs) 560 have a single bit that indicates that a preset amount of data has 561 been lost, but this is a non-quantitative indicator. CBR PWs have 562 the advantage that there is a constant two way data flow, while other 563 PW types do not have the constant symmetric flow of payload on which 564 to piggyback the congestion notification. Most PW types therefore 565 provide no way for a transmitter to determine (or even to make an 566 educated guess as to) whether any data has been lost. 568 Thus we need to add a mechanism for determining whether data packets 569 on a PW have become lost. There are several possible methods for 570 doing this: 572 o Detect Congestion Using PW Sequence Numbers 574 o Detect Congestion Using Modified VCCV Packets [RFC5085] 576 o Rely on Explicit Congestion Notification (ECN) [RFC3168] 578 We discuss each option in turn in the following sections. 580 5.1. Using Sequence Numbers to Detect Congestion 582 When the optional sequencing feature is in use on a PW [RFC4385], it 583 is necessary for the receiver to maintain a "next expected sequence 584 number" for the PW. If a packet arrives with a sequence number that 585 is earlier than the next expected (a "misordered packet"), the packet 586 is discarded; if it arrives with a sequence number that is greater 587 than or equal to the next expected, the packet is delivered, and the 588 next expected sequence number becomes the sequence number of the 589 current packet plus 1. 591 It is easy to tell when there is one or more missing packets (i.e., 592 there is a "gap" in the sequence space) -- that is the case when a 593 packet arrives whose sequence number is greater than the next 594 expected. What is difficult to tell is whether any misordered 595 packets that arrive after the gap are indeed the missing packets. 596 One could imagine that the receiver remembers the sequence number of 597 each missing packet for a period of time, and then checks off each 598 such sequence number if a misordered packet carrying that sequence 599 number later arrives. The difficulty is doing this in a manner which 600 is efficient enough to be done by the microcoded hardware handling 601 the PW data path. This approach does not really seem feasible. 603 One could make certain simplifying assumptions, such as assuming that 604 the presence of any gaps at all indicates congestion. While this 605 assumption makes it feasible to use the sequence numbers to "detect 606 congestion", it also throttles the PW unnecessarily if there is 607 really just misordering and no congestion. Such an approach would be 608 considerably more likely to misinterpret misordering as congestion 609 than would TCP's approach. 611 An intermediate approach would be to keep track of the number of 612 missing packets and the number of misordered packets for each PW. 613 One could "detect congestion" if the number of missing packets is 614 significantly larger than the number of misordered packets over some 615 sampling period. However, gaps occurring near the end of a sampling 616 period would tend to result in false indications of congestion. To 617 avoid this one might try to smooth the results over several sampling 618 periods; While this would tend to decrease the responsiveness, it is 619 inevitable that there will be a trade-off between the rapidity of 620 responsiveness and the rate of false alarms. 622 One would not expect the hardware or microcode to keep track of the 623 sampling period; presumably software would read the necessary 624 counters from hardware at the necessary intervals. 626 Such a scheme would have the advantage of being based on existing PW 627 mechanisms. However, it has the disadvantage of requiring 628 sequencing, which introduces a fairly complicated interaction between 629 the control processing and the data path, and precludes the use of 630 some pipelined forwarding designs. 632 5.2. Using VCCV to Detect Congestion 634 It is reasonable to suppose that the hardware keeps counts of the 635 number of packets sent and received on each PW. Suppose that the PW 636 periodically inserts VCCV packets into the PE data stream, where each 637 VCCV packet carries: 639 o A sequence number, increasing by 1 for each successive VCCV 640 packet; 642 o The current value of the transmission counter for the PW 644 We assume that the size of the counter is such that it cannot wrap 645 during the interval between n VCCV packets, for some n > 1. 647 When the receiver gets one of these VCCV packets on a PW, it inserts 648 into the packet, or packet metadata the count of received packets for 649 that PW, and then delivers the VCCV packet to the software. The 650 receiving software can now compute, for the inter-VCCV intervals, the 651 count of packets transmitted and the count of packets received. The 652 presence of congestion can be inferred if the count of packets 653 transmitted is significantly greater than the count of packets 654 received during the most recent interval. Even the loss rate could 655 be calculated. The loss rate calculated in this way could be used as 656 input to the TFRC rate equation. 658 VCCV messages would not need to be sent on a PW (for the purpose of 659 detecting congestion) in the absence of traffic on that PW. 661 Of course, misordered packets that are sent during one interval but 662 arrive during the next will throw off the loss rate calculation; 663 hence the difference between sent traffic and received traffic should 664 be "significant" before the presence of congestion is inferred. The 665 value of "significance" can be made larger or smaller depending on 666 the probability of misordering. 668 Note that congestion can cause a VCCV packet to go missing, and 669 anything that misorders packets can misorder a VCCV packet as well as 670 any other. One may not want to infer the presence of congestion if a 671 single VCCV packet does not arrive when expected, as it may just be 672 delayed in the network, even if it hasn't been misordered. However, 673 failure to receive a VCCV packet after a certain amount of time has 674 elapsed since the last VCCV was received (on a particular PW) may be 675 taken as evidence of congestion. This scheme has the disadvantage of 676 requiring periodic VCCV packets, and it requires VCCV packet formats 677 to be modified to include the necessary counts. However, the 678 interaction between the control path and the data path is very 679 simple, as there is no polling of counters, no need for timers in the 680 data path, and no need for the control path to do read-modify-write 681 operations on the data path hardware. A bigger disadvantage may 682 arise from the possible inability to ensure that the transmit counts 683 in the VCCVs are exactly correct. The transmitting hardware may not 684 be able to insert a packet count in the VCCV IMMEDIATELY before 685 transmission of the VCCV on the wire, and if it cannot, the count of 686 transmit packets will only be approximate. 688 Neither scheme can provide the same type of continuous feedback that 689 TCP gets. TCP gets a continuous stream of acknowledgments, whereas 690 this type of PW congestion detection mechanism would only be able to 691 say whether congestion occurred during a particular interval. If the 692 interval is about 1 RTT, the PW congestion control would be 693 approximately as responsive as TCP congestion control, and there does 694 not seem to be any advantage to making it smaller. However, sampling 695 at an interval of 1 RTT might generate excessive amounts of overhead. 696 Sampling at longer intervals would reduce responsiveness to 697 congestion but would not necessarily render the congestion control 698 mechanism "TCP-unfriendly". 700 5.3. Explicit Congestion Notification 702 In networks that support explicit congestion notification (ECN) 703 [RFC3168] the ECN notification provides congestion information to the 704 PEs before the onset of congestion discard. This is particularly 705 useful to PWs that are sensitive to packet loss, since it gives the 706 PE the opportunity to intelligently reduce the offered load. ECN 707 marking rates of packets received on a PW could be used to calculate 708 the TFRC rate for a PW. However ECN is not widely deployed at the 709 time of writing; hence it seems that PEs must also be capable of 710 operating in a network where packet loss is the only indicator of 711 congestion. 713 6. Feedback from Receiver to Transmitter 715 Given that the receiver can tell, for each sampling interval, whether 716 or not a PW's traffic has encountered congestion, the receiver must 717 provide this information as feedback to the transmitter, so that the 718 transmitter can adjust its transmission rate appropriately. The 719 feedback could be as simple as a bit stating whether or not there was 720 any packet loss during the specified interval. Alternatively, the 721 actual loss rate could be provided in the feedback, if that 722 information turns out to be useful to the transmitter (e.g. to enable 723 it to calculate an appropriate rate at which to send). There are a 724 number of possible ways in which the feedback can be provided: 725 control plane, reverse data traffic, or VCCV messages. We discuss 726 each in turn below. 728 6.1. Control Plane Feedback 730 A control message can be sent periodically to indicate the presence 731 or absence of congestion. For example, when LDP is the control 732 protocol [RFC4447], the control message would of course be delivered 733 reliably by TCP. (The same considerations apply for any protocol 734 which has a reliable control channel.) When congestion is detected, 735 a control message can be sent indicating that fact. No further 736 congestion control messages would need to be sent until congestion is 737 no longer detected. If the loss rate is being sent, changes in the 738 loss rate would need to be sent as well. When there is no longer any 739 congestion, a message indicating the absence of congestion would have 740 to be sent. 742 Since congestion in the reverse direction can prevent the delivery of 743 these control messages, periodic "no congestion detected" messages 744 would need to be sent whenever there is no congestion. Failure to 745 receive these in a timely manner would lead the control protocol peer 746 to infer that there is congestion. (Actually, there might or might 747 not be congestion in the transmitting direction, but in the absence 748 of any feedback one cannot assume that everything is fine.) If 749 control messages really cannot get through at all, control protocol 750 keep-alives will fail and the control connection will go down anyway. 752 If the control messages simply say whether or not congestion was 753 detected, then given a reliable control channel, periodic messages 754 are not needed during periods of congestion. Of course, if the 755 control messages carry more data, such as the loss rate, then they 756 need to be sent whenever that data changes. 758 If it is desired to control congestion on a per-tunnel basis, these 759 control messages will simply say that there was congestion on some PW 760 (one or more) within the tunnel. If it is desired to control 761 congestion on a per-PW basis, the control message can list the PWs 762 which have experienced congestion, most likely by listing the 763 corresponding labels. If the VCCV method of detecting congestion is 764 used, one could even include the sent/received statistics for 765 particular VCCV intervals. 767 This method is very simple, as one does not have to worry about the 768 congestion control messages themselves getting lost or out of 769 sequence. Feedback traffic is minimized, as a single control message 770 relays feedback about an entire tunnel. 772 6.2. Using Reverse Data Packets for Feedback 774 If a receiver detects congestion on a particular PW, it can set a bit 775 in the data packets that are travelling on that PW in the reverse 776 direction; when no congestion is detected, the bit would be clear. 777 The bit would be ignored on any packet that is received out of 778 sequence, of course. There are several disadvantages to this 779 technique: 781 o There may be no (or insufficient) data traffic in the reverse 782 direction 784 o Sequencing of the data stream is required 786 o The transmission of the congestion indications is not reliable 788 o The most one could hope to convey is one bit of information per PW 789 (if there is even a bit available in the encapsulation). 791 6.3. Reverse VCCV Traffic 793 Congestion indications for a particular PW could be carried in VCCV 794 packets travelling in the reverse direction on that PW. Of course, 795 this would require that the VCCV packets be sent periodically in the 796 reverse direction whether or not there is reverse direction traffic. 797 For congestion feedback purposes they might need to be sent more 798 frequently than they'd need to be sent for OAM purposes. It would 799 also be necessary for the VCCVs to be sequenced (with respect to each 800 other, not necessarily with respect to the data stream). Since VCCV 801 transmission is unreliable, one would want to send multiple VCCVs 802 within whatever period we want to be able to respond in. Further, 803 this method provides no means of aggregating congestion information 804 into information about the tunnel. 806 7. Responding to Congestion 808 In TCP, one tends to think of the transmission rate in terms of MTUs 809 per RTT, which defines the maximum number of unacknowledged packets 810 that TCP is allowed to maintain "in flight". Upon detection of a 811 lost packet, this rate is halved ("multiplicative decrease"). It 812 will be halved again approximately every RTT until the missing data 813 gets through. Once all missing data has gotten through, the 814 transmission rate is increased by one MTU per RTT. Every time a new 815 acknowledgment (i.e., not a duplicate acknowledgment) is received, 816 the rate is similarly increased (additive increase). Thus TCP can 817 adjust its transmit rate very rapidly, i.e., it responds on the order 818 of a RTT. By contrast, TCP-friendly rate control adjusts its rate 819 rather more gradually. 821 For simplicity, this discussion only covers the "congestion 822 avoidance" phase of TCP congestion control. The analogy of TCP's 823 "slow start phase" would also be needed. 825 TCP can easily estimate the RTT, since all its transmissions are 826 acknowledged. In PWE3, the best way to estimate the RTT might be via 827 the control protocol. In fact, if the control protocol is TCP-based, 828 getting the RTT estimate from TCP might be a good option. 830 TCP's rate control is window-based, expressed as a number of bytes 831 that can be in flight. PWE3's rate control would need to be rate- 832 based. The TFRC specification [RFC3448] provides the equation for 833 the TCP-friendly rate for a given loss rate, RTT, and MTU. Given 834 some means of determining the loss rate, as described in Section 5, 835 the TCP friendly rate for a PW or a tunnel can be calculated at the 836 ingress PE. However as we noted earlier, a PW may be carrying many 837 flows, in which case the use of a single flow TCP rate would be a 838 significant underestimate of the true fair rate application, and 839 hence damaging to the operation of the PW. 841 If the congestion detection mechanism only produces an approximate 842 result, the probability of a "false alarm" (thinking that there is 843 congestion when there really is not) for some interval becomes 844 significant. It would be better then to have some algorithm which 845 smoothes the result over several intervals. The TFRC procedures, 846 which tend to generate a smoother and less abrupt change in the 847 transmission rate than the AIMD procedures, may also be more 848 appropriate in this case. 850 Once a PE has determined the appropriate rate at which to transmit 851 traffic on a given PW or tunnel, it needs some means to enforce that 852 rate via policing, shaping, or selective shutting down of PWs. There 853 are tradeoffs to be made among these options, depending on various 854 factors including the higher layer service that is carried. The 855 effect of different mechanisms when the higher layer traffic is 856 already using TCP is discussed below. 858 7.1. Interaction with TCP 860 Ideally there should be no PW-specific congestion control mechanism 861 used when the higher layer traffic is already running over TCP and is 862 thus subject to TCP's existing congestion control. However it may be 863 difficult to determine what the higher layer is on any given PW. 864 Thus, interaction between PW-specific congestion control and TCP's 865 congestion control needs to be considered. 867 As noted in Section 3.2, a PW-specific congestion control mechanism 868 may interact poorly with the "outer" control loop of TCP if the PW 869 carries TCP traffic. A well-documented example of such poor 870 interaction is a token bucket policer that drops packets outside the 871 token bucket. TCP has difficulty finding the "bottleneck" bandwidth 872 in such an environment and tends to overshoot, incurring heavy losses 873 and consequent loss of throughput. 875 A shaper that queues packets at the PE and only injects them into the 876 network at the appropriate rate may be a better choice, but may still 877 interact unpredictably with the "outer control loop" of TCP flows 878 that happen to traverse the PW. This issue warrants further study. 880 Another possibility is simply to shut down a PW when the rate of 881 traffic on the PW significantly exceeds the appropriate rate that has 882 been determined for the PW. While this might be viewed as draconian, 883 it does ensure that any PW that is allowed to stay up will behave in 884 a predictable manner. Note that this would also be the most likely 885 choice of action for CBR PWs (as discussed in Section 9). Thus all 886 PWs would be treated alike and there would be no need to try to 887 determine what sort of upper layer payload a PW is carrying. 889 8. Rate Control per Tunnel vs. per PW 891 Rate controls can be applied on a per-tunnel basis or on a per-PW 892 basis. Applying them on a per-tunnel basis (and obtaining congestion 893 feedback on a per-tunnel basis) would seem to provide the most 894 efficient and most scalable system. Achieving fairness among the PWs 895 then becomes a local issue for the transmitter. However, if the 896 different PWs follow different paths through the network (e.g. 897 because of ECMP over the tunnel), it is possible that some PWs will 898 encounter congestion while some will not. If rate controls are 899 applied on a per-tunnel basis, then if any PW in a tunnel is affected 900 by congestion, all the PWs in the tunnel will be throttled. While 901 this is sub-optimal, it is not clear that this would be a significant 902 problem in practise, and it may still be the best trade-off. 904 Per-tunnel rate control also has some desirable properties if the 905 action taken during congestion is to selectively shut down certain 906 PWs. Since a tunnel will typically carry many PWs, it will be 907 possible to make relatively small adjustments in the total bandwidth 908 consumed by the tunnel by selectively shutting down or bringing up 909 one or more PWs. 911 Note again the issue of estimating the correct rate. We know how 912 many PWs there per tunnel, but we do not know how many flows there 913 are per PW. 915 9. Constant Bit Rate Services 917 As noted above, some PW services may require a fixed rate of 918 transmission, and it may be impossible to provide the service while 919 throttling the transmission rate. To provide such services, the 920 network paths must be engineered so that congestion is impossible; 921 providing such services over the Internet is thus not very likely. 922 In fact, as congestion control cannot be applied to such services, it 923 may be necessary to prohibit these services from being provided in 924 the Internet, except in the case where the payload is known to 925 consist of TCP connections or other traffic that is congestion- 926 controlled by the end-points. It is not clear how such a prohibition 927 could be enforced. 929 The only feasible mechanism for handling congestion affecting CBR 930 services would appear to be to selectively turn off PWs, or channels 931 with the PW, when congestion occurs. Clearly it is important to 932 avoid "false alarms" in this case. It is also important to avoid 933 bringing PWs (or channels within the PW) back up too quickly and re- 934 introducing congestion. 936 The idea of controlling rate per tunnel rather than PW, discussed 937 above, seems particularly attractive when some of the PWs are CBR. 938 First, it provides the possibility that non-CBR PWs could be 939 throttled before it is necessary to shut down the CBR PWs. Second, 940 with the aggregation of multiple PWs on a single rate-controlled 941 tunnel, it becomes possible to gradually increase or decrease the 942 total offered load on the tunnel by selectively bringing up or 943 shutting down PWs. As noted above, local policies at a PE could be 944 used to determine which PWs to shut down or bring up first. Similar 945 approaches would apply if the CBR PW offers a channelized service, 946 with selected channels being shut down and brought up to control the 947 total rate of the PW. 949 10. Managed vs Unmanaged Deployment 951 As discussed in Section 2, there are a significant set of scenarios 952 in which PW-specific congestion control may not be necessary. One 953 might therefore argue that it doesn't seem to make sense to require 954 PW-specific congestion control to be used on all PWs at all times. 955 On the other hand, if the option of turning off PW-specific 956 congestion control is available, there is nothing to stop a provider 957 from turning it off in inappropriate situations. As this may 958 contribute to congestive collapse outside the provider's own network, 959 it may not be advisable to allow this. 961 The circumstance in which it is most vocally argued that congestion 962 control is not needed are where the PW is part of the service 963 providers own network infrastructure. In cases where the PW is 964 deployed in a managed, well-engineered network it is probably 965 necessary to permit the operator to take the congestion risk upon 966 themselves if they desire to do so by disabling any congestion 967 control mechanism. Ironically work in progress on the design of an 968 MPLS Transport Profile indicates that some of these users require 969 that an OAM is run (end-to-end, or over part of the tunnel (known as 970 Tandem monitoring)), and when the OAM detects that the performance 971 parameters are breached (delay, jitter or packet loss) this is used 972 to trigger a failover to a backup path. When the backup path 973 mechanism operates in 1 to 1 mode, this moves the congestive traffic 974 to another network path, which is the characteristic we require. [** 975 add reference **] 977 Further in such an environment it is likely that the PW will be used 978 to carry many TCP equivalent flows. In these managed circumstances 979 the operator will have to make a judgement call on a fair estimate of 980 the number of equivalent flows that the PW represents and set the 981 congestion parameters accordingly. 983 The counterpoint to a well managed network is a plug and play 984 network, of the type typically found in a consumer environment. 985 Whilst PWs have been designed to provide an infrastructure tool to 986 the service provider, we cannot be sure that they will not find some 987 consumer or similar plug and play application. In such an 988 environment a conservative approach seems appropriate, and the 989 default PW configuration MUST enable the congestion avoidance 990 mechanism, with parameters set to the equivalent of a single TCP 991 flow. 993 11. Related Work: Pre-Congestion Notification 995 It has been suggested that Pre-congestion Notification (PCN) 996 [I-D.ietf-pcn-architecture] might provide a basis for addressing the 997 PW congestion control problem. Using PCN, it would potentially be 998 possible to determine if the level of congestion currently existing 999 between an ingress and an egress PE was sufficiently low to safely 1000 allow a new PW to be established. PCN's pre-emption mechanisms could 1001 be used to notify a PE that one or more PWs need to be brought down, 1002 which again could be coupled with local policies to determine exactly 1003 which PWs should be shut down first. This approach certainly merits 1004 further examination, but we note that PCN is considerably further 1005 away from deployment in the Internet than ECN, and thus cannot be 1006 considered as a near-term solution to the problem of PW-induced 1007 congestion in the Internet. 1009 12. IANA Considerations 1011 This section may be removed by the RFC Editor. 1013 There are no IANA actions required by this document. 1015 13. Security Considerations 1017 Security is a good thing (TM). 1019 14. Conclusion 1021 Pseudowires are sometimes used to emulate services network that carry 1022 non-TCP data flows. In these circumstances the service payload will 1023 not react to network congestion by reducing its offered load. 1024 Pseudowires SHOULD therefore reduces their network bandwidth demands 1025 in the face of significant packet loss, including if necessary 1026 completely ceasing transmission. In some service provider network 1027 environments the network operator may choose to not deploy congestion 1028 avoidance mechanisms, but they should make that choice in the full 1029 knowledge that they are avoiding a network safety valve. In an 1030 unmanaged environment a conservative approach to congestion is 1031 REQUIRED. 1033 Since it is difficult to determine a priori the number of equivalent 1034 TCP flow that a pseudowire represents, a suitably "fair" rate of 1035 back-off cannot easily be pre-determined. In a managed environment 1036 setting the congestion parameters will require some level of informed 1037 judgement. In an unmanaged environment the equivalent TCP flow 1038 should be set to one. 1040 The selection of the appropriate mechanism(s) to implement congestion 1041 avoidance is work in progress in the IETF PWE3 working group. 1043 15. Informative References 1045 [I-D.briscoe-tsvarea-fair] 1046 Briscoe, B., "Flow Rate Fairness: Dismantling a Religion", 1047 draft-briscoe-tsvarea-fair-02 (work in progress), 1048 July 2007. 1050 [I-D.ietf-pcn-architecture] 1051 Eardley, P., "Pre-Congestion Notification (PCN) 1052 Architecture", draft-ietf-pcn-architecture-11 (work in 1053 progress), April 2009. 1055 [I-D.ietf-pwe3-fc-flow] 1056 Roth, M., Solomon, R., and M. Tsurusawa, "Reliable Fibre 1057 Channel Transport Over MPLS Networks", 1058 draft-ietf-pwe3-fc-flow-00 (work in progress), 1059 January 2009. 1061 [RFC2001] Stevens, W., "TCP Slow Start, Congestion Avoidance, Fast 1062 Retransmit, and Fast Recovery Algorithms", RFC 2001, 1063 January 1997. 1065 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1066 Requirement Levels", BCP 14, RFC 2119, March 1997. 1068 [RFC2581] Allman, M., Paxson, V., and W. Stevens, "TCP Congestion 1069 Control", RFC 2581, April 1999. 1071 [RFC2914] Floyd, S., "Congestion Control Principles", BCP 41, 1072 RFC 2914, September 2000. 1074 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1075 of Explicit Congestion Notification (ECN) to IP", 1076 RFC 3168, September 2001. 1078 [RFC3448] Handley, M., Floyd, S., Padhye, J., and J. Widmer, "TCP 1079 Friendly Rate Control (TFRC): Protocol Specification", 1080 RFC 3448, January 2003. 1082 [RFC3758] Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P. 1083 Conrad, "Stream Control Transmission Protocol (SCTP) 1084 Partial Reliability Extension", RFC 3758, May 2004. 1086 [RFC3985] Bryant, S. and P. Pate, "Pseudo Wire Emulation Edge-to- 1087 Edge (PWE3) Architecture", RFC 3985, March 2005. 1089 [RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram 1090 Congestion Control Protocol (DCCP)", RFC 4340, March 2006. 1092 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 1093 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 1094 Use over an MPLS PSN", RFC 4385, February 2006. 1096 [RFC4447] Martini, L., Rosen, E., El-Aawar, N., Smith, T., and G. 1097 Heron, "Pseudowire Setup and Maintenance Using the Label 1098 Distribution Protocol (LDP)", RFC 4447, April 2006. 1100 [RFC4553] Vainshtein, A. and YJ. Stein, "Structure-Agnostic Time 1101 Division Multiplexing (TDM) over Packet (SAToP)", 1102 RFC 4553, June 2006. 1104 [RFC4842] Malis, A., Pate, P., Cohen, R., and D. Zelig, "Synchronous 1105 Optical Network/Synchronous Digital Hierarchy (SONET/SDH) 1106 Circuit Emulation over Packet (CEP)", RFC 4842, 1107 April 2007. 1109 [RFC4960] Stewart, R., "Stream Control Transmission Protocol", 1110 RFC 4960, September 2007. 1112 [RFC5085] Nadeau, T. and C. Pignataro, "Pseudowire Virtual Circuit 1113 Connectivity Verification (VCCV): A Control Channel for 1114 Pseudowires", RFC 5085, December 2007. 1116 [RFC5086] Vainshtein, A., Sasson, I., Metz, E., Frost, T., and P. 1117 Pate, "Structure-Aware Time Division Multiplexed (TDM) 1118 Circuit Emulation Service over Packet Switched Network 1119 (CESoPSN)", RFC 5086, December 2007. 1121 [RFC5087] Stein, Y(J)., Shashoua, R., Insler, R., and M. Anavi, 1122 "Time Division Multiplexing over IP (TDMoIP)", RFC 5087, 1123 December 2007. 1125 Authors' Addresses 1127 Stewart Bryant 1128 Cisco Systems, Inc. 1129 250 Longwater 1130 Green Park, Reading RG2 6GB 1131 U.K. 1133 Phone: 1134 Fax: 1135 Email: stbryant@cisco.com 1136 URI: 1138 Bruce Davie 1139 Cisco Systems, Inc. 1140 1414 Mass. Ave. 1141 Boxborough, MA 01719 1142 USA 1144 Email: bsd@cisco.com 1145 Luca Martini 1146 Cisco Systems, Inc. 1147 9155 East Nichols Avenue, Suite 400. 1148 Englewood, CO 80112 1149 USA 1151 Email: lmartini@cisco.com 1153 Eric Rosen 1154 Cisco Systems, Inc. 1155 1414 Mass. Ave. 1156 Boxborough, MA 01719 1157 USA 1159 Email: erosen@cisco.com