idnits 2.17.1 draft-ietf-diffserv-pdb-vw-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** The document seems to lack a 1id_guidelines paragraph about Internet-Drafts being working documents. ** The document seems to lack a 1id_guidelines paragraph about 6 months document validity -- however, there's a paragraph with a matching beginning. Boilerplate error? ** The document is more than 15 pages and seems to lack a Table of Contents. == No 'Intended status' indicated for this document; assuming Proposed Standard == The page length should not exceed 58 lines per page, but there was 21 longer pages, the longest (page 1) being 59 lines Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. ** There are 31 instances of lines with control characters in the document. Miscellaneous warnings: ---------------------------------------------------------------------------- == Line 228 has weird spacing: '...il time for p...' == Line 250 has weird spacing: '...by so, effect...' == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- Couldn't find a document date in the document -- date freshness check skipped. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'RFC2474' is defined on line 517, but no explicit reference was found in the text == Unused Reference: 'RFC2597' is defined on line 525, but no explicit reference was found in the text == Unused Reference: 'RFC2638' is defined on line 532, but no explicit reference was found in the text ** Downref: Normative reference to an Informational RFC: RFC 2475 ** Obsolete normative reference: RFC 2598 (Obsoleted by RFC 3246) ** Downref: Normative reference to an Informational RFC: RFC 2638 == Outdated reference: A later version (-03) exists of draft-ietf-diffserv-pdb-def-00 ** Downref: Normative reference to an Informational draft: draft-ietf-diffserv-pdb-def (ref. 'PDBDEF') -- Possible downref: Non-RFC (?) normative reference: ref. 'CAIDA' -- Possible downref: Non-RFC (?) normative reference: ref. 'NS2' -- Possible downref: Non-RFC (?) normative reference: ref. 'FBK' ** Downref: Normative reference to an Informational RFC: RFC 2415 Summary: 12 errors (**), 0 flaws (~~), 9 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Internet Engineering Task Force Van Jacobson 2 Differentiated Services Working Group Kathleen Nichols 3 Packet Design, Inc. 4 Internet Draft Kedar Poduri 5 Expires Jan., 2001 Cisco Systems, Inc. 6 draft-ietf-diffserv-pdb-vw-00.txt July, 2000 8 The `Virtual Wire' Per-Domain Behavior 9 11 Status of this Memo 13 This document is an Internet-Draft and is in full conformance with 14 all provisions of Section 10 of RFC2026. Internet-Drafts are 15 working documents of the Internet Engineering Task Force (IETF), 16 its areas, and its working groups. Note that other groups may also 17 distribute working documents as Internet-Drafts. 19 Internet-Drafts are draft documents valid for a maximum of six 20 months and may be updated, replaced, or obsoleted by other docu- 21 ments at any time. It is inappropriate to use Internet-Drafts as 22 reference material or to cite them other than as "work in 23 progress." 25 The list of current Internet-Drafts can be accessed at http:// 26 www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft 27 Shadow Directories can be accessed at http://www.ietf.org/ 28 shadow.html. Distribution of this memo is unlimited. 30 Abstract 32 This document describes an edge-to-edge behavior, in diffserv 33 terminology a per-domain behavior, called `Virtual Wire' (VW) 34 that can be constructed in any domain supporting the diffserv EF 35 PHB plus appropriate domain ingress policers. The VW behavior is 36 essentially indistinguishable from a dedicated circuit and can be 37 used anywhere it is desired to replace dedicated circuits with IP 38 transport. Although one attribute of VW is the delivery of a peak 39 rate, in VW this is explicitly coupled with a bounded jitter 40 attribute. 42 The document is a edited version of the earlier draft-ietf-diff- 43 serv-ba-vw-00.txt with a new name to reflect a change in Diffserv 44 WG terminology. 46 A pdf version of this document is available at ftp://ftp.packet- 47 design.com/ietf/vw_pdb_0.pdf 49 1.0 Introduction 51 [RFC2598] describes a diffserv PHB called expedited forwarding 52 (EF) intended for use in building a scalable, low loss, low 53 latency, low jitter, assured bandwidth, end-to-end service that 54 appears to the endpoints like an unshared, point-to-point connec- 56 Jacobson, Nichols and Poduri Expires: January, 2001 [page 1 ] 57 tion or `virtual wire.' For scalability, a diffserv domain sup- 58 plying this service must be completely unaware of the individual 59 endpoints using it and sees instead only the aggregate EF marked 60 traffic entering and transiting the domain. This document pro- 61 vides the specifications necessary on that aggregated traffic (in 62 diffserv terminology, a per-domain behavior or PDB) in order to 63 meet these requirements and thus defines a new pdb, the Virtual 64 Wire per-domain behavior or VW PDB. Despite the lack of per-flow 65 state, if the aggregate input rates are appropriately policed and 66 the EF service rates on interior links are appropriately config- 67 ured, the edge-to-edge service supplied by the domain will be 68 indistinguishable from that supplied by dedicated wires between 69 the endpoints. This note gives a quantitative definition of what 70 is meant by `appropriately policed and configured'. 72 Network hardware has become sufficiently reliable that the over- 73 whelming majority of network loss, latency and jitter are due to 74 the queues traffic experiences while transiting the network. 75 Therefore providing low loss, latency and jitter to a traffic 76 aggregate means ensuring that the packets of the aggregate see no 77 (or very small) queues. Queues arise when short-term traffic 78 arrival rate exceeds departure rate at some node(s). Thus ensur- 79 ing no queues for a particular traffic aggregate is equivalent to 80 bounding rates such that, at every transit node, the aggregate's 81 maximum arrival rate is less than that aggregate's minimum depar- 82 ture rate. These attributes can be ensured for a traffic aggregate 83 by using the VW PDB. 85 Creating the VW PDB has two parts: 87 1. Configuring individual nodes so that the aggregate has a 88 well-defined minimum departure rate. (`Well-defined' means 89 independent of the dynamic state of the node. In particular, 90 independent of the intensity of other traffic at the node.) 92 2. Conditioning the entire DS domain's aggregate (via policing 93 and shaping) so that its arrival rate at any node is always 94 less than that node's configured minimum departure rate. 96 [RFC2598] provides the first part. This document describes how 97 one configures the EF PHBs in the collection of nodes that make up 98 a DS domain and the domain's boundary traffic conditioners 99 (described in [RFC2475]) to provide the second part. This 100 description results in a diffserv per-domain behavior, as 101 described in [PDBDEF]. 103 This document introduces and describes VW informally via pictures 104 and examples rather than by derivation and formal proof. The 105 intended audience is ISPs and router builders and the authors feel 106 this community is best served by aids to developing a strong intu- 107 ition for how and why VW works. However, VW has a simple, formal 108 description and its properties can and have been derived quite 109 rigorously. Such papers may prove interesting, but are outside 111 Jacobson, Nichols and Poduri Expires: January, 2001 [page 2 ] 112 the intent of this document. 114 The VW PDB has two major attributes: an assured peak rate and a 115 bounded jitter. It is possible to define a different PDB with only 116 the first of these, a "constant bit-rate" PDB, but this is not the 117 objective of this document. 119 The next sections describe the VW PDB in detail and give examples 120 of how it might be implemented. The keywords "MUST", "MUST NOT", 121 "REQUIRED", "SHOULD", "SHOULD NOT", and "MAY" that appear in this 122 document are to be interpreted as described in [RFC2119]. 124 2.0 Description of the Virtual Wire PDB 126 2.1 Applicability 128 A Virtual Wire (VW) PDB is intended to send "circuit replacement" 129 traffic across a diffserv network. That is, this PDB is intended 130 to mimic, from the point of view of the originating and terminat- 131 ing nodes, the behavior of a hard-wired circuit of some fixed 132 capacity. It does this in a scalable (aggregatable) way that 133 doesn't require `per-circuit' state to exist anywhere but the 134 ingress router adjacent to the originator. This PDB should be 135 suitable for any packetizable traffic that currently uses fixed 136 circuits (e.g., telephony, telephone trunking, broadcast video 137 distribution, leased data lines) and packet traffic that has sim- 138 ilar delivery requirements (e.g., IP telephony or video confer- 139 encing). Thus the conceptual model of the VW PDB is as shown in 140 Figure 1: some portion (possibly all) of a physical wire between a 141 sender and receiver is replaced by a (higher bandwidth) DS domain 142 implementing VW in a way that is invisible to S, R or the circuit 143 infrastructure outside of the cloud. 145 Figure 1: VW conceptual model 147 2.2 Rules 149 The VW PDB uses the EF PHB to implement a transit behavior with 150 the required attributes. Each node in the domain MUST implement 151 the EF PHB as described in section 2 of [RFC2598] but with the 152 SHOULDs of that section taken as MUSTs. Specifically, RFC2598 153 states "The EF PHB is defined as a forwarding treatment for a par- 154 ticular diffserv aggregate where the departure rate of the aggre- 155 gate's packets from any diffserv node must equal or exceed a 156 configurable rate." The EF traffic SHOULD receive this rate inde- 157 pendent of the intensity of any other traffic attempting to tran- 158 sit the node. It SHOULD average at least the configured rate when 159 measured over any time interval equal to or longer than the time 160 it takes to send an output link MTU sized packet at the configured 161 rate." This leads to an "EF bound" on the delay that EF-marked 162 packets can experience at each node that is inversely propor- 163 tional to the configured EF rate for that link. 165 Jacobson, Nichols and Poduri Expires: January, 2001 [page 3 ] 166 The bandwidth limit of each output interface SHOULD be configured 167 as described in Section 2.4 of this document. In addition, each 168 domain boundary input interface that can be the ingress for EF 169 marked traffic MUST strictly police that traffic as described in 170 Section 2.4. Each domain boundary output interface that can be the 171 egress for EF marked traffic MUST strictly shape that traffic as 172 described in Section 2.4. 174 2.3 Attributes 176 Colloquially, "the same as a wire." That is, as long as packets 177 are sourced at a rate <= the virtual wire's configured rate, they 178 will be delivered with a high degree of assurance and with almost 179 no distortion of the interpacket timing imposed by the source. 180 However, any packets sourced at a rate greater than the VW config- 181 ured rate, measured over any time scale longer than a packet time 182 at that rate, will be unconditionally discarded. 184 2.4 Parameters 186 This section develops a parameterization of VW in terms of measur- 187 able properties of the traffic (i.e., the packet size and physical 188 wire bandwidth) and domain (the link bandwidths and EF transit 189 bound of each of the domain's routers). We will show that: 191 1. There is a simple formula relating the circuit bandwidth, 192 domain link bandwidths, packet size and maximum tolerable `jit- 193 ter' across the domain, and this jitter bound holds for each 194 packet individually - there is no interpacket dependence. 196 2. This formula and the EF bound described in [RFC2598] deter- 197 mine the maximum VW bandwidth that can be allocated between 198 some ingress and egress of the domain. (This is because the EF 199 bound is essentially a bound on the worst-case jitter that will 200 be seen by EF marked packets transiting some router and the 201 formula says that any three parameters from the set determine the fourth.) 204 3. When the ingress VW flows are allocated and policed so as to 205 ensure that this maximum VW bandwidth is not exceeded at any 206 node of the domain, the EF BA on a link exiting a node can con- 207 sist of an arbitrary aggregate of VW flows (i.e., no per-flow 208 state is needed) because there is no perturbation of the aggre- 209 gate's service order that will cause any of the constituent 210 flows to exceed its domain transit jitter bound. 212 2.4.1 The Jitter bound and `Jitter Window' for a single VW 213 circuit (flow) 215 Figure 2: Time structure of packets of a CBR stream at a high to 216 low bandwidth transition 218 Jacobson, Nichols and Poduri Expires: January, 2001 [page 4 ] 219 Figure 2 shows a CBR stream of size S packets being sourced at 220 rate R. At the domain egress border router, the packets arrive on 221 a link of bandwidth B (= nR) and depart to their destination on a 222 link of bandwidth R. 224 Figure 3: Details of arrival / departure relationships 226 Figure 3 shows the detailed timing of events at the router. At 227 time T0 the last bit of packet 0 arrives so output is started on 228 the egress link. It will take until time for packet 0 229 to be completely output. As long as the last bit of packet 1 230 arrives at the border router before , the destination node will 231 find the traffic indistinguishable from a stream carried the 232 entire way on a dedicated wire of bandwidth R. This means that 233 packets can be jittered or displaced in time (due to queue waits) 234 as they cross the domain and that there is a jitter window at the 235 border router of duration 237 (EQ 1) 239 that must bound the sum of all the queue waits seen be a packet as 240 it transits the domain. As long as this sum is less than , the 241 destination will see service identical to a dedicated wire. Note 242 that the jitter window is (implicitly) computed relative to the 243 first packet of the flight of packets departing the boundary 244 router and, thus, can only include variable delays. Any transit 245 delay experienced by all the packets, be it propagation time, 246 router forwarding latency, or even average queue waits, is 247 removed by the relative measure so the sum described in this para- 248 graph is not sensitive to delay but only to delay variation. Also 249 note that when packets enter the domain they are already separated 250 by so, effectively, everything is pushed to the left edge of 251 the jitter window and there's no slack time to absorb delay vari- 252 ation in the domain. However by simply delaying the output of the 253 first packet to arrive at E by one packet time (S/R), the phase 254 reference for all the traffic is reset so that all subsequent 255 packets enter at the right of their jitter window and have maximum 256 slack time during transit. 258 Figure 4: Packet timing structure edge-to-edge 260 Figure 4 shows the edge-to-edge path from the source to the desti- 261 nation. The links from S to I and E to D run at the virtual wire 262 rate R (or the traffic is shaped to rate R if the links run at a 263 higher rate). The solid rectangles on these links indicate the 264 packet time S/R. The dotted lines carry the packet times across 265 the domain since the time boundaries of these virtual packets form 266 the jitter window boundaries of the actual packets (whose dura- 267 tion and spacing are shown by the solid rectangles below the 268 intra-domain link). Note that each packet's jitter is indepen- 269 dent. E.g., even though the two packets about to arrive at E have 270 been displaced in opposite directions so that the total time 272 Jacobson, Nichols and Poduri Expires: January, 2001 [page 5 ] 273 between them is almost , neither has gone out of its jitter win- 274 dow so the output from E to D will be smooth and continuous. 276 The preceding derives the jitter window in terms of the bandwidths 277 of the ingress circuit and intra-domain links. In practice, the 278 `givens' are more likely to be the intradomain link bandwidths and 279 the jitter window (which can be no less than the EF bound associ- 280 ated with each output link that might be traversed by some VW 281 flow(s)). Rearranging Equation 1 based on this gives R, the maxi- 282 mum amount of bandwidth that can be allocated to the aggregate of 283 all VW circuits, as a function of the EF bound (= jitter window = 284 ): 286 (EQ 2) 288 Note that the upper bound on VW traffic that can be handled by any 289 output link is simply the MTU divided by the link's EF bound. 291 2.4.2 Jitter independence under aggregation 293 Figure 5: Three VW customers forming an aggregate 295 This jitter independence is what allows multiple `virtual wires' 296 to be transparently aggregated into a single VW PDB. Figure 5 297 shows three independent VW customers, blue, yellow and red, 298 entering the domain at I. Assume that their traffic has worst-case 299 phasing, i.e., that one packet from each stream arrives simulta- 300 neously at I. Even if the output link scheduler makes a random 301 choice of which packet to send from its EF queue, no packet will 302 get pushed outside its jitter window. For example, in Figure 5 303 node I ships a different perturbation of the 3 customer aggregate 304 in every window yet this has no effect on the edge-to-edge VW 305 properties). 307 The jitter independence means that we only have to compare the 308 jitter window of Equation 1 to the worst case of the total queue 309 wait that can be seen by a single VW packet as it crosses the 310 domain. There are three potential sources of queue wait for a VW 311 packet: 313 1. it can queue behind non-EF packets (if any) 315 2. it can queue behind another VW packet from the same customer 317 3. it can queue behind VW packet(s) from other customers 319 For case (1), the EF `priority queuing' model says that the VW 320 traffic will never wait while a non-EF queue is serviced so the 321 only delay it can experience from non-EF traffic is if it has to 322 wait for the finish of a packet that was being sent at the time it 323 arrived. For an output link of bandwidth B, this can impose a 325 Jacobson, Nichols and Poduri Expires: January, 2001 [page 6 ] 326 worst-case delay of S/B. Note that this implies that if the (low 327 bandwidth) links of a network are carrying both VW and other traf- 328 fic, then n in Equation 1 must be at least 2 (i.e., the EF bound 329 can be at most half the link bandwidth) in order to make the jit- 330 ter window large enough to absorb this delay. 332 Case (2) can only happen if the previous packet hasn't completely 333 departed at the time the next packet arrives. Since each ingress 334 VW stream is strictly shaped to a rate R, any two packets will be 335 separated by at least time S/R so having leftovers is equivalent 336 to saying the departure rate on some link is R over any time scale of S/R or longer so this can't hap- 339 pen for any legal VW/EF configuration. Or, to put it another way, 340 if case (2) happens, either the VW policer is set too loosely or 341 some link's EF bound is set too tight. 343 Case (3) is a simple generalization of (2). If there are a total 344 of n customers, the worst possible queue occurs if all n arrive 345 simultaneously at some output link. Since each customer is indi- 346 vidually shaped to rate R, when this happens then no new packets 347 from any stream can arrive for at least time S/R. At the end of 348 this time, there can only be leftover packets in the queue if the 349 departure rate < nR over this time scale. Conforming to the EF 350 property (restated in section 2.2) means that any link capable of 351 handling the aggregate traffic must have a departure rate > nR 352 over any time scale longer than S/(nR) so, again, this can't hap- 353 pen in any legal VW/EF configuration. 355 For case (1), a packet could be displaced by non-EF traffic once 356 per hop so the edge-to-edge jitter is a function of the path 357 length. But this isn't true for case (3): The strict ingress 358 policing implies that a packet from any given VW stream can meet 359 any other VW stream in a queue at most once. This means the worst 360 case jitter caused by aggregating VW customers is a linear func- 361 tion of the number of customers in the aggregate but completely 362 independent of topology. 364 2.4.3 Topological effects on allocation 366 Although the jitter caused by aggregating VW customers is inde- 367 pendent of topology, the number of customers and/or bandwidth per 368 customer is very sensitive to topology and the topological 369 effects may be subtle. Equation 2 gives the aggregate VW that can 370 traverse any link but if there are n customers, the topology and 371 their (possible) paths through it determine how this bandwidth 372 divided among them. 374 Figure 6: Cumulative jitter from spatially distinct flows 376 The first thought is to simply divide the aggregate bandwidth by 377 the customer in-degree at some link. E.g., in Figure 5 there are 378 three customers aggregated into the I>E link so each should get a 380 Jacobson, Nichols and Poduri Expires: January, 2001 [page 7 ] 381 third of the available VW bandwidth. But Figure 6 shows that this 382 is too optimistic. Note that the yellow flow (I>E) gets jittered 383 one packet time when it meets the blue flow (I>2) then an addi- 384 tional packet time when it meets the red flow (3>4). So even 385 though the maximum in-degree is two and the largest possible 386 aggregate has only two components, the combined interactions 387 require that each customer get at most 1/3 of the available VW 388 bandwidth rather than 1/2. In general, if nj is the total number 389 of other customers encountered (or potentially encountered) by 390 customer j as it traverses the domain, then the bandwidth shares 391 can be at most . 393 It is possible to `tune' the topology to trade off total VW band- 394 width vs. reliability. For example in Figure 7, if the SF>NY VW 395 traffic is constrained (via route pinning or tunneling) to only 396 follow the northern path and LA>DC to only follow the southern 397 path, then each customer gets the entire VW bandwidth on their 398 path but at the expense of neither being able to use the alternate 399 path in the event of a link failure. If they want to take advan- 400 tage of the redundancy, only half the bandwidth can be allocated 401 to each even though the full bandwidth will be available most of 402 the time. Mixed strategies are also possible. For example the 403 SF>NY customer could get an expensive SLA that guaranteed the full 404 VW bandwidth even under a link failure and LA>DC could get a cheap 405 SLA that gave full bandwidth unless there was a link failure in 406 which case it gave nothing. 408 Figure 7: bandwidth vs. redundancy trade-off 410 2.4.4 Per-customer bandwidth and/or packet size variation 412 Figure 8: Aggregating VW flows with different bandwidth shares 414 In the last example, customers could get different VW shares 415 because their traffic was engineered to be disjoint. But when cus- 416 tomers with different shares transit the same link(s), there can 417 be problems. For example, Figure 8 shows blue and red customers 418 allocated 1/4 share each while yellow gets a 1/2 share. Since yel- 419 low's share is larger, its jitter window is smaller (the dotted 420 yellow line). Since the packets from a customer can appear any- 421 where in the jitter window, it's perfectly possible for packets 422 from red, blue and yellow to arrive simultaneously at I. Since I 423 has no per-customer state, the serving order for the three is ran- 424 dom and it's entirely possible that blue and red will be served 425 before yellow. But since yellow's jitter window is only two link 426 packet times wide, this results in no packets in its first window 427 and two in its second so its jitter bound is violated. There are 428 at least three different ways to deal with this: 430 1. Make all customers use the smallest jitter window. This is 431 equivalent to provisioning based on the number of customers 432 times the max customer rate rather than on the sum of the cus- 433 tomer rates. In bandwidth rich regions of the cloud this is 435 Jacobson, Nichols and Poduri Expires: January, 2001 [page 8 ] 436 probably the simplest solution but in bandwidth poor regions it 437 can result in denying customers that there is capacity to 438 accept. 440 2. Utilize a set of LU DSCPs to create an EF-like code point per 441 rate and service them in rate magnitude order. For a small set 442 of customer rates this makes full use of the capacity at the 443 expense of additional router queues (one per code point). 445 3. Separate rate and jitter bounds in the SLA. I.e., base the 446 jitter bound on the packet time of the minimum customer rate 447 (which is equivalent to making all customers use the largest 448 jitter window). This effectively treats the larger customers as 449 if their traffic were an aggregate of min rate flows which may 450 be the appropriate choice if the flow is indeed an aggregate, 451 e.g., a trunk containing many voice calls. 453 2.5 Assumptions 455 The topology independence of the VW PDB holds only while routing 456 is relatively stable. Since packets can be duplicated while rout- 457 ing is converging, and since path lengths can be shorter after a 458 routing change, it is possible to violate the VW traffic bounds 459 and thus jitter stream(s) more than their jitter window for a 460 small time during and just after a routing change. 462 The derivation in the preceding section assumed that the VW PDB 463 was the only user of EF in the domain. If this is not true, the 464 provisioning and allocation calculations must be modified to 465 account for the other users of EF. 467 2.6 Example uses 469 An enterprise could use VW to provision a large scale, internal 470 VoIP telephony system. Say for example that the internal links are 471 all Fast Ethernet (100Mb/s) or faster and arranged in a 3 level 472 hierarchy (switching/aggregation/routing) so the network diameter 473 is 5 hops. Typical telephone audio codecs deliver a packet every 474 20ms. At this codec rate, RTP encapsulated G.711 voice is 200 byte 475 packets & G.729 voice is 60 byte packets. 477 20ms at 100 Mb/s is 250 Kbytes (~150 MTUs, ~1200 G.711 calls or 478 ~4,000 G.729 calls) which would be the capacity if the net were 479 carrying only VW telephony traffic. 481 Worse case jitter from other traffic through a diameter 5 482 enterprise is 5 MTU times or 0.6 ms leaving between 19 ms 483 (optimistic) to 10 ms (ultra conservative - see scaling notes in 484 the appendix) for VW. 10ms at 100Mb/s is 125Kbytes so using the 485 most conservative assumptions we can admit ~600 G.711 or ~2000 486 G.729 calls if the ingress can simultaneously police both packet & 487 bit rate. If the ingress can police only one of these, we can only 488 admit ~75 calls because each packet might be as long as an MTU. 490 Jacobson, Nichols and Poduri Expires: January, 2001 [page 9 ] 491 2.7 Environmental concerns 493 Routing instability will generally translate directly into VW 494 service degradation. 496 Multipath routing of VW will, in general, increase the jitter and 497 degrade the service unless either the paths are exactly the same 498 length (so there is no effect on jitter) and/or the routing deci- 499 sion is such that it always sends any particular customer down the 500 same path. 502 The analysis in Section 2.4 would hold in a world where traffic 503 policers and link schedulers are perfect and mathematically 504 exact. When computing parameters for our world, 5-10% fudge fac- 505 tors should be used. 507 3.0 Security Considerations 509 There are no security considerations for the VW PDB other than 510 those associated with the EF PHB which are described in [RFC2598]. 512 4.0 References 514 [RFC2119] "Key words for use in RFCs to Indicate Requirement Lev- 515 els", S.Bradner, www.ietf.org/rfc/rfc2119 517 [RFC2474] "Definition of the Differentiated Services Field (DS 518 Field) in the IPv4 and IPv6 Headers", K.Nichols, S. 519 Blake, F. Baker, D. Black, www.ietf.org/rfc/rfc2474.txt 521 [RFC2475] "An Architecture for Differentiated Services", S. Blake, 522 D. Black, M.Carlson,E.Davies,Z.Wang,W.Weiss, 523 www.ietf.org/rfc/rfc2475.txt 525 [RFC2597] "Assured Forwarding PHB Group", F. Baker, J. Heinanen, 526 W. Weiss, J. Wroclawski, ftp://ftp.isi.edu/in-notes/ 527 rfc2597.txt 529 [RFC2598] "An Expedited Forwarding PHB", V.Jacobson, K.Nichols, 530 K.Poduri, ftp://ftp.isi.edu/in-notes/rfc2598.txt 532 [RFC2638] "A Two-bit Differentiated Services Architecture for the 533 Internet", K. Nichols, V. Jacobson, and L. Zhang, 534 www.ietf.org/rfc/rfc2638.{txt,ps} 536 [PDBDEF] "Definition of Differentiated Services Per-domain Behav- 537 iors and Rules for their Specification", K.Nichols, 538 B.Carpenter, draft-ietf-diffserv-pdb-def-00.[txt, pdf] 540 [CAIDA] The nature of the beast: recent traffic measurements from 541 an Internet backbone. K Claffy, Greg Miller and Kevin 542 Thompson. http://www.caida.org/Papers/Inet98/index.html 544 Jacobson, Nichols and Poduri Expires: January, 2001 [page 10 ] 545 [NS2] The simulator ns-2, available at: http://www-mash.cs.ber- 546 keley.edu/ns/. 548 [FBK] K. Nichols, "Improving Network Simulation with Feedback", 549 Proceedings of LCN'98, October, 1998. 551 [RFC2415] RFC 2415, K. Poduri and K. Nichols, "Simulation Studies 552 of Increased Initial TCP Window Size", September, 1998. 554 5.0 Authors' Addresses 556 Van Jacobson 557 Packet Design, Inc. 558 66 Willow Place 559 Menlo Park, CA 94025 560 van@packetdesign.com 562 Kathleen Nichols 563 Packet Design, Inc. 564 66 Willow Place 565 Menlo Park, CA 94025 566 nichols@packetdesign.com 568 Kedar Poduri 569 Cisco Systems, Inc. 570 170 W. Tasman Drive 571 San Jose, CA 95134-1706 572 poduri@cisco.com 574 6.0 Appendix: On Jitter for the VW PDB 576 The VW PDB's bounded jitter translates into the generally useful 577 properties of network bandwidth limits and buffer resource lim- 578 its. These properties make VW useful for a variety of statically 579 and dynamically provisioned services, many of which have no 580 intrinsic need for jitter bounds. IP telephony is an important 581 application for the VW PDB where expected and worst-case jitter 582 for rate-controlled streams of packets is of interest; thus this 583 appendix is primarily focused on voice jitter. 585 Rather than the "phase jitter" used in the body of this document, 586 this appendix used "interpacket jitter" for a variety of reasons. 587 This might be changed in a future version. Note that, as shown in 588 section 2.4.1, the phase jitter can correct for a larger inter- 589 packet jitter. 591 The appendix focuses on jitter for individual flows aggregated in 592 a VW PDB, derives worst-case bounds on the jitter, and gives sim- 593 ulation results for jitter. 595 6.1 Jitter and Delay 597 Jacobson, Nichols and Poduri Expires: January, 2001 [page 11 ] 598 The VW PDB is sufficiently restrictive in its rules to preserve 599 the required EF per-hop behavior under aggregation. These proper- 600 ties also make it useful as a basis for Internet telephony, to get 601 low jitter and delay. Since a VW PDB will have link arrival rates 602 that do not exceed departure rates over fairly small time scales, 603 end-to-end delay is based on the transmission time of a packet on 604 a wire and the handling time of individual network elements and 605 thus is a function of the number of hops in a path, the bandwidth 606 of the links, and the properties of the particular piece of equip- 607 ment used. Unless the end-to-end delay is excessive due to very 608 slow links or very slow equipment, it is usually the jitter, or 609 variation of delay, of a voice stream that is more critical than 610 the delay. 612 We derive the worst case jitter for a a VW PDB in a DS domain 613 using it to carry a number of rate-controlled flows. For this we 614 use inter-packet jitter, defined as the absolute value of the dif- 615 ference between the arrival time difference of two adjacent pack- 616 ets and their departure time difference, that is: 618 (EQ 3)jitter = |(ak-aj) - (dk-dj)| 620 The maximum jitter will occur if one packet waits for no other 621 packets at any hop of its path and the adjacent packet waits for 622 the maximum amount of packets possible. There are two sources of 623 jitter, one from waiting for other EF packets that may have accu- 624 mulated in a queue due to simultaneous arrivals of EF packets on 625 several input lines feeding the same queue and another from wait- 626 ing for non-EF packets to complete. The first type is strictly 627 bounded by the properties of the VW PDB and the EF PHB. The second 628 type is minimized by using a Priority Queuing mechanism to sched- 629 ule the output link and giving priority to EF packets and this 630 value can be approached by using a non-bursty weighted round- 631 robin packet scheduler and giving the EF queue a large weight. The 632 total jitter is the sum of these two. 634 Maximum jitter will be given across the domain in terms of T, the 635 virtual packet time or cycle time. It is important to recall the 636 analysis of section 2.0 showing that this jitter across the DS 637 domain is completely invisible to the end-to-end flow using the VW 638 PDB if it is within the jitter window at the egress router. 640 6.1.1 Jitter from other VW packets 642 The jitter from meeting other packets of the VW aggregate comes 643 from (near) simultaneous arrival of packets at different input 644 ports all destined for the same output queue that can be com- 645 pletely rearranged at the next packet arrival to that queue. This 646 jitter has a strict bound which we will show here. 648 It will be helpful to remember that, from RFC 2598, a PDB using 649 the EF PHB will get its configured share of each link at all time 650 scales above the time to send an MTU at the rate corresponding to 652 Jacobson, Nichols and Poduri Expires: January, 2001 [page 12 ] 653 that share of that link. 655 Focus on the DS domain of Figure 9. Unless otherwise stated, in 656 this section assume M Boundary Routers, each having N inputs and 657 outputs. We assume that each of the BR's ingress ports receives a 658 flow of EF-marked packets that are being policed to a peak rate R. 659 If each flow sends a fixed size packet, then it's possible to cal- 660 culate the fixed time, T, between packets of each of these MxN 661 flows that enters the DS domain, a quite reasonable assumption for 662 packets carrying voice. For example, assume a domain traversed by 663 MxN flows of 68 byte voice packets sent at 20 ms time intervals. 664 Note we assume all ingress links have some packets that will be 665 marked for the VW aggregate. Thus the total number of ingress EF- 666 marked streams to the VW aggregate is I = MxN. 668 To construct a network where the maximum jitter can occur, a sin- 669 gle flow traversing the network must be able to meet packets from 670 all the other flows marked for the EF PHB and it should be possi- 671 ble to meet them in the worst possible way. 673 Figure 9: A DS domain 675 Unless otherwise stated, assume that all the routers of the domain 676 have N inputs and outputs and that all links have the same band- 677 width B. Although there are a number of ways that the individual 678 streams from different egress ports might combine in the interior 679 of the network, in a properly configured network, the arrival rate 680 of the VW PDB must not exceed the departure rate at any network 681 node. Consider a particular flow from A to Z and how to ensure 682 that packets entering the VW PDB at A meet every other flow enter- 683 ing the domain from all egress points as they traverse the domain 684 to Z. Consider three cases: the first is a single bottleneck, the 685 second makes no assumptions about routing in the network and the 686 third assumes that the paths of individual flows can be completely 687 specified. 689 Assume there are H hops from A to Z and that delay is the minimum 690 time it takes for a packet to go from A to Z in the absence of 691 queuing. Both packets experience delay and thus it subtracts in 692 the jitter calculation. Recall that the packets of the flow are 693 separated in time by T, then (normalizing to a start time of 0): 695 (EQ 4)dj = 0 697 (EQ 5)dk = T 699 (EQ 6)aj = delay 701 (EQ 7)ak = time spent waiting behind all other packets +delay+T 703 Then we can use: 705 (EQ 8)jitter = time spent waiting behind all other packets 707 Jacobson, Nichols and Poduri Expires: January, 2001 [page 13 ] 708 as we explore calculating worst case jitter for different topolo- 709 gies. That is, the worst case queueing delay can be used to bound 710 the jitter. 712 The next step is to establish some useful relationships between 713 parameters. First, assume that some fraction, f, of a link's 714 capacity is allocated to EF-marked packets. Since we are assuming 715 that all the flows that are admitted into this DS domain's VW 716 aggregate generate packets at a spacing of T, this can be 717 expressed in time as fxT. Then the amount of time to send an EF 718 packet on each link can be written as fxT/(total number of EF- 719 marked flow crossing the link). Note that f should be less than 720 0.5 in order that an MTU-sized non-EF packet will not cause the EF 721 condition to be violated. In the subsequent analysis, we will, in 722 general, assume that the entire fraction f of EF traffic is 723 present in order to calculate worst case bounds. 725 6.1.1.1 Worst case jitter in a network with a dumbbell bottleneck 727 Consider a DS domain topology shown in Figure 10. In order for a 728 packet of the (A,Z) flow to wait behind packets of all the other 729 MxN - 1 flows, packets from each of these flows must be sitting in 730 the router queue for the bottleneck link L when the (A,Z) packet 731 arrives. Since N flows arrive on each of the M links, the lowest 732 bound on the bandwidth B occurs when the N packets arrive in 733 bursts. In this case, B must be large enough (relative to L) so 734 that the packets are still sitting in L's queue when our (A,Z) 735 packet arrives at the end of a burst of N packets, that is B > 736 NxL. Then the 738 (EQ 9)jitterworst case = MxNx(time to send an EF packet on L) 740 Since we expressed the EF aggregate's allocation on L as fxL, the 741 time to send an EF packet on L is (at most) fxT/(MxN), so 743 (EQ 10)jitterworst case = fxT 745 Figure 10: A dumbbell bottleneck 747 This result shows that the worst case jitter will be less than 748 half a packet time for any VW-compliant allocation on this topol- 749 ogy. For the worst case to occur, all N packets must arrive at 750 each of the M border routers within the time it takes to transmit 751 them all on B (from above, this is bounded by fxT/N). By assuming 752 independence, an interested person should be able to get some 753 insight on the likelihood of this happening. Simulation results 754 are included in a later section. 756 6.1.1.2 Worst case jitter in an arbitrary network 758 Jacobson, Nichols and Poduri Expires: January, 2001 [page 14 ] 759 Consider the network of Figure 9 and in this case, one packet of 760 the (A,Z) flow must arrive at the same time, but be queued behind 761 a packet from each of the other flows as it passes through the 762 network. This can happen at any link of the network and even at 763 the same link. Assume all links have bandwidth B and that we don't 764 know the path the individual EF packets or flows of the aggregate 765 will follow. Then the worst case jitter is 767 (EQ 11)jitterworst case = Ix(fxT/I) = fxT 769 the same as the bottleneck case. In a somewhat pathological con- 770 struct where two flows pass through the same link more than once, 771 but take different paths between those links, we assume the pack- 772 ets are serialized when they first meet and are not retimed by the 773 disjoint paths to meet again. Although one could construct a case 774 where the a particular packet queues behind another multiple 775 times, a bit of thought should show that this is unlikely in the 776 realm of applicability of the VW PDB. 778 If the allocation can have knowledge that not all flows of the 779 aggregate will take the same path, then one could allocate each 780 link to a smaller number of flows, but this would also imply that 781 the number of flows that it's possible to meet and be jittered by 782 is smaller. Allocation can be kept to under 0.5 times the band- 783 width of a core link, while the existence of multiple paths offers 784 both fault tolerance and an expectation that the actual load on 785 any link will be less than 0.5. 787 How likely is this case to happen? One packet of the (A,Z) flow 788 must encounter and queue behind every other individual shaped 789 flow that makes up the domain's VW aggregate as it crosses the 790 domain. 792 6.1.1.3 Maximal jitter in a network with "pinned" paths per flow 794 Then at each hop the (A,Z) packet has to arrive at the same time 795 as an EF packet from the (N-1) other inputs and the (A,Z) packet 796 has to be able to end up anywhere within that burst of N packets. 797 In particular, for two adjacent packets of the (A,Z) flow, one 798 must arrive at the front of every hop's burst and the other at the 799 end of every hop's burst. This clearly requires an unrealistic 800 form of path pinning or route selection by every individual EF- 801 marked flow entering the DS domain. This unidirectional path is 802 shown in Figure 11 where all routers have N inputs and at each of 803 the H routers on the path from A to Z, N-1 flows are sent to other 804 output queues, while N-1 of the shaped input flows that have not 805 yet crossed the A to Z path enter the router at the other input 806 ports. 808 Figure 11: Example path for maximal jitter across DS domain from 809 A to Z 811 It should be noted that if the number of hops from A to Z is not 813 Jacobson, Nichols and Poduri Expires: January, 2001 [page 15 ] 814 large enough, it won't be possible for one of its packets to meet 815 all the other shaped flows and if the number of hops is larger 816 than what's required there won't be any other shaped flows to meet 817 there. For the flow from A to B to meet every other ingress stream 818 as it traverses a path of H hops: 820 (EQ 12) Hx(N-1) = MxN - 1 822 then compute the maximum jitter as: 824 (EQ 13)jitter = Hx(N-1)x(time to send an EF packet on each link) 826 If the total number of ingress streams exceeds Hx(N-1) + 1, then 827 it's not possible to meet all the other streams and the maximum 828 jitter is 830 (EQ 14)jitterworst case = H x (N-1) x fT/(number of ingress-shaped 831 EF flows on each link) 833 Otherwise the max jitter is 835 (EQ 15)jitterworst case = (MxN - 1) x fT/(number of ingress-shaped 836 EF flows on each link) 838 Then the maximum jitter depends on the number of hops or the num- 839 ber of border routers. In this construction, the number of 840 ingress-shaped EF flows on each link is N, thus: 842 (EQ 16)jitterworst case < smaller of (HxfT, MxfxT) 844 Dividing out T gives jitter in terms of the number of ingress 845 flow cycle times (or virtual packet times). Then, for the jitter 846 to exceed the cycle time (or 20 ms for our VoIP example), 848 (EQ 17)fxH > 1 and fxM > 1 850 If f were at its maximum of 0.5, then it appears to be easy to 851 exceed a cycle time of jitter across a domain. However, it's use- 852 ful to examine what f might typically be. Note that for this con- 853 struction: 855 (EQ 18)f= NxR/B 857 For our example voice flows, a reasonable R is 28-32 Kbps. Then, 858 for a link of 128 Kbps, f = 0.25xN; for 1.5 Mbps, f = 0.02xN; for 859 10 Mbps, f = 0.003xN; for 45 Mbps, f = 0.0007xN; and for 100 Mbps, 860 f = 0.0003xN. Then such a network of DS3 links can handle almost 861 1500 individual shaped flows at this rate. Another way to look at 862 this is that the hop count times the number of ingress ports of 863 each router must exceed the link bandwidth divided by the VoIP 864 rate in order to have a maximum jitter of a packet time at the 865 VoIP rate. 867 Jacobson, Nichols and Poduri Expires: January, 2001 [page 16 ] 868 (EQ 19)HxN > B/R 870 For a network of all T1 links, this becomes HxN > 50 and for 871 larger bandwidth links, it increases. 873 Suppose that the ingress flows are not the same rate. If the allo- 874 cation, f, is at its maximum, then this means the number of 875 ingress flows must decrease. For example, if the A to Z flow is 876 10xR, then it will meet 9 fewer packets as it traverses the net- 877 work. Even though the assumptions behind this case are not realis- 878 tic, we can see that the jitter can be kept to a reasonable 879 amount. The rules of the EF PHB and the VW PDB should make it easy 880 to compute the worst case jitter for any topology and allocation. 882 6.1.1.4 Achievability of the maximum 884 Now that we've examined how to compute the worst case jitter, we 885 look at how likely it is that this worst case happens and how it 886 relates to the jitter window. 888 In addition to the topological and allocation assumptions that 889 were made in order to allow a flow to have the opportunity of 890 meeting every other flow, events must align so that the meeting 891 actually happens at each hop. If we could assume independence of 892 the timing of each flow's arrival within an interval of T, then 893 that probability is on the order of (fT/N)N For this to happen at 894 every hop we need the joint probability of this happening at all H 895 nodes. Further we need the joint probability of that event in com- 896 bination with an adjacent packet not meeting any other packets. 897 For each additional hop, the number of ways the packets can com- 898 bine increases exponentially, thus the probability of that par- 899 ticular worst case combination decreases. 901 6.1.1.5 Jitter from non-VW packets 903 The worst case occurs when one packet of a flow waits for no other 904 packets to complete and the adjacent packet arrives at every hop 905 just as an MTU-sized non-EF packet has begun transmission. That 906 worst case jitter is the sum of the times to send MTU-sized pack- 907 ets at the link bandwidth of all the hops in the path or, for 908 equal bandwidth paths, 910 (EQ 20)jitter = HxMTU/B 912 Note that if one link has a bandwidth much smaller than the oth- 913 ers, that term will dominate the jitter. 915 If we assume that the MTU is on the order of 10-20 times the voice 916 packet size in our example, then the time to send an MTU on a link 917 is 10 or 20 times fxT/N so that our jitter bound becomes 20 x H x 918 fxT/N. 920 What has to happen in order to achieve the worst case? For jitter 922 Jacobson, Nichols and Poduri Expires: January, 2001 [page 17 ] 923 against the default traffic, one packet waits for no default traf- 924 fic and the adjacent packet arrives just as an MTU of the default 925 type begins transmission on the link. 927 The worst case is linear in the number of hops, but since the 928 joint probability of an EF packet arriving at each queue precisely 929 at the start of a non-EF packet on the link decreases in hop 930 count, measured or simulated jitter will be seen to grow as a neg- 931 ative exponential of the number of hops in a path, even at very 932 high percentiles of probability. The reason for this is that the 933 number of ways that the packets can arrive at the EF queue grows 934 as pH so the probability is on the order of p-H. When the link 935 bandwidth is small, it may be necessary to fragment non-EF packet 936 to control jitter. 938 How should we relate jitter in terms of source cycle times or vir- 939 tual packet times to the jitter window defined in section 2.0? 940 Note that we can write 942 (EQ 21)jitter window = Sx(1/R - 1/((nxR)/f)) 944 and noting that T = S/R, we get: 946 (EQ 22)jitter window = Tx(n-f/n) 948 So that, in many cases, the jitter window can be approximated by 949 T. 951 6.2 Quantifying Jitter through Simulation 953 Section 1 derived and discussed the worst-case jitter for indi- 954 vidual flows of a diffserv per-domain behavior (PDB) based on the 955 EF PHB. We showed that the worst case jitter can be bounded and 956 calculated these theoretical bounds. The worst case bounds repre- 957 sent possible, but not likely, values. Thus, to get a better feel 958 for the likely worst jitter values, we used simulation. 960 We use the ns-2 network simulator; our use of this simulator has 961 been described in a number of documents [NS2,FBK,RFC2415]. The 962 following subsections describe the simulation set-up for these 963 particular experiments. 965 6.2.1 Topology 967 Figure 12 shows the topology we used in the simulations. A and Z 968 are edge routers through which traffic from various customers 969 enters and exits the Diffserv cloud. We vary the topology within 970 the Diffserv cloud to explore the worst-case jitter for EF traffic 971 in various scenarios. Jitter is measured on a flow or set of flows 972 that transit the network from A to Z. To avoid per hop synchroni- 973 zations, half the DE traffic at each hop is new to the path while 974 half of the DE traffic exits the path. For the mixed EF and DE 975 simulations, half the EF flows go from A to Z while, at each hop, 977 Jacobson, Nichols and Poduri Expires: January, 2001 [page 18 ] 978 the other half of the 10% rate only crosses the path at that hop. 979 As discussed in section 1, this is an unlikely construction but we 980 undertake it to give a more pessimistic jitter For the EF-only 981 simulations, we emulate the case analyzed in section 1.1.3 by mea- 982 suring jitter on one end to end flow and having (N-1) new EF flows 983 meet that flow at every hop. Note that N is determined by the max- 984 imum number of 28 kbps flows that can fit in the EF share of each 985 link, so N=share x bandwidth/28 kbps. 987 Figure 12: Simulated topology 989 6.2.2 Traffic 991 Traffic is generated to emulate G.729 voice flows with packet size 992 (B) of 68 Bytes and a 20 ms packetization rate. The resultant 993 flows have a rate of 27 Kbps. As previously discussed, jitter 994 experienced by the voice flows has two main components; jitter 995 caused by meeting others flows in the EF queue, and jitter due to 996 traffic in other low priority classes. To analyze the first compo- 997 nent, we vary the multiplexing level of voice flows that are 998 admitted into the DS domain and for the second, we generate data 999 traffic for the default or DE PHB. Since we are interested in 1000 exploring the worst case jitter, data traffic is generated as 1001 long-lived TCP connections with 1500 Byte MTU segments. Current 1002 measurements show real Internet traffic consists of a mixture of 1003 packet sizes, over 50% of which are minimum-sized packets of 40 1004 bytes and over 80% of which are much smaller than 1500 Bytes 1005 [CAIDA]. Thus a realistic traffic mix would only improve the jit- 1006 ter that we see in the simulations. 1008 6.2.3 Schedulers and Queues 1010 All the nodes(routers) in the network have the same configura- 1011 tion: a simple Priority Queue (PQ) scheduler with two queues. 1012 Voice traffic is queued in the high priority queue while the data 1013 traffic is queued in the queue with the lower priority. The sched- 1014 uler empties all the packets in the high priority queue before 1015 servicing the data packets in a lower priority queue. However, if 1016 the scheduler is busy servicing a data packet at the time of 1017 arrival of a voice packet, the voice packet is served only after 1018 the data packet is serviced completely, i.e., the scheduler is 1019 non-preemptive. For priority queuing where the low priority queue 1020 is kept congested, simulating two queues is adequate. 1022 Figure 13: Link scheduling in the simulations 1024 6.2.4 Results 1026 In the following simulations, three bandwidth values were used 1027 for the DS domain links: 1.5 Mbps, 10 Mbps, and 45 Mbps. Unless 1028 otherwise stated, the aggregate of EF traffic was allocated 10% of 1029 the link bandwidth. The hops per path was varied from 1 to 24. 1030 Then, the 1.5 Mbps links can carry about 5 voice flows, the 10 1032 Jacobson, Nichols and Poduri Expires: January, 2001 [page 19 ] 1033 Mbps about 36 voice flows, and 45 Mbps about 160. 1035 6.2.4.1 Jitter due to other voice traffic only 1037 To see the jitter that comes only from meeting other EF-marked 1038 packets, we simulated voice traffic only and found this to be 1039 quite negligible. For example, with a 10 Mbps link and 10% of the 1040 link share assigned to the voice flows, a single bottleneck link 1041 in a dumbbell has a worst case jitter of 2 ms. In simulation the 1042 99.97th percentile jitter for one to 25 hops never exceeds a third 1043 of a millisecond. This source of jitter is quite small, particu- 1044 larly compared to the jitter from traffic in other queue(s) as we 1045 will see in the next section. 1047 6.2.4.2 Jitter in a voice flow where there is a congested default 1048 class 1050 Our traffic model for the DE queue keeps it full with mostly 1500 1051 byte packets. From section 1, the worst case jitter is equal to 1052 the number of hops times the time to transmit a packet at the link 1053 rate. The likelihood of this worst case occurring goes down expo- 1054 nentially in hop count, and the simulations confirm this. Figure 1055 15 shows several percentiles of the jitter for 10 Mbps links where 1056 the time to transmit an MTU at link speed is 1.2 ms. 1058 Figure 14: Various percentile jitter values for 10 Mbps links and 1059 10% allocation 1061 Recall that the period of the voice streams is 20 ms and note 1062 that, the jitter does not even reach half a period. The median 1063 jitter gets quite flat with number of hops. Although the higher 1064 percentile values increase at a somewhat higher rate with number 1065 of hops, it still does not approach the calculated worst case. The 1066 data is also shown normalized by the MTU transmission time at 10 1067 Mbps. Now the vertical axis value is the number of MTU sized pack- 1068 ets of jitter that the flow experiences. This normalization is 1069 presented to make it easier to relate the results to the analysis, 1070 though it obscures the impact (or lack thereof) of the jitter on 1071 the 20 ms flows. 1073 Figure 16 shows the same results for 1.5 Mbps links and Figure 17 1074 for 45 Mbps links. 1076 Figure 15: Various percentiles of jitter for 1.5 Mbps links and 1077 10% share 1079 Notice that the worst case jitter for the 1.5 Mbps link is on the 1080 order of two cycle times while, for 45 Mbps, it is less than 10% 1081 of the cycle time. However, the behavior in terms of number of 1082 MTUs is similar. 1084 Figure 16: Various percentiles of jitter for 45 Mbps links and 1085 10% share 1087 Jacobson, Nichols and Poduri Expires: January, 2001 [page 20 ] 1088 The jitter in time and thus as a fraction of the virtual packet 1089 time of the flow being measured clearly increases with decreasing 1090 bandwidth. Even the smallest bandwidth, 1.5 Mbps can handle 1091 nearly all jitter with a jitter buffer of 2 packets. The two 1092 higher bandwidths don't even jitter by one virtual packet time, 1093 thus staying within the jitter window. Figures 18, 19, and 20 com- 1094 pare the median, 99th percentile and 99.97th percentile (essen- 1095 tially the worst case). It's also interesting to normalize the 1096 results of each experiment by the MTU transmission time at that 1097 link bandwidth. The normalized values show that all scenarios 1098 experience the same behavior relative to the MTU transmission 1099 time. 1101 Figure 17: Median jitter for all three bandwidths by time and 1102 normalized 1104 Figure 18: 99th percentile of jitter for the three bandwidths; 1105 absolute time and normalized 1107 Figure 19: 99.97th percentile of jitter; absolute time and 1108 normalized by MTU transmission times 1110 The simulation experiments are not yet complete, but they clearly 1111 show the probability of achieving the worst case jitter decreas- 1112 ing with hop count and show that jitter can be controlled. The 1113 normalization shows that the jitter behavior is the same regard- 1114 less of bandwidth. The absolute times differ by scale factors that 1115 depend on the bandwidth. 1117 6.2.4.3 Jitter with an increased allocation 1119 In the following, the experiments of the last section are 1120 repeated, but using a 20% link share, rather than a 10% link share 1121 Figure 21 shows the jitter percentiles for 10 Mbps links and a 20% 1122 share. The values are also plotted with the 10% share results (on 1123 the right hand side) to show how similar they are. 1125 Figure 20: Jitter percentiles for 10 Mbps links and 20% EF share 1127 Using the previous section, we would believe that the results for 1128 other bandwidths would have the same shape, but be scaled by the 1129 bandwidth difference. Figure 22 shows this to indeed be the case. 1130 Thus it is sufficient to simulated only a single bandwidth. 1132 Figure 21: Jitter for 1.5 Mbps links (on left) and 45 Mbps links 1133 (on right) 1135 In all the experiments, it can be clearly seen that the shape of 1136 the jitter vs. hops curve flattens because the probability of the 1137 worst case occurring at each hop decreases exponentially in hops. 1138 To see if there is an allocation level at which the jitter behav- 1139 ior diverges, we simulated and show results for allocations of 10, 1141 Jacobson, Nichols and Poduri Expires: January, 2001 [page 21 ] 1142 20, 30, 40, and 50 percent, all for 10 Mbps links in Figure 23. 1144 Figure 22: Median and 99th percentile jitter for various 1145 allocations and 10 Mbps links 1147 What may not be obvious from figure 19 is that the similarity 1148 between the five allocation levels shows that jitter from other EF 1149 traffic is negligible compared to the jitter from waiting for DE 1150 packets to complete. Clearly, the probability of jitter from 1151 other EF traffic goes up with increasing allocation level, but it 1152 is so small compared to the DE-induced jitter that it isn't visi- 1153 ble except for the highest percentiles and the largest hop count.