idnits 2.17.1 draft-ietf-ippm-6man-pdm-option-10.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (May 9, 2017) is 2536 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Obsolete normative reference: RFC 2460 (Obsoleted by RFC 8200) Summary: 1 error (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 INTERNET-DRAFT N. Elkins 3 Inside Products 4 R. Hamilton 5 Chemical Abstracts Service 6 M. Ackermann 7 Intended Status: Proposed Standard BCBS Michigan 8 Expires: November 10, 2017 May 9, 2017 10 IPv6 Performance and Diagnostic Metrics (PDM) Destination Option 11 draft-ietf-ippm-6man-pdm-option-10 13 Abstract 15 To assess performance problems, this document describes optional 16 headers embedded in each packet that provide sequence numbers and 17 timing information as a basis for measurements. Such measurements 18 may be interpreted in real-time or after the fact. An implementation 19 of the existing IPv6 Destination Options extension header, the 20 Performance and Diagnostic Metrics (PDM) Destination Options 21 extension header as well as the field limits, calculations, and usage 22 of the PDM in measurement are included in this document. 24 Status of this Memo 26 This Internet-Draft is submitted to IETF in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF), its areas, and its working groups. Note that 31 other groups may also distribute working documents as 32 Internet-Drafts. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 The list of current Internet-Drafts can be accessed at 40 http://www.ietf.org/1id-abstracts.html 42 The list of Internet-Draft Shadow Directories can be accessed at 43 http://www.ietf.org/shadow.html 45 Copyright and License Notice 47 Copyright (c) 2017 IETF Trust and the persons identified as the 48 document authors. All rights reserved. 50 IETF Trust Legal Provisions of 28-dec-2009, Section 6.b(i), paragraph 51 3: This document is subject to BCP 78 and the IETF Trust's Legal 52 Provisions Relating to IETF Documents 53 (http://trustee.ietf.org/license-info) in effect on the date of 54 publication of this document. Please review these documents 55 carefully, as they describe your rights and restrictions with respect 56 to this document. Code Components extracted from this document must 57 include Simplified BSD License text as described in Section 4.e of 58 the Trust Legal Provisions and are provided without warranty as 59 described in the Simplified BSD License. 61 Table of Contents 63 1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . 5 64 1.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . 5 65 1.2 Rationale for defined solution . . . . . . . . . . . . . . . 5 66 1.3 IPv6 Transition Technologies . . . . . . . . . . . . . . . . 6 67 2 Measurement Information Derived from PDM . . . . . . . . . . . . 6 68 2.1 Round-Trip Delay . . . . . . . . . . . . . . . . . . . . . . 6 69 2.2 Server Delay . . . . . . . . . . . . . . . . . . . . . . . . 7 70 3 Performance and Diagnostic Metrics Destination Option Layout . . 7 71 3.1 Destination Options Header . . . . . . . . . . . . . . . . . 7 72 3.2 Performance and Diagnostic Metrics Destination Option . . . 7 73 3.2.1 PDM Layout . . . . . . . . . . . . . . . . . . . . . . . 7 74 3.2.2 Base Unit for Time Measurement . . . . . . . . . . . . . 9 75 3.3 Header Placement . . . . . . . . . . . . . . . . . . . . . . 10 76 3.4 Header Placement Using IPSec ESP Mode . . . . . . . . . . . 10 77 3.4.1 Using ESP Transport Mode . . . . . . . . . . . . . . . . 10 78 3.4.2 Using ESP Tunnel Mode . . . . . . . . . . . . . . . . . 10 79 3.5 Implementation Considerations . . . . . . . . . . . . . . . 11 80 3.5.1 PDM Activation . . . . . . . . . . . . . . . . . . . . . 11 81 3.5.2 PDM Timestamps . . . . . . . . . . . . . . . . . . . . . 11 82 3.6 Dynamic Configuration Options . . . . . . . . . . . . . . . 11 83 3.7 Information Access and Storage . . . . . . . . . . . . . . . 11 84 4 Security Considerations . . . . . . . . . . . . . . . . . . . . 12 85 4.1 Resource Consumption and Resource Consumption Attacks . . . 12 86 4.2 Pervasive monitoring . . . . . . . . . . . . . . . . . . . . 12 87 4.3 PDM as a Covert Channel . . . . . . . . . . . . . . . . . . 13 88 4.4 Timing Attacks . . . . . . . . . . . . . . . . . . . . . . . 13 89 5 IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 14 90 6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 91 6.1 Normative References . . . . . . . . . . . . . . . . . . . . 14 92 6.2 Informative References . . . . . . . . . . . . . . . . . . . 15 93 Appendix A: Context for PDM . . . . . . . . . . . . . . . . . . . 15 94 A.1 End User Quality of Service (QoS) . . . . . . . . . . . . . 15 95 A.2 Need for a Packet Sequence Number (PSN) . . . . . . . . . . 15 96 A.3 Rationale for Defined Solution . . . . . . . . . . . . . . . 16 97 A.4 Use PDM with Other Headers . . . . . . . . . . . . . . . . . 16 98 Appendix B : Timing Considerations . . . . . . . . . . . . . . . . 17 99 B.1 Timing Differential Calculations . . . . . . . . . . . . . . 17 100 B.2 Considerations of this time-differential representation . . 18 101 B.2.1 Limitations with this encoding method . . . . . . . . . 18 102 B.2.2 Loss of precision induced by timer value truncation . . 19 103 Appendix C: Sample Packet Flows . . . . . . . . . . . . . . . . . 20 104 C.1 PDM Flow - Simple Client Server . . . . . . . . . . . . . . 20 105 C.1.1 Step 1 . . . . . . . . . . . . . . . . . . . . . . . . . 21 106 C.1.2 Step 2 . . . . . . . . . . . . . . . . . . . . . . . . . 21 107 C.1.3 Step 3 . . . . . . . . . . . . . . . . . . . . . . . . . 22 108 C.1.4 Step 4 . . . . . . . . . . . . . . . . . . . . . . . . . 23 109 C.1.5 Step 5 . . . . . . . . . . . . . . . . . . . . . . . . . 24 110 C.2 Other Flows . . . . . . . . . . . . . . . . . . . . . . . . 24 111 C.2.1 PDM Flow - One Way Traffic . . . . . . . . . . . . . . . 24 112 C.2.2 PDM Flow - Multiple Send Traffic . . . . . . . . . . . . 26 113 C.2.3 PDM Flow - Multiple Send with Errors . . . . . . . . . . 27 114 Appendix D: Potential Overhead Considerations . . . . . . . . . . 28 115 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 29 116 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 30 118 1 Background 120 To assess performance problems, measurements based on optional 121 sequence numbers and timing may be embedded in each packet. Such 122 measurements may be interpreted in real-time or after the fact. 124 As defined in RFC2460 [RFC2460], destination options are carried by 125 the IPv6 Destination Options extension header. Destination options 126 include optional information that need be examined only by the IPv6 127 node given as the destination address in the IPv6 header, not by 128 routers or other "middle boxes". This document specifies a new 129 destination option, the Performance and Diagnostic Metrics (PDM) 130 destination option. This document specifies the layout, field 131 limits, calculations, and usage of the PDM in measurement. 133 1.1 Terminology 135 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 136 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 137 document are to be interpreted as described in RFC 2119 [RFC2119]. 139 1.2 Rationale for defined solution 141 The current IPv6 specification does not provide timing nor a similar 142 field in the IPv6 main header or in any extension header. The IPv6 143 Performance and Diagnostic Metrics destination option (PDM) provides 144 such fields. 146 Advantages include: 148 1. Real measure of actual transactions. 150 2. Independence from transport layer protocols. 152 3. Ability to span organizational boundaries with consistent 153 instrumentation. 155 4. No time synchronization needed between session partners 157 5. Ability to handle all transport protocols (TCP, UDP, SCTP, etc) in 158 a uniform way 160 The PDM provides the ability to determine quickly if the (latency) 161 problem is in the network or in the server (application). That is, 162 it is a fast way to do triage. For more information on background 163 and usage of PDM, see Appendix A. 165 1.3 IPv6 Transition Technologies 167 In the path to full implementation of IPv6, transition technologies 168 such as translation or tunneling may be employed. The PDM header is 169 not expected to work in such scenarios. It is likely that an IPv6 170 packet containing PDM will be dropped if using IPv6 transition 171 technologies. 173 2 Measurement Information Derived from PDM 175 Each packet contains information about the sender and receiver. In IP 176 protocol, the identifying information is called a "5-tuple". 178 The 5-tuple consists of: 180 SADDR : IP address of the sender 181 SPORT : Port for sender 182 DADDR : IP address of the destination 183 DPORT : Port for destination 184 PROTC : Protocol for upper layer (ex. TCP, UDP, ICMP, etc.) 186 The PDM contains the following base fields: 188 PSNTP : Packet Sequence Number This Packet 189 PSNLR : Packet Sequence Number Last Received 190 DELTATLR : Delta Time Last Received 191 DELTATLS : Delta Time Last Sent 193 Other fields for storing time scaling factors are also in the PDM and 194 will be described in section 3. 196 This information, combined with the 5-tuple, allows the measurement 197 of the following metrics: 199 1. Round-trip delay 200 2. Server delay 202 2.1 Round-Trip Delay 204 Round-trip *Network* delay is the delay for packet transfer from a 205 source host to a destination host and then back to the source host. 206 This measurement has been defined, and the advantages and 207 disadvantages discussed in "A Round-trip Delay Metric for IPPM" 208 [RFC2681]. 210 2.2 Server Delay 212 Server delay is the interval between when a packet is received by a 213 device and the first corresponding packet is sent back in response. 214 This may be "Server Processing Time". It may also be a delay caused 215 by acknowledgements. Server processing time includes the time taken 216 by the combination of the stack and application to return the 217 response. The stack delay may be related to network performance. If 218 this aggregate time is seen as a problem, and there is a need to make 219 a clear distinction between application processing time and stack 220 delay, including that caused by the network, then more client based 221 measurements are needed. 223 3 Performance and Diagnostic Metrics Destination Option Layout 225 3.1 Destination Options Header 227 The IPv6 Destination Options Header is used to carry optional 228 information that needs to be examined only by a packet's destination 229 node(s). The Destination Options Header is identified by a Next 230 Header value of 60 in the immediately preceding header and is defined 231 in RFC2460 [RFC2460]. The IPv6 Performance and Diagnostic Metrics 232 Destination Option (PDM) is an implementation of the Destination 233 Options Header. The PDM does not require time synchronization. 235 3.2 Performance and Diagnostic Metrics Destination Option 237 3.2.1 PDM Layout 239 The IPv6 Performance and Diagnostic Metrics Destination Option (PDM) 240 contains the following fields: 242 SCALEDTLR: Scale for Delta Time Last Received 243 SCALEDTLS: Scale for Delta Time Last Sent 244 PSNTP : Packet Sequence Number This Packet 245 PSNLR : Packet Sequence Number Last Received 246 DELTATLR : Delta Time Last Received 247 DELTATLS : Delta Time Last Sent 249 The PDM destination option is encoded in type-length-value (TLV) 250 format as follows: 252 0 1 2 3 253 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 254 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 255 | Option Type | Option Length | ScaleDTLR | ScaleDTLS | 256 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 257 | PSN This Packet | PSN Last Received | 258 |-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 259 | Delta Time Last Received | Delta Time Last Sent | 260 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 262 Option Type 264 TBD = 0xXX (TBD) [To be assigned by IANA] [RFC2780] 266 In keeping with RFC2460[RFC2460], the two high order bits of the 267 Option Type field are encoded to indicate specific processing of the 268 option; for the PDM destination option, these two bits MUST be set to 269 00. 271 The third high order bit of the Option Type specifies whether or not 272 the Option Data of that option can change en-route to the packet's 273 final destination. 275 In the PDM, the value of the third high order bit MUST be 0. 277 Option Length 279 8-bit unsigned integer. Length of the option, in octets, excluding 280 the Option Type and Option Length fields. This field MUST be set to 281 16. 283 Scale Delta Time Last Received (SCALEDTLR) 285 8-bit unsigned integer. This is the scaling value for the Delta Time 286 Last Received (DELTATLR) field. The possible values are from 0-255. 287 See Section 4 for further discussion on Timing Considerations and 288 formatting of the scaling values. 290 Scale Delta Time Last Sent (SCALEDTLS) 292 8-bit signed integer. This is the scaling value for the Delta Time 293 Last Sent (DELTATLS) field. The possible values are from 0 to 255. 295 Packet Sequence Number This Packet (PSNTP) 297 16-bit unsigned integer. This field will wrap. It is intended for 298 use while analyzing packet traces. 300 Initialized at a random number and incremented monotonically for each 301 packet of the session flow of the 5-tuple. The random number 302 initialization is intended to make it harder to spoof and insert such 303 packets. 305 Operating systems MUST implement a separate packet sequence number 306 counter per 5-tuple. 308 Packet Sequence Number Last Received (PSNLR) 310 16-bit unsigned integer. This is the PSNTP of the packet last 311 received on the 5-tuple. 313 This field is initialized to 0. 315 Delta Time Last Received (DELTATLR) 317 A 16-bit unsigned integer field. The value is set according to the 318 scale in SCALEDTLR. 320 Delta Time Last Received = (Send time packet n - Receive time packet 321 n-1) 323 Delta Time Last Sent (DELTATLS) 325 A 16-bit unsigned integer field. The value is set according to the 326 scale in SCALEDTLS. 328 Delta Time Last Sent = (Receive time packet n - Send time packet n-1) 330 3.2.2 Base Unit for Time Measurement 332 A time differential is always a whole number in a CPU; it is the unit 333 specification -- hours, seconds, nanoseconds -- that determine what 334 the numeric value means. For PDM, the base time unit is 1 attosecond 335 (asec). This allows for a common unit and scaling of the time 336 differential among all IP stacks and hardware implementations. 338 Note that PDM provides the ability to measure both time differentials 339 that are extremely small, and time differentials in a DTN-type 340 environment where the delays may be very great. To store a time 341 differential in just 16 bits that must range in this way will require 342 some scaling of the time differential value. 344 One issue is the conversion from the native time base in the CPU 345 hardware of whatever device is in use to some number of attoseconds. 346 It might seem this will be an astronomical number, but the conversion 347 is straightforward. It involves multiplication by an appropriate 348 power of 10 to change the value into a number of attoseconds. Then, 349 to scale the value so that it fits into DELTATLR or DELTATLS, the 350 value is shifted by of a number of bits, retaining the 16 high-order 351 or most significant bits. The number of bits shifted becomes the 352 scaling factor, stored as SCALEDTLR or SCALEDTLS, respectively. For a 353 full description of this process, including examples, please see 354 Appendix A. 356 3.3 Header Placement 358 The PDM Destination Option is placed as defined in RFC2460 [RFC2460]. 359 There may be a choice of where to place the Destination Options 360 header. If using ESP mode, please see section 3.4 of this document 361 for placement of the PDM Destination Options header. 363 For each IPv6 packet header, the PDM MUST NOT appear more than once. 364 However, an encapsulated packet MAY contain a separate PDM associated 365 with each encapsulated IPv6 header. 367 3.4 Header Placement Using IPSec ESP Mode 369 IPSec Encapsulating Security Payload (ESP) is defined in [RFC4303] 370 and is widely used. Section 3.1.1 of [RFC4303] discusses placement 371 of Destination Options Headers. 373 The placement of PDM is different depending on if ESP is used in 374 tunnel or transport mode. 376 3.4.1 Using ESP Transport Mode 378 Note that Destination Options MAY be placed before or after ESP or 379 both. If using PDM in ESP transport mode, PDM MUST be placed after 380 the ESP header so as not to leak information. 382 3.4.2 Using ESP Tunnel Mode 384 Note that Destination Options MAY be placed before or after ESP or 385 both in both the outer set of IP headers and the inner set of IP 386 headers. A tunnel endpoint that creates a new packet may decide to 387 use PDM independent of the use of PDM of the original packet to 388 enable delay measurements between the two tunnel endpoints 390 3.5 Implementation Considerations 392 3.5.1 PDM Activation 394 An implementation should provide an interface to enable or disable 395 the use of PDM. This specification recommends having PDM off by 396 default. 398 PDM MUST NOT be turned on merely if a packet is received with a PDM 399 header. The received packet could be spoofed by another device. 401 3.5.2 PDM Timestamps 403 The PDM timestamps are intended to isolate wire time from server or 404 host time, but may necessarily attribute some host processing time to 405 network latency. 407 RFC2330 [RFC2330] "Framework for IP Performance Metrics" describes 408 two notions of wire time in section 10.2. These notions are only 409 defined in terms of an Internet host H observing an Internet link L 410 at a particular location: 412 + For a given IP packet P, the 'wire arrival time' of P at H on L 413 is the first time T at which any bit of P has appeared at H's 414 observational position on L. 416 + For a given IP packet P, the 'wire exit time' of P at H on L is 417 the first time T at which all the bits of P have appeared at H's 418 observational position on L. 420 This specification does not define the exact H's observing position 421 on L. That is left for the deployment setups to define. However, the 422 position where PDM timestamps are taken SHOULD be as close to the 423 physical network interface as possible. Not all implementations will 424 be able to achieve the ideal level of measurement. 426 3.6 Dynamic Configuration Options 428 If the PDM destination options extension header is used, then it MAY 429 be turned on for all packets flowing through the host, applied to an 430 upper-layer protocol (TCP, UDP, SCTP, etc), a local port, or IP 431 address only. These are at the discretion of the implementation. 433 3.7 Information Access and Storage 435 Measurement information provided by PDM may be made accessible for 436 higher layers or the user itself. Similar to activating the use of 437 PDM, the implementation may also provide an interface to indicate if 438 received 440 PDM information may be stored, if desired. If a packet with PDM 441 information is received and the information should be stored, the 442 upper layers may be notified. Furthermore, the implementation should 443 define a configurable maximum lifetime after which the information 444 can be removed as well as a configurable maximum amount of memory 445 that should be allocated for PDM information. 447 4 Security Considerations 449 PDM may introduce some new security weaknesses. 451 4.1 Resource Consumption and Resource Consumption Attacks 453 PDM needs to calculate the deltas for time and keep track of the 454 sequence numbers. This means that control blocks which reside in 455 memory may be kept at the end hosts per 5-tuple. 457 A limit on how much memory is being used SHOULD be implemented. 459 Without a memory limit, any time a control block is kept in memory, 460 an attacker can try to mis-use the control blocks to cause excessive 461 resource consumption. This could be used to compromise the end host. 463 PDM is used only at the end hosts and memory is used only at the end 464 host and not at routers or middle boxes. 466 4.2 Pervasive monitoring 468 Since PDM passes in the clear, a concern arises as to whether the 469 data can be used to fingerprint the system or somehow obtain 470 information about the contents of the payload. 472 Let us discuss fingerprinting of the end host first. It is possible 473 that seeing the pattern of deltas or the absolute values could give 474 some information as to the speed of the end host - that is, if it is 475 a very fast system or an older, slow device. This may be useful to 476 the attacker. However, if the attacker has access to PDM, the 477 attacker also has access to the entire packet and could make such a 478 deduction based merely on the time frames elapsed between packets 479 WITHOUT PDM. 481 As far as deducing the content of the payload, it appears to us that 482 PDM is quite unhelpful in this regard. 484 4.3 PDM as a Covert Channel 486 PDM provides a set of fields in the packet which could be used to 487 leak data. But, there is no real reason to suspect that PDM would 488 be chosen rather than another part of the payload or another 489 Extension Header. 491 A firewall or another device could sanity check the fields within the 492 PDM but randomly assigned sequence numbers and delta times might be 493 expected to vary widely. The biggest problem though is how an 494 attacker would get access to PDM in the first place to leak data. 495 The attacker would have to either compromise the end host or have Man 496 in the Middle (MitM). It is possible that either one could change 497 the fields. But, then the other end host would get sequence numbers 498 and deltas that don't make any sense. 500 It is conceivable that someone could compromise an end host and make 501 it start sending packets with PDM without the knowledge of the host. 502 But, again, the bigger problem is the compromise of the end host. 503 Once that is done, the attacker probably has better ways to leak 504 data. 506 Having said that, if a PDM aware middle box or an implementation 507 detects some number of "nonsensical" sequence numbers it could take 508 action to block (or alert on) this traffic. 510 4.4 Timing Attacks 512 The fact that PDM can help in the separation of node processing time 513 from network latency brings value to performance monitoring. Yet, it 514 is this very characteristic of PDM which may be misused to make 515 certain new type of timing attacks against protocols and 516 implementations possible. 518 Depending on the nature of the cryptographic protocol used, it may be 519 possible to leak the long term credentials of the device. For 520 example, if an attacker is able to create an attack which causes the 521 enterprise to turn on PDM to diagnose the attack, then the attacker 522 might use PDM during that debugging time to launch a timing attack 523 against the long term keying material used by the cryptographic 524 protocol. 526 An implementation may want to be sure that PDM is enabled only for 527 certain ip addresses, or only for some ports. Additionally, the 528 implementation SHOULD require an explicit restart of monitoring after 529 a certain time period (for example for 1 hour), to make sure that PDM 530 is not accidentally left on after debugging has been done etc. 532 Even so, if using PDM, a user "Consent to be Measured" SHOULD be a 533 pre-requisite for using PDM. Consent is common in enterprises and 534 with some subscription services. The actual content of "Consent to 535 be Measured" will differ by site but it SHOULD make clear that the 536 traffic is being measured for quality of service and to assist in 537 diagnostics as well as to make clear that there may be potential 538 risks of certain vulnerabilities if the traffic is captured during a 539 diagnostic session 541 5 IANA Considerations 543 This draft requests an Option Type assignment in the Destination 544 Options and Hop-by-Hop Options sub-registry of Internet Protocol 545 Version 6 (IPv6) Parameters [ref to RFCs and URL below]. 547 http://www.iana.org/assignments/ipv6-parameters/ipv6- 548 parameters.xhtml#ipv6-parameters-2 550 Hex Value Binary Value Description Reference 551 act chg rest 552 ------------------------------------------------------------------- 553 TBD TBD Performance and [This draft] 554 Diagnostic Metrics 555 (PDM) 557 6 References 559 6.1 Normative References 561 [RFC1122] Braden, R., "Requirements for Internet Hosts -- 562 Communication Layers", RFC 1122, October 1989. 564 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 565 Requirement Levels", BCP 14, RFC 2119, March 1997. 567 [RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 568 (IPv6) Specification", RFC 2460, December 1998. 570 [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip 571 Delay Metric for IPPM", RFC 2681, September 1999. 573 [RFC2780] Bradner, S. and V. Paxson, "IANA Allocation Guidelines 574 For Values In the Internet Protocol and Related Headers", BCP 37, RFC 575 2780, March 2000. 577 [RFC4303] Kent, S, "IP Encapsulating Security Payload (ESP)", RFC 578 4303, December 2005. 580 6.2 Informative References 582 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 583 "Framework for IP Performance Metrics", RFC 2330, May 1998. 585 [TRAM-TCPM] Trammel, B., "Encoding of Time Intervals for the TCP 586 Timestamp Option-01", Internet Draft, July 2013. [Work in Progress] 588 Appendix A: Context for PDM 590 A.1 End User Quality of Service (QoS) 592 The timing values in the PDM embedded in the packet will be used to 593 estimate QoS as experienced by an end user device. 595 For many applications, the key user performance indicator is response 596 time. When the end user is an individual, he is generally 597 indifferent to what is happening along the network; what he really 598 cares about is how long it takes to get a response back. But this is 599 not just a matter of individuals' personal convenience. In many 600 cases, rapid response is critical to the business being conducted. 602 Low, reliable and acceptable response times are not just "nice to 603 have". On many networks, the impact can be financial hardship or can 604 endanger human life. In some cities, the emergency police contact 605 system operates over IP; law enforcement, at all levels, use IP 606 networks; transactions on our stock exchanges are settled using IP 607 networks. The critical nature of such activities to our daily lives 608 and financial well-being demand a simple solution to support response 609 time measurements. 611 A.2 Need for a Packet Sequence Number (PSN) 613 While performing network diagnostics of an end-to-end connection, it 614 often becomes necessary to isolate the factors along the network path 615 responsible for problems. Diagnostic data may be collected at 616 multiple places along the path (if possible), or at the source and 617 destination. Then, in post-collection processing, the diagnostic 618 data corresponding to each packet at different observation points 619 must be matched for proper measurements. A sequence number in each 620 packet provides sufficient basis for the matching process. If need 621 be, the timing fields may be used along with the sequence number to 622 ensure uniqueness. 624 This method of data collection along the path is of special use to 625 determine where packet loss or packet corruption is happening. 627 The packet sequence number needs to be unique in the context of the 628 session (5-tuple). 630 A.3 Rationale for Defined Solution 632 One of the important functions of PDM is to allow you to do quickly 633 dispatch the right set of diagnosticians. Within network or server 634 latency, there may be many components. The job of the diagnostician 635 is to rule each one out until the culprit is found. 637 How PDM fits into this diagnostic picture is that PDM will quickly 638 tell you how to escalate. PDM will point to either the network area 639 or the server area. Within the server latency, PDM does not tell 640 you if the bottleneck is in the IP stack or the application or buffer 641 allocation. Within the network latency, PDM does not tell you which 642 of the network segments or middle boxes is at fault. 644 What PDM does tell you is whether the problem is in the network or 645 the server. 647 A.4 Use PDM with Other Headers 649 For diagnostics, one my want to use PDM with other headers (L2, L3, 650 etc). For example, if PDM is used is by a technician (or tool) 651 looking at a packet capture, within the packet capture, they would 652 have available to them the layer 2 header, IP header (v6 or v4), TCP, 653 UCP, ICMP, SCTP or other headers. All information would be looked 654 at together to make sense of the packet flow. The technician or 655 processing tool could analyze, report or ignore the data from PDM, as 656 necessary. 658 For an example of how PDM can help with TCP retransmit problems, 659 please look at Appendix C. 661 Appendix B : Timing Considerations 663 B.1 Timing Differential Calculations 665 The time counter in a CPU is a binary whole number, representing a 666 number of milliseconds (msec), microseconds (usec) or even 667 picoseconds (psec). Representing one of these values as attoseconds 668 (asec) means multiplying by 10 raised to some exponent. Refer to this 669 table of equalities: 671 Base value = # of sec = # of asec 1000s of asec 672 --------------- ------------- ------------- ------------- 673 1 second 1 sec 10**18 asec 1000**6 asec 674 1 millisecond 10**-3 sec 10**15 asec 1000**5 asec 675 1 microsecond 10**-6 sec 10**12 asec 1000**4 asec 676 1 nanosecond 10**-9 sec 10**9 asec 1000**3 asec 677 1 picosecond 10**-12 sec 10**6 asec 1000**2 asec 678 1 femtosecond 10**-15 sec 10**3 asec 1000**1 asec 680 For example, if you have a time differential expressed in 681 microseconds, since each microsecond is 10**12 asec, you would 682 multiply your time value by 10**12 to obtain the number of 683 attoseconds. If you time differential is expressed in nanoseconds, 684 you would multiply by 10**9 to get the number of attoseconds. 686 The result is a binary value that will need to be shortened by a 687 number of bits so it will fit into the 16-bit PDM DELTA field. 689 The next step is to divide by 2 until the value is contained in just 690 16 significant bits. The exponent of the value in the last column of 691 of the table is useful here; the initial scaling factor is that 692 exponent multiplied by 10. This is the minimum number of low-order 693 bits to be shifted-out or discarded. It represents dividing the time 694 value by 1024 raised to that exponent. 696 The resulting value may still be too large to fit into 16 bits, but 697 can be normalized by shifting out more bits (dividing by 2) until the 698 value fits into the 16-bit DELTA field. The number of extra bits 699 shifted out is then added to the scaling factor. The scaling factor, 700 the total number of low-order bits dropped, is the SCALEDTL value. 702 For example: say an application has these start and finish timer 703 values (hexadecimal values, in microseconds): 705 Finish: 27C849234 usec (02:57:58.997556) 706 -Start: 27C83F696 usec (02:57:58.957718) 707 ========== ========= =============== 708 Difference 9B9E usec 00.039838 sec or 39838 usec 710 To convert this differential value to binary attoseconds, multiply 711 the number of microseconds by 10**12. Divide by 1024**4, or simply 712 discard 40 bits from the right. The result is 36232, or 8D88 in hex, 713 with a scaling factor or SCALEDTL value of 40. 715 For another example, presume the time differential is larger, say 716 32.311072 seconds, which is 32311072 usec. Each microsecond is 10**12 717 asec, so multiply by 10**12, giving the hexadecimal value 718 1C067FCCAE8120000. Using the initial scaling factor of 40, drop the 719 last 10 characters (40 bits) from that string, giving 1C067FC. This 720 will not fit into a DELTA field, as it is 25 bits long. Shifting the 721 value to the right another 9 bits results in a DELTA value of E033, 722 with a resulting scaling factor of 49. 724 When the time differential value is a small number, regardless of the 725 time unit, the exponent trick given above is not useful in 726 determining the proper scaling value. For example, if the time 727 differential is 3 seconds and you want to convert that directly, you 728 would follow this path: 730 3 seconds = 3*10**18 asec (decimal) 731 = 29A2241AF62C0000 asec (hexadecimal) 733 If you just truncate the last 60 bits, you end up with a delta value 734 of 2 and a scaling factor of 60, when what you really wanted was a 735 delta value with more significant digits. The most precision with 736 which you can store this value in 16 bits is A688, with a scaling 737 factor of 46. 739 B.2 Considerations of this time-differential representation 741 There are a few considerations to be taken into account with this 742 representation of a time differential. The first is whether there are 743 any limitations on the maximum or minimum time differential that can 744 be expressed using method of a delta value and a scaling factor. The 745 second is the amount of imprecision introduced by this method. 747 B.2.1 Limitations with this encoding method 749 The DELTATLS and DELTATLR fields store only the 16 most-significant 750 bits of the time differential value. Thus the range, excluding the 751 scaling factor, is from 0 to 65535, or a maximum of 2**16-1. This 752 method is further described in [TRAM-TCPM]. 754 The actual magnitude of the time differential is determined by the 755 scaling factor. SCALEDTLR and SCALEDTLS are 8-bit unsigned integers, 756 so the scaling factor ranges from 0 to 255. The smallest number that 757 can be represented would have a value of 1 in the delta field and a 758 value of 0 in the associated scale field. This is the representation 759 for 1 attosecond. Clearly this allows PDM to measure extremely small 760 time differentials. 762 On the other end of the scale, the maximum delta value is 65535, or 763 FFFF in hexadecimal. If the maximum scale value of 255 is used, the 764 time differential represented is 65535*2**255, which is over 3*10**55 765 years, essentially, forever. So there appears to be no real 766 limitation to the time differential that can be represented. 768 B.2.2 Loss of precision induced by timer value truncation 770 As PDM specifies the DELTATLR and DELTATLS values as 16-bit unsigned 771 integers, any time the precision is greater than those 16 bits, there 772 will be truncation of the trailing bits, with an accompanying loss of 773 precision in the value. 775 Any time differential value smaller than 65536 asec can be stored 776 exactly in DELTATLR or DELTATLS, because the representation of this 777 value requires at most 16 bits. 779 Since the time differential values in PDM are measured in 780 attoseconds, the range of values that would be truncated to the same 781 encoded value is 2**(Scale)-1 asec. 783 For example, the smallest time differential that would be truncated 784 to fit into a delta field is 786 1 0000 0000 0000 0000 = 65536 asec 788 This value would be encoded as a delta value of 8000 (hexadecimal) 789 with a scaling factor of 1. The value 791 1 0000 0000 0000 0001 = 65537 asec 793 would also be encoded as a delta value of 8000 with a scaling factor 794 of 1. This actually is the largest value that would be truncated to 795 that same encoded value. When the scale value is 1, the value range 796 is calculated as 2**1 - 1, or 1 asec, which you can see is the 797 difference between these minimum and maximum values. 799 The scaling factor is defined as the number of low-order bits 800 truncated to reduce the size of the resulting value so it fits into a 801 16-bit delta field. If, for example, you had to truncate 12 bits, the 802 loss of precision would depend on the bits you truncated. The range 803 of these values would be 805 0000 0000 0000 = 0 asec 807 to 808 1111 1111 1111 = 4095 asec 810 So the minimum loss of precision would be 0 asec, where the delta 811 value exactly represents the time differential, and the maximum loss 812 of precision would be 4095 asec. As stated above, the scaling factor 813 of 12 means the maximum loss of precision is 2**12-1 asec, or 4095 814 asec. 816 Compare this loss of precision to the actual time differential. The 817 range of actual time differential values that would incur this loss 818 of precision is from 820 1000 0000 0000 0000 0000 0000 0000 = 2**27 asec or 134217728 asec 821 to 822 1111 1111 1111 1111 1111 1111 1111 = 2**28-1 asec or 268435455 asec 824 Granted, these are small values, but the point is, any value between 825 these two values will have a maximum loss of precision of 4095 asec, 826 or about 0.00305% for the first value, as encoded, and at most 827 0.001526% for the second. These maximum-loss percentages are 828 consistent for all scaling values. 830 Appendix C: Sample Packet Flows 832 C.1 PDM Flow - Simple Client Server 834 Following is a sample simple flow for the PDM with one packet sent 835 from Host A and one packet received by Host B. The PDM does not 836 require time synchronization between Host A and Host B. The 837 calculations to derive meaningful metrics for network diagnostics are 838 shown below each packet sent or received. 840 C.1.1 Step 1 842 Packet 1 is sent from Host A to Host B. The time for Host A is set 843 initially to 10:00AM. 845 The time and packet sequence number are saved by the sender 846 internally. The packet sequence number and delta times are sent in 847 the packet. 849 Packet 1 851 +----------+ +----------+ 852 | | | | 853 | Host | ----------> | Host | 854 | A | | B | 855 | | | | 856 +----------+ +----------+ 858 PDM Contents: 860 PSNTP : Packet Sequence Number This Packet: 25 861 PSNLR : Packet Sequence Number Last Received: - 862 DELTATLR : Delta Time Last Received: - 863 SCALEDTLR: Scale of Delta Time Last Received: 0 864 DELTATLS : Delta Time Last Sent: - 865 SCALEDTLS: Scale of Delta Time Last Sent: 0 867 Internally, within the sender, Host A, it must keep: 869 Packet Sequence Number of the last packet sent: 25 870 Time the last packet was sent: 10:00:00 872 Note, the initial PSNTP from Host A starts at a random number. In 873 this case, 25. The time in these examples is shown in seconds for 874 the sake of simplicity. 876 C.1.2 Step 2 878 Packet 1 is received at Host B. Its time is set to one hour later 879 than Host A. In this case, 11:00AM 881 Internally, within the receiver, Host B, it must note: 883 Packet Sequence Number of the last packet received: 25 884 Time the last packet was received : 11:00:03 886 Note, this timestamp is in Host B time. It has nothing whatsoever to 887 do with Host A time. The Packet Sequence Number of the last packet 888 received will become PSNLR which will be sent out in the packet sent 889 by Host B in the next step. The time last received will be used to 890 calculate the DELTALR value to be sent out in the packet sent by Host 891 B in the next step. 893 C.1.3 Step 3 895 Packet 2 is sent by Host B to Host A. Note, the initial packet 896 sequence number (PSNTP) from Host B starts at a random number. In 897 this case, 12. Before sending the packet, Host B does a calculation 898 of deltas. Since Host B knows when it is sending the packet, and it 899 knows when it received the previous packet, it can do the following 900 calculation: 902 Sending time : packet 2 - receive time : packet 1 904 The result of this calculation is called: Delta Time Last Received 905 (DELTATLR) 907 Note, both sending time and receive time are saved internally in Host 908 B. They do not travel in the packet. Only the Delta is in the 909 packet. 911 Assume that within Host B is the following: 913 Packet Sequence Number of the last packet received: 25 914 Time the last packet was received: 11:00:03 915 Packet Sequence Number of this packet: 12 916 Time this packet is being sent: 11:00:07 918 Now a delta value to be sent out in the packet can be calculated. 919 DELTATLR becomes: 921 4 seconds = 11:00:07 - 11:00:03 = 3782DACE9D900000 asec 923 This is the derived metric: Server Delay. The time and scaling 924 factor must be converted; in this case, the time differential is 925 DE0B, and the scaling factor is 2E, or 46 in decimal. Then, these 926 values, along with the packet sequence numbers will be sent to Host A 927 as follows: 929 Packet 2 931 +----------+ +----------+ 932 | | | | 933 | Host | <---------- | Host | 934 | A | | B | 935 | | | | 936 +----------+ +----------+ 938 PDM Contents: 940 PSNTP : Packet Sequence Number This Packet: 12 941 PSNLR : Packet Sequence Number Last Received: 25 942 DELTATLR : Delta Time Last Received: DE0B (4 seconds) 943 SCALEDTLR: Scale of Delta Time Last Received: 2E (46 decimal) 944 DELTATLS : Delta Time Last Sent: - 945 SCALEDTLS: Scale of Delta Time Last Sent: 0 947 The metric left to be calculated is the Round-Trip Delay. This will 948 be calculated by Host A when it receives Packet 2. 950 C.1.4 Step 4 952 Packet 2 is received at Host A. Remember, its time is set to one 953 hour earlier than Host B. Internally, it must note: 955 Packet Sequence Number of the last packet received: 12 956 Time the last packet was received : 10:00:12 958 Note, this timestamp is in Host A time. It has nothing whatsoever to 959 do with Host B time. 961 So, now, Host A can calculate total end-to-end time. That is: 963 End-to-End Time = Time Last Received - Time Last Sent 965 For example, packet 25 was sent by Host A at 10:00:00. Packet 12 was 966 received by Host A at 10:00:12 so: 968 End-to-End time = 10:00:12 - 10:00:00 or 12 (Server and Network RT 969 delay combined). This time may also be called total Overall Round- 970 Trip Time (RTT) which includes Network RTT and Host Response Time. 972 This derived metric we will call Delta Time Last Sent (DELTATLS) 974 Round trip delay can now be calculated. The formula is: 976 Round trip delay = (Delta Time Last Sent - Delta Time Last Received) 977 Or: 979 Round trip delay = 12 - 4 or 8 981 Now, the only problem is that at this point all metrics are in Host A 982 only and not exposed in a packet. To do that, we need a third packet. 984 Note: this simple example assumes one send and one receive. That 985 is done only for purposes of explaining the function of the PDM. In 986 cases where there are multiple packets returned, one would take the 987 time in the last packet in the sequence. The calculations of such 988 timings and intelligent processing is the function of post-processing 989 of the data. 991 C.1.5 Step 5 993 Packet 3 is sent from Host A to Host B. 995 +----------+ +----------+ 996 | | | | 997 | Host | ----------> | Host | 998 | A | | B | 999 | | | | 1000 +----------+ +----------+ 1002 PDM Contents: 1004 PSNTP : Packet Sequence Number This Packet: 26 1005 PSNLR : Packet Sequence Number Last Received: 12 1006 DELTATLR : Delta Time Last Received: 0 1007 SCALEDTLS: Scale of Delta Time Last Received 0 1008 DELTATLS : Delta Time Last Sent: A688 (scaled value) 1009 SCALEDTLR: Scale of Delta Time Last Received: 30 (48 decimal) 1011 To calculate Two-Way Delay, any packet capture device may look at 1012 these packets and do what is necessary. 1014 C.2 Other Flows 1016 What has been discussed so far is a simple flow with one packet sent 1017 and one returned. Let's look at how PDM may be useful in other 1018 types of flows. 1020 C.2.1 PDM Flow - One Way Traffic 1022 The flow on a particular session may not be a send-receive paradigm. 1023 Let us consider some other situations. In the case of a one-way 1024 flow, one might see the following: 1026 Note: The time is expressed in generic units for simplicity. That 1027 is, these values do not represent a number of attoseconds, 1028 microseconds or any other real units of time. 1030 Packet Sender PSN PSN Delta Time Delta Time 1031 This Packet Last Recvd Last Recvd Last Sent 1032 ===================================================================== 1033 1 Server 1 0 0 0 1034 2 Server 2 0 0 5 1035 3 Server 3 0 0 12 1036 4 Server 4 0 0 20 1038 What does this mean and how is it useful? 1040 In a one-way flow, only the Delta Time Last Sent will be seen as 1041 used. Recall, Delta Time Last Sent is the difference between the 1042 send of one packet from a device and the next. This is a measure of 1043 throughput for the sender - according to the sender's point of view. 1044 That is, it is a measure of how fast is the application itself (with 1045 stack time included) able to send packets. 1047 How might this be useful? If one is having a performance issue at 1048 the client and sees that packet 2, for example, is sent after 5 time 1049 units from the server but takes 10 times that long to arrive at the 1050 destination, then one may safely conclude that there are delays in 1051 the path other than at the server which may be causing the delivery 1052 issue of that packet. Such delays may include the network links, 1053 middle-boxes, etc. 1055 Now, true one-way traffic is quite rare. What people often mean by 1056 "one-way" traffic is an application such as FTP where a group of 1057 packets (for example, a TCP window size worth) is sent, then the 1058 sender waits for acknowledgment. This type of flow would actually 1059 fall into the "multiple-send" traffic model. 1061 C.2.2 PDM Flow - Multiple Send Traffic 1063 Assume that two packets are sent for each ACK from the server. For 1064 example, a TCP flow will do this, per RFC1122 [RFC1122] Section- 1065 4.2.3. 1067 Packet Sender PSN PSN Delta Time Delta Time 1068 This Packet Last Recvd Last Recvd Last Sent 1069 ===================================================================== 1070 1 Server 1 0 0 0 1071 2 Server 2 0 0 5 1072 3 Client 1 2 20 0 1073 4 Server 3 1 10 15 1075 How might this be used? 1077 Notice that in packet 3, the client has a value of Delta Time Last 1078 received of 20. Recall that Delta Time Last Received is the Send 1079 time of packet 3 - receive time of packet 2. So, what does one know 1080 now? In this case, Delta Time Last Received is the processing time 1081 for the Client to send the next packet. 1083 How to interpret this depends on what is actually being sent. 1084 Remember, PDM is not being used in isolation, but to supplement the 1085 fields found in other headers. Let's take some examples: 1087 1. Client is sending a standalone TCP ACK. One would find this by 1088 looking at the payload length in the IPv6 header and the TCP 1089 Acknowledgement field in the TCP header. So, in this case, the 1090 client is taking 20 units to send back the ACK. This may or may not 1091 be interesting. 1093 2. Client is sending data with the packet. Again, one would find 1094 this by looking at the payload length in the IPv6 header and the TCP 1095 Acknowledgement field in the TCP header. So, in this case, the 1096 client is taking 20 units to send back data. This may represent 1097 "User Think Time". Again, this may or may not be interesting, in 1098 isolation. But, if there is a performance problem receiving data at 1099 the server, then taken in conjunction with RTT or other packet timing 1100 information, this information may be quite interesting. 1102 Of course, one also needs to look at the PSN Last Received field to 1103 make sure of the interpretation of this data. That is, to make 1104 sure that the Delta Last Received corresponds to the packet of 1105 interest. 1107 The benefits of PDM are that such information is now available in a 1108 uniform manner for all applications and all protocols without 1109 extensive changes required to applications. 1111 C.2.3 PDM Flow - Multiple Send with Errors 1113 Let us now look at a case of how PDM may be able to help in a case of 1114 TCP retransmission and add to the information that is sent in the TCP 1115 header. 1117 Assume that three packets are sent with each send from the server. 1119 From the server, this is what is seen. 1121 Pkt Sender PSN PSN Delta Time Delta Time TCP Data 1122 This Pkt LastRecvd LastRecvd LastSent SEQ Bytes 1123 ===================================================================== 1124 1 Server 1 0 0 0 123 100 1125 2 Server 2 0 0 5 223 100 1126 3 Server 3 0 0 5 333 100 1128 The client, however, does not receive all the packets. From the 1129 client, this is what is seen for the packets sent from the server. 1131 Pkt Sender PSN PSN Delta Time Delta Time TCP Data 1132 This Pkt LastRecvd LastRecvd LastSent SEQ Bytes 1133 ===================================================================== 1134 1 Server 1 0 0 0 123 100 1135 2 Server 3 0 0 5 333 100 1137 Let's assume that the server now retransmits the packet. (Obviously, 1138 a duplicate acknowledgment sequence for fast retransmit or a 1139 retransmit timeout would occur. To illustrate the point, these 1140 packets are being left out.) 1142 So, then if a TCP retransmission is done, then from the client, this 1143 is what is seen for the packets sent from the server. 1145 Pkt Sender PSN PSN Delta Time Delta Time TCP Data 1146 This Pkt LastRecvd LastRecvd LastSent SEQ Bytes 1147 ===================================================================== 1148 1 Server 4 0 0 30 223 100 1150 The server has resent the old packet 2 with TCP sequence number of 1151 223. The retransmitted packet now has a PSN This Packet value of 4. 1153 The Delta Last Sent is 30 - the time between sending the packet with 1154 PSN of 3 and this current packet. 1156 Let's say that packet 4 is lost again. Then, after some amount of 1157 time (RTO) then the packet with TCP sequence number of 223 is resent. 1159 From the client, this is what is seen for the packets sent from the 1160 server. 1162 Pkt Sender PSN PSN Delta Time Delta Time TCP Data 1163 This Pkt LastRecvd LastRecvd LastSent SEQ Bytes 1164 ===================================================================== 1165 1 Server 5 0 0 60 223 100 1167 If now, this packet arrives at the destination, one has a very good 1168 idea that packets exist which are being sent from the server as 1169 retransmissions and not arriving at the client. This is because the 1170 PSN of the resent packet from the server is 5 rather than 4. If we 1171 had used TCP sequence number alone, we would never have seen this 1172 situation. The TCP sequence number in all situations is 223. 1174 This situation would be experienced by the user of the application 1175 (the human being actually sitting somewhere) as a "hangs" or long 1176 delay between packets. On large networks, to diagnose problems such 1177 as these where packets are lost somewhere on the network, one has to 1178 take multiple traces to find out exactly where. 1180 The first thing is to start with doing a trace at the client and the 1181 server. So, we can see if the server sent a particular packet and 1182 the client received it. If the client did not receive it, then we 1183 start tracking back to trace points at the router right after the 1184 server and the router right before the client. Did they get these 1185 packets which the server has sent? This is a time consuming 1186 activity. 1188 With PDM, we can speed up the diagnostic time because we may be able 1189 to use only the trace taken at the client to see what the server is 1190 sending. 1192 Appendix D: Potential Overhead Considerations 1194 One might wonder as to the potential overhead of PDM. First, PDM is 1195 entirely optional. That is, a site may choose to implement PDM or 1196 not as they wish. If they are happy with the costs of PDM vs. the 1197 benefits, then the choice should be theirs. 1199 Below is a table outlining the potential overhead in terms of 1200 additional time to deliver the response to the end user for various 1201 assumed RTTs. 1203 Bytes RTT Bytes Bytes New Overhead 1204 in Packet Per Millisec in PDM RTT 1205 ===================================================================== 1206 1000 1000 milli 1 16 1016.000 16.000 milli 1207 1000 100 milli 10 16 101.600 1.600 milli 1208 1000 10 milli 100 16 10.160 .160 milli 1209 1000 1 milli 1000 16 1.016 .016 milli 1211 Below are some examples of actual RTTs for packets traversing large 1212 enterprise networks. The first example is for packets going to 1213 multiple business partners. 1215 Bytes RTT Bytes Bytes New Overhead 1216 in Packet Per Millisec in PDM RTT 1217 ===================================================================== 1218 1000 17 milli 58 16 17.360 .360 milli 1220 The second example is for packets at a large enterprise customer 1221 within a data center. Notice that the scale is now in microseconds 1222 rather than milliseconds. 1224 Bytes RTT Bytes Bytes New Overhead 1225 in Packet Per Microsec in PDM RTT 1226 ===================================================================== 1227 1000 20 micro 50 16 20.320 .320 micro 1229 As with other diagnostic tools, such as packet traces, a certain 1230 amount of processing time will be required to create and process PDM. 1231 Since PDM is lightweight (has only a few variables), we expect the 1232 processing time to be minimal. 1234 Acknowledgments 1236 The authors would like to thank Keven Haining, Al Morton, Brian 1237 Trammel, David Boyes, Bill Jouris, Richard Scheffenegger, and Rick 1238 Troth for their comments and assistance. We would also like to thank 1239 Tero Kivinen and Jouni Korhonen for their detailed and perceptive 1240 reviews. 1242 Authors' Addresses 1244 Nalini Elkins 1245 Inside Products, Inc. 1246 36A Upper Circle 1247 Carmel Valley, CA 93924 1248 United States 1249 Phone: +1 831 659 8360 1250 Email: nalini.elkins@insidethestack.com 1251 http://www.insidethestack.com 1253 Robert M. Hamilton 1254 Chemical Abstracts Service 1255 A Division of the American Chemical Society 1256 2540 Olentangy River Road 1257 Columbus, Ohio 43202 1258 United States 1259 Phone: +1 614 447 3600 x2517 1260 Email: rhamilton@cas.org 1261 http://www.cas.org 1263 Michael S. Ackermann 1264 Blue Cross Blue Shield of Michigan 1265 P.O. Box 2888 1266 Detroit, Michigan 48231 1267 United States 1268 Phone: +1 310 460 4080 1269 Email: mackermann@bcbsm.com 1270 http://www.bcbsm.com