idnits 2.17.1 draft-ietf-ippm-route-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). (Using the creation date from RFC2330, updated by this document, for RFC5378 checks: 1998-05-01) -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (July 2, 2018) is 2124 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'RFC2460' is defined on line 1004, but no explicit reference was found in the text == Unused Reference: 'RFC2675' is defined on line 1008, but no explicit reference was found in the text == Unused Reference: 'RFC4494' is defined on line 1021, but no explicit reference was found in the text == Unused Reference: 'RFC5644' is defined on line 1041, but no explicit reference was found in the text == Unused Reference: 'RFC6282' is defined on line 1050, but no explicit reference was found in the text == Unused Reference: 'RFC6437' is defined on line 1055, but no explicit reference was found in the text == Unused Reference: 'RFC6564' is defined on line 1060, but no explicit reference was found in the text == Unused Reference: 'RFC7045' is defined on line 1069, but no explicit reference was found in the text == Unused Reference: 'I-D.brockners-inband-oam-data' is defined on line 1106, but no explicit reference was found in the text == Outdated reference: A later version (-17) exists of draft-ietf-ippm-ioam-data-03 ** Downref: Normative reference to an Informational RFC: RFC 2330 ** Obsolete normative reference: RFC 2460 (Obsoleted by RFC 8200) ** Downref: Normative reference to an Informational RFC: RFC 2991 ** Downref: Normative reference to an Informational RFC: RFC 5835 ** Downref: Normative reference to an Informational RFC: RFC 7312 ** Downref: Normative reference to an Informational RFC: RFC 7799 ** Downref: Normative reference to an Experimental RFC: RFC 7820 Summary: 7 errors (**), 0 flaws (~~), 12 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group J. Alvarez-Hamelin 3 Internet-Draft Universidad de Buenos Aires 4 Updates: 2330 (if approved) A. Morton 5 Intended status: Standards Track AT&T Labs 6 Expires: January 3, 2019 J. Fabini 7 TU Wien 8 C. Pignataro 9 Cisco Systems, Inc. 10 July 2, 2018 12 Advanced Unidirectional Route Assessment (AURA) 13 draft-ietf-ippm-route-02 15 Abstract 17 This memo introduces an advanced unidirectional route assessment 18 (AURA) metric and associated measurement methodology, based on the IP 19 Performance Metrics (IPPM) Framework RFC 2330. This memo updates RFC 20 2330 in the areas of path-related terminology and path description, 21 primarily to include the possibility of parallel subpaths between a 22 given Source and Destination pair, owing to the presence of multi- 23 path technologies. 25 Requirements Language 27 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 28 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 29 "OPTIONAL" in this document are to be interpreted as described in BCP 30 14[RFC2119] [RFC8174] when, and only when, they appear in all 31 capitals, as shown here. 33 Status of This Memo 35 This Internet-Draft is submitted in full conformance with the 36 provisions of BCP 78 and BCP 79. 38 Internet-Drafts are working documents of the Internet Engineering 39 Task Force (IETF). Note that other groups may also distribute 40 working documents as Internet-Drafts. The list of current Internet- 41 Drafts is at https://datatracker.ietf.org/drafts/current/. 43 Internet-Drafts are draft documents valid for a maximum of six months 44 and may be updated, replaced, or obsoleted by other documents at any 45 time. It is inappropriate to use Internet-Drafts as reference 46 material or to cite them other than as "work in progress." 48 This Internet-Draft will expire on January 3, 2019. 50 Copyright Notice 52 Copyright (c) 2018 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents 57 (https://trustee.ietf.org/license-info) in effect on the date of 58 publication of this document. Please review these documents 59 carefully, as they describe your rights and restrictions with respect 60 to this document. Code Components extracted from this document must 61 include Simplified BSD License text as described in Section 4.e of 62 the Trust Legal Provisions and are provided without warranty as 63 described in the Simplified BSD License. 65 Table of Contents 67 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 68 1.1. Issues with Earlier Work to define Route . . . . . . . . 3 69 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 70 3. Route Metric Terms and Definitions . . . . . . . . . . . . . 5 71 3.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 6 72 3.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 6 73 3.3. Metric Definitions . . . . . . . . . . . . . . . . . . . 7 74 3.4. Related Round-Trip Delay and Loss Definitions . . . . . . 8 75 3.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . 9 76 3.6. Reporting the Metric . . . . . . . . . . . . . . . . . . 9 77 4. Route Assessment Methodologies . . . . . . . . . . . . . . . 10 78 4.1. Active Methodologies . . . . . . . . . . . . . . . . . . 10 79 4.1.1. Temporal Composition for Route Metrics . . . . . . . 12 80 4.1.2. Routing Class C Identification . . . . . . . . . . . 13 81 4.2. Hybrid Methodologies . . . . . . . . . . . . . . . . . . 14 82 4.3. Combining Different Methods . . . . . . . . . . . . . . . 15 83 5. Background on Round-Trip Delay Measurement Goals . . . . . . 16 84 6. Tools to Measure Delays in the Internet . . . . . . . . . . . 16 85 7. RTD Measurements Statistics . . . . . . . . . . . . . . . . . 18 86 8. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 19 87 9. Security Considerations . . . . . . . . . . . . . . . . . . . 19 88 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 89 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 20 90 12. Appendix I MPLS Methods for Route Assessment . . . . . . . . 20 91 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 92 13.1. Normative References . . . . . . . . . . . . . . . . . . 21 93 13.2. Informative References . . . . . . . . . . . . . . . . . 24 94 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 25 96 1. Introduction 98 The IETF IP Performance Metrics (IPPM) working group first created a 99 framework for metric development in [RFC2330]. This framework has 100 stood the test of time and enabled development of many fundamental 101 metrics. It has been updated in the area of metric composition 102 [RFC5835], and in several areas related to active stream measurement 103 of modern networks with reactive properties [RFC7312]. 105 The [RFC2330] framework motivated the development of "performance and 106 reliability metrics for paths through the Internet," and Section 5 of 107 [RFC2330] defines terms that support description of a path under 108 test. However, metrics for assessment of path components and related 109 performance aspects had not been attempted in IPPM when the [RFC2330] 110 framework was written. 112 This memo takes-up the route measurement challenge and specifies a 113 new route metric, two practical frameworks for methods of measurement 114 (using either active or hybrid active-passive methods [RFC7799]), and 115 round-trip delay and link information discovery using the results of 116 measurements. All route measurements are limited by the willingness 117 of hosts along the path to be discovered, to cooperate with the 118 methods used, or to recognize that the measurement operation is 119 taking place (such as when tunnels are present). 121 1.1. Issues with Earlier Work to define Route 123 Section 7 of [RFC2330] presented a simple example of a "route" metric 124 along with several other examples. The example is reproduced below 125 (where the reference is to Section 5 of [RFC2330]): 127 "route: The path, as defined in Section 5, from A to B at a given 128 time." 130 This example provides a starting point to develop a more complete 131 definition of route. Areas needing clarification include: 133 Time: In practice, the route will be assessed over a time interval, 134 because active path detection methods like [PT] rely on TTL limits 135 for their operation and cannot accomplish discovery of all hosts 136 using a single packet. 138 Type-P: The legacy route definition lacks the option to cater for 139 packet-dependent routing. In this memo, we assess the route for a 140 specific packet of Type-P, and reflect this in the metric 141 definition. The methods of measurement determine the specific 142 Type-P used. 144 Parallel Paths: This a reality of Internet paths and a strength of 145 advanced route assessment methods, so the metric must acknowledge 146 this possibility. Use of Equal Cost Multi-Path (ECMP) and Unequal 147 Cost Multi-Path (UCMP) technologies are common sources of parallel 148 subpaths. 150 Cloud Subpath: May contain hosts that do not decrement TTL or Hop 151 Limit, but may have two or more exchange links connecting 152 "discoverable" hosts or routers. Parallel subpaths contained 153 within clouds cannot be discovered. The assessment methods only 154 discover hosts or routers on the path that decrement TTL or Hop 155 Count, or cooperate with interrogation protocols. The presence of 156 tunnels and nested tunnels further complicate assessment by hiding 157 hops. 159 Hop: Although the [RFC2330] definition was a link-host pair, only 160 hosts are discoverable or have the capability to cooperate with 161 interrogation protocols where link information may be exposed. 163 The refined definition of Route metrics begins in the sections that 164 follow. 166 2. Scope 168 The purpose of this memo is to add new route metrics and methods of 169 measurement to the existing set of IPPM metrics. 171 The scope is to define route metrics that can identify the path taken 172 by a packet or a flow traversing the Internet between two hosts. 173 Although primarily intended for hosts communicating on the Internet 174 with IP, the definitions and metrics are constructed to be applicable 175 to other network domains, if desired. The methods of measurement to 176 assess the path may not be able to discover all hosts comprising the 177 path, but such omissions are often deterministic and explainable 178 sources of error. 180 Also, to specify a framework for active methods of measurement which 181 use the techniques described in [PT] at a minimum, and a framework 182 for hybrid active-passive methods of measurement, such as the Hybrid 183 Type I method [RFC7799] described in 184 [I-D.ietf-ippm-ioam-data](intended only for single administrative 185 domains), which do not rely on ICMP and provide a protocol for 186 explicit interrogation of nodes on a path. Combinations of active 187 methods and hybrid active-passive methods are also in-scope. 189 Further, this memo provides additional analysis of the round-trip 190 delay measurements made possible by the methods, in an effort to 191 discover more details about the path, such as the link technology in 192 use. 194 This memo updates Section 5 of [RFC2330] in the areas of path-related 195 terminology and path description, primarily to include the 196 possibility of parallel subpaths between a given Source and 197 Destination address pair (possibly resulting from Equal Cost Multi- 198 Path (ECMP) and Unequal Cost Multi-Path (UCMP) technologies). 200 There are several simple non-goals of this memo. There is no attempt 201 to assess the reverse path from any host on the path to the host 202 attempting the path measurement. The reverse path contribution to 203 delay will be that experienced by ICMP packets (in active methods), 204 and may be different from delays experienced by UDP or TCP packets. 205 Also, the round trip delay will include an unknown contribution of 206 processing time at the host that generates the ICMP response. 207 Therefore, the ICMP-based active methods are not supposed to yield 208 accurate, reproducible estimations of the round-trip delay that UDP 209 or TCP packets will experience. 211 3. Route Metric Terms and Definitions 213 This section sets requirements for the following components to 214 support the Route Metric: 216 Host Identity The unique address for hosts communicating within the 217 network domain. For hosts communicating on the Internet with IP, 218 it is the globally routable IP address(es) which the host uses 219 when communicating with other hosts under normal or error 220 conditions. The Host Identity revealed (and its connection to a 221 Host Name through reverse DNS) determines whether interfaces to 222 parallel links can be associated with a single host, or appear to 223 identify unique hosts. 225 Discoverable Host Hosts that convey their Host Identity according to 226 the requirements of their network domain, such as when error 227 conditions are detected by that host. For hosts communicating 228 with IP packets, compliance with Section 3.2.2.4 of [RFC1122] when 229 discarding a packet due to TTL or Hop Limit Exceeded condition, 230 MUST result in sending the corresponding Time Exceeded message 231 (containing a form of host identity) to the source. This 232 requirement is also consistent with section 5.3.1 of [RFC1812] for 233 routers. 235 Cooperating Host Hosts MUST respond to direct queries for their host 236 identity as part of a previously agreed and established 237 interrogation protocol. Hosts SHOULD also provide information 238 such as arrival/departure interface identification, arrival 239 timestamp, and any relevant information about the host or specific 240 link which delivered the query to the host. 242 Hop A Hop MUST contain a Host Identity, and MAY contain arrival and/ 243 or departure interface identification. 245 3.1. Formal Name 247 Type-P-Route-Ensemble-Method-Variant, abbreviated as Route Ensemble. 249 Note that Type-P depends heavily on the chosen method and variant. 251 3.2. Parameters 253 This section lists the REQUIRED input factors to specify a Route 254 metric. 256 o Src, the address of a host (such as the globally routable IP 257 address). 259 o Dst, the address of a host (such as the globally routable IP 260 address). 262 o i, the limit on the number of Hops a specific packet may visit as 263 it traverses from the host at Src to the host at Dst (such as the 264 TTL or Hop Limit). 266 o MaxHops, the maximum value of i used, (i=1,2,3,...MaxHops). 268 o T0, a time (start of measurement interval) 270 o Tf, a time (end of measurement interval) 272 o T, the host time of a packet as measured at MP(Src), meaning 273 Measurement Point at the Source. 275 o Ta, the host time of a reply packet's *arrival* as measured at 276 MP(Src), assigned to packets that arrive within a "reasonable" 277 time (see parameter below). 279 o Tmax, a maximum waiting time for reply packets to return to the 280 source, set sufficiently long to disambiguate packets with long 281 delays from packets that are discarded (lost), such that the 282 distribution of round-trip delay is not truncated. 284 o F, the number of different flows simulated by the method and 285 variant. 287 o flow, the stream of packets with the same n-tuple of designated 288 header fields that (when held constant) result in identical 289 treatment in a multi-path decision (such as the decision taken in 290 load balancing). 292 o Type-P, the complete description of the packets for which this 293 assessment applies (including the flow-defining fields). 295 3.3. Metric Definitions 297 This section defines the REQUIRED measurement components of the Route 298 metrics (unless otherwise indicated): 300 M, the total number of packets sent between T0 and Tf. 302 N, the smallest value of i needed for a packet to be received at Dst 303 (sent between T0 and Tf). 305 Nmax, the largest value of i needed for a packet to be received at 306 Dst (sent between T0 and Tf). Nmax may be equal to N. 308 Next, define a *singleton* definition for a Hop on the path, with 309 sufficient indexes to identify all Hops identified in a measurement 310 interval. 312 A Hop, designated h(i,j), the IP address and/or identity of one of j 313 Discoverable Hosts (or Cooperating Hosts) that are i hops away from 314 the host with address = Src during the measurement interval, T0 to 315 Tf. As defined above, a Hop singleton measurement MUST contain a 316 Host Identity, hid(i,j), and MAY contain one or more of the following 317 attributes: 319 o a(i,j) Arrival Interface ID 321 o d(i,j) Departure Interface ID 323 o t(i,j) Arrival Timestamp (where t(i,j) is ideally supplied by the 324 hop, or approximated from the sending time of the packet that 325 revealed the hop) 327 o Measurements of Round Trip Delay (for each packet that reveals the 328 same Host Identity and attributes, but not timestamp of course, 329 see next section) 331 Now that Host Identities and related information can be positioned 332 according to their distance from the host with address Src in hops, 333 we introduce two forms of Routes: 335 A Route Ensemble is defined as the combination of all routes 336 traversed by different flows from the host at Src address to the host 337 at Dst address. The route traversed by each flow (with addresses Src 338 and Dst, and other fields which constitute flow criteria) is a member 339 of the ensemble and called a Member Route. 341 Using h(i,j) and components and parameters, further define: 343 A Member Route is an ordered graph {h(1,j), ... h(Nj, j)} in the 344 context of a single flow, where h(i-1, j) and h(i, j) are by 1 hop 345 away from each other and Nj=Dst is the minimum count of hops needed 346 by the packet on Member Route j to reach Dst. Member Routes must be 347 unique. The uniqueness property requires that any two Member routes 348 j and k that are part of the same Route Ensemble differ either in 349 terms of minimum hop count Nj and Nk to reach the destination Dst, 350 or, in the case of identical hop count Nj=Nk, they have at least one 351 distinct hop: h(i,j) != h(i, k) for at least one i (i=1..Nj). 353 The Route Ensemble from Src to Dst, during the measurement interval 354 T0 to Tf, is the aggregate of all m distinct Member Routes discovered 355 between the two hosts with Src and Dst addresses. More formally, 356 with the host having address Src omitted: 358 Route Ensemble = { 359 {h(1,1), h(2,1), h(3,1), ... h(N1,1)=Dst}, 360 {h(1,2), h(2,2), h(3,2),..., h(N2,2)=Dst}, 361 ... 362 {h(1,m), h(2,m), h(3,m), ....h(Nm,m)=Dst} 363 } 365 where the following conditions apply: i <= Nj <= Nmax (j=1..m) 367 Note that some h(i,j) may be empty (null) in the case that systems do 368 not reply (not discoverable, or not cooperating). 370 h(i-1,j) and h(i,j) are the Hops on the same Member Route one hop 371 away from each other. 373 Hop h(i,j) may be identical with h(k,l) for i!=k and j!=l ; which 374 means there may be portions shared among different Member Routes 375 (parts of various routes may overlap). 377 3.4. Related Round-Trip Delay and Loss Definitions 379 RTD(i,j,T) is defined as a singleton of the [RFC2681] Round-trip 380 Delay between the host with address = Src and the host at Hop h(i,j) 381 at time T. 383 RTL(i,j,T) is defined as a singleton of the [RFC6673] Round-trip Loss 384 between the host with address = Src and the host at Hop h(i,j) at 385 time T. 387 3.5. Discussion 389 Depending on the way that Host Identity is revealed, it may be 390 difficult to determine parallel subpaths between the same pair of 391 hosts (i.e. multiple parallel links). It is easier to detect 392 parallel subpaths involving different hosts. 394 o If a pair of discovered hosts identify two different addresses, 395 then they will appear to be different hosts. 397 o If a pair of discovered hosts identify two different IP addresses, 398 and the IP addresses resolve to the same host name (in the DNS), 399 then they will appear to be the same hosts. 401 o If a discovered host always replies using the same network 402 address, regardless of the interface a packet arrives on, then 403 multiple parallel links cannot be detected in that network domain. 405 o If parallel links between routers are aggregated below the IP 406 layer, In other words, all links share the same pair of IP 407 addresses, then the existence of these parallel links can't be 408 detected at IP layer. This applies to other network domains with 409 layers below them, as well. 411 @@@@ This paragraph on Temporal Composition moved to support a more 412 complete section on Methodology (section 4). 414 When a route assessment employs IP packets (for example), the reality 415 of flow assignment to parallel subpaths involves layers above IP. 416 Thus, the measured Route Ensemble is applicable to IP and higher 417 layers (as described in the methodology's packet of Type-P and flow 418 parameters). 420 @@@@ The Temporal Measurement and Route Class C (unrelated to address 421 classes of the past) is now partly addressed in Section 4. 423 3.6. Reporting the Metric 425 @@@@ now partly addressed, based on feedback at IETF-101: 427 An Information Model and an XML Data Model for Storing Traceroute 428 Measurements is available in [RFC5388]. The measured information at 429 each hop includes four pieces of information: a one-dimensional hop 430 index, host symbolic address, host IP address, and RTD for each 431 response. 433 The description of Hop information that may be collected according to 434 this memo covers more dimensions, as defined in Section 3.3 above. 435 For example, the Hop index is two-dimensional to capture the 436 complexity of a Route Ensemble, and it contains corresponding host 437 identities at a minimum. The models need to be expanded to include 438 these features, as well as Arrival Interface ID, Departure Interface 439 ID, and Arrival Timestamp, when available. 441 @@@@ can we leave updates to RFC 5388 for further work? Or, do we 442 need to take-on this topic in an Appendix here? 444 4. Route Assessment Methodologies 446 There are two classes of methods described in this section, active 447 methods relying on the reaction to TTL or Hop Limit Exceeded 448 condition to discover hosts on a path, and Hybrid active-passive 449 methods that involve direct interrogation of cooperating hosts 450 (usually within a single domain). Description of these methods 451 follow. 453 @@@@ Editor's Note: We need to incorporate description of Type-P 454 packets (with the flow parameters) used in each method below (done 455 for Active). 457 4.1. Active Methodologies 459 We have chosen to describe the method based on that employed in 460 current open source tools, thereby providing a practical framework 461 for further advanced techniques to be included as method variants. 462 This method is applicable to use across multiple administrative 463 domains. 465 Paris-traceroute [PT] provides some measure of protection from path 466 variation generated by ECMP load balancing, and it ensures traceroute 467 packets will follow the same path in 98% of cases according to 468 [SCAMPER]. If it is necessary to find every path possible between 469 two hosts, Paris-traceroute provides "exhaustive" mode while scamper 470 provides "tracelb" (stands for traceroute load balance). 472 The Type-P of packets used could be ICMP (as in the original 473 traceroute), UDP or TCP. The later are used when a particular 474 characteristic needs to be to verified, such as filtering or traffic 475 shaping on specific ports (i.e., services). [SCAMPER] supports IPv6 476 traceroute measurements, keeping the FlowLable constant in all 477 packets. 479 The advanced route assessment methods used in Paris-traceroute [PT] 480 keep the critical fields constant for every packet to maintain the 481 appearance of the same flow. Since route assessment can be conducted 482 using TCP, UDP or ICMP packets, this method REQUIRES the Diffserv 483 field, the protocol number, IP source and destination addresses, and 484 the port settings for TCP or UDP kept constant. For ICMP probes, the 485 method additionally REQUIRES keeping the type, code, and ICMP 486 checksum constant; which occupy the corresponding positions in the 487 header of an IP packet, e.g., bytes 20 to 23 when the header IP has 488 no options. 490 Maintaining a constant checksum in ICMP is most challenging because 491 the ICMP Sequence Number is part of the calculation. The advanced 492 traceroute method requires calculations using the IP Sequence Number 493 Field and the Identifier Field, yielding a constant ICMP checksum in 494 successive packets. For an example of calculations to maintain a 495 constant checksum, see Appendix A of [RFC7820], where revision of a 496 timestamp field is complemented by modifying the 2 octet checksum 497 complement field (these fields take the roles of the ICMP Sequence 498 Number and Identifier Fields, respectively). 500 For TCP and UDP packets, the checksum must also be kept constant. 501 Therefore, the first four bytes of UDP (or TCP) data field are 502 modified to compensate for fields that change from packet to packet. 504 @@@@ Note: other variants of advanced traceroute are planned be 505 described. 507 Finally, the return path is also important to check. Taking into 508 account that it is an ICMP time exceeded (during transit) packet, the 509 source and destination IP are constant for every reply. Then, we 510 should consider the fields in the first 32 bits of the protocol on 511 the top of IP: the type and code of ICMP packet, and its checksum. 512 Again, to maintain the ICMP checksum constant for the returning 513 packets, we need to consider the whole ICMP message. It contains the 514 IP header of the discarded packet plus the first 8 bytes of the IP 515 payload; that is some of the fields of TCP header, the UDP header 516 plus four data bytes, the ICMP header plus four bytes. Therefore, 517 for UDP case the data field is used to maintain the ICMP checksum 518 constant in the returning packet. For the ICMP case, the identifier 519 and sequence fields of the sent ICMP probe are manipulated to be 520 constant. The TCP case presents no problem because its first eight 521 bytes will be the same for every packet probe. 523 Formally, to maintain the same flow in the measurements to a certain 524 hop, the Type-P-Route-Ensemble-Method-Variant packets should be[PT]: 526 o TCP case: Fields Src, Dst, port-Src, port_Dst, and Diffserv Field 527 should be the same. 529 o UDP case: Fields Src, Dst, port-Src, port-Dst, and Diffserv Field 530 should be the same, the UDP-checksum should change to maintain 531 constant the IP checksum of the ICMP time exceeded reply. Then, 532 the data length should be fixed, and the data field is used to 533 fixing it (consider that ICMP checksum uses its data field, which 534 contains the original IP header plus 8 bytes of UDP, where TTL, IP 535 identification, IP checksum, and UDP checksum changes). 537 o ICMP case: The Data field should compensate variations on TTL, IP 538 identification, and IP checksum for every packet. 540 Then, the way to identify different hops and attempts of the same 541 flow is: 543 o TCP case: The IP identification field. 545 o UDP case: The IP identification field. 547 o ICMP case: The IP identification field, and ICMP Sequence number. 549 4.1.1. Temporal Composition for Route Metrics 551 The Active Route Assessment Methods described above have the ability 552 to discover portions of a path where ECMP load balancing is present, 553 observed as two or more unique Member Routes having one or more 554 distinct Hops which are part of the Route Ensemble. Likewise, 555 attempts to deliberately vary the flow characteristics to discover 556 all Member Routes will reveal portions of the path which are flow- 557 invariant. 559 Section 9.2 of [RFC2330] describes Temporal Composition of metrics, 560 and introduces the possibility of a relationship between earlier 561 measurement results and the results for measurement at the current 562 time (for a given metric). There is value in establishing a Temporal 563 Composition relationship for Route Metrics. However, this 564 relationship does not represent a forecast of future route conditions 565 in any way. 567 For Route Metric measurements, the value of Temporal Composition is 568 to reduce the measurement iterations required with repeated 569 measurements. Reduced iterations are possible by inferring that 570 current measurements using fixed and previously measured flow 571 characteristics: 573 o will have many common hops with previous measurements. 575 o will have relatively time-stable results at the ingress and egress 576 portions of the path when measured from user locations, as opposed 577 to measurements of backbone networks and across inter-domain 578 gateways. 580 o may have greater potential for time-variation in path portions 581 where ECMP load balancing is observed (because increasing or 582 decreasing the pool of links changes the hash calculations). 584 Optionally, measurement systems may take advantage of the inferences 585 above when seeking to reduce measurement iterations, after exhaustive 586 measurements indicate that the time-stable properties are present. 587 Repetitive Active Route measurement systems: 589 1. SHOULD occasionally check path portions which have exhibited 590 stable results over time, particularly ingress and egress 591 portions of the path. 593 2. SHOULD continue testing portions of the path that have previously 594 exhibited ECMP load balancing. 596 3. SHALL trigger re-assessment of the complete path and Route 597 Ensemble, if any change in hops is observed for a specific (and 598 previously tested) flow. 600 @@@@ Comments on this new material are very welcome! 602 4.1.2. Routing Class C Identification 604 There is an opportunity to apply the [RFC2330] notion of equal 605 treatment for a class of packets, "...very useful to know if a given 606 Internet component treats equally a class C of different types of 607 packets", as it applies to Route measurements. Knowledge of "class 608 C" parameters (unrelated to address classes of the past) on a path 609 potentially reduces the number of flows required for a given method 610 to assess a Route Ensemble over time. 612 First, recognize that each Member Route of a Route Ensemble will have 613 a corresponding Routing Class C. Class C can be discovered by 614 testing with multiple flows, all of which traverse the unique set of 615 hops that comprise a specific Member Route. 617 Second, recognize that the different Routing Classes depend primarily 618 on the hash functions used at each instance of ECMP load balancing on 619 the path. 621 Third, recognize the synergy with Temporal Composition methods 622 (described above) where evaluation intends to discover time-stable 623 portions of each Member Route so that more emphasis can be placed on 624 ECMP portions that also determine Class C. 626 The methods to assess the various Routing Class C characteristics 627 benefit from the following measurement capabilities: 629 o flows designed to determine which n-tuple header fields are 630 considered by a given hash function and ECMP hop on the path, and 631 which are not. This operation immediately narrows the search 632 space, where possible, and partially defines a Routing Class C. 634 o a priori knowledge of the possible types of hash functions in use 635 also helps to design the flows for testing (major router vendors 636 publish information about these hash functions, examples are here 637 https://www.researchgate.net/ 638 publication/281571413_COMPARISON_OF_HASH_STRATEGIES_FOR_FLOW- 639 BASED_LOAD_BALANCING ). 641 o ability to direct the emphasis of current measurements on ECMP 642 portions of the path, based on recent past measurement results 643 (the Class C of some portions of the path is essentially "all 644 packets"). 646 @@@@ Comments on this new material are very welcome! Especially 647 suggestions for tools that might lend themselves to support these 648 measurements. 650 4.2. Hybrid Methodologies 652 The Hybrid Type I methods provide an alternative method for Route 653 Member assessment. As mentioned in the Scope section, 654 [I-D.ietf-ippm-ioam-data] provides a possible set of data fields that 655 would support route identification. 657 In general, nodes in the measured domain would be equipped with 658 specific abilities: 660 1. The ingress node adds one or more fields to the measurement 661 packets, and identifies to other nodes in the domain that a route 662 assessment will be conducted using one or more specific packets. 663 The packets typically originate from a host outside the domain, 664 and constitute normal traffic on the domain. 666 2. Each node visited by the specific packet within in the domain 667 identifies itself in a data field of the packet (the field has 668 been added for this purpose). 670 3. When a measurement packet reaches the edge node of the domain, 671 the edge node adds its identity to the list, removes all the 672 identities from the packet, forwards the packet onward, and 673 communicates the ordered list of node identities to the intended 674 receiver. 676 In addition to node identity, nodes may also identify the ingress and 677 egress interfaces utilized by the tracing packet, the time of day 678 when the packet was processed, and other generic data (as described 679 in section 4 of [I-D.ietf-ippm-ioam-data]). 681 4.3. Combining Different Methods 683 In principle, there are advantages if the entity conducting Route 684 measurements can utilize both forms of advanced methods (active and 685 hybrid), and combine the results. For example, if there are hosts 686 involved in the path that qualify as Cooperating Hosts, but not as 687 Discoverable Hosts, then a more complete view of hops on the path is 688 possible when a hybrid method (or interrogation protocol) is applied 689 and the results are combined with the active method results collected 690 across all other domains. 692 In order to combine the results of active and hybrid/interrogation 693 methods, the network hosts that are part of a domain supporting an 694 interrogation protocol have the following attributes: 696 1. Hosts at the ingress to the domain SHOULD be both Discoverable 697 and Cooperating, and SHOULD reveal the same Host Identity in 698 response to both active and hybrid methods. 700 2. Any Hosts within the domain that are both Discoverable and 701 Cooperating SHOULD reveal the same Host Identity in response to 702 both active and hybrid methods. 704 3. Hosts at the egress to the domain SHOULD be both Discoverable and 705 Cooperating, and SHOULD reveal the same Host Identity in response 706 to both active and hybrid methods. 708 When Hosts follow these requirements, it becomes a simple matter to 709 match single domain measurements with the overlapping results from a 710 multidomain measurement. 712 In practice, Internet users do not typically have the ability to 713 utilize the OAM capabilities of networks that their packets traverse, 714 so the results from a remote domain supporting an interrogation 715 protocol would not normally be accessible. However, a network 716 operator could combine interrogation results from their access domain 717 with other measurements revealing the path outside their domain. 719 5. Background on Round-Trip Delay Measurement Goals 721 The aim of this method is to use packet probes to unveil the paths 722 between any two end-hosts of the network. Moreover, information 723 derived from RTD measurements might be meaningful to identify: 725 1. Intercontinental submarine links 727 2. Satellite communications 729 3. Congestion 731 4. Inter-domain paths 733 This categorization is widely accepted in the literature and among 734 operators alike, and it can be trusted with empirical data and 735 several sources as ground of truth (e.g., [RTTSub] [bdrmap][IDCong]). 737 The first two categories correspond to the physical distance 738 dependency on Round Trip Delay (RTD) while the last one binds RTD 739 with queueing delay on routers. Due to the significant contribution 740 of propagation delay in long distance hops, RTD will be at least 741 100ms on transatlantic hops, depending on the geolocation of the 742 vantage points. Moreover, RTD is typically greater than 480ms when 743 two hops are connected using geostationary satellite technology 744 (i.e., their orbit is at 36000km). Detecting congestion with latency 745 implies deeper mathematical understanding since network traffic load 746 is not stationary. Nonetheless, as the first approach, a link seems 747 to be congested if after sending several traceroute probes, it is 748 possible to detect congestion observing different statistics 749 parameters (e.g., see [IDCong]). 751 6. Tools to Measure Delays in the Internet 753 Internet routing is complex because it depends on the policies of 754 thousands Autonomous Systems (AS). While most of the routers perform 755 load balancing on flows using Equal Cost Multiple Path (ECMP), a few 756 still divide the workload through packet-based techniques. The 757 former scenario is defined according to [RFC2991] while the latter 758 generates a round-robin scheme to deliver every new outgoing packet. 759 ECMP keeps flow state in the router to ensure every packet of a flow 760 is delivered by the same path, and this avoids increasing the packet 761 delay variation and possibly producing overwhelming packet reordering 762 in TCP flows. 764 Taking into account that Internet protocol was designed under the 765 "end-to-end" principle, the IP payload and its header do not provide 766 any information about the routes or path necessary to reach some 767 destination. For this reason, the well-known tool traceroute was 768 developed to gather the IP addresses of each hop along a path using 769 the ICMP protocol [RFC0792]. Besides, traceroute adds the measured 770 RTD from each hop. However, the growing complexity of the Internet 771 makes it more challenging to develop accurate traceroute 772 implementation. For instance, the early traceroute tools would be 773 inaccurate in the current network, mainly because they were not 774 designed to retain flow state. However, evolved traceroute tools, 775 such as Paris-traceroute [PT] [MLB] and Scamper [SCAMPER], expect to 776 encounter ECMP and achieve more accurate results when they do. 778 Paris-traceroute-like tools operate in the following way: every 779 packet should follow the same path because the sensitive fields of 780 the header are controlled to appear as the same flow. This means 781 that source and destination IP addresses, source and destination port 782 numbers are the same in every packet. Additionally, Differentiated 783 Services Code Point (DSCP), checksum and ICMP code should remain 784 constant since they may affect the path selection. 786 Today's traceroute tools can send either UDP, TCP or ICMP packet 787 probes. Since ICMP header does not include transport layer 788 information, there are no fields for source and destination port 789 numbers. For this reason, these tools keep constant ICMP type, code, 790 and checksum fields to generate a kind of flow. However, the 791 checksum may vary in every packet, therefore when probes use ICMP 792 packets, ICMP Identifier and Sequence Number are manipulated to 793 maintain constant checksum in every packet. On the other hand, when 794 UDP probes are generated, the expected variation in the checksum of 795 each packet is again compensated by manipulating the payload. 797 Paris-traceroute allows its users to measure RTD in every hop of the 798 path for a particular flow. Furthermore, either Paris-traceroute or 799 Scamper is capable of unveiling the many available paths between a 800 source and destination (which are visible to this method). This task 801 is accomplished by repeating complete traceroute measurements with 802 different flow parameters for each measurement. The Framework for IP 803 Performance Metrics (IPPM) ([RFC2330] updated by[RFC7312]) has the 804 flexibility to require that the round-trip delay measurement 805 [RFC2681] uses packets with the constraints to assure that all 806 packets in a single measurement appear as the same flow. This 807 flexibility covers ICMP, UDP, and TCP. The accompanying methodology 808 of [RFC2681] needs to be expanded to report the sequential hop 809 identifiers along with RTD measurements, but no new metric definition 810 is needed. 812 7. RTD Measurements Statistics 814 Several articles have shown that network traffic presents a self- 815 similar nature [SSNT] [MLRM] which is accountable for filling the 816 queues of the routers. Moreover, router queues are designed to 817 handle traffic bursts, which is one of the most remarkable features 818 of self-similarity. Naturally, while queue length increases, the 819 delay to traverse the queue increases as well and leads to an 820 increase on RTD. Due to traffic bursts generate short-term overflow 821 on buffers (spiky patterns), every RTD only depicts the queueing 822 status on the instant when that packet probe was in transit. For 823 this reason, several RTD measurements during a time window could 824 begin to describe the random behavior of latency. Loss must also be 825 accounted for in the methodology. 827 To understand the ongoing process, examining the quartiles provides a 828 non-parametric way of analysis. Quartiles are defined by five 829 values: minimum RTD (m), RTD value of the 25% of the Empirical 830 Cumulative Distribution Function (ECDF) (Q1), the median value (Q2), 831 the RTD value of the 75% of the ECDF (Q3) and the maximum RTD (M). 832 Congestion can be inferred when RTD measurements are spread apart, 833 and consequently, the Inter-Quartile Range (IQR), the distance 834 between Q3 and Q1, increases its value. 836 This procedure requires to compute quartile values "on the fly" using 837 the algorithm presented in [P2]. 839 This procedure allow us to update the quartiles value whenever a new 840 measurement arrives, which is radically different from classic 841 methods of computing quartiles because they need to use the whole 842 dataset to compute the values. This way of calculus provides savings 843 in memory and computing time. 845 To sum up, the proposed measurement procedure consists in performing 846 traceroutes several times to obtain samples of the RTD in every hop 847 from a path, during a time window (W) and compute the quantiles for 848 every hop. This could be done for a single path flow or for every 849 detected path flow. 851 Even though a particular hop may be understood as the amount of hops 852 away from the source, a more detailed classification could be used. 853 For example, a possible classification may be identify ICMP Time 854 Exceeded packets coming from the same routers to those who have the 855 same hop distance, IP address of the router which is replying and TTL 856 value of the received ICMP packet. 858 Thus, the proposed methodology is based on this algorithm: 860 ================================================================ 861 1 input: W (window time of the measurement) 862 2 i_t (time between two measurements) 863 3 E (True: exhaustive, False: a single path) 864 4 Dst (destination IP address) 865 5 output: Qs (quartiles for every hop and alt in the path(s) to Dst) 866 ---------------------------------------------------------------- 867 6 T . 985 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - 986 Communication Layers", STD 3, RFC 1122, 987 DOI 10.17487/RFC1122, October 1989, 988 . 990 [RFC1812] Baker, F., Ed., "Requirements for IP Version 4 Routers", 991 RFC 1812, DOI 10.17487/RFC1812, June 1995, 992 . 994 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 995 Requirement Levels", BCP 14, RFC 2119, 996 DOI 10.17487/RFC2119, March 1997, 997 . 999 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 1000 "Framework for IP Performance Metrics", RFC 2330, 1001 DOI 10.17487/RFC2330, May 1998, 1002 . 1004 [RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 1005 (IPv6) Specification", RFC 2460, DOI 10.17487/RFC2460, 1006 December 1998, . 1008 [RFC2675] Borman, D., Deering, S., and R. Hinden, "IPv6 Jumbograms", 1009 RFC 2675, DOI 10.17487/RFC2675, August 1999, 1010 . 1012 [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip 1013 Delay Metric for IPPM", RFC 2681, DOI 10.17487/RFC2681, 1014 September 1999, . 1016 [RFC2991] Thaler, D. and C. Hopps, "Multipath Issues in Unicast and 1017 Multicast Next-Hop Selection", RFC 2991, 1018 DOI 10.17487/RFC2991, November 2000, 1019 . 1021 [RFC4494] Song, JH., Poovendran, R., and J. Lee, "The AES-CMAC-96 1022 Algorithm and Its Use with IPsec", RFC 4494, 1023 DOI 10.17487/RFC4494, June 2006, 1024 . 1026 [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. 1027 Zekauskas, "A One-way Active Measurement Protocol 1028 (OWAMP)", RFC 4656, DOI 10.17487/RFC4656, September 2006, 1029 . 1031 [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. 1032 Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", 1033 RFC 5357, DOI 10.17487/RFC5357, October 2008, 1034 . 1036 [RFC5388] Niccolini, S., Tartarelli, S., Quittek, J., Dietz, T., and 1037 M. Swany, "Information Model and XML Data Model for 1038 Traceroute Measurements", RFC 5388, DOI 10.17487/RFC5388, 1039 December 2008, . 1041 [RFC5644] Stephan, E., Liang, L., and A. Morton, "IP Performance 1042 Metrics (IPPM): Spatial and Multicast", RFC 5644, 1043 DOI 10.17487/RFC5644, October 2009, 1044 . 1046 [RFC5835] Morton, A., Ed. and S. Van den Berghe, Ed., "Framework for 1047 Metric Composition", RFC 5835, DOI 10.17487/RFC5835, April 1048 2010, . 1050 [RFC6282] Hui, J., Ed. and P. Thubert, "Compression Format for IPv6 1051 Datagrams over IEEE 802.15.4-Based Networks", RFC 6282, 1052 DOI 10.17487/RFC6282, September 2011, 1053 . 1055 [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, 1056 "IPv6 Flow Label Specification", RFC 6437, 1057 DOI 10.17487/RFC6437, November 2011, 1058 . 1060 [RFC6564] Krishnan, S., Woodyatt, J., Kline, E., Hoagland, J., and 1061 M. Bhatia, "A Uniform Format for IPv6 Extension Headers", 1062 RFC 6564, DOI 10.17487/RFC6564, April 2012, 1063 . 1065 [RFC6673] Morton, A., "Round-Trip Packet Loss Metrics", RFC 6673, 1066 DOI 10.17487/RFC6673, August 2012, 1067 . 1069 [RFC7045] Carpenter, B. and S. Jiang, "Transmission and Processing 1070 of IPv6 Extension Headers", RFC 7045, 1071 DOI 10.17487/RFC7045, December 2013, 1072 . 1074 [RFC7312] Fabini, J. and A. Morton, "Advanced Stream and Sampling 1075 Framework for IP Performance Metrics (IPPM)", RFC 7312, 1076 DOI 10.17487/RFC7312, August 2014, 1077 . 1079 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1080 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1081 May 2016, . 1083 [RFC7820] Mizrahi, T., "UDP Checksum Complement in the One-Way 1084 Active Measurement Protocol (OWAMP) and Two-Way Active 1085 Measurement Protocol (TWAMP)", RFC 7820, 1086 DOI 10.17487/RFC7820, March 2016, 1087 . 1089 [RFC8029] Kompella, K., Swallow, G., Pignataro, C., Ed., Kumar, N., 1090 Aldrin, S., and M. Chen, "Detecting Multiprotocol Label 1091 Switched (MPLS) Data-Plane Failures", RFC 8029, 1092 DOI 10.17487/RFC8029, March 2017, 1093 . 1095 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1096 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1097 May 2017, . 1099 13.2. Informative References 1101 [bdrmap] Luckie, M., Dhamdhere, A., Huffaker, B., Clark, D., and 1102 KC. Claffy, "bdrmap: Inference of Borders Between IP 1103 Networks", In Proceedings of the 2016 ACM on Internet 1104 Measurement Conference, pp. 381-396. ACM, 2016. 1106 [I-D.brockners-inband-oam-data] 1107 Brockners, F., Bhandari, S., Pignataro, C., Gredler, H., 1108 Leddy, J., Youell, S., Mizrahi, T., Mozes, D., Lapukhov, 1109 P., Chang, R., and d. daniel.bernier@bell.ca, "Data Fields 1110 for In-situ OAM", draft-brockners-inband-oam-data-07 (work 1111 in progress), July 2017. 1113 [IDCong] Luckie, M., Dhamdhere, A., Clark, D., and B. Huffaker, 1114 "Challenges in inferring Internet interdomain congestion", 1115 In Proceedings of the 2014 Conference on Internet 1116 Measurement Conference, pp. 15-22. ACM, 2014. 1118 [MLB] Augustin, B., Friedman, T., and R. Teixeira, "Measuring 1119 load-balanced paths in the Internet", Proceedings of the 1120 7th ACM SIGCOMM conference on Internet measurement, pp. 1121 149-160. ACM, 2007., 2007. 1123 [MLRM] Fontugne, R., Mazel, J., and K. Fukuda, "An empirical 1124 mixture model for large-scale RTT measurements", 2015 1125 IEEE Conference on Computer Communications (INFOCOM), pp. 1126 2470-2478. IEEE, 2015., 2015. 1128 [P2] Jain, R. and I. Chlamtac, "The P 2 algorithm for dynamic 1129 calculation of quantiles and histograms without storing 1130 observations", Communications of the ACM 28.10 (1985): 1131 1076-1085, 2015. 1133 [PT] Augustin, B., Cuvellier, X., Orgogozo, B., Viger, F., 1134 Friedman, T., Latapy, M., Magnien, C., and R. Teixeira, 1135 "Avoiding traceroute anomalies with Paris traceroute", 1136 Proceedings of the 6th ACM SIGCOMM conference on Internet 1137 measurement, pp. 153-158. ACM, 2006., 2006. 1139 [RFC7594] Eardley, P., Morton, A., Bagnulo, M., Burbridge, T., 1140 Aitken, P., and A. Akhter, "A Framework for Large-Scale 1141 Measurement of Broadband Performance (LMAP)", RFC 7594, 1142 DOI 10.17487/RFC7594, September 2015, 1143 . 1145 [RTTSub] Bischof, Z., Rula, J., and F. Bustamante, "In and out of 1146 Cuba: Characterizing Cuba's connectivity", In Proceedings 1147 of the 2015 ACM Conference on Internet Measurement 1148 Conference, pp. 487-493. ACM, 2015. 1150 [SCAMPER] Matthew Luckie, M., "Scamper: a scalable and extensible 1151 packet prober for active measurement of the Internet", 1152 Proceedings of the 10th ACM SIGCOMM conference on 1153 Internet measurement, pp. 239-245. ACM, 2010., 2010. 1155 [SSNT] Park, K. and W. Willinger, "Self-Similar Network Traffic 1156 and Performance Evaluation (1st ed.)", John Wiley & Sons, 1157 Inc., New York, NY, USA, 2000. 1159 Authors' Addresses 1161 Jose Ignacio Alvarez-Hamelin 1162 Universidad de Buenos Aires 1163 Av. Paseo Colon 850 1164 Buenos Aires C1063ACV 1165 Argentine 1167 Phone: +54 11 5285-0716 1168 Email: ihameli@cnet.fi.uba.ar 1169 URI: http://cnet.fi.uba.ar/ignacio.alvarez-hamelin/ 1171 Al Morton 1172 AT&T Labs 1173 200 Laurel Avenue South 1174 Middletown, NJ 07748 1175 USA 1177 Phone: +1 732 420 1571 1178 Fax: +1 732 368 1192 1179 Email: acm@research.att.com 1181 Joachim Fabini 1182 TU Wien 1183 Gusshausstrasse 25/E389 1184 Vienna 1040 1185 Austria 1187 Phone: +43 1 58801 38813 1188 Fax: +43 1 58801 38898 1189 Email: Joachim.Fabini@tuwien.ac.at 1190 URI: http://www.tc.tuwien.ac.at/about-us/staff/joachim-fabini/ 1191 Carlos Pignataro 1192 Cisco Systems, Inc. 1193 7200-11 Kit Creek Road 1194 Research Triangle Park, NC 27709 1195 USA 1197 Email: cpignata@cisco.com