idnits 2.17.1 draft-ietf-ippm-route-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). (Using the creation date from RFC2330, updated by this document, for RFC5378 checks: 1998-05-01) -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (December 11, 2019) is 1598 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'Dst' is mentioned on line 906, but not defined == Missing Reference: 'Hop' is mentioned on line 906, but not defined == Missing Reference: 'Alt' is mentioned on line 906, but not defined == Unused Reference: 'RFC2460' is defined on line 1043, but no explicit reference was found in the text == Unused Reference: 'RFC2675' is defined on line 1047, but no explicit reference was found in the text == Unused Reference: 'RFC4494' is defined on line 1060, but no explicit reference was found in the text == Unused Reference: 'RFC5644' is defined on line 1080, but no explicit reference was found in the text == Unused Reference: 'RFC6282' is defined on line 1094, but no explicit reference was found in the text == Unused Reference: 'RFC6437' is defined on line 1099, but no explicit reference was found in the text == Unused Reference: 'RFC6564' is defined on line 1109, but no explicit reference was found in the text == Unused Reference: 'RFC7045' is defined on line 1118, but no explicit reference was found in the text == Outdated reference: A later version (-17) exists of draft-ietf-ippm-ioam-data-08 ** Downref: Normative reference to an Informational RFC: RFC 2330 ** Obsolete normative reference: RFC 2460 (Obsoleted by RFC 8200) ** Downref: Normative reference to an Informational RFC: RFC 2991 ** Downref: Normative reference to an Informational RFC: RFC 5835 ** Downref: Normative reference to an Informational RFC: RFC 7312 ** Downref: Normative reference to an Informational RFC: RFC 7799 ** Downref: Normative reference to an Experimental RFC: RFC 7820 Summary: 7 errors (**), 0 flaws (~~), 14 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group J. Alvarez-Hamelin 3 Internet-Draft Universidad de Buenos Aires 4 Updates: 2330 (if approved) A. Morton 5 Intended status: Standards Track AT&T Labs 6 Expires: June 13, 2020 J. Fabini 7 TU Wien 8 C. Pignataro 9 Cisco Systems, Inc. 10 R. Geib 11 Deutsche Telekom 12 December 11, 2019 14 Advanced Unidirectional Route Assessment (AURA) 15 draft-ietf-ippm-route-07 17 Abstract 19 This memo introduces an advanced unidirectional route assessment 20 (AURA) metric and associated measurement methodology, based on the IP 21 Performance Metrics (IPPM) Framework RFC 2330. This memo updates RFC 22 2330 in the areas of path-related terminology and path description, 23 primarily to include the possibility of parallel subpaths between a 24 given Source and Destination pair, owing to the presence of multi- 25 path technologies. 27 Requirements Language 29 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 30 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 31 "OPTIONAL" in this document are to be interpreted as described in BCP 32 14[RFC2119] [RFC8174] when, and only when, they appear in all 33 capitals, as shown here. 35 Status of This Memo 37 This Internet-Draft is submitted in full conformance with the 38 provisions of BCP 78 and BCP 79. 40 Internet-Drafts are working documents of the Internet Engineering 41 Task Force (IETF). Note that other groups may also distribute 42 working documents as Internet-Drafts. The list of current Internet- 43 Drafts is at https://datatracker.ietf.org/drafts/current/. 45 Internet-Drafts are draft documents valid for a maximum of six months 46 and may be updated, replaced, or obsoleted by other documents at any 47 time. It is inappropriate to use Internet-Drafts as reference 48 material or to cite them other than as "work in progress." 49 This Internet-Draft will expire on June 13, 2020. 51 Copyright Notice 53 Copyright (c) 2019 IETF Trust and the persons identified as the 54 document authors. All rights reserved. 56 This document is subject to BCP 78 and the IETF Trust's Legal 57 Provisions Relating to IETF Documents 58 (https://trustee.ietf.org/license-info) in effect on the date of 59 publication of this document. Please review these documents 60 carefully, as they describe your rights and restrictions with respect 61 to this document. Code Components extracted from this document must 62 include Simplified BSD License text as described in Section 4.e of 63 the Trust Legal Provisions and are provided without warranty as 64 described in the Simplified BSD License. 66 Table of Contents 68 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 69 1.1. Issues with Earlier Work to define Route . . . . . . . . 3 70 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 71 3. Route Metric Terms and Definitions . . . . . . . . . . . . . 5 72 3.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 6 73 3.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 6 74 3.3. Metric Definitions . . . . . . . . . . . . . . . . . . . 7 75 3.4. Related Round-Trip Delay and Loss Definitions . . . . . . 9 76 3.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . 9 77 3.6. Reporting the Metric . . . . . . . . . . . . . . . . . . 10 78 4. Route Assessment Methodologies . . . . . . . . . . . . . . . 10 79 4.1. Active Methodologies . . . . . . . . . . . . . . . . . . 10 80 4.1.1. Temporal Composition for Route Metrics . . . . . . . 12 81 4.1.2. Routing Class C Identification . . . . . . . . . . . 13 82 4.1.3. Intermediate Observation Point Route Measurement . . 14 83 4.2. Hybrid Methodologies . . . . . . . . . . . . . . . . . . 15 84 4.3. Combining Different Methods . . . . . . . . . . . . . . . 15 85 5. Background on Round-Trip Delay Measurement Goals . . . . . . 16 86 6. Tools to Measure Delays in the Internet . . . . . . . . . . . 17 87 7. RTD Measurements Statistics . . . . . . . . . . . . . . . . . 18 88 8. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 20 89 9. Security Considerations . . . . . . . . . . . . . . . . . . . 20 90 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 91 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 21 92 12. Appendix I MPLS Methods for Route Assessment . . . . . . . . 21 93 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 94 13.1. Normative References . . . . . . . . . . . . . . . . . . 22 95 13.2. Informative References . . . . . . . . . . . . . . . . . 25 96 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 26 98 1. Introduction 100 The IETF IP Performance Metrics (IPPM) working group first created a 101 framework for metric development in [RFC2330]. This framework has 102 stood the test of time and enabled development of many fundamental 103 metrics. It has been updated in the area of metric composition 104 [RFC5835], and in several areas related to active stream measurement 105 of modern networks with reactive properties [RFC7312]. 107 The [RFC2330] framework motivated the development of "performance and 108 reliability metrics for paths through the Internet," and Section 5 of 109 [RFC2330] defines terms that support description of a path under 110 test. However, metrics for assessment of path components and related 111 performance aspects had not been attempted in IPPM when the [RFC2330] 112 framework was written. 114 This memo takes-up the route measurement challenge and specifies a 115 new route metric, two practical frameworks for methods of measurement 116 (using either active or hybrid active-passive methods [RFC7799]), and 117 Round-Trip Delay and link information discovery using the results of 118 measurements. All route measurements are limited by the willingness 119 of hosts along the path to be discovered, to cooperate with the 120 methods used, or to recognize that the measurement operation is 121 taking place (such as when tunnels are present). 123 1.1. Issues with Earlier Work to define Route 125 Section 7 of [RFC2330] presented a simple example of a "route" metric 126 along with several other examples. The example is reproduced below 127 (where the reference is to Section 5 of [RFC2330]): 129 "route: The path, as defined in Section 5, from A to B at a given 130 time." 132 This example provides a starting point to develop a more complete 133 definition of route. Areas needing clarification include: 135 Time: In practice, the route will be assessed over a time interval, 136 because active path detection methods like [PT] rely on TTL limits 137 for their operation and cannot accomplish discovery of all hosts 138 using a single packet. 140 Type-P: The legacy route definition lacks the option to cater for 141 packet-dependent routing. In this memo, we assess the route for a 142 specific packet of Type-P, and reflect this in the metric 143 definition. The methods of measurement determine the specific 144 Type-P used. 146 Parallel Paths: Parallel paths are a reality of the Internet and a 147 strength of advanced route assessment methods, so the metric must 148 acknowledge this possibility. Use of Equal Cost Multi-Path (ECMP) 149 and Unequal Cost Multi-Path (UCMP) technologies are common sources 150 of parallel subpaths. 152 Cloud Subpath: May contain hosts that do not decrement TTL or Hop 153 Limit, but may have two or more exchange links connecting 154 "discoverable" hosts or routers. Parallel subpaths contained 155 within clouds cannot be discovered. The assessment methods only 156 discover hosts or routers on the path that decrement TTL or Hop 157 Count, or cooperate with interrogation protocols. The presence of 158 tunnels and nested tunnels further complicate assessment by hiding 159 hops. 161 Hop: Although the [RFC2330] definition of a hop was a link-host 162 pair, only hosts are discoverable or have the capability to 163 cooperate with interrogation protocols where link information may 164 be exposed. 166 The refined definition of Route metrics begins in the sections that 167 follow. 169 2. Scope 171 The purpose of this memo is to add new route metrics and methods of 172 measurement to the existing set of IPPM metrics. 174 The scope is to define route metrics that can identify the path taken 175 by a packet or a flow traversing the Internet between two hosts. 176 Although primarily intended for hosts communicating on the Internet 177 with IP, the definitions and metrics are constructed to be applicable 178 to other network domains, if desired. The methods of measurement to 179 assess the path may not be able to discover all hosts comprising the 180 path, but such omissions are often deterministic and explainable 181 sources of error. 183 Also, to specify a framework for active methods of measurement which 184 use the techniques described in [PT] at a minimum, and a framework 185 for hybrid active-passive methods of measurement, such as the Hybrid 186 Type I method [RFC7799] described in 187 [I-D.ietf-ippm-ioam-data](intended only for single administrative 188 domains), which do not rely on ICMP and provide a protocol for 189 explicit interrogation of nodes on a path. Combinations of active 190 methods and hybrid active-passive methods are also in-scope. 192 Further, this memo provides additional analysis of the round-trip 193 delay measurements made possible by the methods, in an effort to 194 discover more details about the path, such as the link technology in 195 use. 197 This memo updates Section 5 of [RFC2330] in the areas of path-related 198 terminology and path description, primarily to include the 199 possibility of parallel subpaths between a given Source and 200 Destination address pair (possibly resulting from Equal Cost Multi- 201 Path (ECMP) and Unequal Cost Multi-Path (UCMP) technologies). 203 There are several simple non-goals of this memo. There is no attempt 204 to assess the reverse path from any host on the path to the host 205 attempting the path measurement. The reverse path contribution to 206 delay will be that experienced by ICMP packets (in active methods), 207 and may be different from delays experienced by UDP or TCP packets. 208 Also, the round trip delay will include an unknown contribution of 209 processing time at the host that generates the ICMP response. 210 Therefore, the ICMP-based active methods are not supposed to yield 211 accurate, reproducible estimations of the Round-Trip Delay that UDP 212 or TCP packets will experience. 214 3. Route Metric Terms and Definitions 216 This section sets requirements for the following components to 217 support the Route Metric: 219 Host A Host as defined in [RFC2330] (a computer capable of IP 220 communication, includes routers), a.k.a. RFC 2330 Host. 222 Node A Node is any network function on the path capable of IP-layer 223 Communication, includes RFC 2330 Hosts. 225 Node Identity The unique address for Nodes communicating within the 226 network domain. For Nodes communicating on the Internet with IP, 227 it is the globally routable IP address(es) which the Node uses 228 when communicating with other Nodes under normal or error 229 conditions. The Node Identity revealed (and its connection to a 230 Node Name through reverse DNS) determines whether interfaces to 231 parallel links can be associated with a single Node, or appear to 232 identify unique Nodes. 234 Discoverable Node Nodes that convey their Node Identity according to 235 the requirements of their network domain, such as when error 236 conditions are detected by that Node. For Nodes communicating 237 with IP packets, compliance with Section 3.2.2.4 of [RFC1122] when 238 discarding a packet due to TTL or Hop Limit Exceeded condition, 239 MUST result in sending the corresponding Time Exceeded message 240 (containing a form of Node identity) to the source. This 241 requirement is also consistent with section 5.3.1 of [RFC1812] for 242 routers. 244 Cooperating Node Nodes that MUST respond to direct queries for their 245 Node identity as part of a previously agreed and established 246 interrogation protocol. Nodes SHOULD also provide information 247 such as arrival/departure interface identification, arrival 248 timestamp, and any relevant information about the Node or specific 249 link which delivered the query to the Node. 251 Hop A Hop MUST contain a Node Identity, and MAY contain arrival and/ 252 or departure interface identification, round trip delay, and an 253 arrival timestamp. 255 3.1. Formal Name 257 Type-P-Route-Ensemble-Method-Variant, abbreviated as Route Ensemble. 259 Note that Type-P depends heavily on the chosen method and variant. 261 3.2. Parameters 263 This section lists the REQUIRED input factors to specify a Route 264 metric. 266 o Src, the address of a Node (such as the globally routable IP 267 address). 269 o Dst, the address of a Node (such as the globally routable IP 270 address). 272 o i, the limit on the number of Hops a specific packet may visit as 273 it traverses from the Node at Src to the Node at Dst (such as the 274 TTL or Hop Limit). 276 o MaxHops, the maximum value of i used, (i=1,2,3,...MaxHops). 278 o T0, a time (start of measurement interval) 280 o Tf, a time (end of measurement interval) 282 o MP(address), Measurement Point at address, such as Src or Dst, 283 usually at the same node stack layer as "address". 285 o T, the Node time of a packet as measured at MP(Src), meaning 286 Measurement Point at the Source. 288 o Ta, the Node time of a reply packet's *arrival* as measured at 289 MP(Src), assigned to packets that arrive within a "reasonable" 290 time (see parameter below). 292 o Tmax, a maximum waiting time for reply packets to return to the 293 source, set sufficiently long to disambiguate packets with long 294 delays from packets that are discarded (lost), such that the 295 distribution of Round-Trip Delay is not truncated. 297 o F, the number of different flows simulated by the method and 298 variant. 300 o flow, the stream of packets with the same n-tuple of designated 301 header fields that (when held constant) result in identical 302 treatment in a multi-path decision (such as the decision taken in 303 load balancing). Note: The IPv6 flow label MAY be included in the 304 flow definition if the MP(Src) is a Tunnel End Point (TEP) 305 complying with [RFC6438] guidelines. 307 o Type-P, the complete description of the packets for which this 308 assessment applies (including the flow-defining fields). 310 3.3. Metric Definitions 312 This section defines the REQUIRED measurement components of the Route 313 metrics (unless otherwise indicated): 315 M, the total number of packets sent between T0 and Tf. 317 N, the smallest value of i needed for a packet to be received at Dst 318 (sent between T0 and Tf). 320 Nmax, the largest value of i needed for a packet to be received at 321 Dst (sent between T0 and Tf). Nmax may be equal to N. 323 Next, define a *singleton* definition for a Hop on the path, with 324 sufficient indexes to identify all Hops identified in a measurement 325 interval. 327 A Hop, designated h(i,j), the IP address and/or identity of one of j 328 Discoverable Nodes (or Cooperating Nodes) that are i hops away from 329 the Node with address = Src during the measurement interval, T0 to 330 Tf. As defined above, a Hop singleton measurement MUST contain a 331 Node Identity, hid(i,j), and MAY contain one or more of the following 332 attributes: 334 o a(i,j) Arrival Interface ID (e.g., when [RFC5837] is supported) 335 o d(i,j) Departure Interface ID (e.g., when [RFC5837] is supported) 337 o t(i,j) Arrival Timestamp (where t(i,j) is ideally supplied by the 338 Hop, or approximated from the sending time of the packet that 339 revealed the Hop) 341 o Measurements of Round-Trip Delay (for each packet that reveals the 342 same Node Identity and flow attributes, but not timestamp of 343 course, see next section) 345 Node Identities and related information can be ordered by their 346 distance from the Node with address Src in Hops h(i,j). Based on 347 this, two forms of Routes are distinguished: 349 A Route Ensemble is defined as the combination of all routes 350 traversed by different flows from the Node at Src address to the Node 351 at Dst address. A single Route traversed by a single flow 352 (determined by an unambiguous tuple of addresses Src and Dst, and 353 other identical flow criteria) is a member of the Route Ensemble and 354 called a Member Route. 356 Using h(i,j) and components and parameters, further define: 358 When considering the set of Hops in the context of a single flow, a 359 Member Route j is an ordered list {h(1,j), ... h(Nj, j)} where h(i-1, 360 j) and h(i, j) are by 1 hop away from each other and Nj satisfying 361 h(Nj,j)=Dst is the minimum count of Hops needed by the packet on 362 Member Route j to reach Dst. Member Routes must be unique. The 363 uniqueness property requires that any two Member routes j and k that 364 are part of the same Route Ensemble differ either in terms of minimum 365 hop count Nj and Nk to reach the destination Dst, or, in the case of 366 identical hop count Nj=Nk, they have at least one distinct Hop: 367 h(i,j) != h(i, k) for at least one i (i=1..Nj). 369 All the optional information collected to describe a Member Route, 370 such as the arrival interface, departure interface, and Round Trip 371 Delay at each Hop, turns each list item into a rich structure. There 372 may be information on the links between Hops, possibly information on 373 the routing (arrival interface and departure interface), an estimate 374 of distance between Hops based on Round-Trip Delay measurements and 375 calculations, and a time stamp indicating when all these additional 376 details were valid. 378 The Route Ensemble from Src to Dst, during the measurement interval 379 T0 to Tf, is the aggregate of all m distinct Member Routes discovered 380 between the two Nodes with Src and Dst addresses. More formally, 381 with the Node having address Src omitted: 383 Route Ensemble = { 384 {h(1,1), h(2,1), h(3,1), ... h(N1,1)=Dst}, 385 {h(1,2), h(2,2), h(3,2),..., h(N2,2)=Dst}, 386 ... 387 {h(1,m), h(2,m), h(3,m), ....h(Nm,m)=Dst} 388 } 390 where the following conditions apply: i <= Nj <= Nmax (j=1..m) 392 Note that some h(i,j) may be empty (null) in the case that systems do 393 not reply (not discoverable, or not cooperating). 395 h(i-1,j) and h(i,j) are the Hops on the same Member Route one hop 396 away from each other. 398 Hop h(i,j) may be identical with h(k,l) for i!=k and j!=l ; which 399 means there may be portions shared among different Member Routes 400 (parts of Member Routes may overlap). 402 3.4. Related Round-Trip Delay and Loss Definitions 404 RTD(i,j,T) is defined as a singleton of the [RFC2681] Round-Trip 405 Delay between the Node with address = Src and the Node at Hop h(i,j) 406 at time T. 408 RTL(i,j,T) is defined as a singleton of the [RFC6673] Round-trip Loss 409 between the Node with address = Src and the Node at Hop h(i,j) at 410 time T. 412 3.5. Discussion 414 Depending on the way that Node Identity is revealed, it may be 415 difficult to determine parallel subpaths between the same pair of 416 Nodes (i.e. multiple parallel links). It is easier to detect 417 parallel subpaths involving different Nodes. 419 o If a pair of discovered Nodes identify two different addresses, 420 then they will appear to be different Nodes. 422 o If a pair of discovered Nodes identify two different IP addresses, 423 and the IP addresses resolve to the same Node name (in the DNS), 424 then they will appear to be the same Nodes. 426 o If a discovered Node always replies using the same network 427 address, regardless of the interface a packet arrives on, then 428 multiple parallel links cannot be detected in that network domain. 429 This condition may apply to traceroute-style methods, but may not 430 apply to other hybrid methods based on In-situ Operations, 431 Administration, and Maintenance (IOAM). 433 o If parallel links between routers are aggregated below the IP 434 layer, then from Node point of view, all these links share the 435 same pair of IP addresses. The existence of these parallel links 436 can't be detected at IP layer. This applies to other network 437 domains with layers below them, as well. This condition may apply 438 to traceroute-style methods, but may not apply to other hybrid 439 methods based on IOAM. 441 When a route assessment employs IP packets (for example), the reality 442 of flow assignment to parallel subpaths involves layers above IP. 443 Thus, the measured Route Ensemble is applicable to IP and higher 444 layers (as described in the methodology's packet of Type-P and flow 445 parameters). 447 3.6. Reporting the Metric 449 An Information Model and an XML Data Model for Storing Traceroute 450 Measurements is available in [RFC5388]. The measured information at 451 each hop includes four pieces of information: a one-dimensional hop 452 index, Node symbolic address, Node IP address, and RTD for each 453 response. 455 The description of Hop information that may be collected according to 456 this memo covers more dimensions, as defined in Section 3.3 above. 457 For example, the Hop index is two-dimensional to capture the 458 complexity of a Route Ensemble, and it contains corresponding Node 459 identities at a minimum. The models need to be expanded to include 460 these features, as well as Arrival Interface ID, Departure Interface 461 ID, and Arrival Timestamp, when available. The original sending 462 Timestamp from the Src Node anchors a particular measurement in time. 464 4. Route Assessment Methodologies 466 There are two classes of methods described in this section, active 467 methods relying on the reaction to TTL or Hop Limit Exceeded 468 condition to discover Nodes on a path, and Hybrid active-passive 469 methods that involve direct interrogation of cooperating Nodes 470 (usually within a single domain). Description of these methods 471 follow. 473 4.1. Active Methodologies 475 This section describes the method employed by current open source 476 tools, thereby providing a practical framework for further advanced 477 techniques to be included as method variants. This method is 478 applicable for use across multiple administrative domains. 480 Paris-traceroute [PT] provides some measure of protection from path 481 variation generated by ECMP load balancing, and it ensures traceroute 482 packets will follow the same path in 98% of cases according to 483 [SCAMPER]. If it is necessary to find every usable path between two 484 Nodes, Paris-traceroute provides "exhaustive" mode while scamper 485 provides "tracelb" (stands for traceroute load balance). 487 The Type-P of packets used could be ICMP (as in the original 488 traceroute), UDP or TCP. The latter are used when a particular 489 characteristic needs to be to verified, such as filtering or traffic 490 shaping on specific ports (i.e., services). [SCAMPER] supports IPv6 491 traceroute measurements, keeping the FlowLabel constant in all 492 packets. 494 The advanced route assessment methods used in Paris-traceroute [PT] 495 keep the critical fields constant for every packet to maintain the 496 appearance of the same flow. Since route assessment can be conducted 497 using TCP, UDP or ICMP packets, this method REQUIRES the Diffserv 498 field, the protocol number, IP source and destination addresses, and 499 the port settings for TCP or UDP to be kept constant. For ICMP 500 probes, the method additionally REQUIRES keeping the type, code, and 501 ICMP checksum constant (the latter occupy the corresponding positions 502 in the header of an IP packet, e.g., bytes 20 to 23 when the header 503 IP has no options). 505 Maintaining a constant checksum in ICMP is most challenging because 506 the ICMP Sequence Number is part of the calculation. The advanced 507 traceroute method requires calculations using the IP Sequence Number 508 Field and the Identifier Field, yielding a constant ICMP checksum in 509 successive packets. For an example of calculations to maintain a 510 constant checksum, see Appendix A of [RFC7820], where revision of a 511 timestamp field is complemented by modifying the 2 octet checksum 512 complement field (these fields take the roles of the ICMP Sequence 513 Number and Identifier Fields, respectively). 515 For TCP and UDP packets, the checksum must also be kept constant. 516 Therefore, the first four bytes of UDP (or TCP) data field are 517 modified to compensate for fields that change from packet to packet. 519 Finally, the return path is also important to check. Taking into 520 account that it is an ICMP time exceeded (during transit) packet, the 521 source and destination IP are constant for every reply. Then, we 522 should consider the fields in the first 32 bits of the protocol on 523 the top of IP: the type and code of ICMP packet, and its checksum. 524 Again, to keep the ICMP checksum constant for the returning packets, 525 we need to consider the whole ICMP message. It contains the IP 526 header of the discarded packet plus the first 8 bytes of the IP 527 payload; that is some of the fields of the TCP header, the UDP header 528 plus four data bytes, the ICMP header plus four bytes. Therefore, 529 for the UDP case the data field is used to maintain the ICMP checksum 530 constant in the returning packet. For the ICMP case, the identifier 531 and sequence fields of the sent ICMP probe are manipulated to be 532 constant. The TCP case presents no problem because its first eight 533 bytes will be the same for every packet probe. 535 Formally, to maintain the same flow in the measurements to a certain 536 hop, the Type-P-Route-Ensemble-Method-Variant packets should be[PT]: 538 o TCP case: Fields Src, Dst, port-Src, port_Dst, and Diffserv Field 539 should be the same. 541 o UDP case: Fields Src, Dst, port-Src, port-Dst, and Diffserv Field 542 should be the same, the UDP-checksum should change to keep the IP 543 checksum of the ICMP time exceeded reply constant. Then, the data 544 length should be fixed, and the data field is used to fixing it 545 (consider that ICMP checksum uses its data field, which contains 546 the original IP header plus 8 bytes of UDP, where TTL or Hop 547 Limit, IP identification, IP checksum, and UDP checksum changes). 549 o ICMP case: The Data field should compensate variations on TTL or 550 Hop Limit, IP identification, and IP checksum for every packet. 552 Then, the way to identify different hops and attempts of the same 553 flow is: 555 o TCP case: The IP identification field. 557 o UDP case: The IP identification field. 559 o ICMP case: The IP identification field, and ICMP Sequence number. 561 4.1.1. Temporal Composition for Route Metrics 563 The Active Route Assessment Methods described above have the ability 564 to discover portions of a path where ECMP load balancing is present, 565 observed as two or more unique Member Routes having one or more 566 distinct Hops which are part of the Route Ensemble. Likewise, 567 attempts to deliberately vary the flow characteristics to discover 568 all Member Routes will reveal portions of the path which are flow- 569 invariant. 571 Section 9.2 of [RFC2330] describes Temporal Composition of metrics, 572 and introduces the possibility of a relationship between earlier 573 measurement results and the results for measurement at the current 574 time (for a given metric). There is value in establishing a Temporal 575 Composition relationship for Route Metrics. However, this 576 relationship does not represent a forecast of future route conditions 577 in any way. 579 For Route Metric measurements, the value of Temporal Composition is 580 to reduce the measurement iterations required with repeated 581 measurements. Reduced iterations are possible by inferring that 582 current measurements using fixed and previously measured flow 583 characteristics: 585 o will have many common hops with previous measurements. 587 o will have relatively time-stable results at the ingress and egress 588 portions of the path when measured from user locations, as opposed 589 to measurements of backbone networks and across inter-domain 590 gateways. 592 o may have greater potential for time-variation in path portions 593 where ECMP load balancing is observed (because increasing or 594 decreasing the pool of links changes the hash calculations). 596 Optionally, measurement systems may take advantage of the inferences 597 above when seeking to reduce measurement iterations, after exhaustive 598 measurements indicate that the time-stable properties are present. 599 Repetitive Active Route measurement systems: 601 1. SHOULD occasionally check path portions which have exhibited 602 stable results over time, particularly ingress and egress 603 portions of the path. 605 2. SHOULD continue testing portions of the path that have previously 606 exhibited ECMP load balancing. 608 3. SHALL trigger re-assessment of the complete path and Route 609 Ensemble, if any change in hops is observed for a specific (and 610 previously tested) flow. 612 4.1.2. Routing Class C Identification 614 There is an opportunity to apply the [RFC2330] notion of equal 615 treatment for a class of packets, "...very useful to know if a given 616 Internet component treats equally a class C of different types of 617 packets", as it applies to Route measurements. Knowledge of "class 618 C" parameters (unrelated to address classes of the past) on a path 619 potentially reduces the number of flows required for a given method 620 to assess a Route Ensemble over time. 622 First, recognize that each Member Route of a Route Ensemble will have 623 a corresponding Routing Class C. Class C can be discovered by 624 testing with multiple flows, all of which traverse the unique set of 625 hops that comprise a specific Member Route. 627 Second, recognize that the different Routing Classes depend primarily 628 on the hash functions used at each instance of ECMP load balancing on 629 the path. 631 Third, recognize the synergy with Temporal Composition methods 632 (described above) where evaluation intends to discover time-stable 633 portions of each Member Route so that more emphasis can be placed on 634 ECMP portions that also determine Class C. 636 The methods to assess the various Routing Class C characteristics 637 benefit from the following measurement capabilities: 639 o flows designed to determine which n-tuple header fields are 640 considered by a given hash function and ECMP hop on the path, and 641 which are not. This operation immediately narrows the search 642 space, where possible, and partially defines a Routing Class C. 644 o a priori knowledge of the possible types of hash functions in use 645 also helps to design the flows for testing (major router vendors 646 publish information about these hash functions, examples are here 647 https://www.researchgate.net/ 648 publication/281571413_COMPARISON_OF_HASH_STRATEGIES_FOR_FLOW- 649 BASED_LOAD_BALANCING ). 651 o ability to direct the emphasis of current measurements on ECMP 652 portions of the path, based on recent past measurement results 653 (the Routing Class C of some portions of the path is essentially 654 "all packets"). 656 4.1.3. Intermediate Observation Point Route Measurement 658 There are many examples where passive monitoring of a flow at an 659 Observation Point within the network can detect unexpected Round Trip 660 Delay or Delay Variation. But how can the cause of the anomalous 661 delay be investigated further --from the Observation Point -- 662 possibly located at an intermediate point on the path? 664 In this case, knowledge that the flow of interest belongs to a 665 specific Routing Class C will enable measurement of the route where 666 anomalous delay has been observed. Specifically, Round-Trip Delay 667 assessment to each Hop on the path between the Observation Point and 668 the Destination for the flow of interest may discover high or 669 variable delay on a specific link and Hop combination. 671 The determination of a Routing Class C which includes the flow of 672 interest is as described in the section above, aided by computation 673 of the relevant hash function output as the target. 675 4.2. Hybrid Methodologies 677 The Hybrid Type I methods provide an alternative method for Route 678 Member assessment. As mentioned in the Scope section, 679 [I-D.ietf-ippm-ioam-data] provides a possible set of data fields that 680 would support route identification. 682 In general, nodes in the measured domain would be equipped with 683 specific abilities: 685 o Support of the "Loopback" Flag (L-bit), where a copy of the packet 686 is returned to the encapsulating node, and the packet is processed 687 like any other IOAM packet on the return transfer. 689 In addition to node identity, nodes may also identify the ingress and 690 egress interfaces utilized by the tracing packet, the time of day 691 when the packet was processed, and other generic data (as described 692 in section 4 of [I-D.ietf-ippm-ioam-data]). Interface identification 693 isn't necessarily limited to IP, i.e. different links in a bundle 694 (LACP) could be identified. Equally well, links without explicit IP 695 addresses can be identified (like with unnumbered interfaces in an 696 IGP deployment). 698 Note that Type-P packet specification for this method will likely be 699 a partial specification, because most of the packet fields are 700 determined by the user traffic. The packet header(s) added by the 701 Hybrid method can certainly be specified in Type-P. 703 4.3. Combining Different Methods 705 In principle, there are advantages if the entity conducting Route 706 measurements can utilize both forms of advanced methods (active and 707 hybrid), and combine the results. For example, if there are Nodes 708 involved in the path that qualify as Cooperating Nodes, but not as 709 Discoverable Nodes, then a more complete view of Hops on the path is 710 possible when a hybrid method (or interrogation protocol) is applied 711 and the results are combined with the active method results collected 712 across all other domains. 714 In order to combine the results of active and hybrid/interrogation 715 methods, the network Nodes that are part of a domain supporting an 716 interrogation protocol have the following attributes: 718 1. Nodes at the ingress to the domain SHOULD be both Discoverable 719 and Cooperating, and SHOULD reveal the same Node Identity in 720 response to both active and hybrid methods. 722 2. Any Nodes within the domain that are both Discoverable and 723 Cooperating SHOULD reveal the same Node Identity in response to 724 both active and hybrid methods. 726 3. Nodes at the egress to the domain SHOULD be both Discoverable and 727 Cooperating, and SHOULD reveal the same Node Identity in response 728 to both active and hybrid methods. 730 When Nodes follow these requirements, it becomes a simple matter to 731 match single domain measurements with the overlapping results from a 732 multidomain measurement. 734 In practice, Internet users do not typically have the ability to 735 utilize the OAM capabilities of networks that their packets traverse, 736 so the results from a remote domain supporting an interrogation 737 protocol would not normally be accessible. However, a network 738 operator could combine interrogation results from their access domain 739 with other measurements revealing the path outside their domain. 741 5. Background on Round-Trip Delay Measurement Goals 743 The aim of this method is to use packet probes to unveil the paths 744 between any two end-Nodes of the network. Moreover, information 745 derived from RTD measurements might be meaningful to identify: 747 1. Intercontinental submarine links 749 2. Satellite communications 751 3. Congestion 753 4. Inter-domain paths 755 This categorization is widely accepted in the literature and among 756 operators alike, and it can be trusted with empirical data and 757 several sources as ground of truth (e.g., [RTTSub] ) but it is an 758 inference measurement nonetheless [bdrmap][IDCong]. 760 The first two categories correspond to the physical distance 761 dependency on Round-Trip Delay (RTD) while the last one binds RTD 762 with queueing delay on routers. Due to the significant contribution 763 of propagation delay in long distance hops, RTD will be on the order 764 of 100ms on transatlantic hops, depending on the geolocation of the 765 vantage points. Moreover, RTD is typically greater than 480ms when 766 two hops are connected using geostationary satellite technology 767 (i.e., their orbit is at 36000km). Detecting congestion with latency 768 implies deeper mathematical understanding since network traffic load 769 is not stationary. Nonetheless, as the first approach, a link seems 770 to be congested if after sending several traceroute probes, it is 771 possible to detect congestion observing different statistics 772 parameters (e.g., see [IDCong]). 774 6. Tools to Measure Delays in the Internet 776 Internet routing is complex because it depends on the policies of 777 thousands Autonomous Systems (AS). While most of the routers perform 778 load balancing on flows using Equal Cost Multiple Path (ECMP), a few 779 still divide the workload through packet-based techniques. The 780 former scenario is defined according to [RFC2991] while the latter 781 generates a round-robin scheme to deliver every new outgoing packet. 782 ECMP uses a hashing function to ensure that every packet of a flow is 783 delivered by the same path, and this avoids increasing the packet 784 delay variation and possibly producing overwhelming packet reordering 785 in TCP flows. 787 Taking into account that Internet protocol was designed under the 788 "end-to-end" principle, the IP payload and its header do not provide 789 any information about the routes or path necessary to reach some 790 destination. For this reason, the well-known tool traceroute was 791 developed to gather the IP addresses of each hop along a path using 792 the ICMP protocol [RFC0792]. Besides, traceroute adds the measured 793 RTD from each hop. However, the growing complexity of the Internet 794 makes it more challenging to develop an accurate traceroute 795 implementation. For instance, the early traceroute tools would be 796 inaccurate in the current network, mainly because they were not 797 designed to retain flow state. However, evolved traceroute tools, 798 such as Paris-traceroute [PT] [MLB] and Scamper [SCAMPER], expect to 799 encounter ECMP and achieve more accurate results when they do. 801 Paris-traceroute-like tools operate in the following way: every 802 packet should follow the same path because the sensitive fields of 803 the header are controlled to appear as the same flow. This means 804 that source and destination IP addresses, source and destination port 805 numbers are the same in every packet. Additionally, Differentiated 806 Services Code Point (DSCP), checksum and ICMP code should remain 807 constant since they may affect the path selection. 809 Today's traceroute tools can send either UDP, TCP or ICMP packet 810 probes. Since ICMP header does not include transport layer 811 information, there are no fields for source and destination port 812 numbers. For this reason, these tools keep constant ICMP type, code, 813 and checksum fields to generate a kind of flow. However, the 814 checksum may vary in every packet, therefore when probes use ICMP 815 packets, ICMP Identifier and sequence number are manipulated to 816 maintain constant checksum in every packet. On the other hand, when 817 UDP probes are generated, the expected variation in the checksum of 818 each packet is again compensated by manipulating the payload. 820 Paris-traceroute allows its users to measure RTD in every hop of the 821 path for a particular flow. Furthermore, either Paris-traceroute or 822 Scamper is capable of unveiling the many available paths between a 823 source and destination (which are visible to this method). This task 824 is accomplished by repeating complete traceroute measurements with 825 different flow parameters for each measurement. The Framework for IP 826 Performance Metrics (IPPM) ([RFC2330] updated by [RFC7312]) has the 827 flexibility to require that the Round-Trip Delay measurement 828 [RFC2681] uses packets with the constraints to assure that all 829 packets in a single measurement appear as the same flow. This 830 flexibility covers ICMP, UDP, and TCP. The accompanying methodology 831 of [RFC2681] needs to be expanded to report the sequential hop 832 identifiers along with RTD measurements, but no new metric definition 833 is needed. 835 7. RTD Measurements Statistics 837 Several articles have shown that network traffic presents a self- 838 similar nature [SSNT] [MLRM] which is accountable for filling the 839 queues of the routers. Moreover, router queues are designed to 840 handle traffic bursts, which is one of the most remarkable features 841 of self-similarity. Naturally, while queue length increases, the 842 delay to traverse the queue increases as well and leads to an 843 increase on RTD. Due to traffic bursts generate short-term overflow 844 on buffers (spiky patterns), every RTD only depicts the queueing 845 status on the instant when that packet probe was in transit. For 846 this reason, several RTD measurements during a time window could 847 begin to describe the random behavior of latency. Loss must also be 848 accounted for in the methodology. 850 To understand the ongoing process, examining the quartiles provides a 851 non-parametric way of analysis. Quartiles are defined by five 852 values: minimum RTD (m), RTD value of the 25% of the Empirical 853 Cumulative Distribution Function (ECDF) (Q1), the median value (Q2), 854 the RTD value of the 75% of the ECDF (Q3) and the maximum RTD (M). 855 Congestion can be inferred when RTD measurements are spread apart, 856 and consequently, the Inter-Quartile Range (IQR), the distance 857 between Q3 and Q1, increases its value. 859 This procedure requires to compute quartile values "on the fly" using 860 the algorithm presented in [P2]. 862 This procedure allows us to update the quartiles value whenever a new 863 measurement arrives, which is radically different from classic 864 methods of computing quartiles because they need to use the whole 865 dataset to compute the values. This way of calculus provides savings 866 in memory and computing time. 868 To sum up, the proposed measurement procedure consists in performing 869 traceroutes several times to obtain samples of the RTD in every hop 870 from a path, during a time window (W) and compute the quantiles for 871 every hop. This could be done for a single Member Route flow or for 872 every detected Route Ensemble flow. 874 The identification of a specific Hop in traceroute is based on the IP 875 origin address of the returned ICMP Time Exceeded packet, and on the 876 distance identified by the value set in the TTL field inserted by 877 traceroute. As this specific Hop can be reached by different paths, 878 also the IP source and destination addresses of the traceroute packet 879 need to be recorded. Finally, different return paths are 880 distinguished by evaluating the ICMP Time Exceeded TTL (of the reply 881 message): if this TTL is constant for different paths containing the 882 same Hop, the return paths have the same distance. Moreover, this 883 distance can be estimated considering that the TTL value is normally 884 initialized with values 64, 128, or 255. The 5-tuple (origin IP, 885 destination IP, reply IP, distance, response TTL) univocally 886 identifies every measurement. 888 This algorithm below runs in the origin of the traceroute. It 889 returns the Qs quartiles for every Hop and Alt (alternative paths 890 because of balancing). The following parameters are used: W window 891 time of the measurement, i_t time between two measurements, Dst 892 destination IP address. 894 ================================================================ 895 1 input: W (window time of the measurement) 896 2 i_t (time between two measurements) 897 3 E (True: exhaustive, False: a single path) 898 4 Dst (destination IP address) 899 5 output: Qs (quartiles for every Hop and Alt) 900 ---------------------------------------------------------------- 901 6 T := start_timer(W) 902 7 while T is not finished do: 903 8 | start_timer(i_t) 904 9 | RTD(Hop,Alt) = advanced-traceroute(Dst,E) 905 10 | for each Hop and Alt in RTD do: 906 11 | | Qs[Dst,Hop,Alt] := ComputeQs(RTD(Hop,Alt)) 907 12 | done 908 13 | wait until i_t timer is expired 909 14 done 910 15 return (Qs) 911 ================================================================ 913 During the time W, lines 6 and 7 assure that the measurement loop is 914 made. Line 8 and 13 set a timer for each cycle of measurements. A 915 cycle comprises the traceroutes packets, considering every possible 916 Hop and the alternatives paths in the Alt variable (ensured in lines 917 9-12). In line 9 the advance-traceroute could be either Paris- 918 traceroute or Scamper, which will use "exhaustive" mode or "tracelb" 919 option if E is set True, respectively. The procedure returns a list 920 of tuples (m,Q1,Q2,Q3,M) for each intermediate hop in the path 921 towards the Dst. Additionally, it could also return path variations 922 using "alt" variable.Finally, lines 10 and 12 stores each measurement 923 into the real-time quartiles computation. 925 8. Conclusions 927 Combining the method proposed in Section 4 and statistics in 928 Section 7, we can measure the performance of paths interconnecting 929 two endpoints in Internet, and attempt the categorization of link 930 types and congestion presence based on RTD. 932 9. Security Considerations 934 The security considerations that apply to any active measurement of 935 live paths are relevant here as well. See [RFC4656] and [RFC5357]. 937 The active measurement process of "changing several fields to keep 938 the checksum of different packets identical" does not require special 939 security considerations because it is part of synthetic traffic 940 generation, and is designed to have minimal to zero impact on network 941 processing (to process the packets for ECMP). 943 For applicable Hybrid methods, the security considerations 944 in[I-D.ietf-ippm-ioam-data] apply. 946 When considering privacy of those involved in measurement or those 947 whose traffic is measured, the sensitive information available to 948 potential observers is greatly reduced when using active techniques 949 which are within this scope of work. Passive observations of user 950 traffic for measurement purposes raise many privacy issues. We refer 951 the reader to the privacy considerations described in the Large Scale 952 Measurement of Broadband Performance (LMAP) Framework [RFC7594], 953 which covers active and passive techniques. 955 10. IANA Considerations 957 This memo makes no requests of IANA. We thank the good folks at IANA 958 for having checked this section anyway. 960 11. Acknowledgements 962 The original 3 authors acknowledge Ruediger Geib, for his penetrating 963 comments on the initial draft, and his initial text for the 964 Appendix on MPLS. Carlos Pignataro challenged the authors to 965 consider a wider scope, and applied his substantial expertise with 966 many technologies and their measurement features in his extensive 967 comments. Frank Brockners also shared useful comments, so did Footer 968 Foote. We thank them all! 970 12. Appendix I MPLS Methods for Route Assessment 972 A Node assessing an MPLS path must be part of the MPLS domain where 973 the path is implemented. When this condition is met, RFC 8029 974 provides a powerful set of mechanisms to detect "correct operation of 975 the data plane, as well as a mechanism to verify the data plane 976 against the control plane" [RFC8029]. 978 MPLS routing is based on the presence of a Forwarding Equivalence 979 Class (FEC) Stack in all visited Nodes. Selecting one of several 980 Equal Cost Multi Path (ECMP) is however based on information hidden 981 deeper in the stack. Early deployments may support a so called 982 "Entropy label" for this purpose. State of the art deployments base 983 their choice of an ECMP member based on the IP addresses (see 984 Section 2.4 of [RFC7325]). Both methods allow load sharing 985 information to be decoupled from routing information. Thus, an MPLS 986 traceroute is able to check how packets with a contiguous number of 987 ECMP relevant addresses (and the same destination) are routed by a 988 particular router. The minimum number of MPLS paths traceable at a 989 router should be 32. Implementations supporting more paths are 990 available. 992 The MPLS echo request and reply messages offering this feature must 993 support the Downstream Detailed Mapping TLV (was Downstream Mapping 994 initially, but the latter has been deprecated). The MPLS echo 995 response includes the incoming interface where a router received the 996 MPLS Echo request. The MPLS Echo reply further informs which of the 997 n addresses relevant for the load sharing decision results in a 998 particular next hop interface and contains the next hop's interface 999 address (if available). This ensures that the next hop will receive 1000 a properly coded MPLS Echo request in the next step route of 1001 assessment. 1003 [RFC8403] explains how a central Path Monitoring System could be used 1004 to detect arbitrary MPLS paths between any routers within a single 1005 MPLS domain. The combination of MPLS forwarding, Segment Routing and 1006 MPLS traceroute offers a simple architecture and a powerful mechanism 1007 to detect and validate (segment routed) MPLS paths. 1009 13. References 1011 13.1. Normative References 1013 [I-D.ietf-ippm-ioam-data] 1014 Brockners, F., Bhandari, S., Pignataro, C., Gredler, H., 1015 Leddy, J., Youell, S., Mizrahi, T., Mozes, D., Lapukhov, 1016 P., remy@barefootnetworks.com, r., daniel.bernier@bell.ca, 1017 d., and J. Lemon, "Data Fields for In-situ OAM", draft- 1018 ietf-ippm-ioam-data-08 (work in progress), October 2019. 1020 [RFC0792] Postel, J., "Internet Control Message Protocol", STD 5, 1021 RFC 792, DOI 10.17487/RFC0792, September 1981, 1022 . 1024 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - 1025 Communication Layers", STD 3, RFC 1122, 1026 DOI 10.17487/RFC1122, October 1989, 1027 . 1029 [RFC1812] Baker, F., Ed., "Requirements for IP Version 4 Routers", 1030 RFC 1812, DOI 10.17487/RFC1812, June 1995, 1031 . 1033 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1034 Requirement Levels", BCP 14, RFC 2119, 1035 DOI 10.17487/RFC2119, March 1997, 1036 . 1038 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 1039 "Framework for IP Performance Metrics", RFC 2330, 1040 DOI 10.17487/RFC2330, May 1998, 1041 . 1043 [RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 1044 (IPv6) Specification", RFC 2460, DOI 10.17487/RFC2460, 1045 December 1998, . 1047 [RFC2675] Borman, D., Deering, S., and R. Hinden, "IPv6 Jumbograms", 1048 RFC 2675, DOI 10.17487/RFC2675, August 1999, 1049 . 1051 [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip 1052 Delay Metric for IPPM", RFC 2681, DOI 10.17487/RFC2681, 1053 September 1999, . 1055 [RFC2991] Thaler, D. and C. Hopps, "Multipath Issues in Unicast and 1056 Multicast Next-Hop Selection", RFC 2991, 1057 DOI 10.17487/RFC2991, November 2000, 1058 . 1060 [RFC4494] Song, JH., Poovendran, R., and J. Lee, "The AES-CMAC-96 1061 Algorithm and Its Use with IPsec", RFC 4494, 1062 DOI 10.17487/RFC4494, June 2006, 1063 . 1065 [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. 1066 Zekauskas, "A One-way Active Measurement Protocol 1067 (OWAMP)", RFC 4656, DOI 10.17487/RFC4656, September 2006, 1068 . 1070 [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. 1071 Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", 1072 RFC 5357, DOI 10.17487/RFC5357, October 2008, 1073 . 1075 [RFC5388] Niccolini, S., Tartarelli, S., Quittek, J., Dietz, T., and 1076 M. Swany, "Information Model and XML Data Model for 1077 Traceroute Measurements", RFC 5388, DOI 10.17487/RFC5388, 1078 December 2008, . 1080 [RFC5644] Stephan, E., Liang, L., and A. Morton, "IP Performance 1081 Metrics (IPPM): Spatial and Multicast", RFC 5644, 1082 DOI 10.17487/RFC5644, October 2009, 1083 . 1085 [RFC5835] Morton, A., Ed. and S. Van den Berghe, Ed., "Framework for 1086 Metric Composition", RFC 5835, DOI 10.17487/RFC5835, April 1087 2010, . 1089 [RFC5837] Atlas, A., Ed., Bonica, R., Ed., Pignataro, C., Ed., Shen, 1090 N., and JR. Rivers, "Extending ICMP for Interface and 1091 Next-Hop Identification", RFC 5837, DOI 10.17487/RFC5837, 1092 April 2010, . 1094 [RFC6282] Hui, J., Ed. and P. Thubert, "Compression Format for IPv6 1095 Datagrams over IEEE 802.15.4-Based Networks", RFC 6282, 1096 DOI 10.17487/RFC6282, September 2011, 1097 . 1099 [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, 1100 "IPv6 Flow Label Specification", RFC 6437, 1101 DOI 10.17487/RFC6437, November 2011, 1102 . 1104 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 1105 for Equal Cost Multipath Routing and Link Aggregation in 1106 Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011, 1107 . 1109 [RFC6564] Krishnan, S., Woodyatt, J., Kline, E., Hoagland, J., and 1110 M. Bhatia, "A Uniform Format for IPv6 Extension Headers", 1111 RFC 6564, DOI 10.17487/RFC6564, April 2012, 1112 . 1114 [RFC6673] Morton, A., "Round-Trip Packet Loss Metrics", RFC 6673, 1115 DOI 10.17487/RFC6673, August 2012, 1116 . 1118 [RFC7045] Carpenter, B. and S. Jiang, "Transmission and Processing 1119 of IPv6 Extension Headers", RFC 7045, 1120 DOI 10.17487/RFC7045, December 2013, 1121 . 1123 [RFC7312] Fabini, J. and A. Morton, "Advanced Stream and Sampling 1124 Framework for IP Performance Metrics (IPPM)", RFC 7312, 1125 DOI 10.17487/RFC7312, August 2014, 1126 . 1128 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1129 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1130 May 2016, . 1132 [RFC7820] Mizrahi, T., "UDP Checksum Complement in the One-Way 1133 Active Measurement Protocol (OWAMP) and Two-Way Active 1134 Measurement Protocol (TWAMP)", RFC 7820, 1135 DOI 10.17487/RFC7820, March 2016, 1136 . 1138 [RFC8029] Kompella, K., Swallow, G., Pignataro, C., Ed., Kumar, N., 1139 Aldrin, S., and M. Chen, "Detecting Multiprotocol Label 1140 Switched (MPLS) Data-Plane Failures", RFC 8029, 1141 DOI 10.17487/RFC8029, March 2017, 1142 . 1144 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1145 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1146 May 2017, . 1148 13.2. Informative References 1150 [bdrmap] Luckie, M., Dhamdhere, A., Huffaker, B., Clark, D., and 1151 KC. Claffy, "bdrmap: Inference of Borders Between IP 1152 Networks", In Proceedings of the 2016 ACM on Internet 1153 Measurement Conference, pp. 381-396. ACM, 2016. 1155 [IDCong] Luckie, M., Dhamdhere, A., Clark, D., and B. Huffaker, 1156 "Challenges in inferring Internet interdomain 1157 congestion", In Proceedings of the 2014 Conference on 1158 Internet Measurement Conference, pp. 15-22. ACM, 2014. 1160 [MLB] Augustin, B., Friedman, T., and R. Teixeira, "Measuring 1161 load-balanced paths in the Internet", Proceedings of the 1162 7th ACM SIGCOMM conference on Internet measurement, pp. 1163 149-160. ACM, 2007., 2007. 1165 [MLRM] Fontugne, R., Mazel, J., and K. Fukuda, "An empirical 1166 mixture model for large-scale RTT measurements", 2015 1167 IEEE Conference on Computer Communications (INFOCOM), pp. 1168 2470-2478. IEEE, 2015., 2015. 1170 [P2] Jain, R. and I. Chlamtac, "The P 2 algorithm for dynamic 1171 calculation of quantiles and histograms without storing 1172 observations", Communications of the ACM 28.10 (1985): 1173 1076-1085, 2015. 1175 [PT] Augustin, B., Cuvellier, X., Orgogozo, B., Viger, F., 1176 Friedman, T., Latapy, M., Magnien, C., and R. Teixeira, 1177 "Avoiding traceroute anomalies with Paris traceroute", 1178 Proceedings of the 6th ACM SIGCOMM conference on Internet 1179 measurement, pp. 153-158. ACM, 2006., 2006. 1181 [RFC7325] Villamizar, C., Ed., Kompella, K., Amante, S., Malis, A., 1182 and C. Pignataro, "MPLS Forwarding Compliance and 1183 Performance Requirements", RFC 7325, DOI 10.17487/RFC7325, 1184 August 2014, . 1186 [RFC7594] Eardley, P., Morton, A., Bagnulo, M., Burbridge, T., 1187 Aitken, P., and A. Akhter, "A Framework for Large-Scale 1188 Measurement of Broadband Performance (LMAP)", RFC 7594, 1189 DOI 10.17487/RFC7594, September 2015, 1190 . 1192 [RFC8403] Geib, R., Ed., Filsfils, C., Pignataro, C., Ed., and N. 1193 Kumar, "A Scalable and Topology-Aware MPLS Data-Plane 1194 Monitoring System", RFC 8403, DOI 10.17487/RFC8403, July 1195 2018, . 1197 [RTTSub] Bischof, Z., Rula, J., and F. Bustamante, "In and out of 1198 Cuba: Characterizing Cuba's connectivity", In Proceedings 1199 of the 2015 ACM Conference on Internet Measurement 1200 Conference, pp. 487-493. ACM, 2015. 1202 [SCAMPER] Matthew Luckie, M., "Scamper: a scalable and extensible 1203 packet prober for active measurement of the 1204 Internet", Proceedings of the 10th ACM SIGCOMM conference 1205 on Internet measurement, pp. 239-245. ACM, 2010., 2010. 1207 [SSNT] Park, K. and W. Willinger, "Self-Similar Network Traffic 1208 and Performance Evaluation (1st ed.)", John Wiley & Sons, 1209 Inc., New York, NY, USA, 2000. 1211 Authors' Addresses 1213 J. Ignacio Alvarez-Hamelin 1214 Universidad de Buenos Aires 1215 Av. Paseo Colon 850 1216 Buenos Aires C1063ACV 1217 Argentine 1219 Phone: +54 11 5285-0716 1220 Email: ihameli@cnet.fi.uba.ar 1221 URI: http://cnet.fi.uba.ar/ignacio.alvarez-hamelin/ 1222 Al Morton 1223 AT&T Labs 1224 200 Laurel Avenue South 1225 Middletown, NJ 07748 1226 USA 1228 Phone: +1 732 420 1571 1229 Fax: +1 732 368 1192 1230 Email: acm@research.att.com 1232 Joachim Fabini 1233 TU Wien 1234 Gusshausstrasse 25/E389 1235 Vienna 1040 1236 Austria 1238 Phone: +43 1 58801 38813 1239 Fax: +43 1 58801 38898 1240 Email: Joachim.Fabini@tuwien.ac.at 1241 URI: http://www.tc.tuwien.ac.at/about-us/staff/joachim-fabini/ 1243 Carlos Pignataro 1244 Cisco Systems, Inc. 1245 7200-11 Kit Creek Road 1246 Research Triangle Park, NC 27709 1247 USA 1249 Email: cpignata@cisco.com 1251 Ruediger Geib 1252 Deutsche Telekom 1253 Heinrich Hertz Str. 3-7 1254 Darmstadt 64295 1255 Germany 1257 Phone: +49 6151 5812747 1258 Email: Ruediger.Geib@telekom.de