idnits 2.17.1 draft-ietf-ippm-route-10.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). (Using the creation date from RFC2330, updated by this document, for RFC5378 checks: 1998-05-01) -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (August 13, 2020) is 1349 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'Dst' is mentioned on line 907, but not defined == Missing Reference: 'Hop' is mentioned on line 907, but not defined == Missing Reference: 'Alt' is mentioned on line 907, but not defined == Outdated reference: A later version (-17) exists of draft-ietf-ippm-ioam-data-10 ** Downref: Normative reference to an Informational RFC: RFC 2330 ** Downref: Normative reference to an Informational RFC: RFC 7799 ** Downref: Normative reference to an Informational RFC: RFC 8468 Summary: 3 errors (**), 0 flaws (~~), 6 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group J. Alvarez-Hamelin 3 Internet-Draft Universidad de Buenos Aires 4 Updates: 2330 (if approved) A. Morton 5 Intended status: Standards Track AT&T Labs 6 Expires: February 14, 2021 J. Fabini 7 TU Wien 8 C. Pignataro 9 Cisco Systems, Inc. 10 R. Geib 11 Deutsche Telekom 12 August 13, 2020 14 Advanced Unidirectional Route Assessment (AURA) 15 draft-ietf-ippm-route-10 17 Abstract 19 This memo introduces an advanced unidirectional route assessment 20 (AURA) metric and associated measurement methodology, based on the IP 21 Performance Metrics (IPPM) Framework RFC 2330. This memo updates RFC 22 2330 in the areas of path-related terminology and path description, 23 primarily to include the possibility of parallel subpaths between a 24 given Source and Destination pair, owing to the presence of multi- 25 path technologies. 27 Status of This Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at https://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on February 14, 2021. 44 Copyright Notice 46 Copyright (c) 2020 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents 51 (https://trustee.ietf.org/license-info) in effect on the date of 52 publication of this document. Please review these documents 53 carefully, as they describe your rights and restrictions with respect 54 to this document. Code Components extracted from this document must 55 include Simplified BSD License text as described in Section 4.e of 56 the Trust Legal Provisions and are provided without warranty as 57 described in the Simplified BSD License. 59 Table of Contents 61 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 62 1.1. Issues with Earlier Work to Define a Route Metric . . . . 3 63 1.2. Requirements Language . . . . . . . . . . . . . . . . . . 4 64 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 65 3. Route Metric Specifications . . . . . . . . . . . . . . . . . 5 66 3.1. Terms and Definitions . . . . . . . . . . . . . . . . . . 5 67 3.2. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 6 68 3.3. Parameters . . . . . . . . . . . . . . . . . . . . . . . 6 69 3.4. Metric Definitions . . . . . . . . . . . . . . . . . . . 7 70 3.5. Related Round-Trip Delay and Loss Definitions . . . . . . 9 71 3.6. Discussion . . . . . . . . . . . . . . . . . . . . . . . 10 72 3.7. Reporting the Metric . . . . . . . . . . . . . . . . . . 10 73 4. Route Assessment Methodologies . . . . . . . . . . . . . . . 11 74 4.1. Active Methodologies . . . . . . . . . . . . . . . . . . 11 75 4.1.1. Temporal Composition for Route Metrics . . . . . . . 13 76 4.1.2. Routing Class Identification . . . . . . . . . . . . 15 77 4.1.3. Intermediate Observation Point Route Measurement . . 16 78 4.2. Hybrid Methodologies . . . . . . . . . . . . . . . . . . 16 79 4.3. Combining Different Methods . . . . . . . . . . . . . . . 17 80 5. Background on Round-Trip Delay Measurement Goals . . . . . . 17 81 6. RTD Measurements Statistics . . . . . . . . . . . . . . . . . 18 82 7. Security Considerations . . . . . . . . . . . . . . . . . . . 20 83 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 84 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 21 85 10. Appendix I MPLS Methods for Route Assessment . . . . . . . . 21 86 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 87 11.1. Normative References . . . . . . . . . . . . . . . . . . 22 88 11.2. Informative References . . . . . . . . . . . . . . . . . 24 89 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 26 91 1. Introduction 93 The IETF IP Performance Metrics (IPPM) working group first created a 94 framework for metric development in [RFC2330]. This framework has 95 stood the test of time and enabled development of many fundamental 96 metrics. It has been updated in the area of metric composition 98 [RFC5835], and in several areas related to active stream measurement 99 of modern networks with reactive properties [RFC7312]. 101 The [RFC2330] framework motivated the development of "performance and 102 reliability metrics for paths through the Internet," and Section 5 of 103 [RFC2330] defines terms that support description of a path under 104 test. However, metrics for assessment of paths and related 105 performance aspects had not been attempted in IPPM when the [RFC2330] 106 framework was written. 108 This memo takes up the route measurement challenge and specifies a 109 new route metric, two practical frameworks for methods of measurement 110 (using either active or hybrid active-passive methods [RFC7799]), and 111 Round-Trip Delay and link information discovery using the results of 112 measurements. All route measurements are limited by the willingness 113 of hosts along the path to be discovered, to cooperate with the 114 methods used, or to recognize that the measurement operation is 115 taking place (such as when tunnels are present). 117 1.1. Issues with Earlier Work to Define a Route Metric 119 Section 7 of [RFC2330] presented a simple example of a "route" metric 120 along with several other examples. The example is reproduced below 121 (where the reference is to Section 5 of [RFC2330]): 123 "route: The path, as defined in Section 5, from A to B at a given 124 time." 126 This example provides a starting point to develop a more complete 127 definition of route. Areas needing clarification include: 129 Time: In practice, the route will be assessed over a time interval, 130 because active path detection methods like Paris Traceroute [PT] 131 rely on hop limits for their operation and cannot accomplish 132 discovery of all hosts using a single packet. 134 Type-P: The legacy route definition lacks the option to cater for 135 packet-dependent routing. In this memo, we assess the route for a 136 specific packet of Type-P, and reflect this in the metric 137 definition. The methods of measurement determine the specific 138 Type-P used. 140 Parallel Paths: Parallel paths are a reality of the Internet and a 141 strength of advanced route assessment methods, so the metric must 142 acknowledge this possibility. Use of Equal Cost Multi-Path (ECMP) 143 and Unequal Cost Multi-Path (UCMP) technologies are common sources 144 of parallel subpaths. 146 Cloud Subpath: May contain hosts that do not decrement hop limit, 147 but may have two or more exchange links connecting "discoverable" 148 hosts or routers. Parallel subpaths contained within clouds 149 cannot be discovered. The assessment methods only discover hosts 150 or routers on the path that decrement hop limit, or cooperate with 151 interrogation protocols. The presence of tunnels and nested 152 tunnels further complicate assessment by hiding hops. 154 Hop: Although the [RFC2330] definition of a hop was a link-host 155 pair, only hosts that are discoverable or have the capability to 156 cooperate with interrogation protocols where link information may 157 be exposed. 159 The refined definition of Route metrics begins in the sections that 160 follow. 162 1.2. Requirements Language 164 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 165 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 166 "OPTIONAL" in this document are to be interpreted as described in BCP 167 14[RFC2119] [RFC8174] when, and only when, they appear in all 168 capitals, as shown here. 170 2. Scope 172 The purpose of this memo is to add new route metrics and methods of 173 measurement to the existing set of IPPM metrics. 175 The scope is to define route metrics that can identify the path taken 176 by a packet or a flow traversing the Internet between two hosts. 177 Although primarily intended for hosts communicating on the Internet, 178 the definitions and metrics are constructed to be applicable to other 179 network domains, if desired. The methods of measurement to assess 180 the path may not be able to discover all hosts comprising the path, 181 but such omissions are often deterministic and explainable sources of 182 error. 184 This memo also specifies a framework for active methods of 185 measurement which uses the techniques described in [PT], as well as a 186 framework for hybrid active-passive methods of measurement such as 187 the Hybrid Type I method [RFC7799] described in 188 [I-D.ietf-ippm-ioam-data]. Methods using [I-D.ietf-ippm-ioam-data] 189 are intended only for single administrative domains that provide a 190 protocol for explicit interrogation of nodes on a path. Combinations 191 of active methods and hybrid active-passive methods are also in- 192 scope. 194 Further, this memo provides additional analysis of the round-trip 195 delay measurements made possible by the methods, in an effort to 196 discover more details about the path, such as the link technology in 197 use. 199 This memo updates Section 5 of [RFC2330] in the areas of path-related 200 terminology and path description, primarily to include the 201 possibility of parallel subpaths between a given Source and 202 Destination address pair (possibly resulting from Equal Cost Multi- 203 Path (ECMP) and Unequal Cost Multi-Path (UCMP) technologies). 205 There are several simple non-goals of this memo. There is no attempt 206 to assess the reverse path from any host on the path to the host 207 attempting the path measurement. The reverse path contribution to 208 delay will be that experienced by ICMP packets (in active methods), 209 and may be different from delays experienced by UDP or TCP packets. 210 Also, the round trip delay will include an unknown contribution of 211 processing time at the host that generates the ICMP response. 212 Therefore, the ICMP-based active methods are not supposed to yield 213 accurate, reproducible estimations of the Round-Trip Delay that UDP 214 or TCP packets will experience. 216 3. Route Metric Specifications 218 This section sets requirements for the components of the Route 219 Metric. 221 3.1. Terms and Definitions 223 Host A Host as defined in [RFC2330] (a computer capable of IP 224 communication, includes routers), a.k.a. RFC 2330 Host. 226 Node A Node is any network function on the path capable of IP-layer 227 Communication, includes RFC 2330 Hosts. 229 Node Identity The unique address for Nodes communicating within the 230 network domain. For Nodes communicating on the Internet with IP, 231 it is the globally routable IP address which the Node uses when 232 communicating with other Nodes under normal or error conditions. 233 The Node Identity revealed (and its connection to a Node Name 234 through reverse DNS) determines whether interfaces to parallel 235 links can be associated with a single Node, or appear to identify 236 unique Nodes. 238 Discoverable Node Nodes that convey their Node Identity according to 239 the requirements of their network domain, such as when error 240 conditions are detected by that Node. For Nodes communicating 241 with IP packets, compliance with Section 3.2.2.4 of [RFC1122] when 242 discarding a packet due to TTL or Hop Limit Exceeded condition, 243 MUST result in sending the corresponding Time Exceeded message 244 (containing a form of Node identity) to the source. This 245 requirement is also consistent with section 5.3.1 of [RFC1812] for 246 routers. 248 Cooperating Node Nodes that respond to direct queries for their Node 249 identity as part of a previously agreed and established 250 interrogation protocol. Nodes SHOULD also provide information 251 such as arrival/departure interface identification, arrival 252 timestamp, and any relevant information about the Node or specific 253 link which delivered the query to the Node. 255 Hop specification A Hop specification MUST contain a Node Identity, 256 and MAY contain arrival and/or departure interface identification, 257 round trip delay, and an arrival timestamp. 259 Routing Class A route that treats equally a class of different types 260 of packets, designated "C" (unrelated to address classes of the 261 past) [RFC2330] [RFC8468]. Knowledge of such a class allows any 262 one of the types of packets within that class to be used for 263 subsequent measurement of the route. The designator "class C" is 264 used for historical reasons, see [RFC2330]. 266 3.2. Formal Name 268 The formal name of the metric is: 270 Type-P-Route-Ensemble-Method-Variant 272 abbreviated as Route Ensemble. 274 Note that Type-P depends heavily on the chosen method and variant. 276 3.3. Parameters 278 This section lists the REQUIRED input factors to define and measure a 279 Route metric, as specified in this memo. 281 o Src, the address of a Node (such as the globally routable IP 282 address). 284 o Dst, the address of a Node (such as the globally routable IP 285 address). 287 o i, the limit on the number of Hops a specific packet may visit as 288 it traverses from the Node at Src to the Node at Dst (such as the 289 TTL or Hop Limit). 291 o MaxHops, the maximum value of i used, (i=1,2,3,...MaxHops). 293 o T0, a time (start of measurement interval) 295 o Tf, a time (end of measurement interval) 297 o MP(address), Measurement Point at address, such as Src or Dst, 298 usually at the same node stack layer as "address". 300 o T, the Node time of a packet as measured at MP(Src), meaning 301 Measurement Point at the Source. 303 o Ta, the Node time of a reply packet's *arrival* as measured at 304 MP(Src), assigned to packets that arrive within a "reasonable" 305 time (see parameter below). 307 o Tmax, a maximum waiting time for reply packets to return to the 308 source, set sufficiently long to disambiguate packets with long 309 delays from packets that are discarded (lost), such that the 310 distribution of Round-Trip Delay is not truncated. 312 o F, the number of different flows simulated by the method and 313 variant. 315 o flow, the stream of packets with the same n-tuple of designated 316 header fields that (when held constant) result in identical 317 treatment in a multi-path decision (such as the decision taken in 318 load balancing). Note: The IPv6 flow label MAY be included in the 319 flow definition if the MP(Src) is a Tunnel End Point (TEP) 320 complying with [RFC6438] guidelines. 322 o Type-P, the complete description of the packets for which this 323 assessment applies (including the flow-defining fields). 325 3.4. Metric Definitions 327 This section defines the REQUIRED measurement components of the Route 328 metrics (unless otherwise indicated): 330 M, the total number of packets sent between T0 and Tf. 332 N, the smallest value of i needed for a packet to be received at Dst 333 (sent between T0 and Tf). 335 Nmax, the largest value of i needed for a packet to be received at 336 Dst (sent between T0 and Tf). Nmax may be equal to N. 338 Next define a *singleton* definition for a Node on the path, with 339 sufficient indexes to identify all Nodes identified in a measurement 340 interval (where *singleton* is part of the IPPM Framework [RFC2330]). 342 A Hop Specification, designated h(i,j), the IP address and/or 343 identity of Discoverable Nodes (or Cooperating Nodes) that are i hops 344 away from the Node with address = Src and part of Route j during the 345 measurement interval, T0 to Tf. As defined here, a Hop singleton 346 measurement MUST contain a Node Identity, hid(i,j), and MAY contain 347 one or more of the following attributes: 349 o a(i,j) Arrival Interface ID (e.g., when [RFC5837] is supported) 351 o d(i,j) Departure Interface ID (e.g., when [RFC5837] is supported) 353 o t(i,j) Arrival Timestamp, where t(i,j) is ideally supplied by the 354 Hop. (Note that t(i,j) might be approximated from the sending time 355 of the packet that revealed the Hop, e.g., when the round trip 356 response time is available and divided by 2.) 358 o Measurements of Round-Trip Delay (for each packet that reveals the 359 same Node Identity and flow attributes, then this attribute is 360 computed, see next section) 362 Node Identities and related information can be ordered by their 363 distance from the Node with address Src in Hops h(i,j). Based on 364 this, two forms of Routes are distinguished: 366 A Route Ensemble is defined as the combination of all routes 367 traversed by different flows from the Node at Src address to the Node 368 at Dst address. A single Route traversed by a single flow 369 (determined by an unambiguous tuple of addresses Src and Dst, and 370 other identical flow criteria) is a member of the Route Ensemble and 371 called a Member Route. 373 Using h(i,j) and components and parameters, further define: 375 When considering the set of Hops in the context of a single flow, a 376 Member Route j is an ordered list {h(1,j), ... h(Nj, j)} where h(i-1, 377 j) and h(i, j) are 1 hop away from each other and Nj satisfying 378 h(Nj,j)=Dst is the minimum count of Hops needed by the packet on 379 Member Route j to reach Dst. Member Routes must be unique. The 380 uniqueness property requires that any two Member routes j and k that 381 are part of the same Route Ensemble differ either in terms of minimum 382 hop count Nj and Nk to reach the destination Dst, or, in the case of 383 identical hop count Nj=Nk, they have at least one distinct Hop: 384 h(i,j) != h(i,k) for at least one i (i=1..Nj). 386 All the optional information collected to describe a Member Route, 387 such as the arrival interface, departure interface, and Round Trip 388 Delay at each Hop, turns each list item into a rich structure. There 389 may be information on the links between Hops, possibly information on 390 the routing (arrival interface and departure interface), an estimate 391 of distance between Hops based on Round-Trip Delay measurements and 392 calculations, and a time stamp indicating when all these additional 393 details were valid. 395 The Route Ensemble from Src to Dst, during the measurement interval 396 T0 to Tf, is the aggregate of all m distinct Member Routes discovered 397 between the two Nodes with Src and Dst addresses. More formally, 398 with the Node having address Src omitted: 400 Route Ensemble = { 401 {h(1,1), h(2,1), h(3,1), ... h(N1,1)=Dst}, 402 {h(1,2), h(2,2), h(3,2),..., h(N2,2)=Dst}, 403 ... 404 {h(1,m), h(2,m), h(3,m), ....h(Nm,m)=Dst} 405 } 407 where the following conditions apply: i <= Nj <= Nmax (j=1..m) 409 Note that some h(i,j) may be empty (null) in the case that systems do 410 not reply (not discoverable, or not cooperating). 412 h(i-1,j) and h(i,j) are the Hops on the same Member Route one hop 413 away from each other. 415 Hop h(i,j) may be identical with h(k,l) for i!=k and j!=l ; which 416 means there may be portions shared among different Member Routes 417 (parts of Member Routes may overlap). 419 3.5. Related Round-Trip Delay and Loss Definitions 421 RTD(i,j,T) is defined as a singleton of the [RFC2681] Round-Trip 422 Delay between the Node with address = Src and the Node at Hop h(i,j) 423 at time T. 425 RTL(i,j,T) is defined as a singleton of the [RFC6673] Round-trip Loss 426 between the Node with address = Src and the Node at Hop h(i,j) at 427 time T. 429 3.6. Discussion 431 Depending on the way that Node Identity is revealed, it may be 432 difficult to determine parallel subpaths between the same pair of 433 Nodes (i.e. multiple parallel links). It is easier to detect 434 parallel subpaths involving different Nodes. 436 o If a pair of discovered Nodes identify two different addresses (IP 437 or not), then they will appear to be different Nodes. See item 438 below. 440 o If a pair of discovered Nodes identify two different IP addresses, 441 and the IP addresses resolve to the same Node name (in the DNS), 442 then they will appear to be the same Nodes. 444 o If a discovered Node always replies using the same network 445 address, regardless of the interface a packet arrives on, then 446 multiple parallel links cannot be detected in that network domain. 447 This condition may apply to traceroute-style methods, but may not 448 apply to other hybrid methods based on In-situ Operations, 449 Administration, and Maintenance (IOAM). For example, if the 450 [RFC5837] ICMP extension mechanism is implemented, then parallel 451 links can be detected with the discovery traceroute-style methods. 453 o If parallel links between routers are aggregated below the IP 454 layer, then from Node point of view, all these links share the 455 same pair of IP addresses. The existence of these parallel links 456 can't be detected at the IP layer. This applies to other network 457 domains with layers below them, as well. This condition may apply 458 to traceroute-style methods, but may not apply to other hybrid 459 methods based on IOAM. 461 When a route assessment employs IP packets (for example), the reality 462 of flow assignment to parallel subpaths involves layers above IP. 463 Thus, the measured Route Ensemble is applicable to IP and higher 464 layers (as described in the methodology's packet of Type-P and flow 465 parameters). 467 3.7. Reporting the Metric 469 An Information Model and an XML Data Model for Storing Traceroute 470 Measurements is available in [RFC5388]. The measured information at 471 each hop includes four pieces of information: a one-dimensional hop 472 index, Node symbolic address, Node IP address, and RTD for each 473 response. 475 The description of Hop information that may be collected according to 476 this memo covers more dimensions, as defined in Section 3.4 above. 478 For example, the Hop index is two-dimensional to capture the 479 complexity of a Route Ensemble, and it contains corresponding Node 480 identities at a minimum. The models need to be expanded to include 481 these features, as well as Arrival Interface ID, Departure Interface 482 ID, and Arrival Timestamp, when available. The original sending 483 Timestamp from the Src Node anchors a particular measurement in time. 485 4. Route Assessment Methodologies 487 There are two classes of methods described in this section, active 488 methods relying on the reaction to TTL or Hop Limit Exceeded 489 condition to discover Nodes on a path, and Hybrid active-passive 490 methods that involve direct interrogation of cooperating Nodes 491 (usually within a single domain). Description of these methods 492 follow. 494 4.1. Active Methodologies 496 This section describes the method employed by current open source 497 tools, thereby providing a practical framework for further advanced 498 techniques to be included as method variants. This method is 499 applicable for use across multiple administrative domains. 501 Internet routing is complex because it depends on the policies of 502 thousands of Autonomous Systems (AS). Most routers perform load 503 balancing on flows using a form of Equal Cost Multiple Path (ECMP). 504 [RFC2991] describes a number of flow-based or hashed approaches 505 (e.g., Modulo-N Hash, Hash-Threshold, Highest Random Weight (HRW)), 506 and makes some good suggestions. Flow-based ECMP avoids increased 507 packet delay variation and possibly overwhelming levels of packet 508 reordering in flows. 510 A few routers still divide the workload through packet-based 511 techniques, such as a round-robin scheme to distribute every new 512 outgoing packet to multiple links, as explained in [RFC2991]. The 513 methods described in this section assume flow-based ECMP. 515 Taking into account that Internet protocol was designed under the 516 "end-to-end" principle, the IP payload and its header do not provide 517 any information about the routes or path necessary to reach some 518 destination. For this reason, the popular tool traceroute was 519 developed to gather the IP addresses of each hop along a path using 520 the ICMP protocol [RFC0792]. Traceroute also measures RTD from each 521 hop. However, the growing complexity of the Internet makes it more 522 challenging to develop an accurate traceroute implementation. For 523 instance, the early traceroute tools would be inaccurate in the 524 current network, mainly because they were not designed to retain a 525 flow state. However, evolved traceroute tools, such as Paris- 526 traceroute [PT] [MLB] and Scamper [SCAMPER], expect to encounter ECMP 527 and achieve more accurate results when they do, where Scamper ensures 528 traceroute packets will follow the same path in 98% of 529 cases[SCAMPER]. 531 Today's traceroute tools send Type-P of packets, either ICMP, UDP, or 532 TCP. UDP and TCP are used when a particular characteristic needs to 533 be verified, such as filtering or traffic shaping on specific ports 534 (i.e., services). UDP and TCP traceroute are also used when ICMP 535 responses are not received. [SCAMPER] supports IPv6 traceroute 536 measurements, keeping the FlowLabel constant in all packets. 538 Paris-traceroute allows its users to measure the RTD to every Node of 539 the path for a particular flow. Furthermore, either Paris-traceroute 540 or Scamper is capable of unveiling the many available paths between a 541 source and destination (which are visible to active methods). This 542 task is accomplished by repeating complete traceroute measurements 543 with different flow parameters for each measurement; Paris-traceroute 544 provides "exhaustive" mode while scamper provides "tracelb" (stands 545 for traceroute load balance). The Framework for IP Performance 546 Metrics (IPPM) ([RFC2330] updated by[RFC7312]) has the flexibility to 547 require that the Round-Trip Delay measurement [RFC2681] uses packets 548 with the constraints to assure that all packets in a single 549 measurement appear as the same flow. This flexibility covers ICMP, 550 UDP, and TCP. The accompanying methodology of [RFC2681] needs to be 551 expanded to report the sequential hop identifiers along with RTD 552 measurements, but no new metric definition is needed. 554 The advanced route assessment methods used in Paris-traceroute [PT] 555 keep the critical fields constant for every packet to maintain the 556 appearance of the same flow. When considering IPv6 headers, it is 557 necessary to ensure that the IP source and destination addresses and 558 the FlowLabel are constant (but note that many routers ignore the 559 FlowLabel field at this time), see [RFC6437]. Use of IPv6 Extension 560 Headers may add critical fields, and SHOULD be avoided. In IPv4, 561 certain fields of the IP header and the first four bytes of the IP 562 payload should remain constant in a flow. In the IPv4 header, the IP 563 source and destination addresses, protocol number, and Diffserv 564 fields identify flows. The first four payload bytes include the UDP 565 and TCP ports, and the ICMP type, code, and checksum fields. 567 Maintaining a constant ICMP checksum in IPv4 is most challenging, as 568 the ICMP sequence number or identifier fields will usually change for 569 different probes of the same path. Probes should use arbitrary bytes 570 in the ICMP data field to offset changes to sequence number and 571 identifier, thus keeping the checksum constant. 573 Finally, it is also essential to route the resulting ICMP Time 574 Exceeded messages along a consistent path. In IPv6, the fields above 575 are sufficient. In IPv4, the ICMP Time Exceeded message will contain 576 the IP header and the first eight bytes of the IP payload, which 577 affects its ICMP checksum. The TCP sequence number, UDP Length, and 578 UDP checksum will affect this value, and should remain constant. 580 Formally, to maintain the same flow in the measurements to a 581 particular hop, the Type-P-Route-Ensemble-Method-Variant packets 582 should be[PT]: 584 o TCP case: For IPv4, the fields Src, Dst, port-Src, port_Dst, 585 sequence number, and Diffserv Field SHOULD be the same. For IPv6, 586 the field FlowLabel, Src and Dst SHOULD be the same. 588 o UDP case: For IPv4, the fields Src, Dst, port-Src, port-Dst, 589 Diffserv should be the same, and the UDP-checksum SHOULD change to 590 keep the IP checksum of the ICMP time exceeded reply constant. 591 Then, the data length should be fixed, and the data field is used 592 to make it so (consider that ICMP checksum uses its data field, 593 which contains the original IP header plus 8 bytes of UDP, where 594 TTL, IP identification, IP checksum, and UDP checksum changes). 595 For IPv6, the field FlowLabel, and Source and Destination 596 addresses SHOULD be the same. 598 o ICMP case: For IPv4, the Data field SHOULD compensate variations 599 on TTL or Hop Limit, IP identification, and IP checksum for every 600 packet. There is no need to consider ICMPv6 because only 601 FlowLabel of IPv6 and Source and Destination addresses are used, 602 and all of them SHOULD be constant. 604 Then, the way to identify different hops and attempts of the same 605 IPv4 flow is: 607 o TCP case: The IP identification field. 609 o UDP case: The IP identification field. 611 o ICMP case: The IP identification field, and ICMP Sequence number. 613 4.1.1. Temporal Composition for Route Metrics 615 The Active Route Assessment Methods described above have the ability 616 to discover portions of a path where ECMP load balancing is present, 617 observed as two or more unique Member Routes having one or more 618 distinct Hops which are part of the Route Ensemble. Likewise, 619 attempts to deliberately vary the flow characteristics to discover 620 all Member Routes will reveal portions of the path which are flow- 621 invariant. 623 Section 9.2 of [RFC2330] describes Temporal Composition of metrics, 624 and introduces the possibility of a relationship between earlier 625 measurement results and the results for measurement at the current 626 time (for a given metric). There is value in establishing a Temporal 627 Composition relationship for Route Metrics. However, this 628 relationship does not represent a forecast of future route conditions 629 in any way. 631 For Route Metric measurements, the value of Temporal Composition is 632 to reduce the measurement iterations required with repeated 633 measurements. Reduced iterations are possible by inferring that 634 current measurements using fixed and previously measured flow 635 characteristics: 637 o will have many common hops with previous measurements. 639 o will have relatively time-stable results at the ingress and egress 640 portions of the path when measured from user locations, as opposed 641 to measurements of backbone networks and across inter-domain 642 gateways. 644 o may have greater potential for time-variation in path portions 645 where ECMP load balancing is observed (because increasing or 646 decreasing the pool of links changes the hash calculations). 648 Optionally, measurement systems may take advantage of the inferences 649 above when seeking to reduce measurement iterations, after exhaustive 650 measurements indicate that the time-stable properties are present. 651 Repetitive Active Route measurement systems: 653 1. SHOULD occasionally check path portions which have exhibited 654 stable results over time, particularly ingress and egress 655 portions of the path (e.g., daily checks if measuring many times 656 during a day). 658 2. SHOULD continue testing portions of the path that have previously 659 exhibited ECMP load balancing. 661 3. SHALL trigger re-assessment of the complete path and Route 662 Ensemble, if any change in hops is observed for a specific (and 663 previously tested) flow. 665 4.1.2. Routing Class Identification 667 There is an opportunity to apply the [RFC2330] notion of equal 668 treatment for a class of packets, "...very useful to know if a given 669 Internet component treats equally a class C of different types of 670 packets", as it applies to Route measurements. The notion of class C 671 was examined further in [RFC8468] as it applied to load-balancing 672 flows over parallel paths, which is the case we develop here. 673 Knowledge of class C parameters (unrelated to address classes of the 674 past) on a path potentially reduces the number of flows required for 675 a given method to assess a Route Ensemble over time. 677 First, recognize that each Member Route of a Route Ensemble will have 678 a corresponding class C. Class C can be discovered by testing with 679 multiple flows, all of which traverse the unique set of hops that 680 comprise a specific Member Route. 682 Second, recognize that the different classes depend primarily on the 683 hash functions used at each instance of ECMP load balancing on the 684 path. 686 Third, recognize the synergy with Temporal Composition methods 687 (described above), where evaluation intends to discover time-stable 688 portions of each Member Route, so that more emphasis can be placed on 689 ECMP portions that also determine class C. 691 The methods to assess the various class C characteristics benefit 692 from the following measurement capabilities: 694 o flows designed to determine which n-tuple header fields are 695 considered by a given hash function and ECMP hop on the path, and 696 which are not. This operation immediately narrows the search 697 space, where possible, and partially defines a class C. 699 o a priori knowledge of the possible types of hash functions in use 700 also helps to design the flows for testing (major router vendors 701 publish information about these hash functions, examples are here 702 [LOAD_BALANCE]. 704 o ability to direct the emphasis of current measurements on ECMP 705 portions of the path, based on recent past measurement results 706 (the Routing Class of some portions of the path is essentially 707 "all packets"). 709 4.1.3. Intermediate Observation Point Route Measurement 711 There are many examples where passive monitoring of a flow at an 712 Observation Point within the network can detect unexpected Round Trip 713 Delay or Delay Variation. But how can the cause of the anomalous 714 delay be investigated further *from the Observation Point* possibly 715 located at an intermediate point on the path? 717 In this case, knowledge that the flow of interest belongs to a 718 specific Routing Class C will enable measurement of the route where 719 anomalous delay has been observed. Specifically, Round-Trip Delay 720 assessment to each Hop on the path between the Observation Point and 721 the Destination for the flow of interest may discover high or 722 variable delay on a specific link and Hop combination. 724 The determination of a Routing Class C which includes the flow of 725 interest is as described in the section above, aided by computation 726 of the relevant hash function output as the target. 728 4.2. Hybrid Methodologies 730 The Hybrid Type I methods provide an alternative method for Route 731 Member assessment. As mentioned in the Scope section, 732 [I-D.ietf-ippm-ioam-data] provides a possible set of data fields that 733 would support route identification. 735 In general, nodes in the measured domain would be equipped with 736 specific abilities: 738 o Store the identity of nodes that a packet has visited in header 739 data fields, in the order the packet visited the nodes. 741 o Support of a "Loopback" capability, where a copy of the packet is 742 returned to the encapsulating node, and the packet is processed 743 like any other IOAM packet on the return transfer. 745 In addition to node identity, nodes may also identify the ingress and 746 egress interfaces utilized by the tracing packet, the absolute time 747 when the packet was processed, and other generic data (as described 748 in section 4 of [I-D.ietf-ippm-ioam-data]). Interface identification 749 isn't necessarily limited to IP, i.e. different links in a bundle 750 (LACP) could be identified. Equally well, links without explicit IP 751 addresses can be identified (like with unnumbered interfaces in an 752 IGP deployment). 754 Note that the Type-P packet specification for this method will likely 755 be a partial specification, because most of the packet fields are 756 determined by the user traffic. The packet (encapsulation) header(s) 757 added by the Hybrid method can certainly be specified in Type-P, in 758 unpopulated form. 760 4.3. Combining Different Methods 762 In principle, there are advantages if the entity conducting Route 763 measurements can utilize both forms of advanced methods (active and 764 hybrid), and combine the results. For example, if there are Nodes 765 involved in the path that qualify as Cooperating Nodes, but not as 766 Discoverable Nodes, then a more complete view of Hops on the path is 767 possible when a hybrid method (or interrogation protocol) is applied 768 and the results are combined with the active method results collected 769 across all other domains. 771 In order to combine the results of active and hybrid/interrogation 772 methods, the network Nodes that are part of a domain supporting an 773 interrogation protocol have the following attributes: 775 1. Nodes at the ingress to the domain SHOULD be both Discoverable 776 and Cooperating. 778 2. Any Nodes within the domain that are both Discoverable and 779 Cooperating SHOULD reveal the same Node Identity in response to 780 both active and hybrid methods. 782 3. Nodes at the egress to the domain SHOULD be both Discoverable and 783 Cooperating, and SHOULD reveal the same Node Identity in response 784 to both active and hybrid methods. 786 When Nodes follow these requirements, it becomes a simple matter to 787 match single domain measurements with the overlapping results from a 788 multidomain measurement. 790 In practice, Internet users do not typically have the ability to 791 utilize the OAM capabilities of networks that their packets traverse, 792 so the results from a remote domain supporting an interrogation 793 protocol would not normally be accessible. However, a network 794 operator could combine interrogation results from their access domain 795 with other measurements revealing the path outside their domain. 797 5. Background on Round-Trip Delay Measurement Goals 799 The aim of this method is to use packet probes to unveil the paths 800 between any two end-Nodes of the network. Moreover, information 801 derived from RTD measurements might be meaningful to identify: 803 1. Intercontinental submarine links 804 2. Satellite communications 806 3. Congestion 808 4. Inter-domain paths 810 This categorization is widely accepted in the literature and among 811 operators alike, and it can be trusted with empirical data and 812 several sources as ground of truth (e.g., [RTTSub] ) but it is an 813 inference measurement nonetheless [bdrmap][IDCong]. 815 The first two categories correspond to the physical distance 816 dependency on Round-Trip Delay (RTD), the next one binds RTD with 817 queuing delay on routers, and the last one helps to identify 818 different ASes using traceroutes. Due to the significant 819 contribution of propagation delay in long-distance hops, RTD will be 820 on the order of 100ms on transatlantic hops, depending on the 821 geolocation of the vantage points. Moreover, RTD is typically higher 822 than 480ms when two hops are connected using geostationary satellite 823 technology (i.e., their orbit is at 36000km). Detecting congestion 824 with latency implies deeper mathematical understanding since network 825 traffic load is not stationary. Nonetheless, as the first approach, 826 a link seems to be congested if observing different/varying 827 statistical results after sending several traceroute probes (e.g., 828 see [IDCong]). Finally, to recognize distinctive ASes in the same 829 traceroute path is challenging, because more data is needed, like AS 830 relationships and RIR delegations among other (for more detail, 831 please consult [bdrmap]). 833 6. RTD Measurements Statistics 835 Several articles have shown that network traffic presents a self- 836 similar nature [SSNT] [MLRM] which is accountable for filling the 837 queues of the routers. Moreover, router queues are designed to 838 handle traffic bursts, which is one of the most remarkable features 839 of self-similarity. Naturally, while queue length increases, the 840 delay to traverse the queue increases as well and leads to an 841 increase on RTD. Due to traffic bursts generating short-term 842 overflow on buffers (spiky patterns), every RTD only depicts the 843 queueing status on the instant when that packet probe was in transit. 844 For this reason, several RTD measurements during a time window could 845 begin to describe the random behavior of latency. Loss must also be 846 accounted for in the methodology. 848 To understand the ongoing process, examining the quartiles provides a 849 non-parametric way of analysis. Quartiles are defined by five 850 values: minimum RTD (m), RTD value of the 25% of the Empirical 851 Cumulative Distribution Function (ECDF) (Q1), the median value (Q2), 852 the RTD value of the 75% of the ECDF (Q3) and the maximum RTD (M). 853 Congestion can be inferred when RTD measurements are spread apart, 854 and consequently, the Inter-Quartile Range (IQR), the distance 855 between Q3 and Q1, increases its value. 857 This procedure requires the algorithm presented in [P2] to compute 858 quartile values "on the fly". 860 This procedure allows us to update the quartiles value whenever a new 861 measurement arrives, which is radically different from classic 862 methods of computing quartiles because they need to use the whole 863 dataset to compute the values. This way of calculus provides savings 864 in memory and computing time. 866 To sum up, the proposed measurement procedure consists of performing 867 traceroutes several times to obtain samples of the RTD in every hop 868 from a path, during a time window (W), and compute the quartiles for 869 every hop. This procedure could be done for a single Member Route 870 flow, a non-exhaustive search with parameter E (defined below) set as 871 False, or for every detected Route Ensemble flow (E=True). 873 The identification of a specific Hop in traceroute is based on the IP 874 origin address of the returned ICMP Time Exceeded packet, and on the 875 distance identified by the value set in the TTL (or Hop Limit) field 876 inserted by traceroute. As this specific Hop can be reached by 877 different paths, also the IP source and destination addresses of the 878 traceroute packet need to be recorded. Finally, different return 879 paths are distinguished by evaluating the ICMP Time Exceeded TTL (or 880 Hop Limit) of the reply message: if this TTL (or Hop Limit) is 881 constant for different paths containing the same Hop, the return 882 paths have the same distance. Moreover, this distance can be 883 estimated considering that the TTL (or Hop Limit) value is normally 884 initialized with values 64, 128, or 255. The 5-tuple (origin IP, 885 destination IP, reply IP, distance, response TTL or Hop Limit) 886 unequivocally identifies every measurement. 888 This algorithm below runs in the origin of the traceroute. It 889 returns the Qs quartiles for every Hop and Alt (alternative paths 890 because of balancing). Notice that the "Alt" parameter condenses the 891 parameters of the 5-tuple (origin IP, destination IP, reply IP, 892 distance, response TTL), i.e., one for each possible combination. 894 ================================================================ 895 0 input: W (window time of the measurement) 896 1 i_t (time between two measurements, set the i_t time 897 2 long enough to avoid incomplete results) 898 3 E (True: exhaustive, False: a single path) 899 4 Dst (destination IP address) 900 5 output: Qs (quartiles for every Hop and Alt) 901 ---------------------------------------------------------------- 902 6 T := start_timer(W) 903 7 while T is not finished do: 904 8 | start_timer(i_t) 905 9 | RTD(Hop,Alt) = advanced-traceroute(Dst,E) 906 10 | for each Hop and Alt in RTD do: 907 11 | | Qs[Dst,Hop,Alt] := ComputeQs(RTD(Hop,Alt)) 908 12 | done 909 13 | wait until i_t timer is expired 910 14 done 911 15 return (Qs) 912 ================================================================ 914 During the time W, lines 6 and 7 assure that the measurement loop is 915 made. Line 8 and 13 set a timer for each cycle of measurements. A 916 cycle comprises the traceroutes packets, considering every possible 917 Hop and the alternatives paths in the Alt variable (ensured in lines 918 9-12). In line 9, the advance-traceroute could be either Paris- 919 traceroute or Scamper, which will use the "exhaustive" mode or 920 "tracelb" option if E is set True, respectively. The procedure 921 returns a list of tuples (m,Q1,Q2,Q3,M) for each intermediate hop, or 922 "Alt" in as a function of the 5-tuple, in the path towards the Dst. 923 Finally, lines 10 through 12 stores each measurement into the real- 924 time quartiles computation. 926 Notice there are cases where the even having a unique hop at distance 927 h from the Src to Dst, the returning path could have several 928 possibilities, yielding in different total paths. In this situation, 929 the algorithm will return more "Alt" for this particular hop. 931 7. Security Considerations 933 The security considerations that apply to any active measurement of 934 live paths are relevant here as well. See [RFC4656] and [RFC5357]. 936 The active measurement process of "changing several fields to keep 937 the checksum of different packets identical" does not require special 938 security considerations because it is part of synthetic traffic 939 generation, and is designed to have minimal to zero impact on network 940 processing (to process the packets for ECMP). 942 Some of the protocols used (e.g., ICMP) do not provide cryptographic 943 protection for the requested/returned data, and there are risks of 944 processing untrusted data in general, but these are limitations of 945 the existing protocols where we are applying new methods. 947 For applicable Hybrid methods, the security considerations 948 in[I-D.ietf-ippm-ioam-data] apply. 950 When considering privacy of those involved in measurement or those 951 whose traffic is measured, the sensitive information available to 952 potential observers is greatly reduced when using active techniques 953 which are within this scope of work. Passive observations of user 954 traffic for measurement purposes raise many privacy issues. We refer 955 the reader to the privacy considerations described in the Large Scale 956 Measurement of Broadband Performance (LMAP) Framework [RFC7594], 957 which covers active and passive techniques. 959 8. IANA Considerations 961 This memo makes no requests of IANA. We thank the good folks at IANA 962 for having checked this section anyway. 964 9. Acknowledgements 966 The original 3 authors (Ignacio, Al, Joachim) acknowledge Ruediger 967 Geib, for his penetrating comments on the initial draft, and his 968 initial text for the Appendix on MPLS. Carlos Pignataro challenged 969 the authors to consider a wider scope, and applied his substantial 970 expertise with many technologies and their measurement features in 971 his extensive comments. Frank Brockners also shared useful comments, 972 so did Footer Foote. We thank them all! 974 10. Appendix I MPLS Methods for Route Assessment 976 A Node assessing an MPLS path must be part of the MPLS domain where 977 the path is implemented. When this condition is met, RFC 8029 978 provides a powerful set of mechanisms to detect "correct operation of 979 the data plane, as well as a mechanism to verify the data plane 980 against the control plane" [RFC8029]. 982 MPLS routing is based on the presence of a Forwarding Equivalence 983 Class (FEC) Stack in all visited Nodes. Selecting one of several 984 Equal Cost Multi Path (ECMP) is however based on information hidden 985 deeper in the stack. Late deployments may support a so called 986 "Entropy label" for this purpose. State of the art deployments base 987 their choice of an ECMP member interface on the complete MPLS label 988 stack and on IP addresses up to the complete 5 tuple IP header 989 information (see Section 2.4 of [RFC7325]). Load Sharing based on IP 990 information decouples this function from the actual MPLS routing 991 information. Thus, an MPLS traceroute is able to check how packets 992 with a contiguous number of ECMP relevant IP addresses (and an 993 identical MPLS label stack) are forwarded by a particular router. 994 The minimum number of equivalent MPLS paths traceable at a router 995 should be 32. Implementations supporting more paths are available. 997 The MPLS echo request and reply messages offering this feature must 998 support the Downstream Detailed Mapping TLV (was Downstream Mapping 999 initially, but the latter has been deprecated). The MPLS echo 1000 response includes the incoming interface where a router received the 1001 MPLS Echo request. The MPLS Echo reply further informs which of the 1002 n addresses relevant for the load sharing decision results in a 1003 particular next hop interface and contains the next hop's interface 1004 address (if available). This ensures that the next hop will receive 1005 a properly coded MPLS Echo request in the next step route of 1006 assessment. 1008 [RFC8403] explains how a central Path Monitoring System could be used 1009 to detect arbitrary MPLS paths between any routers within a single 1010 MPLS domain. The combination of MPLS forwarding, Segment Routing and 1011 MPLS traceroute offers a simple architecture and a powerful mechanism 1012 to detect and validate (segment routed) MPLS paths. 1014 11. References 1016 11.1. Normative References 1018 [I-D.ietf-ippm-ioam-data] 1019 Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields 1020 for In-situ OAM", draft-ietf-ippm-ioam-data-10 (work in 1021 progress), July 2020. 1023 [RFC0792] Postel, J., "Internet Control Message Protocol", STD 5, 1024 RFC 792, DOI 10.17487/RFC0792, September 1981, 1025 . 1027 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - 1028 Communication Layers", STD 3, RFC 1122, 1029 DOI 10.17487/RFC1122, October 1989, 1030 . 1032 [RFC1812] Baker, F., Ed., "Requirements for IP Version 4 Routers", 1033 RFC 1812, DOI 10.17487/RFC1812, June 1995, 1034 . 1036 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1037 Requirement Levels", BCP 14, RFC 2119, 1038 DOI 10.17487/RFC2119, March 1997, 1039 . 1041 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 1042 "Framework for IP Performance Metrics", RFC 2330, 1043 DOI 10.17487/RFC2330, May 1998, 1044 . 1046 [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip 1047 Delay Metric for IPPM", RFC 2681, DOI 10.17487/RFC2681, 1048 September 1999, . 1050 [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. 1051 Zekauskas, "A One-way Active Measurement Protocol 1052 (OWAMP)", RFC 4656, DOI 10.17487/RFC4656, September 2006, 1053 . 1055 [RFC5388] Niccolini, S., Tartarelli, S., Quittek, J., Dietz, T., and 1056 M. Swany, "Information Model and XML Data Model for 1057 Traceroute Measurements", RFC 5388, DOI 10.17487/RFC5388, 1058 December 2008, . 1060 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 1061 for Equal Cost Multipath Routing and Link Aggregation in 1062 Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011, 1063 . 1065 [RFC6673] Morton, A., "Round-Trip Packet Loss Metrics", RFC 6673, 1066 DOI 10.17487/RFC6673, August 2012, 1067 . 1069 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1070 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1071 May 2016, . 1073 [RFC8029] Kompella, K., Swallow, G., Pignataro, C., Ed., Kumar, N., 1074 Aldrin, S., and M. Chen, "Detecting Multiprotocol Label 1075 Switched (MPLS) Data-Plane Failures", RFC 8029, 1076 DOI 10.17487/RFC8029, March 2017, 1077 . 1079 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1080 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1081 May 2017, . 1083 [RFC8468] Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V. 1084 Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for 1085 the IP Performance Metrics (IPPM) Framework", RFC 8468, 1086 DOI 10.17487/RFC8468, November 2018, 1087 . 1089 11.2. Informative References 1091 [bdrmap] Luckie, M., Dhamdhere, A., Huffaker, B., Clark, D., and 1092 KC. Claffy, "bdrmap: Inference of Borders Between IP 1093 Networks", In Proceedings of the 2016 ACM on Internet 1094 Measurement Conference, pp. 381-396. ACM, 2016. 1096 [IDCong] Luckie, M., Dhamdhere, A., Clark, D., and B. Huffaker, 1097 "Challenges in inferring Internet interdomain 1098 congestion", In Proceedings of the 2014 Conference on 1099 Internet Measurement Conference, pp. 15-22. ACM, 2014. 1101 [LOAD_BALANCE] 1102 Sanguanpong, S., Pittayapitak, W., and K. Kasom Koht-Arsa, 1103 "COMPARISON OF HASH STRATEGIES FOR FLOW-BASED LOAD 1104 BALANCING", International Journal of Electronic Commerce 1105 Studies, Vol.6, No.2, pp.259-268. 1106 http://dx.doi.org/10.7903/ijecs.1346, 2015. 1108 [MLB] Augustin, B., Friedman, T., and R. Teixeira, "Measuring 1109 load-balanced paths in the Internet", Proceedings of the 1110 7th ACM SIGCOMM conference on Internet measurement, pp. 1111 149-160. ACM, 2007., 2007. 1113 [MLRM] Fontugne, R., Mazel, J., and K. Fukuda, "An empirical 1114 mixture model for large-scale RTT measurements", 2015 1115 IEEE Conference on Computer Communications (INFOCOM), pp. 1116 2470-2478. IEEE, 2015., 2015. 1118 [P2] Jain, R. and I. Chlamtac, "The P 2 algorithm for dynamic 1119 calculation of quartiles and histograms without storing 1120 observations", Communications of the ACM 28.10 (1985): 1121 1076-1085, 2015. 1123 [PT] Augustin, B., Cuvellier, X., Orgogozo, B., Viger, F., 1124 Friedman, T., Latapy, M., Magnien, C., and R. Teixeira, 1125 "Avoiding traceroute anomalies with Paris traceroute", 1126 Proceedings of the 6th ACM SIGCOMM conference on Internet 1127 measurement, pp. 153-158. ACM, 2006., 2006. 1129 [RFC2991] Thaler, D. and C. Hopps, "Multipath Issues in Unicast and 1130 Multicast Next-Hop Selection", RFC 2991, 1131 DOI 10.17487/RFC2991, November 2000, 1132 . 1134 [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. 1135 Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", 1136 RFC 5357, DOI 10.17487/RFC5357, October 2008, 1137 . 1139 [RFC5835] Morton, A., Ed. and S. Van den Berghe, Ed., "Framework for 1140 Metric Composition", RFC 5835, DOI 10.17487/RFC5835, April 1141 2010, . 1143 [RFC5837] Atlas, A., Ed., Bonica, R., Ed., Pignataro, C., Ed., Shen, 1144 N., and JR. Rivers, "Extending ICMP for Interface and 1145 Next-Hop Identification", RFC 5837, DOI 10.17487/RFC5837, 1146 April 2010, . 1148 [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, 1149 "IPv6 Flow Label Specification", RFC 6437, 1150 DOI 10.17487/RFC6437, November 2011, 1151 . 1153 [RFC7312] Fabini, J. and A. Morton, "Advanced Stream and Sampling 1154 Framework for IP Performance Metrics (IPPM)", RFC 7312, 1155 DOI 10.17487/RFC7312, August 2014, 1156 . 1158 [RFC7325] Villamizar, C., Ed., Kompella, K., Amante, S., Malis, A., 1159 and C. Pignataro, "MPLS Forwarding Compliance and 1160 Performance Requirements", RFC 7325, DOI 10.17487/RFC7325, 1161 August 2014, . 1163 [RFC7594] Eardley, P., Morton, A., Bagnulo, M., Burbridge, T., 1164 Aitken, P., and A. Akhter, "A Framework for Large-Scale 1165 Measurement of Broadband Performance (LMAP)", RFC 7594, 1166 DOI 10.17487/RFC7594, September 2015, 1167 . 1169 [RFC8403] Geib, R., Ed., Filsfils, C., Pignataro, C., Ed., and N. 1170 Kumar, "A Scalable and Topology-Aware MPLS Data-Plane 1171 Monitoring System", RFC 8403, DOI 10.17487/RFC8403, July 1172 2018, . 1174 [RTTSub] Bischof, Z., Rula, J., and F. Bustamante, "In and out of 1175 Cuba: Characterizing Cuba's connectivity", In Proceedings 1176 of the 2015 ACM Conference on Internet Measurement 1177 Conference, pp. 487-493. ACM, 2015. 1179 [SCAMPER] Matthew Luckie, M., "Scamper: a scalable and extensible 1180 packet prober for active measurement of the 1181 Internet", Proceedings of the 10th ACM SIGCOMM conference 1182 on Internet measurement, pp. 239-245. ACM, 2010., 2010. 1184 [SSNT] Park, K. and W. Willinger, "Self-Similar Network Traffic 1185 and Performance Evaluation (1st ed.)", John Wiley & Sons, 1186 Inc., New York, NY, USA, 2000. 1188 Authors' Addresses 1190 J. Ignacio Alvarez-Hamelin 1191 Universidad de Buenos Aires 1192 Av. Paseo Colon 850 1193 Buenos Aires C1063ACV 1194 Argentina 1196 Phone: +54 11 5285-0716 1197 Email: ihameli@cnet.fi.uba.ar 1198 URI: http://cnet.fi.uba.ar/ignacio.alvarez-hamelin/ 1200 Al Morton 1201 AT&T Labs 1202 200 Laurel Avenue South 1203 Middletown, NJ 07748 1204 USA 1206 Phone: +1 732 420 1571 1207 Fax: +1 732 368 1192 1208 Email: acm@research.att.com 1210 Joachim Fabini 1211 TU Wien 1212 Gusshausstrasse 25/E389 1213 Vienna 1040 1214 Austria 1216 Phone: +43 1 58801 38813 1217 Fax: +43 1 58801 38898 1218 Email: Joachim.Fabini@tuwien.ac.at 1219 URI: http://www.tc.tuwien.ac.at/about-us/staff/joachim-fabini/ 1220 Carlos Pignataro 1221 Cisco Systems, Inc. 1222 7200-11 Kit Creek Road 1223 Research Triangle Park, NC 27709 1224 USA 1226 Email: cpignata@cisco.com 1228 Ruediger Geib 1229 Deutsche Telekom 1230 Heinrich Hertz Str. 3-7 1231 Darmstadt 64295 1232 Germany 1234 Phone: +49 6151 5812747 1235 Email: Ruediger.Geib@telekom.de