idnits 2.17.1 draft-ietf-ippm-route-09.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). (Using the creation date from RFC2330, updated by this document, for RFC5378 checks: 1998-05-01) -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (July 9, 2020) is 1385 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'Dst' is mentioned on line 884, but not defined == Missing Reference: 'Hop' is mentioned on line 884, but not defined == Missing Reference: 'Alt' is mentioned on line 884, but not defined == Outdated reference: A later version (-17) exists of draft-ietf-ippm-ioam-data-09 ** Downref: Normative reference to an Informational RFC: RFC 2330 ** Downref: Normative reference to an Informational RFC: RFC 2991 ** Downref: Normative reference to an Informational RFC: RFC 5835 ** Downref: Normative reference to an Informational RFC: RFC 7312 ** Downref: Normative reference to an Informational RFC: RFC 7799 ** Downref: Normative reference to an Informational RFC: RFC 8468 Summary: 6 errors (**), 0 flaws (~~), 6 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group J. Alvarez-Hamelin 3 Internet-Draft Universidad de Buenos Aires 4 Updates: 2330 (if approved) A. Morton 5 Intended status: Standards Track AT&T Labs 6 Expires: January 10, 2021 J. Fabini 7 TU Wien 8 C. Pignataro 9 Cisco Systems, Inc. 10 R. Geib 11 Deutsche Telekom 12 July 9, 2020 14 Advanced Unidirectional Route Assessment (AURA) 15 draft-ietf-ippm-route-09 17 Abstract 19 This memo introduces an advanced unidirectional route assessment 20 (AURA) metric and associated measurement methodology, based on the IP 21 Performance Metrics (IPPM) Framework RFC 2330. This memo updates RFC 22 2330 in the areas of path-related terminology and path description, 23 primarily to include the possibility of parallel subpaths between a 24 given Source and Destination pair, owing to the presence of multi- 25 path technologies. 27 Requirements Language 29 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 30 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 31 "OPTIONAL" in this document are to be interpreted as described in BCP 32 14[RFC2119] [RFC8174] when, and only when, they appear in all 33 capitals, as shown here. 35 Status of This Memo 37 This Internet-Draft is submitted in full conformance with the 38 provisions of BCP 78 and BCP 79. 40 Internet-Drafts are working documents of the Internet Engineering 41 Task Force (IETF). Note that other groups may also distribute 42 working documents as Internet-Drafts. The list of current Internet- 43 Drafts is at https://datatracker.ietf.org/drafts/current/. 45 Internet-Drafts are draft documents valid for a maximum of six months 46 and may be updated, replaced, or obsoleted by other documents at any 47 time. It is inappropriate to use Internet-Drafts as reference 48 material or to cite them other than as "work in progress." 49 This Internet-Draft will expire on January 10, 2021. 51 Copyright Notice 53 Copyright (c) 2020 IETF Trust and the persons identified as the 54 document authors. All rights reserved. 56 This document is subject to BCP 78 and the IETF Trust's Legal 57 Provisions Relating to IETF Documents 58 (https://trustee.ietf.org/license-info) in effect on the date of 59 publication of this document. Please review these documents 60 carefully, as they describe your rights and restrictions with respect 61 to this document. Code Components extracted from this document must 62 include Simplified BSD License text as described in Section 4.e of 63 the Trust Legal Provisions and are provided without warranty as 64 described in the Simplified BSD License. 66 Table of Contents 68 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 69 1.1. Issues with Earlier Work to define Route . . . . . . . . 3 70 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 71 3. Route Metric Terms and Definitions . . . . . . . . . . . . . 5 72 3.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 6 73 3.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 6 74 3.3. Metric Definitions . . . . . . . . . . . . . . . . . . . 7 75 3.4. Related Round-Trip Delay and Loss Definitions . . . . . . 9 76 3.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . 9 77 3.6. Reporting the Metric . . . . . . . . . . . . . . . . . . 10 78 4. Route Assessment Methodologies . . . . . . . . . . . . . . . 10 79 4.1. Active Methodologies . . . . . . . . . . . . . . . . . . 11 80 4.1.1. Temporal Composition for Route Metrics . . . . . . . 13 81 4.1.2. Routing Class Identification . . . . . . . . . . . . 14 82 4.1.3. Intermediate Observation Point Route Measurement . . 15 83 4.2. Hybrid Methodologies . . . . . . . . . . . . . . . . . . 15 84 4.3. Combining Different Methods . . . . . . . . . . . . . . . 16 85 5. Background on Round-Trip Delay Measurement Goals . . . . . . 17 86 6. RTD Measurements Statistics . . . . . . . . . . . . . . . . . 18 87 7. Security Considerations . . . . . . . . . . . . . . . . . . . 20 88 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 89 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 20 90 10. Appendix I MPLS Methods for Route Assessment . . . . . . . . 20 91 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 92 11.1. Normative References . . . . . . . . . . . . . . . . . . 21 93 11.2. Informative References . . . . . . . . . . . . . . . . . 24 94 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 25 96 1. Introduction 98 The IETF IP Performance Metrics (IPPM) working group first created a 99 framework for metric development in [RFC2330]. This framework has 100 stood the test of time and enabled development of many fundamental 101 metrics. It has been updated in the area of metric composition 102 [RFC5835], and in several areas related to active stream measurement 103 of modern networks with reactive properties [RFC7312]. 105 The [RFC2330] framework motivated the development of "performance and 106 reliability metrics for paths through the Internet," and Section 5 of 107 [RFC2330] defines terms that support description of a path under 108 test. However, metrics for assessment of path components and related 109 performance aspects had not been attempted in IPPM when the [RFC2330] 110 framework was written. 112 This memo takes up the route measurement challenge and specifies a 113 new route metric, two practical frameworks for methods of measurement 114 (using either active or hybrid active-passive methods [RFC7799]), and 115 Round-Trip Delay and link information discovery using the results of 116 measurements. All route measurements are limited by the willingness 117 of hosts along the path to be discovered, to cooperate with the 118 methods used, or to recognize that the measurement operation is 119 taking place (such as when tunnels are present). 121 1.1. Issues with Earlier Work to define Route 123 Section 7 of [RFC2330] presented a simple example of a "route" metric 124 along with several other examples. The example is reproduced below 125 (where the reference is to Section 5 of [RFC2330]): 127 "route: The path, as defined in Section 5, from A to B at a given 128 time." 130 This example provides a starting point to develop a more complete 131 definition of route. Areas needing clarification include: 133 Time: In practice, the route will be assessed over a time interval, 134 because active path detection methods like [PT] rely on TTL limits 135 for their operation and cannot accomplish discovery of all hosts 136 using a single packet. 138 Type-P: The legacy route definition lacks the option to cater for 139 packet-dependent routing. In this memo, we assess the route for a 140 specific packet of Type-P, and reflect this in the metric 141 definition. The methods of measurement determine the specific 142 Type-P used. 144 Parallel Paths: Parallel paths are a reality of the Internet and a 145 strength of advanced route assessment methods, so the metric must 146 acknowledge this possibility. Use of Equal Cost Multi-Path (ECMP) 147 and Unequal Cost Multi-Path (UCMP) technologies are common sources 148 of parallel subpaths. 150 Cloud Subpath: May contain hosts that do not decrement TTL or Hop 151 Limit, but may have two or more exchange links connecting 152 "discoverable" hosts or routers. Parallel subpaths contained 153 within clouds cannot be discovered. The assessment methods only 154 discover hosts or routers on the path that decrement TTL or Hop 155 Count, or cooperate with interrogation protocols. The presence of 156 tunnels and nested tunnels further complicate assessment by hiding 157 hops. 159 Hop: Although the [RFC2330] definition of a hop was a link-host 160 pair, only hosts that are discoverable or have the capability to 161 cooperate with interrogation protocols where link information may 162 be exposed. 164 The refined definition of Route metrics begins in the sections that 165 follow. 167 2. Scope 169 The purpose of this memo is to add new route metrics and methods of 170 measurement to the existing set of IPPM metrics. 172 The scope is to define route metrics that can identify the path taken 173 by a packet or a flow traversing the Internet between two hosts. 174 Although primarily intended for hosts communicating on the Internet 175 with IP, the definitions and metrics are constructed to be applicable 176 to other network domains, if desired. The methods of measurement to 177 assess the path may not be able to discover all hosts comprising the 178 path, but such omissions are often deterministic and explainable 179 sources of error. 181 Also, to specify a framework for active methods of measurement which 182 use the techniques described in [PT] at a minimum, and a framework 183 for hybrid active-passive methods of measurement, such as the Hybrid 184 Type I method [RFC7799] described in 185 [I-D.ietf-ippm-ioam-data](intended only for single administrative 186 domains), which do not rely on ICMP and provide a protocol for 187 explicit interrogation of nodes on a path. Combinations of active 188 methods and hybrid active-passive methods are also in-scope. 190 Further, this memo provides additional analysis of the round-trip 191 delay measurements made possible by the methods, in an effort to 192 discover more details about the path, such as the link technology in 193 use. 195 This memo updates Section 5 of [RFC2330] in the areas of path-related 196 terminology and path description, primarily to include the 197 possibility of parallel subpaths between a given Source and 198 Destination address pair (possibly resulting from Equal Cost Multi- 199 Path (ECMP) and Unequal Cost Multi-Path (UCMP) technologies). 201 There are several simple non-goals of this memo. There is no attempt 202 to assess the reverse path from any host on the path to the host 203 attempting the path measurement. The reverse path contribution to 204 delay will be that experienced by ICMP packets (in active methods), 205 and may be different from delays experienced by UDP or TCP packets. 206 Also, the round trip delay will include an unknown contribution of 207 processing time at the host that generates the ICMP response. 208 Therefore, the ICMP-based active methods are not supposed to yield 209 accurate, reproducible estimations of the Round-Trip Delay that UDP 210 or TCP packets will experience. 212 3. Route Metric Terms and Definitions 214 This section sets requirements for the following components to 215 support the Route Metric: 217 Host A Host as defined in [RFC2330] (a computer capable of IP 218 communication, includes routers), a.k.a. RFC 2330 Host. 220 Node A Node is any network function on the path capable of IP-layer 221 Communication, includes RFC 2330 Hosts. 223 Node Identity The unique address for Nodes communicating within the 224 network domain. For Nodes communicating on the Internet with IP, 225 it is the globally routable IP address(es) which the Node uses 226 when communicating with other Nodes under normal or error 227 conditions. The Node Identity revealed (and its connection to a 228 Node Name through reverse DNS) determines whether interfaces to 229 parallel links can be associated with a single Node, or appear to 230 identify unique Nodes. 232 Discoverable Node Nodes that convey their Node Identity according to 233 the requirements of their network domain, such as when error 234 conditions are detected by that Node. For Nodes communicating 235 with IP packets, compliance with Section 3.2.2.4 of [RFC1122] when 236 discarding a packet due to TTL or Hop Limit Exceeded condition, 237 MUST result in sending the corresponding Time Exceeded message 238 (containing a form of Node identity) to the source. This 239 requirement is also consistent with section 5.3.1 of [RFC1812] for 240 routers. 242 Cooperating Node Nodes that MUST respond to direct queries for their 243 Node identity as part of a previously agreed and established 244 interrogation protocol. Nodes SHOULD also provide information 245 such as arrival/departure interface identification, arrival 246 timestamp, and any relevant information about the Node or specific 247 link which delivered the query to the Node. 249 Hop A Hop MUST contain a Node Identity, and MAY contain arrival and/ 250 or departure interface identification, round trip delay, and an 251 arrival timestamp. 253 Routing Class A route that treats equally a class C of different 254 types of packets (unrelated to address classes of the past). 255 Knowledge of such a class allows any one of the types of packets 256 within that class to be used for subsequent measurement of the 257 route. 259 3.1. Formal Name 261 Type-P-Route-Ensemble-Method-Variant, abbreviated as Route Ensemble. 263 Note that Type-P depends heavily on the chosen method and variant. 265 3.2. Parameters 267 This section lists the REQUIRED input factors to specify a Route 268 metric. 270 o Src, the address of a Node (such as the globally routable IP 271 address). 273 o Dst, the address of a Node (such as the globally routable IP 274 address). 276 o i, the limit on the number of Hops a specific packet may visit as 277 it traverses from the Node at Src to the Node at Dst (such as the 278 TTL or Hop Limit). 280 o MaxHops, the maximum value of i used, (i=1,2,3,...MaxHops). 282 o T0, a time (start of measurement interval) 284 o Tf, a time (end of measurement interval) 285 o MP(address), Measurement Point at address, such as Src or Dst, 286 usually at the same node stack layer as "address". 288 o T, the Node time of a packet as measured at MP(Src), meaning 289 Measurement Point at the Source. 291 o Ta, the Node time of a reply packet's *arrival* as measured at 292 MP(Src), assigned to packets that arrive within a "reasonable" 293 time (see parameter below). 295 o Tmax, a maximum waiting time for reply packets to return to the 296 source, set sufficiently long to disambiguate packets with long 297 delays from packets that are discarded (lost), such that the 298 distribution of Round-Trip Delay is not truncated. 300 o F, the number of different flows simulated by the method and 301 variant. 303 o flow, the stream of packets with the same n-tuple of designated 304 header fields that (when held constant) result in identical 305 treatment in a multi-path decision (such as the decision taken in 306 load balancing). Note: The IPv6 flow label MAY be included in the 307 flow definition if the MP(Src) is a Tunnel End Point (TEP) 308 complying with [RFC6438] guidelines. 310 o Type-P, the complete description of the packets for which this 311 assessment applies (including the flow-defining fields). 313 3.3. Metric Definitions 315 This section defines the REQUIRED measurement components of the Route 316 metrics (unless otherwise indicated): 318 M, the total number of packets sent between T0 and Tf. 320 N, the smallest value of i needed for a packet to be received at Dst 321 (sent between T0 and Tf). 323 Nmax, the largest value of i needed for a packet to be received at 324 Dst (sent between T0 and Tf). Nmax may be equal to N. 326 Next define a *singleton* definition for a Hop on the path, with 327 sufficient indexes to identify all Hops identified in a measurement 328 interval. 330 A Hop, designated h(i,j), the IP address and/or identity of 331 Discoverable Nodes (or Cooperating Nodes) that are i hops away from 332 the Node with address = Src and part of Route j during the 333 measurement interval, T0 to Tf. As defined here, a Hop singleton 334 measurement MUST contain a Node Identity, hid(i,j), and MAY contain 335 one or more of the following attributes: 337 o a(i,j) Arrival Interface ID (e.g., when [RFC5837] is supported) 339 o d(i,j) Departure Interface ID (e.g., when [RFC5837] is supported) 341 o t(i,j) Arrival Timestamp (where t(i,j) is ideally supplied by the 342 Hop, or approximated from the sending time of the packet that 343 revealed the Hop) 345 o Measurements of Round-Trip Delay (for each packet that reveals the 346 same Node Identity and flow attributes, then this attribute is 347 computed, see next section) 349 Node Identities and related information can be ordered by their 350 distance from the Node with address Src in Hops h(i,j). Based on 351 this, two forms of Routes are distinguished: 353 A Route Ensemble is defined as the combination of all routes 354 traversed by different flows from the Node at Src address to the Node 355 at Dst address. A single Route traversed by a single flow 356 (determined by an unambiguous tuple of addresses Src and Dst, and 357 other identical flow criteria) is a member of the Route Ensemble and 358 called a Member Route. 360 Using h(i,j) and components and parameters, further define: 362 When considering the set of Hops in the context of a single flow, a 363 Member Route j is an ordered list {h(1,j), ... h(Nj, j)} where h(i-1, 364 j) and h(i, j) are 1 hop away from each other and Nj satisfying 365 h(Nj,j)=Dst is the minimum count of Hops needed by the packet on 366 Member Route j to reach Dst. Member Routes must be unique. The 367 uniqueness property requires that any two Member routes j and k that 368 are part of the same Route Ensemble differ either in terms of minimum 369 hop count Nj and Nk to reach the destination Dst, or, in the case of 370 identical hop count Nj=Nk, they have at least one distinct Hop: 371 h(i,j) != h(i,k) for at least one i (i=1..Nj). 373 All the optional information collected to describe a Member Route, 374 such as the arrival interface, departure interface, and Round Trip 375 Delay at each Hop, turns each list item into a rich structure. There 376 may be information on the links between Hops, possibly information on 377 the routing (arrival interface and departure interface), an estimate 378 of distance between Hops based on Round-Trip Delay measurements and 379 calculations, and a time stamp indicating when all these additional 380 details were valid. 382 The Route Ensemble from Src to Dst, during the measurement interval 383 T0 to Tf, is the aggregate of all m distinct Member Routes discovered 384 between the two Nodes with Src and Dst addresses. More formally, 385 with the Node having address Src omitted: 387 Route Ensemble = { 388 {h(1,1), h(2,1), h(3,1), ... h(N1,1)=Dst}, 389 {h(1,2), h(2,2), h(3,2),..., h(N2,2)=Dst}, 390 ... 391 {h(1,m), h(2,m), h(3,m), ....h(Nm,m)=Dst} 392 } 394 where the following conditions apply: i <= Nj <= Nmax (j=1..m) 396 Note that some h(i,j) may be empty (null) in the case that systems do 397 not reply (not discoverable, or not cooperating). 399 h(i-1,j) and h(i,j) are the Hops on the same Member Route one hop 400 away from each other. 402 Hop h(i,j) may be identical with h(k,l) for i!=k and j!=l ; which 403 means there may be portions shared among different Member Routes 404 (parts of Member Routes may overlap). 406 3.4. Related Round-Trip Delay and Loss Definitions 408 RTD(i,j,T) is defined as a singleton of the [RFC2681] Round-Trip 409 Delay between the Node with address = Src and the Node at Hop h(i,j) 410 at time T. 412 RTL(i,j,T) is defined as a singleton of the [RFC6673] Round-trip Loss 413 between the Node with address = Src and the Node at Hop h(i,j) at 414 time T. 416 3.5. Discussion 418 Depending on the way that Node Identity is revealed, it may be 419 difficult to determine parallel subpaths between the same pair of 420 Nodes (i.e. multiple parallel links). It is easier to detect 421 parallel subpaths involving different Nodes. 423 o If a pair of discovered Nodes identify two different addresses, 424 then they will appear to be different Nodes. 426 o If a pair of discovered Nodes identify two different IP addresses, 427 and the IP addresses resolve to the same Node name (in the DNS), 428 then they will appear to be the same Nodes. 430 o If a discovered Node always replies using the same network 431 address, regardless of the interface a packet arrives on, then 432 multiple parallel links cannot be detected in that network domain. 433 This condition may apply to traceroute-style methods, but may not 434 apply to other hybrid methods based on In-situ Operations, 435 Administration, and Maintenance (IOAM). 437 o If parallel links between routers are aggregated below the IP 438 layer, then from Node point of view, all these links share the 439 same pair of IP addresses. The existence of these parallel links 440 can't be detected at IP layer. This applies to other network 441 domains with layers below them, as well. This condition may apply 442 to traceroute-style methods, but may not apply to other hybrid 443 methods based on IOAM. 445 When a route assessment employs IP packets (for example), the reality 446 of flow assignment to parallel subpaths involves layers above IP. 447 Thus, the measured Route Ensemble is applicable to IP and higher 448 layers (as described in the methodology's packet of Type-P and flow 449 parameters). 451 3.6. Reporting the Metric 453 An Information Model and an XML Data Model for Storing Traceroute 454 Measurements is available in [RFC5388]. The measured information at 455 each hop includes four pieces of information: a one-dimensional hop 456 index, Node symbolic address, Node IP address, and RTD for each 457 response. 459 The description of Hop information that may be collected according to 460 this memo covers more dimensions, as defined in Section 3.3 above. 461 For example, the Hop index is two-dimensional to capture the 462 complexity of a Route Ensemble, and it contains corresponding Node 463 identities at a minimum. The models need to be expanded to include 464 these features, as well as Arrival Interface ID, Departure Interface 465 ID, and Arrival Timestamp, when available. The original sending 466 Timestamp from the Src Node anchors a particular measurement in time. 468 4. Route Assessment Methodologies 470 There are two classes of methods described in this section, active 471 methods relying on the reaction to TTL or Hop Limit Exceeded 472 condition to discover Nodes on a path, and Hybrid active-passive 473 methods that involve direct interrogation of cooperating Nodes 474 (usually within a single domain). Description of these methods 475 follow. 477 4.1. Active Methodologies 479 This section describes the method employed by current open source 480 tools, thereby providing a practical framework for further advanced 481 techniques to be included as method variants. This method is 482 applicable for use across multiple administrative domains. 484 Internet routing is complex because it depends on the policies of 485 thousands of Autonomous Systems (AS). While most of the routers 486 perform load balancing on flows using Equal Cost Multiple Path 487 (ECMP), a few still divide the workload through packet-based 488 techniques. The former scenario is defined according to [RFC2991], 489 while the latter generates a round-robin scheme to deliver every new 490 outgoing packet. ECMP uses a hashing function to ensure that every 491 packet of a flow is delivered by the same path, and this avoids 492 increasing the packet delay variation and possibly producing 493 overwhelming packet reordering in TCP flows. 495 Taking into account that Internet protocol was designed under the 496 "end-to-end" principle, the IP payload and its header do not provide 497 any information about the routes or path necessary to reach some 498 destination. For this reason, the popular tool traceroute was 499 developed to gather the IP addresses of each hop along a path using 500 the ICMP protocol [RFC0792]. Traceroute also measures RTD from each 501 hop. However, the growing complexity of the Internet makes it more 502 challenging to develop an accurate traceroute implementation. For 503 instance, the early traceroute tools would be inaccurate in the 504 current network, mainly because they were not designed to retain a 505 flow state. However, evolved traceroute tools, such as Paris- 506 traceroute [PT] [MLB] and Scamper [SCAMPER], expect to encounter ECMP 507 and achieve more accurate results when they do, where Scamper ensures 508 traceroute packets will follow the same path in 98% of 509 cases[SCAMPER]. 511 Today's traceroute tools send Type-P of packets, either ICMP, UDP, or 512 TCP. UDP and TCP are used when a particular characteristic needs to 513 be verified, such as filtering or traffic shaping on specific ports 514 (i.e., services). [SCAMPER] supports IPv6 traceroute measurements, 515 keeping the FlowLabel constant in all packets. 517 Paris-traceroute allows its users to measure RTD in every hop of the 518 path for a particular flow. Furthermore, either Paris-traceroute or 519 Scamper is capable of unveiling the many available paths between a 520 source and destination (which are visible to this method). This task 521 is accomplished by repeating complete traceroute measurements with 522 different flow parameters for each measurement; Paris-traceroute 523 provides "exhaustive" mode while scamper provides "tracelb" (stands 524 for traceroute load balance). The Framework for IP Performance 525 Metrics (IPPM) ([RFC2330] updated by[RFC7312]) has the flexibility to 526 require that the Round-Trip Delay measurement [RFC2681] uses packets 527 with the constraints to assure that all packets in a single 528 measurement appear as the same flow. This flexibility covers ICMP, 529 UDP, and TCP. The accompanying methodology of [RFC2681] needs to be 530 expanded to report the sequential hop identifiers along with RTD 531 measurements, but no new metric definition is needed. 533 The advanced route assessment methods used in Paris-traceroute [PT] 534 keep the critical fields constant for every packet to maintain the 535 appearance of the same flow. In IPv6, it is sufficient to be routed 536 identically if the IP source and destination addresses and the 537 FlowLabel are constant, see [RFC6437]. In IPv4, certain fields of 538 the IP header and the first four bytes of the IP payload should 539 remain constant in a flow. In the IPv4 header, the IP source and 540 destination addresses, protocol number, and Diffserv fields identify 541 flows. The first four payload bytes include the UDP and TCP ports, 542 and the ICMP type, code, and checksum fields. 544 Maintaining a constant ICMP checksum in IPv4 is most challenging, as 545 the ICMP sequence number or identifier fields will usually change for 546 different probes of the same path. Probes should use arbitrary bytes 547 in the ICMP data field to offset changes to sequence number and 548 identifier, thus keeping the checksum constant. 550 Finally, it is also essential to route the resulting ICMP Time 551 Exceeded messages along a consistent path. In IPv6, the fields above 552 are sufficient. In IPv4, the ICMP Time Exceeded message will contain 553 the IP header and the first eight bytes of the IP payload, which 554 affects its ICMP checksum. The TCP sequence number, UDP Length, and 555 UDP checksum will affect this value, and should remain constant. 557 Formally, to maintain the same flow in the measurements to a 558 particular hop, the Type-P-Route-Ensemble-Method-Variant packets 559 should be[PT]: 561 o TCP case: For IPv4, the fields Src, Dst, port-Src, port_Dst, 562 sequence number, and Diffserv Field SHOULD be the same. For IPv6, 563 the field FlowLabel, Src and Dst SHOULD be the same. 565 o UDP case: For IPv4, the fields Src, Dst, port-Src, port-Dst, 566 Diffserv should be the same, and the UDP-checksum SHOULD change to 567 keep the IP checksum of the ICMP time exceeded reply constant. 568 Then, the data length should be fixed, and the data field is used 569 to fixing it (consider that ICMP checksum uses its data field, 570 which contains the original IP header plus 8 bytes of UDP, where 571 TTL, IP identification, IP checksum, and UDP checksum changes). 573 For IPv6, the field FlowLabel, and Source and Destination 574 addresses SHOULD be the same. 576 o ICMP case: For IPv4, the Data field SHOULD compensate variations 577 on TTL or Hop Limit, IP identification, and IP checksum for every 578 packet. There is no need to consider ICMPv6 because only 579 FlowLabel of IPv6 and Source and Destination addresses are used, 580 and all of them SHOULD be constant. 582 Then, the way to identify different hops and attempts of the same 583 flow is: 585 o TCP case: The IP identification field. 587 o UDP case: The IP identification field. 589 o ICMP case: The IP identification field, and ICMP Sequence number. 591 4.1.1. Temporal Composition for Route Metrics 593 The Active Route Assessment Methods described above have the ability 594 to discover portions of a path where ECMP load balancing is present, 595 observed as two or more unique Member Routes having one or more 596 distinct Hops which are part of the Route Ensemble. Likewise, 597 attempts to deliberately vary the flow characteristics to discover 598 all Member Routes will reveal portions of the path which are flow- 599 invariant. 601 Section 9.2 of [RFC2330] describes Temporal Composition of metrics, 602 and introduces the possibility of a relationship between earlier 603 measurement results and the results for measurement at the current 604 time (for a given metric). There is value in establishing a Temporal 605 Composition relationship for Route Metrics. However, this 606 relationship does not represent a forecast of future route conditions 607 in any way. 609 For Route Metric measurements, the value of Temporal Composition is 610 to reduce the measurement iterations required with repeated 611 measurements. Reduced iterations are possible by inferring that 612 current measurements using fixed and previously measured flow 613 characteristics: 615 o will have many common hops with previous measurements. 617 o will have relatively time-stable results at the ingress and egress 618 portions of the path when measured from user locations, as opposed 619 to measurements of backbone networks and across inter-domain 620 gateways. 622 o may have greater potential for time-variation in path portions 623 where ECMP load balancing is observed (because increasing or 624 decreasing the pool of links changes the hash calculations). 626 Optionally, measurement systems may take advantage of the inferences 627 above when seeking to reduce measurement iterations, after exhaustive 628 measurements indicate that the time-stable properties are present. 629 Repetitive Active Route measurement systems: 631 1. SHOULD occasionally check path portions which have exhibited 632 stable results over time, particularly ingress and egress 633 portions of the path. 635 2. SHOULD continue testing portions of the path that have previously 636 exhibited ECMP load balancing. 638 3. SHALL trigger re-assessment of the complete path and Route 639 Ensemble, if any change in hops is observed for a specific (and 640 previously tested) flow. 642 4.1.2. Routing Class Identification 644 There is an opportunity to apply the [RFC2330] notion of equal 645 treatment for a class of packets, "...very useful to know if a given 646 Internet component treats equally a class C of different types of 647 packets", as it applies to Route measurements. The notion of class C 648 was examined further in [RFC8468] as it applied to load-balancing 649 flows over parallel paths, which is the case we develop here. 650 Knowledge of class C parameters (unrelated to address classes of the 651 past) on a path potentially reduces the number of flows required for 652 a given method to assess a Route Ensemble over time. 654 First, recognize that each Member Route of a Route Ensemble will have 655 a corresponding class C. Class C can be discovered by testing with 656 multiple flows, all of which traverse the unique set of hops that 657 comprise a specific Member Route. 659 Second, recognize that the different classes depend primarily on the 660 hash functions used at each instance of ECMP load balancing on the 661 path. 663 Third, recognize the synergy with Temporal Composition methods 664 (described above), where evaluation intends to discover time-stable 665 portions of each Member Route, so that more emphasis can be placed on 666 ECMP portions that also determine class C. 668 The methods to assess the various class C characteristics benefit 669 from the following measurement capabilities: 671 o flows designed to determine which n-tuple header fields are 672 considered by a given hash function and ECMP hop on the path, and 673 which are not. This operation immediately narrows the search 674 space, where possible, and partially defines a class C. 676 o a priori knowledge of the possible types of hash functions in use 677 also helps to design the flows for testing (major router vendors 678 publish information about these hash functions, examples are here 679 [LOAD_BALANCE]. 681 o ability to direct the emphasis of current measurements on ECMP 682 portions of the path, based on recent past measurement results 683 (the Routing Class of some portions of the path is essentially 684 "all packets"). 686 4.1.3. Intermediate Observation Point Route Measurement 688 There are many examples where passive monitoring of a flow at an 689 Observation Point within the network can detect unexpected Round Trip 690 Delay or Delay Variation. But how can the cause of the anomalous 691 delay be investigated further --from the Observation Point -- 692 possibly located at an intermediate point on the path? 694 In this case, knowledge that the flow of interest belongs to a 695 specific Routing Class C will enable measurement of the route where 696 anomalous delay has been observed. Specifically, Round-Trip Delay 697 assessment to each Hop on the path between the Observation Point and 698 the Destination for the flow of interest may discover high or 699 variable delay on a specific link and Hop combination. 701 The determination of a Routing Class C which includes the flow of 702 interest is as described in the section above, aided by computation 703 of the relevant hash function output as the target. 705 4.2. Hybrid Methodologies 707 The Hybrid Type I methods provide an alternative method for Route 708 Member assessment. As mentioned in the Scope section, 709 [I-D.ietf-ippm-ioam-data] provides a possible set of data fields that 710 would support route identification. 712 In general, nodes in the measured domain would be equipped with 713 specific abilities: 715 o Store the identity of nodes that a packet has visited in header 716 data fields, in the order the packet visited the nodes. 718 o Support of a "Loopback" capability, where a copy of the packet is 719 returned to the encapsulating node, and the packet is processed 720 like any other IOAM packet on the return transfer. 722 In addition to node identity, nodes may also identify the ingress and 723 egress interfaces utilized by the tracing packet, the time of day 724 when the packet was processed, and other generic data (as described 725 in section 4 of [I-D.ietf-ippm-ioam-data]). Interface identification 726 isn't necessarily limited to IP, i.e. different links in a bundle 727 (LACP) could be identified. Equally well, links without explicit IP 728 addresses can be identified (like with unnumbered interfaces in an 729 IGP deployment). 731 Note that the Type-P packet specification for this method will likely 732 be a partial specification, because most of the packet fields are 733 determined by the user traffic. The packet (encapsulation) header(s) 734 added by the Hybrid method can certainly be specified in Type-P, in 735 unpopulated form. 737 4.3. Combining Different Methods 739 In principle, there are advantages if the entity conducting Route 740 measurements can utilize both forms of advanced methods (active and 741 hybrid), and combine the results. For example, if there are Nodes 742 involved in the path that qualify as Cooperating Nodes, but not as 743 Discoverable Nodes, then a more complete view of Hops on the path is 744 possible when a hybrid method (or interrogation protocol) is applied 745 and the results are combined with the active method results collected 746 across all other domains. 748 In order to combine the results of active and hybrid/interrogation 749 methods, the network Nodes that are part of a domain supporting an 750 interrogation protocol have the following attributes: 752 1. Nodes at the ingress to the domain SHOULD be both Discoverable 753 and Cooperating, and SHOULD reveal the same Node Identity in 754 response to both active and hybrid methods. 756 2. Any Nodes within the domain that are both Discoverable and 757 Cooperating SHOULD reveal the same Node Identity in response to 758 both active and hybrid methods. 760 3. Nodes at the egress to the domain SHOULD be both Discoverable and 761 Cooperating, and SHOULD reveal the same Node Identity in response 762 to both active and hybrid methods. 764 When Nodes follow these requirements, it becomes a simple matter to 765 match single domain measurements with the overlapping results from a 766 multidomain measurement. 768 In practice, Internet users do not typically have the ability to 769 utilize the OAM capabilities of networks that their packets traverse, 770 so the results from a remote domain supporting an interrogation 771 protocol would not normally be accessible. However, a network 772 operator could combine interrogation results from their access domain 773 with other measurements revealing the path outside their domain. 775 5. Background on Round-Trip Delay Measurement Goals 777 The aim of this method is to use packet probes to unveil the paths 778 between any two end-Nodes of the network. Moreover, information 779 derived from RTD measurements might be meaningful to identify: 781 1. Intercontinental submarine links 783 2. Satellite communications 785 3. Congestion 787 4. Inter-domain paths 789 This categorization is widely accepted in the literature and among 790 operators alike, and it can be trusted with empirical data and 791 several sources as ground of truth (e.g., [RTTSub] ) but it is an 792 inference measurement nonetheless [bdrmap][IDCong]. 794 The first two categories correspond to the physical distance 795 dependency on Round-Trip Delay (RTD), the next one binds RTD with 796 queueing delay on routers, and the last one helps to identify 797 different ASes using traceroutes. Due to the significant 798 contribution of propagation delay in long-distance hops, RTD will be 799 on the order of 100ms on transatlantic hops, depending on the 800 geolocation of the vantage points. Moreover, RTD is typically higher 801 than 480ms when two hops are connected using geostationary satellite 802 technology (i.e., their orbit is at 36000km). Detecting congestion 803 with latency implies deeper mathematical understanding since network 804 traffic load is not stationary. Nonetheless, as the first approach, 805 a link seems to be congested if, after sending several traceroute 806 probes, it is possible to detect congestion observing different 807 statistics parameters (e.g., see [IDCong]). Finally, to recognize 808 distinctive ASes in the same traceroute path is challenging, because 809 more data is needed, like AS relationships and RIR delegations among 810 other (for more detail, please consult [bdrmap]). 812 6. RTD Measurements Statistics 814 Several articles have shown that network traffic presents a self- 815 similar nature [SSNT] [MLRM] which is accountable for filling the 816 queues of the routers. Moreover, router queues are designed to 817 handle traffic bursts, which is one of the most remarkable features 818 of self-similarity. Naturally, while queue length increases, the 819 delay to traverse the queue increases as well and leads to an 820 increase on RTD. Due to traffic bursts generating short-term 821 overflow on buffers (spiky patterns), every RTD only depicts the 822 queueing status on the instant when that packet probe was in transit. 823 For this reason, several RTD measurements during a time window could 824 begin to describe the random behavior of latency. Loss must also be 825 accounted for in the methodology. 827 To understand the ongoing process, examining the quartiles provides a 828 non-parametric way of analysis. Quartiles are defined by five 829 values: minimum RTD (m), RTD value of the 25% of the Empirical 830 Cumulative Distribution Function (ECDF) (Q1), the median value (Q2), 831 the RTD value of the 75% of the ECDF (Q3) and the maximum RTD (M). 832 Congestion can be inferred when RTD measurements are spread apart, 833 and consequently, the Inter-Quartile Range (IQR), the distance 834 between Q3 and Q1, increases its value. 836 This procedure requires the algorithm presented in [P2] to compute 837 quartile values "on the fly". 839 This procedure allows us to update the quartiles value whenever a new 840 measurement arrives, which is radically different from classic 841 methods of computing quartiles because they need to use the whole 842 dataset to compute the values. This way of calculus provides savings 843 in memory and computing time. 845 To sum up, the proposed measurement procedure consists of performing 846 traceroutes several times to obtain samples of the RTD in every hop 847 from a path, during a time window (W), and compute the quartiles for 848 every hop. This procedure could be done for a single Member Route 849 flow, with parameter E set as False, or for every detected Route 850 Ensemble flow (E=True). 852 The identification of a specific Hop in traceroute is based on the IP 853 origin address of the returned ICMP Time Exceeded packet, and on the 854 distance identified by the value set in the TTL field inserted by 855 traceroute. As this specific Hop can be reached by different paths, 856 also the IP source and destination addresses of the traceroute packet 857 need to be recorded. Finally, different return paths are 858 distinguished by evaluating the ICMP Time Exceeded TTL (of the reply 859 message): if this TTL is constant for different paths containing the 860 same Hop, the return paths have the same distance. Moreover, this 861 distance can be estimated considering that the TTL value is normally 862 initialized with values 64, 128, or 255. The 5-tuple (origin IP, 863 destination IP, reply IP, distance, response TTL) univocally 864 identifies every measurement. 866 This algorithm below runs in the origin of the traceroute. It 867 returns the Qs quartiles for every Hop and Alt (alternative paths 868 because of balancing). Notice that the "Alt" parameter condenses the 869 parameters of the 5-tuple (origin IP, destination IP, reply IP, 870 distance, response TTL), i.e., one for each possible combination. 872 ================================================================ 873 1 input: W (window time of the measurement) 874 2 i_t (time between two measurements) 875 3 E (True: exhaustive, False: a single path) 876 4 Dst (destination IP address) 877 5 output: Qs (quartiles for every Hop and Alt) 878 ---------------------------------------------------------------- 879 6 T := start_timer(W) 880 7 while T is not finished do: 881 8 | start_timer(i_t) 882 9 | RTD(Hop,Alt) = advanced-traceroute(Dst,E) 883 10 | for each Hop and Alt in RTD do: 884 11 | | Qs[Dst,Hop,Alt] := ComputeQs(RTD(Hop,Alt)) 885 12 | done 886 13 | wait until i_t timer is expired 887 14 done 888 15 return (Qs) 889 ================================================================ 891 During the time W, lines 6 and 7 assure that the measurement loop is 892 made. Line 8 and 13 set a timer for each cycle of measurements. A 893 cycle comprises the traceroutes packets, considering every possible 894 Hop and the alternatives paths in the Alt variable (ensured in lines 895 9-12). In line 9, the advance-traceroute could be either Paris- 896 traceroute or Scamper, which will use the "exhaustive" mode or 897 "tracelb" option if E is set True, respectively. The procedure 898 returns a list of tuples (m,Q1,Q2,Q3,M) for each intermediate hop, or 899 "Alt" in as a function of the 5-tuple, in the path towards the Dst. 900 Finally, lines 10 through 12 stores each measurement into the real- 901 time quartiles computation. 903 Notice there are cases where the even having a unique hop at distance 904 h from the Src to Dst, the returning path could have several 905 possibilities, yielding in different total paths. In this situation, 906 the algorithm will return more "Alt" for this particular hop. 908 7. Security Considerations 910 The security considerations that apply to any active measurement of 911 live paths are relevant here as well. See [RFC4656] and [RFC5357]. 913 The active measurement process of "changing several fields to keep 914 the checksum of different packets identical" does not require special 915 security considerations because it is part of synthetic traffic 916 generation, and is designed to have minimal to zero impact on network 917 processing (to process the packets for ECMP). 919 For applicable Hybrid methods, the security considerations 920 in[I-D.ietf-ippm-ioam-data] apply. 922 When considering privacy of those involved in measurement or those 923 whose traffic is measured, the sensitive information available to 924 potential observers is greatly reduced when using active techniques 925 which are within this scope of work. Passive observations of user 926 traffic for measurement purposes raise many privacy issues. We refer 927 the reader to the privacy considerations described in the Large Scale 928 Measurement of Broadband Performance (LMAP) Framework [RFC7594], 929 which covers active and passive techniques. 931 8. IANA Considerations 933 This memo makes no requests of IANA. We thank the good folks at IANA 934 for having checked this section anyway. 936 9. Acknowledgements 938 The original 3 authors acknowledge Ruediger Geib, for his penetrating 939 comments on the initial draft, and his initial text for the 940 Appendix on MPLS. Carlos Pignataro challenged the authors to 941 consider a wider scope, and applied his substantial expertise with 942 many technologies and their measurement features in his extensive 943 comments. Frank Brockners also shared useful comments, so did Footer 944 Foote. We thank them all! 946 10. Appendix I MPLS Methods for Route Assessment 948 A Node assessing an MPLS path must be part of the MPLS domain where 949 the path is implemented. When this condition is met, RFC 8029 950 provides a powerful set of mechanisms to detect "correct operation of 951 the data plane, as well as a mechanism to verify the data plane 952 against the control plane" [RFC8029]. 954 MPLS routing is based on the presence of a Forwarding Equivalence 955 Class (FEC) Stack in all visited Nodes. Selecting one of several 956 Equal Cost Multi Path (ECMP) is however based on information hidden 957 deeper in the stack. Late deployments may support a so called 958 "Entropy label" for this purpose. State of the art deployments base 959 their choice of an ECMP member interface on the complete MPLS label 960 stack and on IP addresses up to the complete 5 tuple IP header 961 information (see Section 2.4 of [RFC7325]). Load Sharing based on IP 962 information decouples this function from the actual MPLS routing 963 information. Thus, an MPLS traceroute is able to check how packets 964 with a contiguous number of ECMP relevant IP addresses (and an 965 identical MPLS label stack) are forwarded by a particular router. 966 The minimum number of equivalent MPLS paths traceable at a router 967 should be 32. Implementations supporting more paths are available. 969 The MPLS echo request and reply messages offering this feature must 970 support the Downstream Detailed Mapping TLV (was Downstream Mapping 971 initially, but the latter has been deprecated). The MPLS echo 972 response includes the incoming interface where a router received the 973 MPLS Echo request. The MPLS Echo reply further informs which of the 974 n addresses relevant for the load sharing decision results in a 975 particular next hop interface and contains the next hop's interface 976 address (if available). This ensures that the next hop will receive 977 a properly coded MPLS Echo request in the next step route of 978 assessment. 980 [RFC8403] explains how a central Path Monitoring System could be used 981 to detect arbitrary MPLS paths between any routers within a single 982 MPLS domain. The combination of MPLS forwarding, Segment Routing and 983 MPLS traceroute offers a simple architecture and a powerful mechanism 984 to detect and validate (segment routed) MPLS paths. 986 11. References 988 11.1. Normative References 990 [I-D.ietf-ippm-ioam-data] 991 Brockners, F., Bhandari, S., Pignataro, C., Gredler, H., 992 Leddy, J., Youell, S., Mizrahi, T., Mozes, D., Lapukhov, 993 P., remy@barefootnetworks.com, r., daniel.bernier@bell.ca, 994 d., and J. Lemon, "Data Fields for In-situ OAM", draft- 995 ietf-ippm-ioam-data-09 (work in progress), March 2020. 997 [RFC0792] Postel, J., "Internet Control Message Protocol", STD 5, 998 RFC 792, DOI 10.17487/RFC0792, September 1981, 999 . 1001 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - 1002 Communication Layers", STD 3, RFC 1122, 1003 DOI 10.17487/RFC1122, October 1989, 1004 . 1006 [RFC1812] Baker, F., Ed., "Requirements for IP Version 4 Routers", 1007 RFC 1812, DOI 10.17487/RFC1812, June 1995, 1008 . 1010 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1011 Requirement Levels", BCP 14, RFC 2119, 1012 DOI 10.17487/RFC2119, March 1997, 1013 . 1015 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 1016 "Framework for IP Performance Metrics", RFC 2330, 1017 DOI 10.17487/RFC2330, May 1998, 1018 . 1020 [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip 1021 Delay Metric for IPPM", RFC 2681, DOI 10.17487/RFC2681, 1022 September 1999, . 1024 [RFC2991] Thaler, D. and C. Hopps, "Multipath Issues in Unicast and 1025 Multicast Next-Hop Selection", RFC 2991, 1026 DOI 10.17487/RFC2991, November 2000, 1027 . 1029 [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. 1030 Zekauskas, "A One-way Active Measurement Protocol 1031 (OWAMP)", RFC 4656, DOI 10.17487/RFC4656, September 2006, 1032 . 1034 [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. 1035 Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", 1036 RFC 5357, DOI 10.17487/RFC5357, October 2008, 1037 . 1039 [RFC5388] Niccolini, S., Tartarelli, S., Quittek, J., Dietz, T., and 1040 M. Swany, "Information Model and XML Data Model for 1041 Traceroute Measurements", RFC 5388, DOI 10.17487/RFC5388, 1042 December 2008, . 1044 [RFC5835] Morton, A., Ed. and S. Van den Berghe, Ed., "Framework for 1045 Metric Composition", RFC 5835, DOI 10.17487/RFC5835, April 1046 2010, . 1048 [RFC5837] Atlas, A., Ed., Bonica, R., Ed., Pignataro, C., Ed., Shen, 1049 N., and JR. Rivers, "Extending ICMP for Interface and 1050 Next-Hop Identification", RFC 5837, DOI 10.17487/RFC5837, 1051 April 2010, . 1053 [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, 1054 "IPv6 Flow Label Specification", RFC 6437, 1055 DOI 10.17487/RFC6437, November 2011, 1056 . 1058 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 1059 for Equal Cost Multipath Routing and Link Aggregation in 1060 Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011, 1061 . 1063 [RFC6673] Morton, A., "Round-Trip Packet Loss Metrics", RFC 6673, 1064 DOI 10.17487/RFC6673, August 2012, 1065 . 1067 [RFC7312] Fabini, J. and A. Morton, "Advanced Stream and Sampling 1068 Framework for IP Performance Metrics (IPPM)", RFC 7312, 1069 DOI 10.17487/RFC7312, August 2014, 1070 . 1072 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1073 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1074 May 2016, . 1076 [RFC8029] Kompella, K., Swallow, G., Pignataro, C., Ed., Kumar, N., 1077 Aldrin, S., and M. Chen, "Detecting Multiprotocol Label 1078 Switched (MPLS) Data-Plane Failures", RFC 8029, 1079 DOI 10.17487/RFC8029, March 2017, 1080 . 1082 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1083 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1084 May 2017, . 1086 [RFC8468] Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V. 1087 Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for 1088 the IP Performance Metrics (IPPM) Framework", RFC 8468, 1089 DOI 10.17487/RFC8468, November 2018, 1090 . 1092 11.2. Informative References 1094 [bdrmap] Luckie, M., Dhamdhere, A., Huffaker, B., Clark, D., and 1095 KC. Claffy, "bdrmap: Inference of Borders Between IP 1096 Networks", In Proceedings of the 2016 ACM on Internet 1097 Measurement Conference, pp. 381-396. ACM, 2016. 1099 [IDCong] Luckie, M., Dhamdhere, A., Clark, D., and B. Huffaker, 1100 "Challenges in inferring Internet interdomain 1101 congestion", In Proceedings of the 2014 Conference on 1102 Internet Measurement Conference, pp. 15-22. ACM, 2014. 1104 [LOAD_BALANCE] 1105 Sanguanpong, S., Pittayapitak, W., and K. Kasom Koht-Arsa, 1106 "COMPARISON OF HASH STRATEGIES FOR FLOW-BASED LOAD 1107 BALANCING", International Journal of Electronic Commerce 1108 Studies, Vol.6, No.2, pp.259-268. 1109 http://dx.doi.org/10.7903/ijecs.1346, 2015. 1111 [MLB] Augustin, B., Friedman, T., and R. Teixeira, "Measuring 1112 load-balanced paths in the Internet", Proceedings of the 1113 7th ACM SIGCOMM conference on Internet measurement, pp. 1114 149-160. ACM, 2007., 2007. 1116 [MLRM] Fontugne, R., Mazel, J., and K. Fukuda, "An empirical 1117 mixture model for large-scale RTT measurements", 2015 1118 IEEE Conference on Computer Communications (INFOCOM), pp. 1119 2470-2478. IEEE, 2015., 2015. 1121 [P2] Jain, R. and I. Chlamtac, "The P 2 algorithm for dynamic 1122 calculation of quartiles and histograms without storing 1123 observations", Communications of the ACM 28.10 (1985): 1124 1076-1085, 2015. 1126 [PT] Augustin, B., Cuvellier, X., Orgogozo, B., Viger, F., 1127 Friedman, T., Latapy, M., Magnien, C., and R. Teixeira, 1128 "Avoiding traceroute anomalies with Paris traceroute", 1129 Proceedings of the 6th ACM SIGCOMM conference on Internet 1130 measurement, pp. 153-158. ACM, 2006., 2006. 1132 [RFC7325] Villamizar, C., Ed., Kompella, K., Amante, S., Malis, A., 1133 and C. Pignataro, "MPLS Forwarding Compliance and 1134 Performance Requirements", RFC 7325, DOI 10.17487/RFC7325, 1135 August 2014, . 1137 [RFC7594] Eardley, P., Morton, A., Bagnulo, M., Burbridge, T., 1138 Aitken, P., and A. Akhter, "A Framework for Large-Scale 1139 Measurement of Broadband Performance (LMAP)", RFC 7594, 1140 DOI 10.17487/RFC7594, September 2015, 1141 . 1143 [RFC8403] Geib, R., Ed., Filsfils, C., Pignataro, C., Ed., and N. 1144 Kumar, "A Scalable and Topology-Aware MPLS Data-Plane 1145 Monitoring System", RFC 8403, DOI 10.17487/RFC8403, July 1146 2018, . 1148 [RTTSub] Bischof, Z., Rula, J., and F. Bustamante, "In and out of 1149 Cuba: Characterizing Cuba's connectivity", In Proceedings 1150 of the 2015 ACM Conference on Internet Measurement 1151 Conference, pp. 487-493. ACM, 2015. 1153 [SCAMPER] Matthew Luckie, M., "Scamper: a scalable and extensible 1154 packet prober for active measurement of the 1155 Internet", Proceedings of the 10th ACM SIGCOMM conference 1156 on Internet measurement, pp. 239-245. ACM, 2010., 2010. 1158 [SSNT] Park, K. and W. Willinger, "Self-Similar Network Traffic 1159 and Performance Evaluation (1st ed.)", John Wiley & Sons, 1160 Inc., New York, NY, USA, 2000. 1162 Authors' Addresses 1164 J. Ignacio Alvarez-Hamelin 1165 Universidad de Buenos Aires 1166 Av. Paseo Colon 850 1167 Buenos Aires C1063ACV 1168 Argentina 1170 Phone: +54 11 5285-0716 1171 Email: ihameli@cnet.fi.uba.ar 1172 URI: http://cnet.fi.uba.ar/ignacio.alvarez-hamelin/ 1174 Al Morton 1175 AT&T Labs 1176 200 Laurel Avenue South 1177 Middletown, NJ 07748 1178 USA 1180 Phone: +1 732 420 1571 1181 Fax: +1 732 368 1192 1182 Email: acm@research.att.com 1183 Joachim Fabini 1184 TU Wien 1185 Gusshausstrasse 25/E389 1186 Vienna 1040 1187 Austria 1189 Phone: +43 1 58801 38813 1190 Fax: +43 1 58801 38898 1191 Email: Joachim.Fabini@tuwien.ac.at 1192 URI: http://www.tc.tuwien.ac.at/about-us/staff/joachim-fabini/ 1194 Carlos Pignataro 1195 Cisco Systems, Inc. 1196 7200-11 Kit Creek Road 1197 Research Triangle Park, NC 27709 1198 USA 1200 Email: cpignata@cisco.com 1202 Ruediger Geib 1203 Deutsche Telekom 1204 Heinrich Hertz Str. 3-7 1205 Darmstadt 64295 1206 Germany 1208 Phone: +49 6151 5812747 1209 Email: Ruediger.Geib@telekom.de