idnits 2.17.1 draft-ietf-ippm-multipoint-alt-mark-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The abstract seems to contain references ([RFC8321], [RFC5644]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (January 7, 2020) is 1572 days in the past. Is this intentional? Checking references for intended status: Experimental ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 8321 (Obsoleted by RFC 9341) == Outdated reference: A later version (-10) exists of draft-ietf-ippm-route-07 == Outdated reference: A later version (-21) exists of draft-song-opsawg-ifit-framework-10 == Outdated reference: A later version (-14) exists of draft-zhou-ippm-enhanced-alternate-marking-04 Summary: 2 errors (**), 0 flaws (~~), 5 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 IPPM Working Group G. Fioccola, Ed. 3 Internet-Draft Huawei Technologies 4 Intended status: Experimental M. Cociglio 5 Expires: July 10, 2020 Telecom Italia 6 A. Sapio 7 R. Sisto 8 Politecnico di Torino 9 January 7, 2020 11 Multipoint Alternate Marking method for passive and hybrid performance 12 monitoring 13 draft-ietf-ippm-multipoint-alt-mark-04 15 Abstract 17 The Alternate Marking method, as presented in RFC 8321 [RFC8321], can 18 be applied only to point-to-point flows because it assumes that all 19 the packets of the flow measured on one node are measured again by a 20 single second node. This document aims to generalize and expand this 21 methodology to measure any kind of unicast flows, whose packets can 22 follow several different paths in the network, in wider terms a 23 multipoint-to-multipoint network. For this reason the technique here 24 described is called Multipoint Alternate Marking. Some definitions 25 here introduced extend the scope of RFC 5644 [RFC5644] in the context 26 of alternate marking schema. 28 Requirements Language 30 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 31 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 32 document are to be interpreted as described in RFC 2119 [RFC2119]. 34 Status of This Memo 36 This Internet-Draft is submitted in full conformance with the 37 provisions of BCP 78 and BCP 79. 39 Internet-Drafts are working documents of the Internet Engineering 40 Task Force (IETF). Note that other groups may also distribute 41 working documents as Internet-Drafts. The list of current Internet- 42 Drafts is at https://datatracker.ietf.org/drafts/current/. 44 Internet-Drafts are draft documents valid for a maximum of six months 45 and may be updated, replaced, or obsoleted by other documents at any 46 time. It is inappropriate to use Internet-Drafts as reference 47 material or to cite them other than as "work in progress." 48 This Internet-Draft will expire on July 10, 2020. 50 Copyright Notice 52 Copyright (c) 2020 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents 57 (https://trustee.ietf.org/license-info) in effect on the date of 58 publication of this document. Please review these documents 59 carefully, as they describe your rights and restrictions with respect 60 to this document. Code Components extracted from this document must 61 include Simplified BSD License text as described in Section 4.e of 62 the Trust Legal Provisions and are provided without warranty as 63 described in the Simplified BSD License. 65 Table of Contents 67 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 68 2. Correlation with RFC5644 . . . . . . . . . . . . . . . . . . 4 69 3. Flow classification . . . . . . . . . . . . . . . . . . . . . 5 70 4. Multipoint Performance Measurement . . . . . . . . . . . . . 7 71 4.1. Monitoring Network . . . . . . . . . . . . . . . . . . . 7 72 5. Multipoint Packet Loss . . . . . . . . . . . . . . . . . . . 9 73 6. Network Clustering . . . . . . . . . . . . . . . . . . . . . 9 74 6.1. Algorithm for Cluster partition . . . . . . . . . . . . . 10 75 7. Timing Aspects . . . . . . . . . . . . . . . . . . . . . . . 13 76 8. Multipoint Delay and Delay Variation . . . . . . . . . . . . 15 77 8.1. Delay measurements on multipoint paths basis . . . . . . 15 78 8.1.1. Single Marking measurement . . . . . . . . . . . . . 15 79 8.2. Delay measurements on single packets basis . . . . . . . 16 80 8.2.1. Single and Double Marking measurement . . . . . . . . 16 81 8.2.2. Hashing selection method . . . . . . . . . . . . . . 16 82 9. An Intelligent Performance Management approach . . . . . . . 18 83 10. Examples of application . . . . . . . . . . . . . . . . . . . 19 84 11. Security Considerations . . . . . . . . . . . . . . . . . . . 20 85 12. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 20 86 13. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 87 14. References . . . . . . . . . . . . . . . . . . . . . . . . . 20 88 14.1. Normative References . . . . . . . . . . . . . . . . . . 20 89 14.2. Informative References . . . . . . . . . . . . . . . . . 21 90 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 92 1. Introduction 94 The alternate marking method, as presented until now, is applicable 95 to a point-to-point path; so the extension proposed in this document 96 explains the most general case of multipoint-to-multipoint path and 97 enables flexible and adaptive performance measurements in a managed 98 network. 100 The Alternate Marking methodology described in RFC 8321 [RFC8321] has 101 the property to synchronize measurements in different points 102 maintaining the coherence of the counters. So it is possible to show 103 what is happening in every marking period for each monitored flow. 104 The monitoring parameters are the packet counter and timestamps of a 105 flow for each marking period. Note that additional details about the 106 Alternate Marking methodology are described in the paper 107 [IEEE-Network-PNPM] 109 There are some applications of the alternate marking method where 110 there are a lot of monitored flows and nodes. Multipoint Alternate 111 Marking aims to reduce these values and makes the performance 112 monitoring more flexible in case a detailed analysis is not needed. 113 For instance, by considering n measurement points and m monitored 114 flows,the order of magnitude of the packet counters for each time 115 interval is n*m*2 (1 per color). If both n and m are high values the 116 packet counters increase a lot and Multipoint Alternate Marking 117 offers a tool to control these parameters. 119 The approach presented in this document is applied only to unicast 120 flows and not to multicast. BUM (Broadcast Unknown Unicast 121 Multicast) traffic is not considered here, because traffic 122 replication is not covered by the Multipoint Alternate Marking 123 method. Furthermore it can be applicable to anycast flows and ECMP 124 (Equal-Cost Multi-Path) paths can also be easily monitored with this 125 technique. 127 In short, RFC 8321 [RFC8321] applies to point-to-point unicast flows 128 and BUM traffic and the Multipoint alternate marking and its 129 Clustering approach is valid for multipoint-to-multipoint unicast 130 flows, anycast and ECMP flows. 132 The Alternate Marking method can therefore be extended to any kind of 133 multipoint to multipoint paths, and the network clustering approach 134 presented in this document is the formalization of how to implement 135 this property and allow a flexible and optimized performance 136 measurement support for network management in every situation. 138 Without network clustering, it is possible to apply alternate marking 139 only for all the network or per single flow. Instead, with network 140 clustering, it is possible to use the network clusters partition at 141 different levels to perform the needed degree of detail. In some 142 circumstances it is possible to monitor a Multipoint Network by 143 analysing the Network Clustering, without examining in depth. In 144 case of problems (packet loss is measured or the delay is too high) 145 the filtering criteria could be specified more in order to perform a 146 detailed analysis by using a different combination of clusters up to 147 a per-flow measurement as described in RFC 8321 [RFC8321]. 149 This approach fits very well with the Intelligent Network and 150 Software Defined Network (SDN) paradigm where the SDN Orchestrator 151 and the SDN Controllers are the brains of the network and can manage 152 flow control to the switches and routers and, in the same way, can 153 calibrate the performance measurements depending on the necessity. 154 An SDN Controller Application can orchestrate how deep the network 155 performance monitoring is setup by applying the Multipoint Alternate 156 Marking as described in this document. 158 It is important to underline that, as extension of RFC 8321 159 [RFC8321], this is a methodology draft, so the mechanism that can be 160 used to transmit the counters and the timestamps is out of scope here 161 and the implementation is open. Several options are possible, e.g. 162 [I-D.zhou-ippm-enhanced-alternate-marking]. 164 2. Correlation with RFC5644 166 RFC 5644 [RFC5644] is limited to active measurements using a single 167 source packet or stream, and observations of corresponding packets 168 along the path (spatial), at one or more destinations (one-to-group), 169 or both. 171 Instead, the scope of this memo is to define multiparty metrics for 172 passive and hybrid measurements in a group-to-group topology with 173 multiple sources and destinations. 175 RFC 5644 [RFC5644] introduces metric names that can be reused also 176 here but have to be extended and rephrased to be applied to the 177 alternate marking schema: 179 a. the multiparty metrics are not only one-to-group metrics but can 180 be also group-to-group metrics; 182 b. the spatial metrics, used for measuring the performance of 183 segments of a source to destination path, are applied here to 184 group-to-group segments (called Clusters). 186 3. Flow classification 188 An unicast flow is identified by all the packets having a set of 189 common characteristics. This definition is inspired by RFC 7011 190 [RFC7011]. 192 As an example, by considering a flow as all the packets sharing the 193 same source IP address or the same destination IP address, it is easy 194 to understand that the resulting pattern will not be a point-to-point 195 connection, but a point-to-multipoint or multipoint-to-point 196 connection. 198 In general a flow can be defined by a set of selection rules used to 199 match a subset of the packets processed by the network device. These 200 rules specify a set of headers fields (Identification Fields) and the 201 relative values that must be found in matching packets. 203 The choice of the identification fields directly affects the type of 204 paths that the flow would follow in the network. In fact, it is 205 possible to relate a set of identification fields with the pattern of 206 the resulting graphs, as listed in Figure 1. 208 A TCP 5-tuple usually identifies flows following either a single path 209 or a point-to-point multipath (in case of load balancing). On the 210 contrary, a single source address selects flows following a point-to- 211 multipoint, while a multipoint-to-point can be the result of a 212 matching on a single destination address. In case a selection rule 213 and its reverse are used for bidirectional measurements, they can 214 correspond to a point-to-multipoint in one direction and a 215 multipoint-to-point in the opposite direction. 217 In this way the flows to be monitored are selected into the 218 monitoring points using packet selection rules, that can also change 219 the pattern of the monitored network. 221 The alternate marking method is applicable only to a single path (and 222 partially to a one-to-one multipath), so the extension proposed in 223 this document is suitable also for the most general case of 224 multipoint-to-multipoint, which embraces all the other patterns of 225 Figure 1. 227 point-to-point single path 228 +------+ +------+ +------+ 229 ---<> R1 <>----<> R2 <>----<> R3 <>--- 230 +------+ +------+ +------+ 232 point-to-point multipath 233 +------+ 234 <> R2 <> 235 / +------+ \ 236 / \ 237 +------+ / \ +------+ 238 ---<> R1 <> <> R4 <>--- 239 +------+ \ / +------+ 240 \ / 241 \ +------+ / 242 <> R3 <> 243 +------+ 245 point-to-multipoint 246 +------+ 247 <> R4 <>--- 248 / +------+ 249 +------+ / 250 <> R2 <> 251 / +------+ \ 252 +------+ / \ +------+ 253 ---<> R1 <> <> R5 <>--- 254 +------+ \ +------+ 255 \ +------+ 256 <> R3 <> 257 +------+ \ 258 \ +------+ 259 <> R6 <>--- 260 +------+ 262 multipoint-to-point 263 +------+ 264 ---<> R1 <> 265 +------+ \ 266 \ +------+ 267 <> R4 <> 268 / +------+ \ 269 +------+ / \ +------+ 270 ---<> R2 <> <> R6 <>--- 271 +------+ / +------+ 272 +------+ / 273 <> R5 <> 274 / +------+ 275 +------+ / 276 ---<> R3 <> 277 +------+ 279 multipoint-to-multipoint 280 +------+ +------+ 281 ---<> R1 <> <> R6 <>--- 282 +------+ \ / +------+ 283 \ +------+ / 284 <> R4 <> 285 +------+ \ 286 +------+ \ +------+ 287 ---<> R2 <> <> R7 <>--- 288 +------+ \ / +------+ 289 \ +------+ / 290 <> R5 <> 291 / +------+ \ 292 +------+ / \ +------+ 293 ---<> R3 <> <> R8 <>--- 294 +------+ +------+ 296 Figure 1: Flow classification 298 The case of unicast flow is considered in the previous figure. 299 Anyway the anycast flow is also in scope because there is no 300 replication and only a single node from the anycast group receives 301 the traffic, so it can be viewed as a special case of unicast flow. 302 Furthermore, an ECMP flow is in scope by definition, since it is a 303 point-to-multipoint unicast flow. 305 4. Multipoint Performance Measurement 307 By Using the "traditional" alternate marking method only point-to- 308 point paths can be monitored. To have an IP (TCP/UDP) flow that 309 follows a point-to-point path we have to define, with a specific 310 value, 5 identification fields (IP Source, IP Destination, Transport 311 Protocol, Source Port, Destination Port). 313 Multipoint Alternate Marking enables the performance measurement for 314 multipoint flows selected by identification fields without any 315 constraints (even the entire network production traffic). It is also 316 possible to use multiple marking points for the same monitored flow. 318 4.1. Monitoring Network 320 The Monitoring Network is deduced from the Production Network, by 321 identifying the nodes of the graph that are the measurement points, 322 and the links that are the connections between measurement points. 324 There are some techniques that can help with the building of the 325 monitoring network (as an example it is possible to mention 327 [I-D.ietf-ippm-route]). In general there are different options: the 328 monitoring network can be obtained by considering all the possible 329 paths for the traffic or also by checking the traffic sometimes and 330 update the graph consequently. 332 So a graph model of the monitoring network can be built according to 333 the alternate marking method: the monitored interfaces and links are 334 identified. Only the measurement points and links where the traffic 335 has flowed have to be represented in the graph. 337 The following figure shows a simple example of a Monitoring Network 338 graph: 340 +------+ 341 <> R6 <>--- 342 / +------+ 343 +------+ +------+ / 344 <> R2 <>---<> R4 <> 345 / +------+ \ +------+ \ 346 / \ \ +------+ 347 +------+ / +------+ \ +------+ <> R7 <>--- 348 ---<> R1 <>---<> R3 <>---<> R5 <> +------+ 349 +------+ \ +------+ \ +------+ \ 350 \ \ \ +------+ 351 \ \ <> R8 <>--- 352 \ \ +------+ 353 \ \ 354 \ \ +------+ 355 \ <> R9 <>--- 356 \ +------+ 357 \ 358 \ +------+ 359 <> R10 <>--- 360 +------+ 362 Figure 2: Monitoring Network Graph 364 Each monitoring point is characterized by the packet counter that 365 refers only to a marking period of the monitored flow. 367 The same is applicable also for the delay but it will be described in 368 the following sections. 370 5. Multipoint Packet Loss 372 Since all the packets of the considered flow leaving the network have 373 previously entered the network, the number of packets counted by all 374 the input nodes is always greater or equal than the number of packets 375 counted by all the output nodes. 377 And in case of no packet loss occurring in the marking period, if all 378 the input and output points of the network domain to be monitored are 379 measurement points, the sum of the number of packets on all the 380 ingress interfaces equals the number on egress interfaces for the 381 monitored flow. In this circumstance, if no packet loss occurs, the 382 intermediate measurement points have only the task to split the 383 measurement. 385 It is possible to define the Network Packet Loss (for 1 monitored 386 flow, for 1 period): <>. This is true for 389 every packet flow in each marking period. 391 The Monitored Network Packet Loss with n input nodes and m output 392 nodes is given by: 394 PL = (PI1 + PI2 +...+ PIn) - (PO1 + PO2 +...+ POm) 396 where: 398 PL is the Network Packet Loss (number of lost packets) 400 PIi is the Number of packets flowed through the i-th Input node in 401 this period 403 POj is the Number of packets flowed through the j-th Output node in 404 this period 406 The equation is applied on a per-time-interval basis. 408 6. Network Clustering 410 The previous Equation can determine the number of packets lost 411 globally in the monitored network, exploiting only the data provided 412 by the counters in the input and output nodes. 414 In addition it is also possible to leverage the data provided by the 415 other counters in the network to converge on the smallest 416 identifiable subnetworks where the losses occur. These subnetworks 417 are named Clusters. 419 A Cluster graph is a subnetwork of the entire Monitoring Network 420 graph that still satisfies the packet loss equation where PL in this 421 case is the number of packets lost in the Cluster. 423 For this reason a Cluster should contain all the arcs emanating from 424 its input nodes and all the arcs terminating at its output nodes. 425 This ensures that we can count all the packets (and only those) 426 exiting an input node again at the output node, whatever path they 427 follow. 429 In a completely monitored network (a network where every network 430 interface is monitored), each network device corresponds to a Cluster 431 and each physical link corresponds to two Clusters (one for each 432 direction). 434 Clusters can have different sizes depending on flow filtering 435 criteria adopted. 437 Moreover, sometimes Clusters can be optionally simplified. For 438 example when two monitored interfaces are divided by a single router 439 (one is the input interface and the other is the output interface and 440 the router has only these two interfaces), instead of counting 441 exactly twice, upon entering and leaving, it is possible to consider 442 a single measurement point (in this case we do not care of the 443 internal packet loss of the router). 445 6.1. Algorithm for Cluster partition 447 A simple algorithm can be applied in order to split our monitoring 448 network into Clusters. It is a two-step algorithm: 450 o Group the links where there is the same starting node; 452 o Join the grouped links with at least one ending node in common. 454 After the application of the previous two steps, each one of the 455 composed sets of links together with the endpoint nodes constitutes a 456 Cluster. 458 In our monitoring network graph example it is possible to identify 459 the Clusters partition by applying this two-step algorithm. 461 The first step identifies the following groups: 463 1. Group 1: (R1-R2), (R1-R3), (R1-R10) 465 2. Group 2: (R2-R4), (R2-R5) 466 3. Group 3: (R3-R5), (R3-R9) 468 4. Group 4: (R4-R6), (R4-R7) 470 5. Group 5: (R5-R8) 472 And then, the second step builds the Clusters partition (in 473 particular we can underline that Group 2 and Group 3 connect 474 together, since R5 is in common): 476 1. Cluster 1: (R1-R2), (R1-R3), (R1-R10) 478 2. Cluster 2: (R2-R4), (R2-R5), (R3-R5), (R3-R9) 480 3. Cluster 3: (R4-R6), (R4-R7) 482 4. Cluster 4: (R5-R8) 484 In the end the following 4 Clusters are obtained: 486 Cluster 1 487 +------+ 488 <> R2 <>--- 489 / +------+ 490 / 491 +------+ / +------+ 492 ---<> R1 <>---<> R3 <>--- 493 +------+ \ +------+ 494 \ 495 \ 496 \ 497 \ 498 \ 499 \ 500 \ 501 \ 502 \ +------+ 503 <> R10 <>--- 504 +------+ 506 Cluster 2 507 +------+ +------+ 508 ---<> R2 <>---<> R4 <>--- 509 +------+ \ +------+ 510 \ 511 +------+ \ +------+ 512 ---<> R3 <>---<> R5 <>--- 513 +------+ \ +------+ 514 \ 515 \ 516 \ 517 \ 518 \ +------+ 519 <> R9 <>--- 520 +------+ 522 Cluster 3 523 +------+ 524 <> R6 <>--- 525 / +------+ 526 +------+ / 527 ---<> R4 <> 528 +------+ \ 529 \ +------+ 530 <> R7 <>--- 531 +------+ 533 Cluster 4 534 +------+ 535 ---<> R5 <> 536 +------+ \ 537 \ +------+ 538 <> R8 <>--- 539 +------+ 541 Figure 3: Clusters example 543 There are Clusters with more than 2 nodes and two-nodes Clusters. In 544 the two-nodes Clusters the loss is on the link (Cluster 4). In more- 545 than-2-nodes Clusters the loss is on the Cluster but we cannot know 546 in which link (Cluster 1, 2, 3). 548 In this way the calculation of packet loss can be made on Cluster 549 basis. Note that CIR(Committed Information Rate) and EIR(Excess 550 Information Rate) can also be deduced on Cluster basis. 552 Obviously, by combining some Clusters in a new connected subnetwork 553 (called Super Cluster) the Packet Loss Rule is still true. 555 In this way in a very large network there is no need to configure 556 detailed filter criteria to inspect the traffic. You can check 557 multipoint network and only in case of problems you can go deep with 558 a step-by-step cluster analysis, but only for the cluster or 559 combination of clusters where the problem happens. 561 The algorithm described above is an Iterative clustering algorithm, 562 but it is also possible to apply a Recursive clustering algorithm by 563 using the node-node adjacency matrix representation. 565 The complete and mathematical analysis of the possible Algorithms for 566 Cluster partition, including the considerations in terms of 567 efficiency and a comparison between the different methods, is in the 568 paper [IEEE-ACM-ToN-MPNPM]. 570 7. Timing Aspects 572 It is important to consider the timing aspects, since out of order 573 packets happen and have to be handled as well as described in RFC 574 8321 [RFC8321]. But, in a multi-source situation an additional issue 575 has to be considered. 577 So, if we analyse a multipoint-to-multipoint path with more than one 578 marking node, it is important to recognize the reference measurement 579 interval. In general the measurement interval for describing the 580 results is the interval of the marking node that is more aligned with 581 the start of the measurement, as reported in the following figure. 583 Note that the mark switching approach based on a fixed timer is 584 considered in this document. 586 time -> start stop 587 T(R1) |-------------| 588 T(R2) |-------------| 589 T(R3) |------------| 591 Figure 4: Measurement Interval 593 T(R1) is the measurement interval and this is essential in order to 594 be compatible and make comparison with other active/passive/hybrid 595 Packet Loss metrics. 597 That is why, when we expand to multipoint-to-multipoint flows, we 598 have to consider that all source nodes mark the traffic. 600 Regarding the timing aspects of the methodology, RFC 8321 [RFC8321] 601 already describes two contributions that are taken into account: the 602 clock error between network devices and the network delay between 603 measurement points. 605 But we should now consider an additional contribution. Since all 606 source nodes mark the traffic, the source measurement intervals can 607 be of different lengths and with different offsets and this mismatch 608 m can be added to d, as shown in figure. 610 ...BBBBBBBBB | AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA | BBBBBBBBB... 611 |<======================================>| 612 | L | 613 ...=========>|<==================><==================>|<==========... 614 | L/2 L/2 | 615 |<=><===>| |<===><=>| 616 m d | | d m 617 |<====================>| 618 available counting interval 620 Figure 5: Timing Aspects for Multipoint paths 622 So the misalignment between the marking source routers gives an 623 additional constraint and the value of m is added to d (that already 624 includes clock error and network delay). 626 Therefore, three different possible constraints are considered: clock 627 error between network devices, network delay between measurement 628 points and the misalignment between the marking source routers. 630 In the end, the condition that must be satisfied to enable the method 631 to function properly is that the available counting interval must be 632 > 0, and that means: L - 2m - 2d > 0 for each measurement point on 633 the multipoint path. Therefore, the mismatch between measurement 634 intervals must satisfy this condition. 636 The timing considerations are valid for both packet loss and delay 637 measurements. 639 8. Multipoint Delay and Delay Variation 641 The same line of reasoning can be applied to Delay and Delay 642 Variation. Similarly to the delay measurements defined in RFC 8321 643 [RFC8321], the marking batches anchor the samples to a particular 644 period and this is the time reference that can be used. It is 645 important to highlight that both delay and delay variation 646 measurements make sense in a multipoint path. The Delay Variation is 647 calculated by considering the same packets selected for measuring the 648 Delay. 650 In general, it is possible to perform delay and delay variation 651 measurements on multipoint paths basis or on single packets basis: 653 o Delay measurements on multipoint paths basis means that the delay 654 value is representative of an entire multipoint path (e.g. whole 655 multipoint network, a cluster or a combination of clusters). 657 o Delay measurements on single packets basis means that you can use 658 multipoint path just to easily couple packets between inputs and 659 output nodes of a multipoint path, as it is described in the 660 following sections. 662 8.1. Delay measurements on multipoint paths basis 664 8.1.1. Single Marking measurement 666 Mean delay and mean delay variation measurements can also be 667 generalized to the case of multipoint flows. It is possible to 668 compute the average one-way delay of packets, in one block, in a 669 cluster or in the entire monitored network. 671 The average latency can be measured as the difference between the 672 weighted averages of the mean timestamps of the sets of output and 673 input nodes. 675 8.2. Delay measurements on single packets basis 677 8.2.1. Single and Double Marking measurement 679 Delay and delay variation measurements relative to only one picked 680 packet per period (both single and double marked) can be performed in 681 the Multipoint scenario with some limitations: 683 Single marking based on the first/last packet of the interval 684 would not work, because it would not be possible to agree on the 685 first packet of the interval. 687 Double marking or multiplexed marking would work, but each 688 measurement would only give information about the delay of a 689 single path. However, by repeating the measurement multiple 690 times, it is possible to get information about all the paths in 691 the multipoint flow. This can be done in case of point-to- 692 multipoint path but it is more difficult to achieve in case of 693 multipoint-to-multipoint path because of the multiple source 694 routers. 696 if we would perform a delay measurement for more than one picked 697 packet in the same marking period and, especially, if we want to get 698 delay measurements on multipoint-to-multipoint basis, both single and 699 double marking method are not useful in the Multipoint scenario, 700 since they would not be representative of the entire flow. The 701 packets can follow different paths with various delays and in general 702 it can be very difficult to recognize marked packets in a multipoint- 703 to-multipoint path especially in case they are more than one per 704 period. 706 A desirable option is to monitor simultaneously all the paths of a 707 multipoint path in the same marking period and, for this purpose, 708 hashing can be used as reported in the next Section. 710 8.2.2. Hashing selection method 712 RFC 5474 [RFC5474] and RFC 5475 [RFC5475] introduce sampling and 713 filtering techniques for IP Packet Selection. 715 The hash-based selection methodologies for delay measurement can work 716 in a multipoint-to-multipoint path and can be used both coupled to 717 mean delay or stand alone. 719 [I-D.mizrahi-ippm-compact-alternate-marking] introduces how to use 720 the Hash method combined with alternate marking method for point-to- 721 point flows. It is also called Mixed Hashed Marking: the coupling of 722 marking method and hashing technique is very useful because the 723 marking batches anchor the samples selected with hashing and this 724 simplifies the correlation of the hashing packets along the path. 726 It is possible to use a basic hash or a dynamic hash method. One of 727 the challenges of the basic approach is that the frequency of the 728 sampled packets may vary considerably. For this reason the dynamic 729 approach has been introduced for point-to-point flow in order to have 730 the desired and almost fixed number of samples for each measurement 731 period. In the hash-based sampling, alternate marking is used to 732 create periods, so that hash-based samples are divided into batches, 733 allowing to anchor the selected samples to their period. Moreover in 734 the dynamic hash-based sampling, by dynamically adapting the length 735 of the hash value, the number of samples is bounded in each marking 736 period. This can be realized by choosing the maximum number of 737 samples (NMAX) to be caught in a marking period. The algorithm 738 starts with only few hash bits, that permit to select a greater 739 percentage of packets (e.g. with 0 bit of hash all the packets are 740 sampled, with 1 bit of hash half of the packets are sampled, and so 741 on). When the number of selected packets reaches NMAX, a hashing bit 742 is added. As a consequence, the sampling proceeds at half of the 743 original rate and also the packets already selected that don't match 744 the new hash are discarded. This step can be repeated iteratively. 745 It is assumed that each sample includes the timestamp (used for delay 746 measurement) and the hash value, allowing the management system to 747 match the samples received from the two measurement points. The 748 dynamic process statistically converges at the end of a marking 749 period and the final number of selected samples is between NMAX/2 and 750 NMAX. Therefore, the dynamic approach paces the sampling rate, 751 allowing to bound the number of sampled packets per sampling period. 753 In a multipoint environment the behaviour is similar to point-to 754 point flow. In particular, in the context of multipoint-to- 755 multipoint flow, the dynamic hash could be the solution to perform 756 delay measurements on specific packets and to overcome the single and 757 double marking limitations. 759 The management system receives the samples including the timestamps 760 and the hash value from all the MPs, and this happens both for point- 761 to-point and for multipoint-to-multipoint flow. Then the longest 762 hash used by MPs is deduced and it is applied to couple timestamps of 763 same packets of 2 MPs of a point-to-point path or of input and output 764 MPs of a Cluster (or a Super Cluster or the entire network). But 765 some considerations are needed: if there isn't packet loss the set of 766 input samples is always equal to the set of output samples. In case 767 of packet loss the set of output samples can be a subset of input 768 samples but the method still works because, at the end, it is easy to 769 couple the input and output timestamps of each caught packet using 770 the hash (in particular the "unused part of the hash" that should be 771 different for each packet). 773 In summary, the basic hash is logically similar to the double marking 774 method, and in case of point-to-point path double marking and basic 775 hash selection are equivalent. The dynamic approach scales the 776 number of measurements per interval, and it would seem that double 777 marking would also work well if we reduced the interval length, but 778 this can be done only for point-to-point path and not for multipoint 779 path, where we cannot couple the picked packets in a multipoint 780 paths. So, in general, if we want to get delay measurements on 781 multipoint-to-multipoint path basis and want to select more than one 782 packet per period, double marking cannot be used because we could not 783 be able to couple the picked packets between input and output nodes. 784 On the other hand we can do that by using hashing selection. 786 9. An Intelligent Performance Management approach 788 The Multipoint Alternate Marking framework that is introduced in this 789 document adds flexibility to PM because it can reduce the order of 790 magnitude of the packet counters. This allows an SDN Orchestrator to 791 supervise, control and manage PM in large networks. 793 The monitoring network can be considered as a whole or can be split 794 in Clusters, that are the smallest subnetworks (group-to-group 795 segments), maintaining the packet loss property for each subnetwork. 796 They can also be combined in new connected subnetworks at different 797 levels depending on the detail we want to achieve. 799 An SDN Controller can calibrate Performance Measurements since it is 800 aware of the network topology. It can start without examining in 801 depth. In case of necessity (packet loss is measured or the delay is 802 too high), the filtering criteria could be immediately specified more 803 in order to perform a partition of the network by using Clusters and/ 804 or different combinations of Clusters. In this way the problem can 805 be localized in a specific Cluster or in a single combination of 806 Clusters and a more detailed analysis can be performed step-by-step 807 by successive approximation up to a point-to-point flow detailed 808 analysis. 810 This approach can be called Network Zooming and can be performed in 811 two different ways: 813 1) change the traffic filter and select more detailed flows; 815 2) activate new measurement points by defining more specified 816 clusters. 818 The Network Zooming approach implies that the some filters or rules 819 are changed and there is a transient time to wait once the new 820 network configuration takes effect and it can be determined by the 821 Network Orchestrator/Controller, based on the network conditions. 823 [I-D.song-opsawg-ifit-framework] defines an architecture where the 824 centralized Data Collector and Network Management can apply the 825 intelligent and flexible Alternate Marking algorithm as previously 826 described. 828 As for RFC 8321 [RFC8321], it is possible to classify the traffic and 829 mark a portion of the total traffic. For each period the packet rate 830 and bandwidth are calculated from the number of packets. In this way 831 the Network Orchestrator becomes aware if the traffic rate overcomes 832 limits. In addition more precision can be obtained by reducing the 833 marking period, indeed some implementations use a marking period of 1 834 sec and less. 836 In addition an SDN Controller could also collect the measurement 837 history. 839 It is important to mention that the Multipoint Alternate Marking 840 framework also helps Traffic Visualization. Indeed this methodology 841 is very useful to identify which path or which cluster is crossed by 842 the flow. 844 10. Examples of application 846 There are application fields where it may be useful to take into 847 consideration the Multipoint Alternate Marking: 849 o VPN: The IP traffic is selected on IP source basis in both 850 directions. At the end point WAN interface all the output traffic 851 is counted in a single flow. The input traffic is composed by all 852 the other flows aggregated for source address. So, by considering 853 n end-points, the monitored flows are n (each flow with 1 ingress 854 point and (n-1) egress points) instead of n*(n-1) flows (each 855 flow, with 1 ingress point and 1 egress point); 857 o Mobile Backhaul: LTE traffic is selected, in the Up direction, by 858 the EnodeB source address and, in Down direction, by the EnodeB 859 destination address because the packets are sent from the Mobile 860 Packet Core to the EnodeB. So the monitored flow is only one per 861 EnodeB in both directions; 863 o OTT(Over The Top) services: The traffic is selected, in the Down 864 direction by the source addresses of the packets sent by OTT 865 Servers. In the opposite direction (Up) by the destination IP 866 addresses of the same Servers. So the monitoring is based on a 867 single flow per OTT Servers in both directions. 869 o Enterprise SD-WAN: SD-WAN allows to connect remote branch offices 870 to Data Centers and build higher-performance WANs. A centralized 871 controller is used to set policies and prioritize traffic. The 872 SD-WAN takes into account these policies and the availability of 873 network bandwidth to route traffic. This helps ensure that 874 application performance meets service level agreements (SLAs). 875 This methodology can also help the path selection for the WAN 876 connection based on per Cluster and per flow performance. 878 11. Security Considerations 880 This document specifies a method to perform measurements that does 881 not directly affect Internet security nor applications that run on 882 the Internet. However, implementation of this method must be mindful 883 of security and privacy concerns, as explained in RFC 8321 [RFC8321]. 885 12. Acknowledgements 887 The authors would like to thank Al Morton, Tal Mizrahi, Rachel Huang 888 for the precious contribution. 890 13. IANA Considerations 892 This memo makes no requests of IANA. 894 14. References 896 14.1. Normative References 898 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 899 Requirement Levels", BCP 14, RFC 2119, 900 DOI 10.17487/RFC2119, March 1997, 901 . 903 [RFC5644] Stephan, E., Liang, L., and A. Morton, "IP Performance 904 Metrics (IPPM): Spatial and Multicast", RFC 5644, 905 DOI 10.17487/RFC5644, October 2009, 906 . 908 [RFC8321] Fioccola, G., Ed., Capello, A., Cociglio, M., Castaldelli, 909 L., Chen, M., Zheng, L., Mirsky, G., and T. Mizrahi, 910 "Alternate-Marking Method for Passive and Hybrid 911 Performance Monitoring", RFC 8321, DOI 10.17487/RFC8321, 912 January 2018, . 914 14.2. Informative References 916 [I-D.ietf-ippm-route] 917 Alvarez-Hamelin, J., Morton, A., Fabini, J., Pignataro, 918 C., and R. Geib, "Advanced Unidirectional Route Assessment 919 (AURA)", draft-ietf-ippm-route-07 (work in progress), 920 December 2019. 922 [I-D.mizrahi-ippm-compact-alternate-marking] 923 Mizrahi, T., Arad, C., Fioccola, G., Cociglio, M., Chen, 924 M., Zheng, L., and G. Mirsky, "Compact Alternate Marking 925 Methods for Passive and Hybrid Performance Monitoring", 926 draft-mizrahi-ippm-compact-alternate-marking-05 (work in 927 progress), July 2019. 929 [I-D.song-opsawg-ifit-framework] 930 Song, H., Qin, F., Chen, H., Jin, J., and J. Shin, "In- 931 situ Flow Information Telemetry", draft-song-opsawg-ifit- 932 framework-10 (work in progress), December 2019. 934 [I-D.zhou-ippm-enhanced-alternate-marking] 935 Zhou, T., Fioccola, G., Li, Z., Lee, S., and M. Cociglio, 936 "Enhanced Alternate Marking Method", draft-zhou-ippm- 937 enhanced-alternate-marking-04 (work in progress), October 938 2019. 940 [IEEE-ACM-ToN-MPNPM] 941 IEEE/ACM TRANSACTION ON NETWORKING, "Multipoint Passive 942 Monitoring in Packet Networks", 943 DOI 10.1109/TNET.2019.2950157, 2019. 945 [IEEE-Network-PNPM] 946 IEEE Network, "AM-PM: Efficient Network Telemetry using 947 Alternate Marking", DOI 10.1109/MNET.2019.1800152, 2019. 949 [RFC5474] Duffield, N., Ed., Chiou, D., Claise, B., Greenberg, A., 950 Grossglauser, M., and J. Rexford, "A Framework for Packet 951 Selection and Reporting", RFC 5474, DOI 10.17487/RFC5474, 952 March 2009, . 954 [RFC5475] Zseby, T., Molina, M., Duffield, N., Niccolini, S., and F. 955 Raspall, "Sampling and Filtering Techniques for IP Packet 956 Selection", RFC 5475, DOI 10.17487/RFC5475, March 2009, 957 . 959 [RFC7011] Claise, B., Ed., Trammell, B., Ed., and P. Aitken, 960 "Specification of the IP Flow Information Export (IPFIX) 961 Protocol for the Exchange of Flow Information", STD 77, 962 RFC 7011, DOI 10.17487/RFC7011, September 2013, 963 . 965 Authors' Addresses 967 Giuseppe Fioccola (editor) 968 Huawei Technologies 969 Riesstrasse, 25 970 Munich 80992 971 Germany 973 Email: giuseppe.fioccola@huawei.com 975 Mauro Cociglio 976 Telecom Italia 977 Via Reiss Romoli, 274 978 Torino 10148 979 Italy 981 Email: mauro.cociglio@telecomitalia.it 983 Amedeo Sapio 984 Politecnico di Torino 985 Corso Duca degli Abruzzi, 24 986 Torino 10129 987 Italy 989 Email: amedeo.sapio@polito.it 991 Riccardo Sisto 992 Politecnico di Torino 993 Corso Duca degli Abruzzi, 24 994 Torino 10129 995 Italy 997 Email: riccardo.sisto@polito.it