idnits 2.17.1 draft-song-opsawg-ifit-framework-17.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (22 February 2022) is 792 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC2119' is defined on line 1002, but no explicit reference was found in the text == Unused Reference: 'RFC8174' is defined on line 1011, but no explicit reference was found in the text == Unused Reference: 'I-D.herbert-ipv4-eh' is defined on line 1040, but no explicit reference was found in the text == Outdated reference: A later version (-03) exists of draft-herbert-ipv4-eh-01 == Outdated reference: A later version (-08) exists of draft-ietf-idr-sr-policy-ifit-03 == Outdated reference: A later version (-05) exists of draft-ietf-ippm-ioam-deployment-00 == Outdated reference: A later version (-11) exists of draft-ietf-ippm-ioam-direct-export-07 == Outdated reference: A later version (-12) exists of draft-ietf-ippm-ioam-ipv6-options-07 == Outdated reference: A later version (-13) exists of draft-ietf-ippm-ioam-yang-03 == Outdated reference: A later version (-09) exists of draft-ietf-mboned-multicast-telemetry-02 == Outdated reference: A later version (-07) exists of draft-li-apn-framework-04 == Outdated reference: A later version (-15) exists of draft-mirsky-ippm-hybrid-two-step-12 == Outdated reference: A later version (-16) exists of draft-song-ippm-postcard-based-telemetry-11 == Outdated reference: A later version (-13) exists of draft-song-mpls-extension-header-06 == Outdated reference: A later version (-07) exists of draft-song-spring-siam-02 == Outdated reference: A later version (-14) exists of draft-zhou-ippm-enhanced-alternate-marking-08 -- Obsolete informational reference (is this intentional?): RFC 8889 (Obsoleted by RFC 9342) Summary: 0 errors (**), 0 flaws (~~), 17 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 OPSAWG H. Song 3 Internet-Draft Futurewei 4 Intended status: Informational F. Qin 5 Expires: 26 August 2022 China Mobile 6 H. Chen 7 China Telecom 8 J. Jin 9 LG U+ 10 J. Shin 11 SK Telecom 12 22 February 2022 14 A Framework for In-situ Flow Information Telemetry 15 draft-song-opsawg-ifit-framework-17 17 Abstract 19 As network scale increases and network operation becomes more 20 sophisticated, existing Operation, Administration, and Maintenance 21 (OAM) methods are no longer sufficient to meet the monitoring and 22 measurement requirements. Emerging data-plane on-path telemetry 23 techniques which provide high-precision flow insight and which issue 24 notifications in real time can supplement existing proactive and 25 reactive methods that run in active and passive modes. These new 26 approaches are collectively known as in-situ flow information 27 telemetry (IFIT). They enable quality of experience for users and 28 applications, and identification of network faults and deficiencies. 30 This document outlines a high-level framework for IFIT to collect and 31 correlate performance measurement information from the network. It 32 identifies the components that coordinate existing protocol tools and 33 telemetry mechanisms, and addresses deployment challenges for flow- 34 oriented on-path telemetry techniques, especially in carrier 35 networks. 37 The document is a guide for system designers applying the referenced 38 techniques. It is also intended to motivate further work to enhance 39 the OAM ecosystem. 41 Status of This Memo 43 This Internet-Draft is submitted in full conformance with the 44 provisions of BCP 78 and BCP 79. 46 Internet-Drafts are working documents of the Internet Engineering 47 Task Force (IETF). Note that other groups may also distribute 48 working documents as Internet-Drafts. The list of current Internet- 49 Drafts is at https://datatracker.ietf.org/drafts/current/. 51 Internet-Drafts are draft documents valid for a maximum of six months 52 and may be updated, replaced, or obsoleted by other documents at any 53 time. It is inappropriate to use Internet-Drafts as reference 54 material or to cite them other than as "work in progress." 56 This Internet-Draft will expire on 26 August 2022. 58 Copyright Notice 60 Copyright (c) 2022 IETF Trust and the persons identified as the 61 document authors. All rights reserved. 63 This document is subject to BCP 78 and the IETF Trust's Legal 64 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 65 license-info) in effect on the date of publication of this document. 66 Please review these documents carefully, as they describe your rights 67 and restrictions with respect to this document. Code Components 68 extracted from this document must include Revised BSD License text as 69 described in Section 4.e of the Trust Legal Provisions and are 70 provided without warranty as described in the Revised BSD License. 72 Table of Contents 74 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 75 1.1. Classification and Modes of On-path Telemetry . . . . . . 4 76 1.2. Requirements and Challenges . . . . . . . . . . . . . . . 6 77 1.3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 8 78 1.4. Relationship with Network Telemetry Framework (NTF) . . . 8 79 1.5. Glossary . . . . . . . . . . . . . . . . . . . . . . . . 9 80 2. Architectural Concepts and Key Components . . . . . . . . . . 9 81 2.1. Reference Deployment . . . . . . . . . . . . . . . . . . 9 82 2.2. Key Components . . . . . . . . . . . . . . . . . . . . . 11 83 2.2.1. Flexible Flow, Packet, and Data Selection . . . . . . 11 84 2.2.2. Flexible Data Export . . . . . . . . . . . . . . . . 13 85 2.2.3. Dynamic Network Probe . . . . . . . . . . . . . . . . 15 86 2.2.4. On-demand Technique Selection and Integration . . . . 17 87 2.3. IFIT for Reflective Telemetry . . . . . . . . . . . . . . 18 88 2.3.1. Intelligent Multipoint Performance Monitoring . . . . 19 89 2.3.2. Intent-based Network Monitoring . . . . . . . . . . . 19 90 3. Guidance for Solution Developers . . . . . . . . . . . . . . 20 91 3.1. Encapsulation in Transport Protocols . . . . . . . . . . 20 92 3.2. Tunneling Support . . . . . . . . . . . . . . . . . . . . 21 93 3.3. Deployment Automation . . . . . . . . . . . . . . . . . . 21 95 4. Security Considerations . . . . . . . . . . . . . . . . . . . 22 96 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 97 6. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 22 98 7. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 22 99 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 100 8.1. Normative References . . . . . . . . . . . . . . . . . . 22 101 8.2. Informative References . . . . . . . . . . . . . . . . . 23 102 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 27 104 1. Introduction 106 Efficient network operation increasingly relies on high-quality data- 107 plane telemetry to provide the necessary visibility into the behavior 108 of traffic flows and network resources. Existing Operation, 109 Administration, and Maintenance (OAM) methods, which include 110 proactive and reactive techniques, running both active and passive 111 modes, are no longer sufficient to meet the monitoring and 112 measurement requirements when networks becomes more autonomous 113 [RFC8993] and application-aware [I-D.li-apn-framework]. The 114 complexity of today's networks and service quality requirements 115 demand new high-precision and real-time OAM techniques. 117 The ability to expedite network failure detection, fault 118 localization, and recovery mechanisms, particularly in the case of 119 soft failures or path degradation is expected, and it must not cause 120 service disruption. Emerging on-path telemetry techniques can 121 provide high-precision flow insight and real-time network issue 122 notification (e.g., jitter, latency, packet loss, significant bit 123 error variations, and unequal load-balancing). On-Path Telemetry 124 (OPT) refers to data-plane telemetry techniques that directly tap and 125 measure network traffic by embedding instructions or metadata into 126 user packets. The data provided by on-path telemetry are especially 127 useful for verifying Service Level Agreement (SLA) compliance, user 128 experience enhancement, service path enforcement, fault diagnosis, 129 and network resource optimization. It is essential to recognize that 130 existing work on this topic includes a variety of on-path telemetry 131 techniques, including In-situ OAM (IOAM) [I-D.ietf-ippm-ioam-data], 132 IOAM Direct Export (DEX) [I-D.ietf-ippm-ioam-direct-export], 133 Marking-based Postcard-based Telemetry (PBT-M) 134 [I-D.song-ippm-postcard-based-telemetry], Enhanced Alternate Marking 135 (EAM) [I-D.zhou-ippm-enhanced-alternate-marking], and Hybrid Two-Step 136 (HTS) [I-D.mirsky-ippm-hybrid-two-step], have been developed or 137 proposed. These techniques can provide flow information on the 138 entire forwarding path on a per-packet basis in real-time. The 139 aforementioned on-path telemetry techniques differ from the active 140 and passive OAM schemes in that they directly modify and monitor the 141 user packets in networks so as to achieve high measurement accuracy. 142 Formally, these on-path telemetry techniques can be classified as the 143 OAM hybrid type I, since they involve "augmentation or modification 144 of the stream of interest, or employment of methods that modify the 145 treatment of the streams", according to [RFC7799]. We name these 146 techniques as "In-situ Flow Information Telemetry" (IFIT). 148 On-path telemetry is useful for application-aware networking 149 operations, not only in data center and enterprise networks, but also 150 in carrier networks which may cross multiple domains. The techniques 151 can provide benefits for carrier network operators in various 152 scenarios. For example, it is critical for the operators who offer 153 high-bandwidth, latency and loss-sensitive services such as video 154 streaming and online gaming to closely monitor the relevant flows in 155 real-time as the basis for any further optimizations. 157 This framework document is intended to guide system designers 158 attempting to use the referenced techniques as well as to motivate 159 further work to enhance the telemetry ecosystem. It highlights 160 requirements and challenges, outlines important techniques that are 161 applicable, and provides examples of how these might be applied for 162 critical use cases. 164 The document scope is discussed in Section 1.3. 166 1.1. Classification and Modes of On-path Telemetry 168 The operation of IFIT differs from both active OAM and passive OAM as 169 defined in [RFC7799]. It does not generate any active probe packets 170 or passively observe unmodified user packets. Instead, it modifies 171 selected user packets in order to collect useful information about 172 them. Therefore, the operation is categorized as the hybrid OAM type 173 I method per [RFC7799]. 175 This hybrid OAM type I method can be further partitioned into two 176 modes [passport-postcard]. In the passport mode, each node on the 177 path can add telemetry data to the user packets (i.e., stamps the 178 passport). The accumulated data trace is exported at a configured 179 end node. In the postcard mode, each node directly exports the 180 telemetry data using an independent packet (i.e., sends a postcard) 181 while the user packets are unmodified. It is possible to combine the 182 two modes together in one solution. We call this the hybrid mode. 184 Figure 1 shows the classification of the on-path telemetry 185 techniques. 187 +-----------+-------------+------------+--------------------+ 188 | Mode | Passport | Postcard | Hybrid | 189 +-----------+-------------+------------+--------------------+ 190 | | IOAM Trace | IOAM DEX | Multicast Telemetry| 191 | Technique | IOAM E2E | PBT-M | HTS | 192 | | | EAM | | 193 +-----------+-------------+------------+--------------------+ 195 Figure 1: On-path Telemetry Technique Classification 197 IOAM Trace and E2E options are described in 198 [I-D.ietf-ippm-ioam-data]. 200 EAM is described in [I-D.zhou-ippm-enhanced-alternate-marking]. 202 IOAM DEX option is described in [I-D.ietf-ippm-ioam-direct-export]. 204 PBT-M is described in [I-D.song-ippm-postcard-based-telemetry]. 206 Multicast Telemetry is described in 207 [I-D.ietf-mboned-multicast-telemetry]. 209 HTS is described in [I-D.mirsky-ippm-hybrid-two-step]. 211 The advantages of the passport mode include: 213 * It automatically retains the telemetry data correlation along the 214 entire path. The self-describing feature simplifies the data 215 consumption. 217 * The on-path data for a packet is only exported once so the data 218 export overhead is low. 220 * Only the head and tail nodes of the paths need to be configured 221 for header insertion and removal, so the configuration overhead is 222 low. 224 The disadvantages of the passport mode include: 226 * The telemetry data carried by user packets inflate the packet 227 size, which may be undesirable or prohibitive. 229 * Approaches for encapsulating the instruction header and data in 230 transport protocols need to be standardized. 232 * Carrying sensitive data along the path is vulnerable to security 233 and privacy breach. 235 * If a packet is dropped on the path, the data collected are also 236 lost. 238 The postcard mode complements the passport mode. The advantages of 239 the postcard mode include: 241 * Either there is no packet header overhead (e.g., PBT-M) or the 242 overhead is small and fixed (e.g., IOAM DEX). 244 * The encapsulation requirement may be avoided (e.g., PBT-M). 246 * The telemetry data can be secured before export. 248 * Even if a packet is dropped on the path, the partial data 249 collected are still available. 251 The disadvantages of the postcard mode include: 253 * Telemetry data are spread in multiple postcards so extra effort is 254 needed to correlate the data. 256 * Every node exports a postcard for a packet which increases the 257 data export overhead. 259 * In case of PBT-M, every node on the path needs to be configured, 260 so the configuration overhead is high. 262 * In case of IOAM DEX, the transport encapsulation requirement 263 remains. 265 The hybrid mode either tailors for some specific application scenario 266 (e.g., Multicast Telemetry) or provides some alternative approach 267 (e.g., HTS). A postcard can be sent per segment of a path or the 268 telemetry data can be carried in a companion packet following each 269 monitored use packet. The hybrid mode combines the advantages of 270 both the passport mode and the postcard mode, but it may incur extra 271 processing complexity. 273 1.2. Requirements and Challenges 275 Although on-path telemetry is beneficial, successfully applying such 276 techniques in carrier networks must consider performance, 277 deployability, and flexibility. Specifically, we need to address the 278 following practical deployment challenges: 280 * C1: On-path telemetry incurs extra packet processing which may 281 cause stress on the network data plane. The potential impact on 282 the forwarding performance creates an unfavorable "observer 283 effect" (where the actions of performing on-path telemetry may 284 change the behavior of the traffic being measured). This will not 285 only damage the fidelity of the measurement, but also defy the 286 purpose of the measurement. 288 * C2: On-path telemetry can generate a considerable amount of data 289 which may claim too much transport bandwidth and inundate the 290 servers for data collection, storage, and analysis. For example, 291 if the technique is applied to all the traffic, one node may 292 collect a few tens of bytes as telemetry data for each packet. 293 The whole forwarding path might accumulate telemetry data with a 294 size similar to or even exceeding that of the original packet. 296 * C3: The collectible data defined currently are essential but 297 limited. This, in turn, limits the management and operational 298 techniques that can be applied. Flexibility and extensibility of 299 data definition, aggregation, acquisition, and filtering, must be 300 considered. 302 * C4: Applying only a single underlying on-path telemetry technique 303 may miss some important events or lead to incorrect results. For 304 example, packet drop can cause the loss of the flow telemetry data 305 and the packet drop location and reason remains unknown if only 306 the In-situ OAM trace option is used. A comprehensive solution 307 needs the flexibility to switch between different underlying 308 techniques and adjust the configurations and parameters at 309 runtime. Thus, system-level orchestration is needed. 311 * C5: We must provide solutions to support an incremental deployment 312 strategy. That is, we need to support established encapsulation 313 schemes for various predominant protocols such as Ethernet, IPv6, 314 and MPLS with backward compatibility and properly handle various 315 transport tunnels. 317 * C6: The development of simplified on-path telemetry primitives and 318 models for configuration and queries is essential. Telemetry 319 models may be utilized via an API-based telemetry service for 320 external applications, for end-to-end performance measurement and 321 application performance monitoring. Standard-based protocols and 322 methods are needed for network configuration and programming, and 323 telemetry data pre-processing and export, to provide 324 interoperability. 326 1.3. Scope 328 Following the network telemetry framework discussed in 329 [I-D.ietf-opsawg-ntf], this document focuses on the on-path 330 telemetry, a specific class of data-plane telemetry techniques, and 331 provides a high-level framework which addresses the challenges for 332 deployment listed in Section 1.2, especially in carrier networks. 334 This document aims to clarify the problem space, essential 335 requirements, and summarizes best practices and general system design 336 considerations. This document provides some examples to show the 337 novel network telemetry applications under the framework. 339 As an informational document, it describes an open framework with a 340 few key components. The framework does not enforce any specific 341 implementation on each component, neither does it define interfaces 342 (e.g., API, protocol) between components. The choice of underlying 343 on-path telemetry techniques and other implementation details is 344 determined by the application implementer. Therefore, the framework 345 is not a solution specification. It only provides a high-level 346 overview and is not necessarily a mandatory recommendation for on- 347 path telemetry applications. 349 The standardization of the underlying techniques and interfaces 350 mentioned in this document is undertaken by various working groups. 351 Due to the limited scope and intended status of this document, it has 352 no overlap or conflict with those works. 354 1.4. Relationship with Network Telemetry Framework (NTF) 356 [I-D.ietf-opsawg-ntf] describes a Network Telemetry Framework (NTF). 357 One dimension used by NTF to partition network telemetry techniques 358 and systems is based on the three planes in networks (i.e., control 359 plane, management plane, and forwarding plane) and external data 360 sources. IFIT fits in the category of forwarding-plane telemetry and 361 deals with the specific on-path technical branch of the forwarding- 362 plane telemetry. 364 According to NTF, an on-path telemetry application mainly subscribes 365 to event-triggered or streaming data. The key functional components 366 of IFIT described in Section 2.2 match the general components in NTF 367 with more specific functions. "On-demand Technique Selection and 368 Integration" is an application layer function, matching the "Data 369 Query, Analysis, and Storage" component in NTF; "Flexible Flow, 370 Packet, and Data Selection" matches the "Data Configuration and 371 Subscription" component; "Flexible Data Export" matches the "Data 372 Encoding and Export" component; "Dynamic Network Probe" matches the 373 "Data Generation and Processing" component. 375 1.5. Glossary 377 This section defines and explains the acronyms and terms used in this 378 document. 380 On-path Telemetry: Remotely acquiring performance and behavior data 381 about network flows on a per-packet basis on the packet's 382 forwarding path. The term refers to a class of data-plane 383 telemetry techniques, including IOAM, PBT, EAM, and HTS. Such 384 techniques may need to mark user packets, or insert instruction/ 385 metadata into the headers of user packets. 387 IFIT: In-situ Flow Information Telemetry is a high-level reference 388 framework that shows how network data-plane monitoring and 389 measurement applications can address the deployment challenges of 390 the flow-oriented on-path telemetry techniques. 392 Reflective Telemetry: The reflective telemetry functions in a 393 dynamic and closed-loop fashion. A new telemetry action is 394 provisioned as a result of self-knowledge acquired through prior 395 telemetry actions. 397 2. Architectural Concepts and Key Components 399 To address the challenges mentioned in Section 1.2, a high-level 400 framework which can help to build a workable and efficient on-path 401 telemetry application is presented. In-situ Flow Information 402 Telemetry (IFIT) is dedicated to on-path telemetry data about user 403 and application traffic flows. It covers a class of on-path 404 telemetry techniques and works at a level higher than any specific 405 underlying technique. The framework is comprised of some key 406 functional components (Section 2.2). By assembling these components, 407 IFIT supports reflective telemetry that enables autonomous network 408 operations (Section 2.3). 410 2.1. Reference Deployment 412 Figure 2 shows a reference deployment scenario of on-path telemetry. 414 On-path Telemetry Application 415 +----------------------------------+ 416 | Controller | 417 | +------------+ +-----------+ | 418 | | Configure | | Collector | | 419 | | & |<----| & | | 420 | | Control | | Analyzer | | 421 | +-----:------+ +-----------+ | 422 | : ^ | 423 +-------:------------------|-------+ 424 :configuration |telemetry data 425 :& action | 426 .............:..................|......... 427 : : : | : 428 : +-------:---+-----------:---+--------:---+ 429 : | : | : | : | 430 V | V | V | V | 431 +------+-+ +-----+--+ +------+-+ +------+-+ +--------+ 432 packets| Head | | Path | | Path | | Tail | |Node out| 433 =>| Node |==>| Node |=//=>| Node |==>| Node |=>|of OPT |=> 434 | | | A | | Z | | | |domain | 435 +--------+ +--------+ +--------+ +--------+ +--------+ 437 |<--- On-path Telemetry Domain --->| 439 Figure 2: Deployment Scenario 441 An on-path telemetry application can conduct network data-plane 442 monitoring and measurement tasks over a limited domain [RFC8799] by 443 applying one or more underlying techniques. The application contains 444 multiple elements, including configuring the network nodes and 445 processing the telemetry data. The application usually uses a 446 logically centralized controller for configuring the network nodes in 447 the domain, and collecting and analyzing telemetry data. The 448 configuration determines which underlying technique is used, what 449 telemetry data are of interest, which flows and packets are concerned 450 with, how the telemetry data are collected, etc. The process can be 451 dynamic and interactive: after the telemetry data processing and 452 analyzing, the application may instruct the controller to modify the 453 configuration of the nodes, which affects the future telemetry data 454 collection. 456 From the system-level view, it is recommended to use standardized 457 configuration and data collection interfaces, regardless of the 458 underlying technique. The specification of these interfaces and the 459 implementation of the controller are out of scope for this document. 461 The on-path telemetry domain encompasses the head nodes and the end 462 nodes, and may cross multiple network domains. The head nodes are 463 responsible for enabling the on-path telemetry functions and the end 464 nodes are responsible for terminating them. All capable nodes in 465 this domain will be capable of executing the instructed on-path 466 telemetry function. It is important to note that any application 467 must, through configuration and policy, guarantee that any packet 468 with on-path telemetry header and metadata will not leak out of the 469 domain. 471 The underlying on-path telemetry techniques covered by the IFIT 472 framework can be of any modes discussed in Section 1.1. 474 2.2. Key Components 476 The key components of IFIT to address the challenges listed in 477 Section 1.2 are as follows. The components are described in more 478 detail in the sections that follow. 480 * Flexible flow, packet, and data selection policy, addressing the 481 challenge C1 described in Section 1; 483 * Flexible data export, addressing the challenge C2; 485 * Dynamic network probe, addressing C3; 487 * On-demand technique selection and integration, addressing C4. 489 Note that the challenges C5 and C6 are mostly standard-related, and 490 are fundamental to IFIT. We discuss the protocol implications and 491 guidance for solution developers in Section 3. 493 2.2.1. Flexible Flow, Packet, and Data Selection 495 In most cases, it is impractical to enable data collection for all 496 the flows and for all the packets in a flow due to the potential 497 performance and bandwidth impact. Therefore, a workable solution 498 usually need to select only a subset of flows and flow packets on 499 which to enable data collection, even though this means the loss of 500 some information and accuracy. 502 In the data plane, a flow filter like those used for an Access 503 Control List (ACL) provides an ideal means to determine the subset of 504 flows. An application can set a sample rate or probability to a flow 505 to allow only a subset of flow packets to be monitored, collect a 506 different set of data for different packets, and disable or enable 507 data collection on any specific network node. An application can 508 further allow any node to accept or deny the data collection process 509 in full or partially. 511 Based on these flexible mechanisms, IFIT allows applications to apply 512 flexible flow and data selection policies to suit their requirements. 513 The applications can dynamically change the policies at any time 514 based on the network load, processing capability, focus of interest, 515 and any other criteria. 517 2.2.1.1. Block Diagram 519 +----------------------------+ 520 | +----------+ +----------+ | 521 | |Flow | |Data | | 522 | |Selection | |Selection | | 523 | +----------+ +----------+ | 524 | +----------+ | 525 | |Packet | | 526 | |Selection | | 527 | +----------+ | 528 +----------------------------+ 530 Figure 3: Flexible Flow, Packet, and Data Selection 532 Figure 3 shows the block diagram of this component. The flow 533 selection block defines the policies to choose target flows for 534 monitoring. Flow has different granularity. A basic flow is defined 535 by 5-tuple IP header fields. Flow can also be aggregated at 536 interface level, tunnel level, protocol level, and so on. The packet 537 selection block defines the policies to choose packets from a target 538 flow. The policy can be either a sampling interval, a sampling 539 probability, or some specific packet signature. The data selection 540 block defines the set of data to be collected. This can be changed 541 on a per-packet or per-flow basis. 543 2.2.1.2. Example: Sketch-guided Elephant Flow Selection 545 Network operators are usually more interested in elephant flows which 546 consume more resource and are sensitive to changes in network 547 conditions. A CountMin Sketch [CMSketch] can be used on the data 548 path of the head nodes, which identifies and reports the elephant 549 flows periodically. The controller maintains a current set of 550 elephant flows and dynamically enables the on-path telemetry for only 551 these flows. 553 2.2.1.3. Example: Adaptive Packet Sampling 555 Applying on-path telemetry on all packets of the selected flows can 556 still be out of reach. A sample rate should be set for these flows 557 and telemetry should only be enabled on the sampled packets. 558 However, the head nodes have no clue on the proper sampling rate. An 559 overly high rate would exhaust the network resource and even cause 560 packet drops; An overly low rate, on the contrary, would result in 561 the loss of information and inaccuracy of measurements. 563 An adaptive approach can be used based on the network conditions to 564 dynamically adjust the sampling rate. Every node gives user traffic 565 forwarding higher priority than telemetry data export. In case of 566 network congestion, the telemetry can sense some signals from the 567 data collected (e.g., deep buffer size, long delay, packet drop, and 568 data loss). The controller may use these signals to adjust the 569 packet sampling rate. In each adjustment period (i.e., RTT of the 570 feedback loop), the sampling rate is either decreased or increased in 571 response of the signals. An Additive Increase/Multiplicative 572 Decrease (AIMD) policy similar to the TCP flow control mechanism for 573 rate adjustment can be used. 575 2.2.2. Flexible Data Export 577 The flow telemetry data can catch the dynamics of the network and the 578 interactions between user traffic and network. Nevertheless, the 579 data may contain redundancy. It is advisable to remove the 580 redundancy from the data in order to reduce the data transport 581 bandwidth and server processing load. 583 In addition to efficient export data encoding (e.g., IPFIX [RFC7011] 584 or protobuf (https://developers.google.com/protocol-buffers/)), nodes 585 have several other ways to reduce the export data by taking advantage 586 of network device's capability and programmability. Nodes can cache 587 the data and send the accumulated data in batches if the data is not 588 time sensitive. Various deduplication and compression techniques can 589 be applied on the batched data. 591 From the application perspective, an application may only be 592 interested in some special events which can be derived from the 593 telemetry data. For example, in the case that the forwarding delay 594 of a packet exceeds a threshold, or a flow changes its forwarding 595 path is of interest, it is unnecessary to send the original raw data 596 to the data collecting and processing servers. Rather, IFIT takes 597 advantage of the in-network computing capability of network devices 598 to process the raw data and only push the event notifications to the 599 subscribing applications. 601 Such events can be expressed as policies. A policy can request data 602 export only on change, on exception, on timeout, or on threshold. 604 2.2.2.1. Block Diagram 606 +-------------------------------------------+ 607 | +-----------+ +-----------+ +-----------+ | 608 | |Data | |Data | |Export | | 609 | |Encoding | |Batching | |Protocol | | 610 | +-----------+ +-----------+ +-----------+ | 611 | +-----------+ +-----------+ +-----------+ | 612 | |Data | |Data | |Data | | 613 | |Compression| |Dedup. | |Filter | | 614 | +-----------+ +-----------+ +-----------+ | 615 | +-----------+ +-----------+ | 616 | |Data | |Data | | 617 | |Computing | |Aggregation| | 618 | +-----------+ +-----------+ | 619 +-------------------------------------------+ 621 Figure 4: Flexible Data Export 623 Figure 4 shows the block diagram of this component. The data 624 encoding block defines the method to encode the telemetry data. The 625 data batching block defines the size of batch data buffered at the 626 device side before export. The export protocol block defines the 627 protocol used for telemetry data export. The data compression block 628 defines the algorithm to compress the raw data. The data 629 deduplication block defines the algorithm to remove the redundancy in 630 the raw data. The data filter block defines the policies to filter 631 the needed data. The data computing block defines the policies to 632 prepocess the raw data and generate some new data. The data 633 aggregation block defines the procedure to combine and synthesize the 634 data. 636 2.2.2.2. Example: Event-based Anomaly Monitor 638 Network operators are interested in anomalies such as path change, 639 network congestion, and packet drop. Such anomalies are hidden in 640 raw telemetry data (e.g., path trace, timestamp). Such anomalies can 641 be described as events and programmed into the device data plane. 642 Only the triggered events are exported. For example, if a new flow 643 appears at any node, a path change event is triggered; if the packet 644 delay exceeds a predefined threshold in a node, the congestion event 645 is triggered; if a packet is dropped due to buffer overflow, a packet 646 drop event is triggered. 648 The export data reduction due to such optimization is substantial. 649 For example, given a single 5-hop 10Gbps path, assume a moderate 650 number of 1 million packets per second are monitored, and the 651 telemetry data plus the export packet overhead consume less than 30 652 bytes per hop. Without such optimization, the bandwidth consumed by 653 the telemetry data can easily exceed 1Gbps (more than 10% of the path 654 bandwidth), When the optimization is used, the bandwidth consumed by 655 the telemetry data is negligible. Moreover, the pre-processed 656 telemetry data greatly simplify the work of data analyzers. 658 2.2.3. Dynamic Network Probe 660 Due to limited data plane resource and network bandwidth, it is 661 unlikely one can monitor all the data all the time. On the other 662 hand, the data needed by applications may be arbitrary but ephemeral. 663 It is critical to meet the dynamic data requirements with limited 664 resource. 666 Fortunately, data plane programmability allows new data probles to be 667 dynamically loaded. These on-demand probes are called Dynamic 668 Network Probes (DNP). DNP is the technique to enable probes for 669 customized data collection in different network planes. When working 670 with an on-path telemetry technique, DNP is loaded into the data 671 plane through incremental programming or configuration. The DNP can 672 effectively conduct data generation, processing, and aggregation. 674 DNP introduces flexibility and extensibility to IFIT. It can 675 implement the optimizations for export data reduction motioned in the 676 previous section. It can also generate custom data as required by 677 today's and tomorrow's applications. 679 2.2.3.1. Block Diagram 680 +----------------------------+ 681 | +----------+ +----------+ | 682 | |Active | |YANG | | 683 | |Packet | |Model | | 684 | |Filter | | | | 685 | +----------+ +----------+ | 686 | +----------+ +----------+ | 687 | |Hardware | |Software | | 688 | |Function | |Function | | 689 | +----------+ +----------+ | 690 +----------------------------+ 692 Figure 5: Dynamic Network Probes 694 Figure 5 shows the block diagram of this component. The active 695 packet filter block is available in most hardware and it defines DNPs 696 through dynamically update the packet filtering policies (including 697 flow selection and action). YANG models can be dynamically deployed 698 to enable different data processing and filtering functions. Some 699 hardware allows dynamically loading hardware-based functions into the 700 forwarding path at runtime through mechanisms such as reserved 701 pipelines and function stubs. Dynamically loadable software 702 functions can be implemented in the control processors in capable 703 nodes. 705 2.2.3.2. Examples 707 Following are some possible DNPs that can be dynamically deployed to 708 support applications. 710 On-demand Flow Sketch: A flow sketch is a compact online data 711 structure (usually a variation of multi-hashing table) for 712 approximate estimation of multiple flow properties. It can be 713 used to facilitate flow selection. The aforementioned CountMin 714 Sketch [CMSketch] is such an example. Since a sketch consumes 715 data plane resources, it should only be deployed when actually 716 needed. 718 Smart Flow Filter: The policies that choose flows and packet 719 sampling rate can change during the lifetime of an application. 721 Smart Statistics: An application may need to count flows based on 722 different flow granularity or maintain hit counters for selected 723 flow table entries. 725 Smart Data Reduction: DNP can be used to program the events that 726 conditionally trigger data export. 728 2.2.4. On-demand Technique Selection and Integration 730 With multiple underlying data collection and export techniques at its 731 disposal, IFIT can flexibly adapt to different network conditions and 732 different application requirements. 734 For example, depending on the types of data that are of interest, 735 IFIT may choose either passport or postcard mode to collect the data; 736 if an application needs to track down where the packets are lost, 737 switching from passport mode to postcard mode should be supported. 739 IFIT can further integrate multiple data plane monitoring and 740 measurement techniques together and present a comprehensive data 741 plane telemetry solution. 743 Based on the application requirements and the real-time telemetry 744 data analysis results, new configurations and actions can be 745 deployed. 747 2.2.4.1. Block Diagram 749 +----------------------------------------------+ 750 | +------------+ +-------------+ +---------+ | 751 | |Application | |Configuration| |Telemetry| | 752 | |Requirements|->|& Action |<-|Data | | 753 | | | | | |Analysis | | 754 | +------------+ +-------------+ +---------+ | 755 +----------------------------------------------+ 756 | Passport Mode: | 757 | +----------+ +----------+ | 758 | |IOAM E2E | |IOAM Trace| | 759 | +----------+ +----------+ | 760 | Postcard Mode: | 761 | +----------+ +----------+ +----------+ | 762 | |PBT-M | |IOAM DEX | |EAM | | 763 | +----------+ +----------+ +----------+ | 764 | Hybrid Mode: | 765 | +----------+ +----------+ | 766 | |HTS | |Multicast | | 767 | | | |Telemetry | | 768 | +----------+ +----------+ | 769 +----------------------------------------------+ 771 Figure 6: Technique Selection and Integration 773 Figure 6 shows the block diagram of this component, which lists the 774 candidate on-path telemetry techniques. 776 Located in the logically centralized controller, this component makes 777 all the control and configuration dynamically to the capable nodes in 778 the domain which will affect the future telemetry data. The 779 configuration and action decisions are based on the inputs from the 780 application requirements and the realtime telemetry data analysis 781 results. Note that here the telemetry data source is not limited to 782 the data plane. The data can come form all the sources mentioned in 783 [I-D.ietf-opsawg-ntf], including external data sources. 785 2.3. IFIT for Reflective Telemetry 787 The components described in Section 2.2 can work together to support 788 reflective telemetry, as shown in Figure 7. 790 +---------------------+ 791 | On-path Telemetry | 792 +------+ Applications |<------+ 793 | | | | 794 | +---------------------+ | 795 | Technique Selection | 796 | and Integration | 797 | | 798 |Flexible Flexible | 799 |Flow, reflection-loop Data | 800 |Packet, Export| 801 |and Data | 802 |Selection +----+----+ 803 V +---------+| 804 +----------+ Encapsulation +---------+|| 805 | Head | and Tunneling | Path ||| 806 | Node |----------------------->| Nodes ||+ 807 | | | |+ 808 +----------+ +---------+ 809 DNP DNP 811 Figure 7: IFIT-based Reflective Telemetry 813 An application may pick a suite of telemetry techniques based on its 814 requirements and apply an initial technique to the data plane. It 815 then configures the head nodes to decide the initial target flows/ 816 packets and telemetry data set, the encapsulation and tunneling 817 scheme based on the underlying network architecture, and the IFIT- 818 capable nodes to decide the initial telemetry data export policy. 819 Based on the network condition and the analysis results of the 820 telemetry data, the application can change the telemetry technique, 821 the flow/data selection policy, and the data export approach in real 822 time without breaking the normal network operation. Many of such 823 dynamic changes can be done through loading and unloading DNPs. 825 The reflective telemetry enabled by the IFIT allows numerous new 826 applications. Two examples are provided below. 828 2.3.1. Intelligent Multipoint Performance Monitoring 830 [RFC8889] describes an intelligent performance management based on 831 the network condition. The idea is to split the monitoring network 832 into clusters. The cluster partition that can be applied to every 833 type of network graph and the possibility to combine clusters at 834 different levels enable the so-called Network Zooming. It allows a 835 controller to calibrate the network telemetry, so that it can start 836 without examining in depth and monitor the network as a whole. In 837 case of necessity (packet loss or too high delay), an immediate 838 detailed analysis can be reconfigured. In particular, the 839 controller, that is aware of the network topology, can set up the 840 most suitable cluster partition by changing the traffic filter or 841 activate new measurement points and the problem can be localized with 842 a step-by-step process. 844 An application on top of the controllers can manage such mechanism, 845 whose dynamic and reflective operations are supported by the IFIT 846 framework. 848 2.3.2. Intent-based Network Monitoring 850 1.User Intents 851 | 852 V 5.Per-packet 853 4.Packet +------------+ Telemetry 854 Filter | | Data 855 +--------+ Controller |<--------+ 856 | | | | 857 | +--+---------+ | 858 | | ^ | 859 | 2.|DNPs 3.|Network | 860 | | |Information| 861 | V | | 862 +------+-------------------+-----------+---+ 863 | | | 864 | V +------+ | 865 | +-------+ +------+| | 866 | | Head | +------+|| | 867 | | Node | |Path ||+ | 868 | | | |Nodes |+ | 869 | +-------+ +------+ | 870 +------------------------------------------+ 872 Figure 8: Intent-based Monitoring 874 In this example, a user can express high level intents for network 875 monitoring. The controller translates an intent and configures the 876 corresponding DNPs in capable nodes which collect necessary network 877 information. Based on the real-time information feedback, the 878 controller runs a local algorithm to determine the suspicious flows. 879 It then deploys specific packet filters to the head node to initiate 880 the high precision per-packet on-path telemetry for these flows. 882 3. Guidance for Solution Developers 884 Having a high-level framework covering a class of related techniques 885 promotes a holistic approach for standard development and helps to 886 avoid duplicated efforts and piecemeal solutions that only focus on a 887 specific technique while omitting the compatibility and extensibility 888 issues, which is important to a healthy ecosystem for network 889 telemetry. 891 A complete IFIT-based solution needs standard interfaces for 892 configuration and data extraction, and standard encapsulation on 893 various transport protocols. It may also need standard API and 894 primitives for application programming and deployment. 895 [I-D.ietf-ippm-ioam-deployment] summarizes some techniques for 896 encapsulation and data export for IOAM. Solution developers need to 897 consider the aspects set out in the following subsections. 899 3.1. Encapsulation in Transport Protocols 901 Since the introduction of IOAM, the IOAM option header encapsulation 902 schemes in various network protocols have been defined (e.g., 903 [I-D.ietf-ippm-ioam-ipv6-options]). Similar encapsulation schemes 904 are needed to cover the other on-path telemetry techniques. 905 Meanwhile, the on-path telemetry header/data encapsulation schemes in 906 some popular protocols, such as MPLS and SRv6, are also needed. 907 PBT-M [I-D.song-ippm-postcard-based-telemetry] does not introduce new 908 headers to the packets so the trouble of encapsulation for a new 909 header is avoided. While there are some proposals which allow new 910 header encapsulation in MPLS packets (e.g., 911 [I-D.song-mpls-extension-header]) or in SRv6 packets (e.g., 912 [I-D.song-spring-siam]), they are still in their infancy stage and 913 require further work. Before standards are available, in a confined 914 domain, pre-standard encapsulation approaches may be applied. 916 3.2. Tunneling Support 918 In carrier networks, it is common for user traffic to traverse 919 various tunnels for QoS, traffic engineering, or security. Both the 920 uniform mode and the pipe mode for tunnel support are required and 921 described in [I-D.song-ippm-ioam-tunnel-mode]. The uniform mode 922 treats the nodes in a tunnel uniformly as the nodes outside of the 923 tunnel on a path. In contrast, the pipe mode abstracts all the nodes 924 between the tunnel ingress and egress as a circuit so no nodes in the 925 tunnel is visible to the nodes outside of the tunnel. With such 926 flexibility, the operator can either gain a true end-to-end 927 visibility or apply a hierarchical approach which isolates the 928 monitoring domain between customer and provider. 930 3.3. Deployment Automation 932 Standard approaches that automate the function configuration, and 933 capability query and advertisement, could either be deployed in a 934 centralized fashion or a distributed fashion. The draft 935 [I-D.ietf-ippm-ioam-yang] provides a YANG model for IOAM 936 configuration. Similar models needs to be defined for other 937 techniques. It is also helpful to provide standards-based approaches 938 for configuration in various network environments. For example, in 939 Segment Routing (SR) networks, extensions to BGP or Path Computation 940 Element Communication Protocol (PCEP) can be defined to distribute SR 941 policies carrying on-path telemetry information, so that telemetry 942 behavior can be enabled automatically when the SR policy is applied. 943 [I-D.chen-pce-sr-policy-ifit] defines extensions to PCEP to configure 944 SR policies for on-path telemetry. [I-D.ietf-idr-sr-policy-ifit] 945 defines extensions to BGP for the same purpose. Additional 946 capability discovery and dissemination will be needed for other types 947 of networks. 949 To realize the potential of on-path telemetry, programming and 950 deploying DNPs are important. ForCES [RFC5810] is a standard 951 protocol for network device programming, which can be used for DNP 952 deployment. Currently some related works such as 953 [I-D.wwx-netmod-event-yang] and [I-D.bwd-netmod-eca-framework] have 954 proposed to use YANG models to define the smart policies which can be 955 used to implement DNPs. In the future, other approaches for hardware 956 and software-based functions can be development to enhance the 957 programmability and flexibility. 959 4. Security Considerations 961 In addition to the specific security issues discussed in each 962 individual document on on-path telemetry, this document considers the 963 overall security issues at the system level. This should serve as a 964 guide to the on-path telemetry application developers and users. 965 General security and privacy considerations for any network telemetry 966 system are also discussed in [I-D.ietf-opsawg-ntf]. 968 Since the on-path telemetry techniques work on the network forwarding 969 plane, the IFIT framework poses some security risks. The important 970 and sensitive information about a network could be exposed to an 971 attacker. Further, the on-path telemetry data might swamp various 972 parts of the network, leading to a possible DoS attack. 974 Fortunately, security measures can be enforced on various parts of 975 the framework to mitigate such threats. For example, the 976 configuration can filter and rate limit the monitored traffic; 977 encryption and authentication can be applied on the exported 978 telemetry data; different underlying techniques can be chosen to 979 adapt to the different network conditions. 981 5. IANA Considerations 983 This document includes no request to IANA. 985 6. Contributors 987 Other major contributors of this document include Giuseppe Fioccola, 988 Daniel King, Zhenqiang Li, Zhenbin Li, Tianran Zhou, and James 989 Guichard. 991 7. Acknowledgments 993 We thank Diego Lopez, Shwetha Bhandari, Joe Clarke, Adrian Farrel, 994 Frank Brockners, Al Morton, Alex Clemm, Alan DeKok, Benoit Claise, 995 and Warren Kumari for their constructive suggestions for improving 996 this document. 998 8. References 1000 8.1. Normative References 1002 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1003 Requirement Levels", BCP 14, RFC 2119, 1004 DOI 10.17487/RFC2119, March 1997, 1005 . 1007 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1008 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1009 May 2016, . 1011 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1012 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1013 May 2017, . 1015 [RFC8799] Carpenter, B. and B. Liu, "Limited Domains and Internet 1016 Protocols", RFC 8799, DOI 10.17487/RFC8799, July 2020, 1017 . 1019 8.2. Informative References 1021 [CMSketch] Cormode, G. and S. Muthukrishnan, "An improved data stream 1022 summary: the count-min sketch and its applications", 2005, 1023 . 1025 [I-D.bwd-netmod-eca-framework] 1026 Boucadair, M., Wu, Q., Wang, M., King, D., and C. Xie, 1027 "Framework for Use of ECA (Event Condition Action) in 1028 Network Self Management", Work in Progress, Internet- 1029 Draft, draft-bwd-netmod-eca-framework-00, 3 November 2019, 1030 . 1033 [I-D.chen-pce-sr-policy-ifit] 1034 Chen, H., Yuan, H., Zhou, T., Li, W., Fioccola, G., and Y. 1035 Wang, "PCEP SR Policy Extensions to Enable IFIT", Work in 1036 Progress, Internet-Draft, draft-chen-pce-sr-policy-ifit- 1037 02, 10 July 2020, . 1040 [I-D.herbert-ipv4-eh] 1041 Herbert, T., "IPv4 Extension Headers and Flow Label", Work 1042 in Progress, Internet-Draft, draft-herbert-ipv4-eh-01, 2 1043 May 2019, . 1046 [I-D.ietf-idr-sr-policy-ifit] 1047 Qin, F., Yuan, H., Zhou, T., Fioccola, G., and Y. Wang, 1048 "BGP SR Policy Extensions to Enable IFIT", Work in 1049 Progress, Internet-Draft, draft-ietf-idr-sr-policy-ifit- 1050 03, 10 January 2022, . 1053 [I-D.ietf-ippm-ioam-data] 1054 Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields 1055 for In-situ OAM", Work in Progress, Internet-Draft, draft- 1056 ietf-ippm-ioam-data-17, 13 December 2021, 1057 . 1060 [I-D.ietf-ippm-ioam-deployment] 1061 Brockners, F., Bhandari, S., Bernier, D., and T. Mizrahi, 1062 "In-situ OAM Deployment", Work in Progress, Internet- 1063 Draft, draft-ietf-ippm-ioam-deployment-00, 19 October 1064 2021, . 1067 [I-D.ietf-ippm-ioam-direct-export] 1068 Song, H., Gafni, B., Zhou, T., Li, Z., Brockners, F., 1069 Bhandari, S., Sivakolundu, R., and T. Mizrahi, "In-situ 1070 OAM Direct Exporting", Work in Progress, Internet-Draft, 1071 draft-ietf-ippm-ioam-direct-export-07, 13 October 2021, 1072 . 1075 [I-D.ietf-ippm-ioam-ipv6-options] 1076 Bhandari, S. and F. Brockners, "In-situ OAM IPv6 Options", 1077 Work in Progress, Internet-Draft, draft-ietf-ippm-ioam- 1078 ipv6-options-07, 6 February 2022, 1079 . 1082 [I-D.ietf-ippm-ioam-yang] 1083 Zhou, T., Guichard, J., Brockners, F., and S. Raghavan, "A 1084 YANG Data Model for In-Situ OAM", Work in Progress, 1085 Internet-Draft, draft-ietf-ippm-ioam-yang-03, 25 January 1086 2022, . 1089 [I-D.ietf-mboned-multicast-telemetry] 1090 Song, H., McBride, M., Mirsky, G., Mishra, G., Asaeda, H., 1091 and T. Zhou, "Multicast On-path Telemetry Solutions", Work 1092 in Progress, Internet-Draft, draft-ietf-mboned-multicast- 1093 telemetry-02, 4 January 2022, 1094 . 1097 [I-D.ietf-opsawg-ntf] 1098 Song, H., Qin, F., Martinez-Julia, P., Ciavaglia, L., and 1099 A. Wang, "Network Telemetry Framework", Work in Progress, 1100 Internet-Draft, draft-ietf-opsawg-ntf-13, 3 December 2021, 1101 . 1104 [I-D.li-apn-framework] 1105 Li, Z., Peng, S., Voyer, D., Li, C., Liu, P., Cao, C., 1106 Mishra, G., Ebisawa, K., Previdi, S., and J. N. Guichard, 1107 "Application-aware Networking (APN) Framework", Work in 1108 Progress, Internet-Draft, draft-li-apn-framework-04, 25 1109 October 2021, . 1112 [I-D.mirsky-ippm-hybrid-two-step] 1113 Mirsky, G., Lingqiang, W., Zhui, G., and H. Song, "Hybrid 1114 Two-Step Performance Measurement Method", Work in 1115 Progress, Internet-Draft, draft-mirsky-ippm-hybrid-two- 1116 step-12, 26 January 2022, 1117 . 1120 [I-D.song-ippm-ioam-tunnel-mode] 1121 Song, H., Li, Z., Zhou, T., and Z. Wang, "In-situ OAM 1122 Processing in Tunnels", Work in Progress, Internet-Draft, 1123 draft-song-ippm-ioam-tunnel-mode-00, 27 June 2018, 1124 . 1127 [I-D.song-ippm-postcard-based-telemetry] 1128 Song, H., Mirsky, G., Filsfils, C., Abdelsalam, A., Zhou, 1129 T., Li, Z., Shin, J., and K. Lee, "In-Situ OAM Marking- 1130 based Direct Export", Work in Progress, Internet-Draft, 1131 draft-song-ippm-postcard-based-telemetry-11, 15 November 1132 2021, . 1135 [I-D.song-mpls-extension-header] 1136 Song, H., Li, Z., Zhou, T., Andersson, L., and Z. Zhang, 1137 "MPLS Extension Header", Work in Progress, Internet-Draft, 1138 draft-song-mpls-extension-header-06, 10 January 2022, 1139 . 1142 [I-D.song-spring-siam] 1143 Song, H. and T. Pan, "SRv6 In-situ Active Measurement", 1144 Work in Progress, Internet-Draft, draft-song-spring-siam- 1145 02, 6 December 2021, . 1148 [I-D.wwx-netmod-event-yang] 1149 Wu, Q., Bryskin, I., Birkholz, H., Liu, X., and B. Claise, 1150 "A YANG Data model for ECA Policy Management", Work in 1151 Progress, Internet-Draft, draft-wwx-netmod-event-yang-10, 1152 1 November 2020, . 1155 [I-D.zhou-ippm-enhanced-alternate-marking] 1156 Zhou, T., Fioccola, G., Liu, Y., Lee, S., Cociglio, M., 1157 and W. Li, "Enhanced Alternate Marking Method", Work in 1158 Progress, Internet-Draft, draft-zhou-ippm-enhanced- 1159 alternate-marking-08, 4 January 2022, 1160 . 1163 [passport-postcard] 1164 Handigol, N., Heller, B., Jeyakumar, V., Mazieres, D., and 1165 N. McKeown, "Where is the debugger for my software-defined 1166 network?", 2012, 1167 . 1169 [RFC5810] Doria, A., Ed., Hadi Salim, J., Ed., Haas, R., Ed., 1170 Khosravi, H., Ed., Wang, W., Ed., Dong, L., Gopal, R., and 1171 J. Halpern, "Forwarding and Control Element Separation 1172 (ForCES) Protocol Specification", RFC 5810, 1173 DOI 10.17487/RFC5810, March 2010, 1174 . 1176 [RFC7011] Claise, B., Ed., Trammell, B., Ed., and P. Aitken, 1177 "Specification of the IP Flow Information Export (IPFIX) 1178 Protocol for the Exchange of Flow Information", STD 77, 1179 RFC 7011, DOI 10.17487/RFC7011, September 2013, 1180 . 1182 [RFC8889] Fioccola, G., Ed., Cociglio, M., Sapio, A., and R. Sisto, 1183 "Multipoint Alternate-Marking Method for Passive and 1184 Hybrid Performance Monitoring", RFC 8889, 1185 DOI 10.17487/RFC8889, August 2020, 1186 . 1188 [RFC8993] Behringer, M., Ed., Carpenter, B., Eckert, T., Ciavaglia, 1189 L., and J. Nobre, "A Reference Model for Autonomic 1190 Networking", RFC 8993, DOI 10.17487/RFC8993, May 2021, 1191 . 1193 Authors' Addresses 1195 Haoyu Song 1196 Futurewei 1197 2330 Central Expressway 1198 Santa Clara, 1199 United States of America 1200 Email: haoyu.song@futurewei.com 1202 Fengwei Qin 1203 China Mobile 1204 No. 32 Xuanwumenxi Ave., Xicheng District 1205 Beijing, 100032 1206 P.R. China 1207 Email: qinfengwei@chinamobile.com 1209 Huanan Chen 1210 China Telecom 1211 Email: chenhuan6@chinatelecom.cn 1213 Jaehwan Jin 1214 LG U+ 1215 South Korea 1216 Email: daenamu1@lguplus.co.kr 1218 Jongyoon Shin 1219 SK Telecom 1220 South Korea 1221 Email: jongyoon.shin@sk.com