idnits 2.17.1 draft-song-opsawg-ifit-framework-14.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (April 2, 2021) is 1092 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '1' on line 1229 == Outdated reference: A later version (-03) exists of draft-brockners-opsawg-ioam-deployment-02 == Outdated reference: A later version (-03) exists of draft-herbert-ipv4-eh-01 == Outdated reference: A later version (-17) exists of draft-ietf-ippm-ioam-data-11 == Outdated reference: A later version (-11) exists of draft-ietf-ippm-ioam-direct-export-02 == Outdated reference: A later version (-13) exists of draft-ietf-opsawg-ntf-06 == Outdated reference: A later version (-15) exists of draft-mirsky-ippm-hybrid-two-step-07 == Outdated reference: A later version (-16) exists of draft-song-ippm-postcard-based-telemetry-08 == Outdated reference: A later version (-13) exists of draft-song-mpls-extension-header-02 == Outdated reference: A later version (-14) exists of draft-zhou-ippm-enhanced-alternate-marking-06 Summary: 0 errors (**), 0 flaws (~~), 11 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 OPSAWG H. Song 3 Internet-Draft Futurewei 4 Intended status: Informational F. Qin 5 Expires: October 4, 2021 China Mobile 6 H. Chen 7 China Telecom 8 J. Jin 9 LG U+ 10 J. Shin 11 SK Telecom 12 April 2, 2021 14 In-situ Flow Information Telemetry 15 draft-song-opsawg-ifit-framework-14 17 Abstract 19 As network scale increases and network operation becomes more 20 sophisticated, traditional Operation, Administration and Maintenance 21 (OAM) methods, which include proactive and reactive techniques, 22 running in active and passive modes, are no longer sufficient to meet 23 the monitoring and measurement requirements. On-path telemetry 24 techniques which provide high-precision flow insight and real-time 25 issue notification are emerging to support suitable quality of 26 experience for users and applications, and fault or network 27 deficiency identification before they become critical. 29 Centering on the new data plane on-path telemetry techniques, this 30 document outlines a high-level framework to provide an operational 31 environment that utilizes these techniques to enable the collection 32 and correlation of performance information from the network. The 33 framework identifies the components that are needed to coordinate the 34 existing protocol tools and telemetry mechanisms, and addresses key 35 deployment challenges for flow-oriented on-path telemetry techniques, 36 especially in carrier networks. 38 The framework is informational and intended to guide system designers 39 attempting to apply the referenced techniques as well as to motivate 40 further work to enhance the ecosystem . 42 Status of This Memo 44 This Internet-Draft is submitted in full conformance with the 45 provisions of BCP 78 and BCP 79. 47 Internet-Drafts are working documents of the Internet Engineering 48 Task Force (IETF). Note that other groups may also distribute 49 working documents as Internet-Drafts. The list of current Internet- 50 Drafts is at https://datatracker.ietf.org/drafts/current/. 52 Internet-Drafts are draft documents valid for a maximum of six months 53 and may be updated, replaced, or obsoleted by other documents at any 54 time. It is inappropriate to use Internet-Drafts as reference 55 material or to cite them other than as "work in progress." 57 This Internet-Draft will expire on October 4, 2021. 59 Copyright Notice 61 Copyright (c) 2021 IETF Trust and the persons identified as the 62 document authors. All rights reserved. 64 This document is subject to BCP 78 and the IETF Trust's Legal 65 Provisions Relating to IETF Documents 66 (https://trustee.ietf.org/license-info) in effect on the date of 67 publication of this document. Please review these documents 68 carefully, as they describe your rights and restrictions with respect 69 to this document. Code Components extracted from this document must 70 include Simplified BSD License text as described in Section 4.e of 71 the Trust Legal Provisions and are provided without warranty as 72 described in the Simplified BSD License. 74 Table of Contents 76 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 77 1.1. Classification and Modes of On-path Telemetry . . . . . . 4 78 1.2. Requirements and Challenges . . . . . . . . . . . . . . . 6 79 1.3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 8 80 1.4. Glossary . . . . . . . . . . . . . . . . . . . . . . . . 8 81 1.5. Requirements Language . . . . . . . . . . . . . . . . . . 9 82 2. IFIT Overview . . . . . . . . . . . . . . . . . . . . . . . . 9 83 2.1. Typical Deployment of IFIT . . . . . . . . . . . . . . . 10 84 2.2. IFIT Architecture . . . . . . . . . . . . . . . . . . . . 11 85 2.3. Relationship with Network Telemetry Framework (NTF) . . . 12 86 3. Key Components of IFIT . . . . . . . . . . . . . . . . . . . 13 87 3.1. Flexible Flow, Packet, and Data Selection . . . . . . . . 13 88 3.1.1. Block Diagram . . . . . . . . . . . . . . . . . . . . 14 89 3.1.2. Example: Sketch-guided Elephant Flow Selection . . . 14 90 3.1.3. Example: Adaptive Packet Sampling . . . . . . . . . . 14 91 3.2. Flexible Data Export . . . . . . . . . . . . . . . . . . 15 92 3.2.1. Block Diagram . . . . . . . . . . . . . . . . . . . . 15 93 3.2.2. Example: Event-based Anomaly Monitor . . . . . . . . 16 94 3.3. Dynamic Network Probe . . . . . . . . . . . . . . . . . . 17 95 3.3.1. Block Diagram . . . . . . . . . . . . . . . . . . . . 17 96 3.3.2. Examples . . . . . . . . . . . . . . . . . . . . . . 18 98 3.4. On-demand Technique Selection and Integration . . . . . . 18 99 3.4.1. Block Diagram . . . . . . . . . . . . . . . . . . . . 19 100 4. IFIT for Reflective Telemetry . . . . . . . . . . . . . . . . 19 101 4.1. Example: Intelligent Multipoint Performance Monitoring . 20 102 4.2. Example: Intent-based Network Monitoring . . . . . . . . 21 103 5. Standard Status and Gaps . . . . . . . . . . . . . . . . . . 22 104 5.1. Encapsulation in Transport Protocols . . . . . . . . . . 22 105 5.2. Tunneling Support . . . . . . . . . . . . . . . . . . . . 22 106 5.3. Deployment Automation . . . . . . . . . . . . . . . . . . 22 107 6. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 108 7. Security Considerations . . . . . . . . . . . . . . . . . . . 24 109 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 24 110 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 24 111 10. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 24 112 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 24 113 11.1. Normative References . . . . . . . . . . . . . . . . . . 24 114 11.2. Informative References . . . . . . . . . . . . . . . . . 25 115 11.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 27 116 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 27 118 1. Introduction 120 Efficient network operation increasingly relies on high-quality data- 121 plane telemetry to provide the necessary visibility. Traditional 122 Operation, Administration and Maintenance (OAM) methods, which 123 include proactive and reactive techniques, running both active and 124 passive modes, are no longer sufficient to meet the monitoring and 125 measurement requirements. The complexity of today's networks and 126 service quality requirements demand new high-precision and real-time 127 techniques. 129 The ability to expedite network failure detection, fault 130 localization, and recovery mechanisms, particularly in the case of 131 soft failures or path degradation is expected, without causing 132 service disruption. Networks also become application-aware. 133 Application-aware networking is an industry term used to describe the 134 capacity of a network to maintain current information about users and 135 application connections which may be used to optimize the network 136 resource usage and improve the quality of service. 138 The emerging on-path telemetry techniques can provide high-precision 139 flow insight and real-time network issue notification (e.g., jitter, 140 latency, packet loss, significant bit error variations, and unequal 141 load-balancing). On-path telemetry refers to the data-plane 142 telemetry techniques that directly tap and measure network traffic by 143 embedding instructions or metadata into user packets. The data 144 provided by on-path telemetry are especially useful for SLA 145 compliance, user experience enhancement, service path enforcement, 146 fault diagnosis, and network resource optimization. It is essential 147 to recognize that existing work on this topic includes a variety of 148 on-path telemetry techniques, including In-situ OAM(IOAM) 149 [I-D.ietf-ippm-ioam-data], IOAM Direct Export (DEX) 150 [I-D.ietf-ippm-ioam-direct-export], Postcard-based Telemetry(PBT) 151 [I-D.song-ippm-postcard-based-telemetry], Enhanced Alternate Marking 152 (EAM) [I-D.zhou-ippm-enhanced-alternate-marking], and Hybrid Two 153 Steps (HTS) [I-D.mirsky-ippm-hybrid-two-step], have been proposed, 154 which can provide flow information on the entire forwarding path on a 155 per-packet basis in real-time. The aforementioned on-path telemetry 156 techniques differ from the active and passive OAM schemes discussed 157 earlier in that, they directly modify and monitor the user packets so 158 as to achieve high measurement accuracy. Formally, these on-path 159 telemetry techniques can be classified as the OAM hybrid type I, 160 since they involve "augmentation or modification of the stream of 161 interest, or employment of methods that modify the treatment of the 162 streams", according to [RFC7799]. 164 On-path telemetry is useful for application-aware networking 165 operations not only in data center and enterprise networks but also 166 in carrier networks which may cross multiple domains. Carrier 167 network operators have shown interest in utilizing such techniques 168 for various purposes. For example, it is critical for the operators 169 who offer high-bandwidth, latency and loss-sensitive services such as 170 video streaming and online gaming to closely monitor the relevant 171 flows in real-time as the basis for any further optimizations. 173 This framework document is intended to guide system designers 174 attempting to use the referenced techniques as well as to motivate 175 further work to enhance the telemetry ecosystem. It highlights 176 requirements and challenges, outlines vital techniques that are 177 applicable, and provides examples of how these might be applied for 178 critical use cases. 180 The document scope is discussed in Section 1.3. 182 1.1. Classification and Modes of On-path Telemetry 184 The operation of on-path telemetry differs from both active OAM and 185 passive OAM as defined in [RFC7799]. It does not generate any active 186 probe packets or passively observes unmodified user packets. 187 Instead, it modifies selected user packets in order to collect useful 188 information about them. Therefore, the operation can be categorized 189 as the hybrid OAM type I mode per [RFC7799], which can provide more 190 flexible and accurate network monitoring and measurement. 192 This hybrid type OAM can be further partitioned into two modes. 193 [passport-postcard] first uses the metaphor of "passport" and 194 "postcard" to describe how the on-path data can be collected and 195 exported. In the passport mode, each node on the path adds the 196 telemetry data to the user packets (i.e., stamp the passport). The 197 accumulated data trace is exported at a configured end node. In the 198 postcard mode, each node directly exports the telemetry data using an 199 independent packet (i.e., send a postcard) while the user packets are 200 intact. It is possible to combine the two modes together in one 201 solution. We call this the hybrid mode. 203 Figure 1 shows the classification of the existing on-path telemetry 204 techniques. 206 +-----------+--------------+--------------+---------------+ 207 | Mode | Passport | Postcard | Hybrid | 208 +-----------+--------------+--------------+---------------+ 209 | | IOAM Trace | IOAM DEX | Multicast Te- | 210 | Technique | IOAM E2E | PBT-M | lemetry | 211 | | EAM | | HTS | 212 +-----------+--------------+--------------+---------------+ 214 Figure 1: ON-path Telemetry Technique Classification 216 IOAM Trace and E2E options are described in 217 [I-D.ietf-ippm-ioam-data]. EAM is described in 218 [I-D.zhou-ippm-enhanced-alternate-marking]. IOAM DEX option is 219 described in [I-D.ietf-ippm-ioam-direct-export] PBT-M is described in 220 [I-D.song-ippm-postcard-based-telemetry]. Multicast Telemetry is 221 described in [I-D.song-multicast-telemetry]. HTS is described in 222 [I-D.mirsky-ippm-hybrid-two-step]. 224 The advantages of the passport mode include : 226 o It naturally retains the telemetry data correlation along the 227 entire path. The self-describing feature eases the data 228 consumption. 230 o The on-path data for a packet is only exported once so the data 231 export overhead is low. 233 o Only the head and end nodes of the paths need to be configured so 234 the configuration overhead is low. 236 The disadvantages of the passport mode include : 238 o The telemetry data carried by user packets inflate the packet 239 size, which is undesirable or prohibitive. 241 o Approaches for encapsulating the instruction header and data in 242 transport protocols need to be standardized, which is challenging. 244 o Carrying sensitive data along the path is vulnerable to security 245 and privacy breach. 247 o If a packet is dropped on the path, the data collected so far are 248 also lost. 250 The postcard mode complements the passport mode by addressing the 251 issues of the passport mode. The advantages of the postcard mode 252 include: 254 o Either there is no packet header overhead (e.g., PBT-M) or the 255 overhead is small and fixed (e.g., IOAM DEX). 257 o The encapsulation requirement can be avoided (e.g., PBT-M). 259 o The telemetry data can be secured. 261 o Even if a packet is dropped on the path, the partial data 262 collected so far are still available. 264 The disadvantages of the postcard mode include: 266 o Telemetry data are spread in multiple postcards so extra effort is 267 needed to correlate the data. 269 o Every node exports a postcard for a packet which increases the 270 data export overhead. 272 o In case of PBT-M, every node on the path needs to be configured, 273 so the configuration overhead is high. 275 o In case of IOAM DEX, the encapsulation requirement remains. 277 The hybrid mode either tailors for some specific application scenario 278 (e.g., Multicast Telemetry) or provides some alternative approach 279 (e.g., HTS). 281 1.2. Requirements and Challenges 283 Although on-path telemetry is beneficial, successfully applying such 284 techniques in carrier networks must consider performance, 285 deployability, and flexibility. Specifically, we need to address the 286 following practical deployment challenges: 288 o C1: On-path telemetry incurs extra packet processing which may 289 cause stress on the network data plane. The potential impact on 290 the forwarding performance creates an unfavorable "observer 291 effect". This will not only damages the fidelity of the 292 measurement but also defies the purpose of the measurement. For 293 example, the growing IOAM data per hop can negatively affect 294 service levels by increasing the serialization delay and header 295 parsing delay. 297 o C2: On-path telemetry can generate a considerable amount of data 298 which may claim too much transport bandwidth and inundate the 299 servers for data collection, storage, and analysis. Increasing 300 the data handling capacity is technically viable but expensive. 301 For example, if IOAM is applied to all the traffic, one node may 302 collect a few tens of bytes as telemetry data for each packet. 303 The whole forwarding path might accumulate a data-trace with a 304 size similar to or even exceeding that of the original packet. 305 Transporting the telemetry data alone is projected to consume 306 almost half of the network bandwidth, plus it creates significant 307 back-end data handling and storage requirements. 309 o C3: The collectible data defined currently are essential but 310 limited. As the network operation evolves to be declarative 311 (intent-based) and automated, and the trends of network 312 virtualization, wireline and wireless convergence, and packet- 313 optical integration continue, more data is needed in an on-demand 314 and interactive fashion. Flexibility and extensibility on data 315 defining, aggregation, acquisition, and filtering, must be 316 considered. 318 o C4: Applying only a single underlying on-path telemetry technique 319 may lead to a defective result. For example, packet drop can 320 cause the loss of the flow telemetry data and the packet drop 321 location and reason remains unknown if only the In-situ OAM trace 322 option is used. A comprehensive solution needs the flexibility to 323 switch between different underlying techniques and adjust the 324 configurations and parameters at runtime. Thus, system-level 325 orchestration is needed. 327 o C5: If we were to apply some on-path telemetry technique in 328 today's carrier operator networks, we must provide solutions to 329 tailor the provider's network deployment base and support an 330 incremental deployment strategy. That is, we need to support 331 established encapsulation schemes for various predominant 332 protocols such as Ethernet, IPv4, IPv6, and MPLS with backward 333 compatibility and properly handle various transport tunnels. 335 o C6: The development of simplified on-path telemetry primitives and 336 models for configuration and queries is essential. Telemetry 337 models may be utilized via an API-based telemetry service for 338 external applications, for end-to-end performance measurement and 339 application performance monitoring. The standard-based protocols 340 and methods are needed for network configuration and programming, 341 and telemetry data processing and export, to provide 342 interoperability. 344 1.3. Scope 346 Following the network telemetry framework discussed in 347 [I-D.ietf-opsawg-ntf], this document focuses on the on-path 348 telemetry, a specific class of data-plane telemetry techniques, and 349 provides a high-level framework which addresses the aforementioned 350 challenges for deployment, especially in carrier operator networks. 352 This document aims to clarify the problem space, essential 353 requirements, and summarizes best practices and general system design 354 considerations. The framework helps to analyze the current standard 355 status and identify gaps, and to motivate new standard works to 356 complete the ecosystem. This document provides some examples to show 357 some novel network telemetry applications under the framework. 359 As an informational document, it describes an open framework with a 360 few key components. The framework does not enforces any specific 361 implementation on each component, neither does it define interfaces 362 (e.g., API, protocol) between components. The choice of underlying 363 on-path telemetry techniques and other implementation details is 364 determined by application implementer. Therefore, the framework is 365 not a solution specification. It only provides a high-level overview 366 and is not necessarily a mandatory recommendation for on-path 367 telemetry applications. Implementation of the framework is 368 implementor specific and may utilize functional components and 369 techniques outlined in this document. 371 The standardization of the underlying techniques and interfaces 372 mentioned in this document is undertaken by various working groups. 373 Due to the limited scope and intended status of this document, it has 374 no overlap or conflict with those works. 376 1.4. Glossary 378 This section defines and explains the acronyms and terms used in this 379 document. 381 On-path Telemetry: Remotely acquiring performance and behavior data 382 about network flows on a per-packet basis on the packet's 383 forwarding path. The term refers to a class of data plane 384 telemetry techniques, including IOAM, PBT, EAM, and HTS. Such 385 techniques may need to mark user packets, or insert instruction or 386 metadata to the headers of user packets. 388 IFIT: In-situ Flow Information Telemetry, pronounced as "I-Fit". 389 The name of a high-level reference framework that shows how 390 network data-plane monitoring applications can address the 391 deployment challenges of the flow-oriented on-path telemetry 392 techniques. 394 IFIT Domain: A network domain in which an on-path telemetry 395 application operates. The network domain contains multiple 396 forwarding devices, such as routers and switches, that are capable 397 of IFIT-specific functions. It also contains a logically 398 centralized controller whose responsibility is to apply IFIT- 399 specific configurations and functions to IFIT-capable forwarding 400 devices, and to collect and analyze the on-path telemetry data 401 from those devices. An IFIT domain contains multiple network node 402 capable of IFIT-specific functions. We name all the entry nodes 403 to an IFIT domain head nodes and all the exit nodes end nodes. A 404 path in an IFIT domain starts from a head node and ends at an end 405 node. Usually the instruction header encapsulation or packet 406 marking, if needed, happens at the head nodes; the instruction 407 header decapsulation or packet unmarking, if needed, happens at 408 the end nodes. 410 Reflective Telemetry: The telemetry functions in a dynamic and 411 interactive fashion. A new telemetry action is provisioned as a 412 result of self-knowledge acquired through prior telemetry actions. 414 1.5. Requirements Language 416 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 417 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 418 "OPTIONAL" in this document are to be interpreted as described in BCP 419 14 [RFC2119][RFC8174] when, and only when, they appear in all 420 capitals, as shown here. 422 2. IFIT Overview 424 To address the challenges mentioned above, we present a high-level 425 framework based on multiple network operators' requirements and 426 common industry practice, which can help to build a workable and 427 efficient on-path telemetry application. We name the framework "In- 428 situ Flow Information Telemetry" (IFIT) to reflect the fact that this 429 framework is dedicated to on-path telemetry data about user and 430 application traffic flows. As a reference framework, IFIT covers a 431 class of on-path telemetry techniques and works a level higher than 432 any specific underlying technique. The framework is comprised of 433 some key functional components (Section 3). By assembling these 434 components, IFIT supports reflective telemetry that enables 435 autonomous network operations (Section 4). 437 2.1. Typical Deployment of IFIT 439 Figure 2 shows a typical deployment scenario of IFIT. 441 Application 442 +-------------------------------------+ 443 | Controller | 444 | +------------+ +-----------+ | 445 | | Configure | | Collector | | 446 | | & |<-------| & | | 447 | | Control | | Analyzer | | 448 | +-----:------+ +-----------+ | 449 | : ^ | 450 +-------:---------------------|-------+ 451 :configuration |telemetry data 452 :& action | 453 ...............:.....................|.......... 454 : : : | : 455 : +---------:---+-------------:---++---------:---+ 456 : | : | : | : | 457 V | V | V | V | 458 +------+-+ +-----+--+ +------+-+ +------+-+ 459 packets| Head | | Path | | Path | | End | 460 ==>| Node |====>| Node |==//==>| Node |====>| Node |==> 461 | | | A | | B | | | 462 +--------+ +--------+ +--------+ +--------+ 464 |<--- IFIT Domain --->| 466 Figure 2: IFIT Deployment Scenario 468 An on-path telemetry application can conduct some network data plane 469 monitoring and measurement tasks over an IFIT domain by applying one 470 or more underlying techniques. The application needs to contains 471 multiple elements, including configuring the network nodes and 472 processing the telemetry data. The application usually runs in a 473 logically centralized controller which is responsible for configuring 474 the network nodes in the IFIT domain, and collecting and analyzing 475 telemetry data. The configuration determines which underlying 476 technique is used, what telemetry data are of interest, which flows 477 and packets are concerned with, how the telemetry data are collected, 478 etc. The process can be dynamic and interactive: after the telemetry 479 data processing and analyzing, the application may instruct the 480 controller to modify the configuration of the nodes in the IFIT 481 domain, which affects the future telemetry data collection. 483 From the system-level view, it is recommended to use the standardized 484 configuration and data collection interfaces, regardless of the 485 underlying technique. However, the specification of these interfaces 486 and the implementation of the controller are out of scope for this 487 document. 489 The IFIT domain encompasses the head nodes and the end nodes. An 490 IFIT domain may cross multiple network domains. The head nodes are 491 responsible for enabling the IFIT-specific functions and the end 492 nodes are responsible for terminating them. All capable nodes in an 493 IFIT domain will be capable of executing the instructed IFIT-specific 494 function. It is important to note that any IFIT application must, 495 through configuration and policy, guarantee that any packet with 496 IFIT-specific header and metadata will not leak out of the IFIT 497 domain. The end nodes must be able to capture all packets with IFIT- 498 specific header and metadata and recover their format before 499 forwarding them out of the IFIT domain. 501 The underlying on-path telemetry techniques covered by IFIT can be of 502 any modes discussed in Section 1.1. 504 2.2. IFIT Architecture 506 The IFIT architecture is shown in Figure 3, which contains several 507 key components. These components aim to address the deployment 508 challenges discussed in Section 1. The detailed block diagram and 509 description for each component are given in Section 3. Here we only 510 provide a high level overview. 512 +------------------------------------+ 513 | On-demand Technique | 514 | Selection & Integration | 515 +------------------------------------+ 516 Control Plane | ^ 517 ---------------------+-------------------+------------- 518 Forwarding Plane V | 519 +-----------------+------------------+ 520 | Flexible Flow, | Flexible | 521 | Packet, & Data | Data Export | 522 | Selection | | 523 +-----------------+------------------| 524 | Dynamic Network Probe | 525 +------------------------------------| 527 Figure 3: IFIT Architecture 529 Based on the monitoring and measurement requirements, an application 530 needs to choose one or more underlying on-path telemetry techniques 531 and decide the policies to apply them. Depending on the forwarding- 532 plane protocol and tunneling configuration, the instruction header 533 and metadata encapsulation method, if needed, is also determined. 534 The encapsulation happens at the head nodes and the decapsulation 535 happens at the end nodes. 537 Based on the network condition and application requirement, the head 538 nodes also need to be able to choose flows and packets to enable the 539 IFIT-specific functions, and decide the set of data to be collected. 540 All the nodes who are responsible for exporting telemetry data are 541 configured with special functions to prepare the data. The IFIT- 542 specific functions can be deployed into the network nodes as dynamic 543 network probes. 545 2.3. Relationship with Network Telemetry Framework (NTF) 547 [I-D.ietf-opsawg-ntf] describes a Network Telemetry Framework (NTF). 548 One dimension used by NTF to partition network telemetry techniques 549 and systems is based on the three planes in networks plus external 550 data sources. IFIT fits in the category of forwarding-plane 551 telemetry and deals with the specific on-path technical branch of the 552 forwarding-plane telemetry. 554 According to NTF, an on-path telemetry application mainly subscribes 555 event-triggered or streaming data. The key functional components of 556 IFIT also match the components in NTF. "On-demand Technique 557 Selection and Integration" is an application layer function, matching 558 the "Data Query, Analysis, and Storage" component in NTF; "Flexible 559 Flow, Packet, and Data Selection" matches the "Data Configuration and 560 Subscription" component; "Flexible Data Export" matches the "Data 561 Encoding and Export" component; "Dynamic Network Probe" matches the 562 "Data Generation and Processing" component. 564 3. Key Components of IFIT 566 As shown in the IFIT architecture, the key components of IFIT are as 567 follows: 569 o Flexible flow, packet, and data selection policy, addressing the 570 challenge C1 described in Section 1; 572 o Flexible data export, addressing the challenge C2; 574 o Dynamic network probe, addressing C3; 576 o On-demand technique selection and integration, addressing C4. 578 Note that the challenges C5 and C6 are mostly standard related, which 579 are fundamental to IFIT. We discuss the standard status and gaps in 580 Section 5. 582 In the following section, we provide a detailed description of each 583 component. 585 3.1. Flexible Flow, Packet, and Data Selection 587 In most cases, it is impractical to enable the data collection for 588 all the flows and for all the packets in a flow due to the potential 589 performance and bandwidth impact. Therefore, a workable solution 590 usually need to select only a subset of flows and flow packets to 591 enable the data collection, even though this means the loss of some 592 information and accuracy. 594 In the data plane, the Access Control List (ACL) provides an ideal 595 means to determine the subset of flow(s). An application can set a 596 sample rate or probability to a flow to allow only a subset of flow 597 packets to be monitored, collect a different set of data for 598 different packets, and disable or enable data collection on any 599 specific network node. An application can further allow any node to 600 accept or deny the data collection process in full or partially. 602 Based on these flexible mechanisms, IFIT allows applications to apply 603 flexible flow and data selection policies to suit the requirements. 604 The applications can dynamically change the policies at any time 605 based on the network load, processing capability, focus of interest, 606 and any other criteria. 608 3.1.1. Block Diagram 610 +----------------------------+ 611 | +----------+ +----------+ | 612 | |Flow | |Data | | 613 | |Selection | |Selection | | 614 | +----------+ +----------+ | 615 | +----------+ | 616 | |Packet | | 617 | |Selection | | 618 | +----------+ | 619 +----------------------------+ 621 Figure 4: Flexible Flow, Packet, and Data Selection 623 Figure 4 shows the block diagram of this component. The flow 624 selection block defines the policies to choose target flows for 625 monitoring. Flow has different granularity. A basic flow is defined 626 by 5-tuple IP header fields. Flow can also be aggregated at 627 interface level, tunnel level, protocol level, and so on. The packet 628 selection block defines the policies to choose packets from a target 629 flow. The policy can be either a sampling interval, a sampling 630 probability, or some specific packet signature. The data selection 631 block defines the set of data to be collected. This can be changed 632 on a per-packet or per-flow basis. 634 3.1.2. Example: Sketch-guided Elephant Flow Selection 636 Network operators are usually more interested in elephant flows which 637 consume more resource and are sensitive to changes in network 638 conditions. A CountMin Sketch [CMSketch] can be used on the data 639 path of the head nodes, which identifies and reports the elephant 640 flows periodically. The controller maintains a current set of 641 elephant flows and dynamically enables the on-path telemetry for only 642 these flows. 644 3.1.3. Example: Adaptive Packet Sampling 646 Applying on-path telemetry on all packets of selected flows can still 647 be out of reach. A sample rate should be set for these flows and 648 only enable telemetry on the sampled packets. However, the head 649 nodes have no clue on the proper sampling rate. An overly high rate 650 would exhaust the network resource and even cause packet drops; An 651 overly low rate, on the contrary, would result in the loss of 652 information and inaccuracy of measurements. 654 An adaptive approach can be used based on the network conditions to 655 dynamically adjust the sampling rate. Every node gives user traffic 656 forwarding higher priority than telemetry data export. In case of 657 network congestion, the telemetry can sense some signals from the 658 data collected (e.g., deep buffer size, long delay, packet drop, and 659 data loss). The controller may use these signals to adjust the 660 packet sampling rate. In each adjustment period (i.e., RTT of the 661 feedback loop), the sampling rate is either decreased or increased in 662 response of the signals. An AIMD policy similar to the TCP flow 663 control mechanism for the rate adjustment can be used. 665 3.2. Flexible Data Export 667 The flow telemetry data can catch the dynamics of the network and the 668 interactions between user traffic and network. Nevertheless, the 669 data inevitably contain redundancy. It is advisable to remove the 670 redundancy from the data in order to reduce the data transport 671 bandwidth and server processing load. 673 In addition to efficient export data encoding (e.g., IPFIX [RFC7011] 674 or protobuf [1]), nodes have several other ways to reduce the export 675 data by taking advantage of network device's capability and 676 programmability. Nodes can cache the data and send the accumulated 677 data in batch if the data is not time sensitive. Various 678 deduplication and compression techniques can be applied on the batch 679 data. 681 From the application perspective, an application may only be 682 interested in some special events which can be derived from the 683 telemetry data. For example, in case that the forwarding delay of a 684 packet exceeds a threshold, or a flow changes its forwarding path is 685 of interest, it is unnecessary to send the original raw data to the 686 data collecting and processing servers. Rather, IFIT takes advantage 687 of the in-network computing capability of network devices to process 688 the raw data and only push the event notifications to the subscribing 689 applications. 691 Such events can be expressed as policies. An policy can request data 692 export only on change, on exception, on timeout, or on threshold. 694 3.2.1. Block Diagram 695 +-------------------------------------------+ 696 | +-----------+ +-----------+ +-----------+ | 697 | |Data | |Data | |Export | | 698 | |Encoding | |Batching | |Protocol | | 699 | +-----------+ +-----------+ +-----------+ | 700 | +-----------+ +-----------+ +-----------+ | 701 | |Data | |Data | |Data | | 702 | |Compression| |Dedup. | |Filter | | 703 | +-----------+ +-----------+ +-----------+ | 704 | +-----------+ +-----------+ | 705 | |Data | |Data | | 706 | |Computing | |Aggregation| | 707 | +-----------+ +-----------+ | 708 +-------------------------------------------+ 710 Figure 5: Flexible Data Export 712 Figure 5 shows the block diagram of this component. The data 713 encoding block defines the method to encode the telemetry data. The 714 data batching block defines the size of batch data buffered at the 715 device side before export. The export protocol block defines the 716 protocol used for telemetry data export. The data compression block 717 defines the algorithm to compress the raw data. The data 718 deduplication block defines the algorithm to remove the redundancy in 719 the raw data. The data filter block defines the policies to filter 720 the needed data. The data computing block defines the policies to 721 prepocess the raw data and generate some new data. The data 722 aggregation block defines the procedure to combine and synthesize the 723 data. 725 3.2.2. Example: Event-based Anomaly Monitor 727 Network operators are interested in the anomalies such as path 728 change, network congestion, and packet drop. Such anomalies are 729 hidden in raw telemetry data (e.g., path trace, timestamp). Such 730 anomalies can be described as events and programmed into the device 731 data plane. Only the triggered events are exported. For example, if 732 a new flow appears at any node, a path change event is triggered; if 733 the packet delay exceeds a predefined threshold in a node, the 734 congestion event is triggered; if a packet is dropped due to buffer 735 overflow, a packet drop event is triggered. 737 The export data reduction due to such optimization is substantial. 738 For example, given a single 5-hop 10Gbps path, assume a moderate 739 number of 1 million packets per second are monitored, and the 740 telemetry data plus the export packet overhead consume less than 30 741 bytes per hop. Without such optimization, the bandwidth consumed by 742 the telemetry data can easily exceed 1Gbps (more than 10% of the path 743 bandwidth), When the optimization is used, the bandwidth consumed by 744 the telemetry data is negligible. Moreover, the pre-processed 745 telemetry data greatly simplify the work of data analyzers. 747 3.3. Dynamic Network Probe 749 Due to limited data plane resource and network bandwidth, it is 750 unlikely one can monitor all the data all the time. On the other 751 hand, the data needed by applications may be arbitrary but ephemeral. 752 It is critical to meet the dynamic data requirements with limited 753 resource. 755 Fortunately, data plane programmability allows IFIT to dynamically 756 load new data probes. These on-demand probes are called Dynamic 757 Network Probes (DNP). DNP is the technique to enable probes for 758 customized data collection in different network planes. When working 759 with IOAM or PBT, DNP is loaded to the data plane through incremental 760 programming or configuration. The DNP can effectively conduct data 761 generation, processing, and aggregation. 763 DNP introduces enough flexibility and extensibility to IFIT. It can 764 implement the optimizations for export data reduction motioned in the 765 previous section. It can also generate custom data as required by 766 today and tomorrow's applications. 768 3.3.1. Block Diagram 770 +----------------------------+ 771 | +----------+ +----------+ | 772 | |ACL | |YANG | | 773 | | | |Model | | 774 | +----------+ +----------+ | 775 | +----------+ +----------+ | 776 | |Hardware | |Software | | 777 | |Function | |Function | | 778 | +----------+ +----------+ | 779 +----------------------------+ 781 Figure 6: Dynamic Network Probes 783 Figure 6 shows the block diagram of this component. The Access 784 Control List (ACL) block is available in most hardware and it defines 785 DNPs through dynamically update the ACL policies (including flow 786 filtering and action). YANG models can be dynamically deployed to 787 enable different data processing and filtering functions. Some 788 hardware allows dynamically loading hardware-based functions into the 789 forwarding path at runtime through mechanisms such as reserved 790 pipelines and function stubs. Dynamically loadable software 791 functions can be implemented in the control processors in IFIT nodes. 793 3.3.2. Examples 795 Following are some possible DNPs that can be dynamically deployed to 796 support applications. 798 On-demand Flow Sketch: A flow sketch is a compact online data 799 structure (usually a variation of multi-hashing table) for 800 approximate estimation of multiple flow properties. It can be 801 used to facilitate flow selection. The aforementioned CountMin 802 Sketch [CMSketch] is such an example. Since a sketch consumes 803 data plane resources, it should only be deployed when actually 804 needed. 806 Smart Flow Filter: The policies that choose flows and packet 807 sampling rate can change during the lifetime of an application. 809 Smart Statistics: An application may need to count flows based on 810 different flow granularity or maintain hit counters for selected 811 flow table entries. 813 Smart Data Reduction: DNP can be used to program the events that 814 conditionally trigger data export. 816 3.4. On-demand Technique Selection and Integration 818 With multiple underlying data collection and export techniques at its 819 disposal, IFIT can flexibly adapt to different network conditions and 820 different application requirements. 822 For example, depending on the types of data that are of interest, 823 IFIT may choose either IOAM or PBT to collect the data; if an 824 application needs to track down where the packets are lost, switching 825 from IOAM to PBT should be supported. 827 IFIT can further integrate multiple data plane monitoring and 828 measurement techniques together and present a comprehensive data 829 plane telemetry solution. 831 Based on the application requirements and the real-time telemetry 832 data analysis results, new configurations and actions can be 833 deployed. 835 3.4.1. Block Diagram 837 +----------------------------------------------+ 838 | +------------+ +-------------+ +---------+ | 839 | |Application | |Configuration| |Telemetry| | 840 | |Requirements|->|& Action |<-|Data | | 841 | | | | | |Analysis | | 842 | +------------+ +-------------+ +---------+ | 843 +----------------------------------------------+ 844 | Passport Mode: | 845 | +----------+ +----------+ +----------+ | 846 | |IOAM E2E | |IOAM Trace| |EAM | | 847 | +----------+ +----------+ +----------+ | 848 | Postcard Mode: | 849 | +----------+ +----------+ | 850 | |PBT-M | |IOAM DEX | | 851 | +----------+ +----------+ | 852 | Hybrid Mode: | 853 | +----------+ +----------+ | 854 | |HTS | |Multicast | | 855 | | | |Telemetry | | 856 | +----------+ +----------+ | 857 +----------------------------------------------+ 859 Figure 7: Technique Selection and Integration 861 Figure 7 shows the block diagram of this component, which lists the 862 candidate on-path telemetry techniques. 864 Located in the logically centralized controller of an IFIT domain, 865 this component makes all the control and configuration dynamically to 866 the capable nodes in the domain which will affect the future 867 telemetry data. The configuration and action decisions are based on 868 the inputs from the application requirements and the realtime 869 telemetry data analysis results. Note that here the telemetry data 870 source is not limited to the data plane. The data can come form all 871 the sources mentioned in [I-D.ietf-opsawg-ntf], including external 872 data sources. 874 4. IFIT for Reflective Telemetry 876 The IFIT components can work together to support reflective 877 telemetry, as shown in Figure 8. 879 +---------------------+ 880 | | 881 +------+ Applications |<------+ 882 | | | | 883 | +---------------------+ | 884 | Technique Selection | 885 | and Integration | 886 | | 887 |Flexible Flexible | 888 |Flow, reflection-loop Data | 889 |Packet, Export| 890 |and Data | 891 |Selection +----+----+ 892 V +---------+| 893 +----------+ Encapsulation +---------+|| 894 | Head | and Tunneling | Path ||| 895 | Node |----------------------->| Nodes ||+ 896 | | | |+ 897 +----------+ +---------+ 898 DNP DNP 900 Figure 8: IFIT-based Reflective Telemetry 902 An application may pick a suite of telemetry techniques based on its 903 requirements and apply an initial technique to the data plane. It 904 then configures the head nodes to decide the initial target flows/ 905 packets and telemetry data set, the encapsulation and tunneling 906 scheme based on the underlying network architecture, and the IFIT- 907 capable nodes to decide the initial telemetry data export policy. 908 Based on the network condition and the analysis results of the 909 telemetry data, the application can change the telemetry technique, 910 the flow/data selection policy, and the data export approach in real 911 time without breaking the normal network operation. Many of such 912 dynamic changes can be done through loading and unloading DNPs. 914 The reflective telemetry enabled by the IFIT allows numerous new 915 applications suitable for future network operation architecture. 917 4.1. Example: Intelligent Multipoint Performance Monitoring 919 [I-D.ietf-ippm-multipoint-alt-mark] describes an intelligent 920 performance management based on the network condition. The idea is 921 to split the monitoring network into clusters. The cluster partition 922 that can be applied to every type of network graph and the 923 possibility to combine clusters at different levels enable the so- 924 called Network Zooming. It allows a controller to calibrate the 925 network telemetry, so that it can start without examining in depth 926 and monitor the network as a whole. In case of necessity (packet 927 loss or too high delay), an immediate detailed analysis can be 928 reconfigured. In particular, the controller, that is aware of the 929 network topology, can set up the most suited cluster partition by 930 changing the traffic filter or activate new measurement points and 931 the problem can be localized with a step-by-step process. 933 An application on top of the controllers can manage such mechanism 934 and IFIT's architecture allows its dynamic and reflective operation. 936 4.2. Example: Intent-based Network Monitoring 938 User Intents 939 | 940 V Per-packet 941 +------------+ Telemetry 942 ACL | | Data 943 +--------+ Controller |<--------+ 944 | | | | 945 | +--+---------+ | 946 | | ^ | 947 | |DNPs |Network | 948 | | |Information| 949 | V | | 950 +------+-------------------+-----------+---+ 951 | | | 952 | V +------+ | 953 | +-------+ +------+| | 954 | | Head | IFIT Domain +------+|| | 955 | | Node | |Path ||+ | 956 | | | |Nodes |+ | 957 | +-------+ +------+ | 958 +------------------------------------------+ 960 Figure 9: Intent-based Monitoring 962 In this example, a user can express high level intents for network 963 monitoring. The controller translates an intent and configure the 964 corresponding DNPs in IFIT-capable nodes which collect necessary 965 network information. Based on the real-time information feedback, 966 the controller runs a local algorithm to determine the suspicious 967 flows. It then deploys ACLs to the head node to initiate the high 968 precision per-packet on-path telemetry for these flows. 970 5. Standard Status and Gaps 972 A complete IFIT-based solution needs standard interfaces for 973 configuration and data extraction, and standard encapsulation on 974 various transport protocols. It may also need standard API and 975 primitives for application programming and deployment. The draft 976 [I-D.brockners-opsawg-ioam-deployment] summarizes some current 977 proposals on encapsulation and data export for IOAM. These works 978 should be extended or modified to support other types of on-path 979 telemetry techniques and other transport protocols. The high-level 980 IFIT helps to develop coherent and universal standard encapsulation 981 and data export approaches. 983 5.1. Encapsulation in Transport Protocols 985 Since the introduction of IOAM, the IOAM option header encapsulation 986 schemes in various network protocols have been proposed. Similar 987 encapsulation schemes need to be extended to cover the other on-path 988 telemetry techniques. On the other hand, the encapsulation scheme 989 for some popular protocols, such as MPLS and IPv4, are noticeably 990 missing. It is important to provide the encapsulation schemes for 991 these protocols because they are still prevalent in carrier networks. 992 IFIT needs to provide solutions to apply the on-path flow telemetry 993 techniques in such networks. PBT-M 994 [I-D.song-ippm-postcard-based-telemetry] does not introduce new 995 headers to the packets so the trouble of encapsulation for a new 996 header is avoided. While there are some proposals which allow new 997 header encapsulation in MPLS packets (e.g., 998 [I-D.song-mpls-extension-header]) or in IPv4 packets (e.g., 999 [I-D.herbert-ipv4-eh]), they are still in their infancy stage and 1000 require significant future work. For the meantime, in a confined 1001 IFIT domain, pre-standard encapsulation approaches may be applied. 1003 5.2. Tunneling Support 1005 In carrier networks, it is common for user traffic to traverse 1006 various tunnels for QoS, traffic engineering, or security. IFIT 1007 supports both the uniform mode and the pipe mode for tunnel support 1008 as described in [I-D.song-ippm-ioam-tunnel-mode]. With such 1009 flexibility, the operator can either gain a true end-to-end 1010 visibility or apply a hierarchical approach which isolates the 1011 monitoring domain between customer and provider. 1013 5.3. Deployment Automation 1015 In addition, standard approaches that automates the function 1016 configuration, and capability query and advertisement, either in a 1017 centralized fashion or a distributed fashion, are still immature. 1019 The draft [I-D.zhou-ippm-ioam-yang] provides the YANG model for IOAM 1020 configuration. Similar models needs to be defined for other 1021 techniques. It is also helpful to provide standards-based approaches 1022 for configuration in various network environments. For example, in 1023 segment routing networks, extensions to BGP or PCEP can be defined to 1024 distribute SR policies carrying IFIT information, so that IFIT 1025 behavior can be enabled automatically when the SR policy is applied. 1026 [I-D.chen-pce-sr-policy-ifit] proposes to extend PCEP policy for IFIT 1027 configuration in segment routing networks. 1028 [I-D.qin-idr-sr-policy-ifit] proposes to extend BGP policy instead 1029 for IFIT configuration in segment routing networks. Additional 1030 capability discovery and dissemination will be needed for other types 1031 of networks. 1033 To realize the potential of IFIT, programming and deploying DNPs are 1034 important. ForCES [RFC5810] is a standard protocol for network 1035 device programming, which can be used for DNP deployment. Currently 1036 some related works such as [I-D.wwx-netmod-event-yang] and 1037 [I-D.bwd-netmod-eca-framework] have proposed to use YANG model to 1038 define the smart policies which can be used to implement DNPs. In 1039 the future, other approaches for hardware and software-based 1040 functions can be development to enhance the programmability and 1041 flexibility. 1043 6. Summary 1045 IFIT is a high-level framework for applying on-path telemetry 1046 techniques, and this document has outlined how the framework may be 1047 used to solve essential use cases. IFIT enables a practical data- 1048 plane telemetry application based on two basic on-path traffic data 1049 collection modes: passport and postcard. 1051 IFIT addresses the key challenges for operators to deploy a complete 1052 on-path telemetry solution. However, as a reference and open 1053 framework, IFIT only describes the basic functions of each identified 1054 component and suggests possible applications. It has no intention of 1055 specifying the implementation of the components and the interfaces 1056 between the components. The compliance of IFIT is by no means 1057 mandatory either. Instead, this informational document aims to 1058 clarify the problem domain, and summarize the best practices and 1059 sensible system design considerations. IFIT can guide the analysis 1060 of the current standard status and gaps, and motivate new works to 1061 complete the ecosystem. IFIT enables data-plane reflective telemetry 1062 applications for advanced network operations. 1064 Having a high-level framework covering a class of related techniques 1065 also promotes a holistic approach for standard development and helps 1066 to avoid duplicated efforts and piecemeal solutions that only focus 1067 on a specific technique while omitting the compatibility and 1068 extensibility issues, which is important to a healthy ecosystem for 1069 network telemetry. 1071 7. Security Considerations 1073 In addition to the specific security issues discussed in each 1074 individual document on on-path telemetry, this document considers the 1075 overall security issues at the IFIT system level. This should serve 1076 as a guide to the on-path telemetry application developers and users. 1078 8. IANA Considerations 1080 This document includes no request to IANA. 1082 9. Contributors 1084 Other major contributors of this document include Giuseppe Fioccola, 1085 Daniel King, Zhenqiang Li, Zhenbin Li, Tianran Zhou, and James 1086 Guichard. 1088 10. Acknowledgments 1090 We thank Diego Lopez, Shwetha Bhandari, Joe Clarke, Adrian Farrel, 1091 Frank Brockners, Al Morton, Alex Clemm, Alan DeKok, and Warren Kumari 1092 for their constructive suggestions for improving this document. 1094 11. References 1096 11.1. Normative References 1098 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1099 Requirement Levels", BCP 14, RFC 2119, 1100 DOI 10.17487/RFC2119, March 1997, 1101 . 1103 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1104 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1105 May 2016, . 1107 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1108 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1109 May 2017, . 1111 11.2. Informative References 1113 [CMSketch] 1114 Cormode, G. and S. Muthukrishnan, "An improved data stream 1115 summary: the count-min sketch and its applications", 2005, 1116 . 1118 [I-D.brockners-opsawg-ioam-deployment] 1119 Brockners, F., Bhandari, S., and d. 1120 daniel.bernier@bell.ca, "In-situ OAM Deployment", draft- 1121 brockners-opsawg-ioam-deployment-02 (work in progress), 1122 September 2020. 1124 [I-D.bwd-netmod-eca-framework] 1125 Boucadair, M., WU, Q., Wang, Z., King, D., and C. Xie, 1126 "Framework for Use of ECA (Event Condition Action) in 1127 Network Self Management", draft-bwd-netmod-eca- 1128 framework-00 (work in progress), November 2019. 1130 [I-D.chen-pce-sr-policy-ifit] 1131 Chen, H., Yuan, H., Zhou, T., Li, W., Fioccola, G., and Y. 1132 Wang, "PCEP SR Policy Extensions to Enable IFIT", draft- 1133 chen-pce-sr-policy-ifit-02 (work in progress), July 2020. 1135 [I-D.herbert-ipv4-eh] 1136 Herbert, T., "IPv4 Extension Headers and Flow Label", 1137 draft-herbert-ipv4-eh-01 (work in progress), May 2019. 1139 [I-D.ietf-ippm-ioam-data] 1140 Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields 1141 for In-situ OAM", draft-ietf-ippm-ioam-data-11 (work in 1142 progress), November 2020. 1144 [I-D.ietf-ippm-ioam-direct-export] 1145 Song, H., Gafni, B., Zhou, T., Li, Z., Brockners, F., 1146 Bhandari, S., Sivakolundu, R., and T. Mizrahi, "In-situ 1147 OAM Direct Exporting", draft-ietf-ippm-ioam-direct- 1148 export-02 (work in progress), November 2020. 1150 [I-D.ietf-ippm-multipoint-alt-mark] 1151 Fioccola, G., Cociglio, M., Sapio, A., and R. Sisto, 1152 "Multipoint Alternate Marking method for passive and 1153 hybrid performance monitoring", draft-ietf-ippm- 1154 multipoint-alt-mark-09 (work in progress), March 2020. 1156 [I-D.ietf-opsawg-ntf] 1157 Song, H., Qin, F., Martinez-Julia, P., Ciavaglia, L., and 1158 A. Wang, "Network Telemetry Framework", draft-ietf-opsawg- 1159 ntf-06 (work in progress), January 2021. 1161 [I-D.mirsky-ippm-hybrid-two-step] 1162 Mirsky, G., Lingqiang, W., Zhui, G., and H. Song, "Hybrid 1163 Two-Step Performance Measurement Method", draft-mirsky- 1164 ippm-hybrid-two-step-07 (work in progress), December 2020. 1166 [I-D.qin-idr-sr-policy-ifit] 1167 Qin, F., Yuan, H., Zhou, T., Fioccola, G., and Y. Wang, 1168 "BGP SR Policy Extensions to Enable IFIT", draft-qin-idr- 1169 sr-policy-ifit-04 (work in progress), October 2020. 1171 [I-D.song-ippm-ioam-tunnel-mode] 1172 Song, H., Li, Z., Zhou, T., and Z. Wang, "In-situ OAM 1173 Processing in Tunnels", draft-song-ippm-ioam-tunnel- 1174 mode-00 (work in progress), June 2018. 1176 [I-D.song-ippm-postcard-based-telemetry] 1177 Song, H., Zhou, T., Li, Z., Mirsky, G., Shin, J., and K. 1178 Lee, "Postcard-based On-Path Flow Data Telemetry using 1179 Packet Marking", draft-song-ippm-postcard-based- 1180 telemetry-08 (work in progress), October 2020. 1182 [I-D.song-mpls-extension-header] 1183 Song, H., Li, Z., Zhou, T., and L. Andersson, "MPLS 1184 Extension Header", draft-song-mpls-extension-header-02 1185 (work in progress), February 2019. 1187 [I-D.song-multicast-telemetry] 1188 Song, H., McBride, M., Mirsky, G., and G. Mishra, 1189 "Multicast On-path Telemetry Solutions", draft-song- 1190 multicast-telemetry-07 (work in progress), January 2021. 1192 [I-D.wwx-netmod-event-yang] 1193 WU, Q., Bryskin, I., Birkholz, H., Liu, X., and B. Claise, 1194 "A YANG Data model for ECA Policy Management", draft-wwx- 1195 netmod-event-yang-10 (work in progress), November 2020. 1197 [I-D.zhou-ippm-enhanced-alternate-marking] 1198 Zhou, T., Fioccola, G., Lee, S., Cociglio, M., and W. Li, 1199 "Enhanced Alternate Marking Method", draft-zhou-ippm- 1200 enhanced-alternate-marking-06 (work in progress), January 1201 2021. 1203 [I-D.zhou-ippm-ioam-yang] 1204 Zhou, T., Guichard, J., Brockners, F., and S. Raghavan, "A 1205 YANG Data Model for In-Situ OAM", draft-zhou-ippm-ioam- 1206 yang-08 (work in progress), July 2020. 1208 [passport-postcard] 1209 Handigol, N., Heller, B., Jeyakumar, V., Mazieres, D., and 1210 N. McKeown, "Where is the debugger for my software-defined 1211 network?", 2012, 1212 . 1214 [RFC5810] Doria, A., Ed., Hadi Salim, J., Ed., Haas, R., Ed., 1215 Khosravi, H., Ed., Wang, W., Ed., Dong, L., Gopal, R., and 1216 J. Halpern, "Forwarding and Control Element Separation 1217 (ForCES) Protocol Specification", RFC 5810, 1218 DOI 10.17487/RFC5810, March 2010, 1219 . 1221 [RFC7011] Claise, B., Ed., Trammell, B., Ed., and P. Aitken, 1222 "Specification of the IP Flow Information Export (IPFIX) 1223 Protocol for the Exchange of Flow Information", STD 77, 1224 RFC 7011, DOI 10.17487/RFC7011, September 2013, 1225 . 1227 11.3. URIs 1229 [1] https://developers.google.com/protocol-buffers/ 1231 Authors' Addresses 1233 Haoyu Song 1234 Futurewei 1235 2330 Central Expressway 1236 Santa Clara 1237 USA 1239 Email: haoyu.song@futurewei.com 1241 Fengwei Qin 1242 China Mobile 1243 No. 32 Xuanwumenxi Ave., Xicheng District 1244 Beijing, 100032 1245 P.R. China 1247 Email: qinfengwei@chinamobile.com 1248 Huanan Chen 1249 China Telecom 1250 P. R. China 1252 Email: chenhuan6@chinatelecom.cn 1254 Jaehwan Jin 1255 LG U+ 1256 South Korea 1258 Email: daenamu1@lguplus.co.kr 1260 Jongyoon Shin 1261 SK Telecom 1262 South Korea 1264 Email: jongyoon.shin@sk.com