idnits 2.17.1 draft-song-opsawg-ifit-framework-11.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 39 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 822 has weird spacing: '...nd Data refl...' == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (March 9, 2020) is 1507 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '1' on line 1177 == Unused Reference: 'RFC2113' is defined on line 1158, but no explicit reference was found in the text == Outdated reference: A later version (-03) exists of draft-brockners-opsawg-ioam-deployment-00 == Outdated reference: A later version (-02) exists of draft-chen-pce-sr-policy-ifit-00 == Outdated reference: A later version (-03) exists of draft-herbert-ipv4-eh-01 == Outdated reference: A later version (-17) exists of draft-ietf-ippm-ioam-data-08 == Outdated reference: A later version (-09) exists of draft-ietf-ippm-multipoint-alt-mark-06 == Outdated reference: A later version (-13) exists of draft-ietf-opsawg-ntf-02 == Outdated reference: A later version (-15) exists of draft-mirsky-ippm-hybrid-two-step-04 == Outdated reference: A later version (-04) exists of draft-qin-idr-sr-policy-ifit-00 == Outdated reference: A later version (-16) exists of draft-song-ippm-postcard-based-telemetry-06 == Outdated reference: A later version (-13) exists of draft-song-mpls-extension-header-02 == Outdated reference: A later version (-07) exists of draft-song-multicast-telemetry-02 == Outdated reference: A later version (-10) exists of draft-wwx-netmod-event-yang-06 == Outdated reference: A later version (-14) exists of draft-zhou-ippm-enhanced-alternate-marking-04 == Outdated reference: A later version (-08) exists of draft-zhou-ippm-ioam-yang-06 Summary: 1 error (**), 0 flaws (~~), 18 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 OPSAWG H. Song, Ed. 3 Internet-Draft Futurewei 4 Intended status: Informational F. Qin 5 Expires: September 10, 2020 China Mobile 6 H. Chen 7 China Telecom 8 J. Jin 9 LG U+ 10 J. Shin 11 SK Telecom 12 March 9, 2020 14 In-situ Flow Information Telemetry 15 draft-song-opsawg-ifit-framework-11 17 Abstract 19 As networks increase in scale and network operations become more 20 sophisticated, traditional Operation, Administration and Maintenance 21 (OAM) methods, which include proactive and reactive techniques, 22 running in active and passive modes, are no longer sufficient to meet 23 the monitoring and measurement requirements. Emerging on-path 24 telemetry techniques which provide high-precision flow insight and 25 real-time issue notification are required to ensure suitable quality 26 of experience for users and applications, and identify faults or 27 network deficiencies before they become critical. 29 This document outlines a high-level framework to provide an 30 operational environment that utilizes existing and emerging on-path 31 telemetry techniques to enable the collection and correlation of 32 performance information from the network. The framework identifies 33 the components that are needed to coordinate the existing protocol 34 tools and telemetry mechanisms, and addresses key deployment 35 challenges for flow-oriented on-path telemetry techniques, especially 36 in carrier networks. 38 The framework is informational and intended to guide system designers 39 attempting to use the referenced techniques as well as to motivate 40 further work to enhance the ecosystem . 42 Status of This Memo 44 This Internet-Draft is submitted in full conformance with the 45 provisions of BCP 78 and BCP 79. 47 Internet-Drafts are working documents of the Internet Engineering 48 Task Force (IETF). Note that other groups may also distribute 49 working documents as Internet-Drafts. The list of current Internet- 50 Drafts is at https://datatracker.ietf.org/drafts/current/. 52 Internet-Drafts are draft documents valid for a maximum of six months 53 and may be updated, replaced, or obsoleted by other documents at any 54 time. It is inappropriate to use Internet-Drafts as reference 55 material or to cite them other than as "work in progress." 57 This Internet-Draft will expire on September 10, 2020. 59 Copyright Notice 61 Copyright (c) 2020 IETF Trust and the persons identified as the 62 document authors. All rights reserved. 64 This document is subject to BCP 78 and the IETF Trust's Legal 65 Provisions Relating to IETF Documents 66 (https://trustee.ietf.org/license-info) in effect on the date of 67 publication of this document. Please review these documents 68 carefully, as they describe your rights and restrictions with respect 69 to this document. Code Components extracted from this document must 70 include Simplified BSD License text as described in Section 4.e of 71 the Trust Legal Provisions and are provided without warranty as 72 described in the Simplified BSD License. 74 Table of Contents 76 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 77 1.1. Requirements and Challenges . . . . . . . . . . . . . . . 4 78 1.2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 6 79 1.3. Glossary . . . . . . . . . . . . . . . . . . . . . . . . 6 80 1.4. Requirements Language . . . . . . . . . . . . . . . . . . 7 81 2. IFIT Overview . . . . . . . . . . . . . . . . . . . . . . . . 7 82 2.1. Typical Deployment of IFIT . . . . . . . . . . . . . . . 8 83 2.1.1. On-path Telemetry Models: Passport vs. Postcard . . . 9 84 2.2. IFIT Architecture . . . . . . . . . . . . . . . . . . . . 10 85 2.3. Relationship with Network Telemetry Framework (NTF) . . . 10 86 3. Key Components of IFIT . . . . . . . . . . . . . . . . . . . 11 87 3.1. Smart Flow, Packet, and Data Selection . . . . . . . . . 11 88 3.1.1. Block Diagram . . . . . . . . . . . . . . . . . . . . 12 89 3.1.2. Example: Sketch-guided Elephant Flow Selection . . . 12 90 3.1.3. Example: Adaptive Packet Sampling . . . . . . . . . . 13 91 3.2. Smart Data Export . . . . . . . . . . . . . . . . . . . . 13 92 3.2.1. Block Diagram . . . . . . . . . . . . . . . . . . . . 14 93 3.2.2. Example: Event-based Anomaly Monitor . . . . . . . . 14 94 3.3. Dynamic Network Probe . . . . . . . . . . . . . . . . . . 15 95 3.3.1. Block Diagram . . . . . . . . . . . . . . . . . . . . 15 96 3.3.2. Examples . . . . . . . . . . . . . . . . . . . . . . 16 98 3.4. On-demand Technique Selection and Integration . . . . . . 16 99 3.4.1. Block Diagram . . . . . . . . . . . . . . . . . . . . 17 100 4. IFIT for Reflective Telemetry . . . . . . . . . . . . . . . . 18 101 4.1. Example: Intelligent Multipoint Performance Monitoring . 19 102 4.2. Example: Intent-based Network Monitoring . . . . . . . . 19 103 5. Standard Status and Gaps . . . . . . . . . . . . . . . . . . 20 104 5.1. Encapsulation in Transport Protocols . . . . . . . . . . 20 105 5.2. Tunneling Support . . . . . . . . . . . . . . . . . . . . 20 106 5.3. Deployment Automation . . . . . . . . . . . . . . . . . . 21 107 6. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 108 7. Security Considerations . . . . . . . . . . . . . . . . . . . 22 109 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 110 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 22 111 10. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 22 112 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 113 11.1. Normative References . . . . . . . . . . . . . . . . . . 22 114 11.2. Informative References . . . . . . . . . . . . . . . . . 23 115 11.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 25 116 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 25 118 1. Introduction 120 Efficient network operation increasingly relies on high-quality data- 121 plane telemetry to provide the necessary visibility. Traditional 122 Operation, Administration and Maintenance (OAM) methods, which 123 include proactive and reactive techniques, running both active and 124 passive modes, are no longer sufficient to meet the monitoring and 125 measurement requirements. The complexity of today's networks and 126 service quality requirements demand new high-precision and real-time 127 techniques. 129 The ability to expedite network failure detection, fault 130 localization, and recovery mechanisms, particularly in the case of 131 soft failures or path degradation is expected, without causing 132 service disruption. Networks also become application-aware. 133 Application-aware networking is an industry term used to describe the 134 capacity of a network to maintain current information about users and 135 application connections which may be used to optimize the network 136 resource usage and improve the quality of service. 138 The emerging on-path telemetry techniques can provide high-precision 139 flow insight and real-time network issue notification (e.g., jitter, 140 latency, packet loss, significant bit error variations, and unequal 141 load-balancing). On-path telemetry refers to the data-plane 142 telemetry techniques that directly tap and measure network traffic by 143 embedding instructions or metadata into user packets. The data 144 provided by on-path telemetry are especially useful for SLA 145 compliance, user experience enhancement, service path enforcement, 146 fault diagnosis, and network resource optimization. It is essential 147 to recognize that existing work on this topic includes a variety of 148 on-path telemetry techniques, including In-situ OAM (IOAM) 149 [I-D.ietf-ippm-ioam-data], Postcard-based Telemetry (PBT) 150 [I-D.song-ippm-postcard-based-telemetry], Enhanced Alternate Marking 151 (EAM) [I-D.zhou-ippm-enhanced-alternate-marking], and Hybrid Two 152 Steps (HTS) [I-D.mirsky-ippm-hybrid-two-step], have been proposed, 153 which can provide flow information on the entire forwarding path on a 154 per-packet basis in real-time. The aforementioned on-path telemetry 155 techniques differ from the active and passive OAM schemes discussed 156 earlier in that, they directly modify the user packets and can 157 guarantee 100% accuracy. These on-path telemetry techniques can be 158 classified as the OAM hybrid type I, since they involve "augmentation 159 or modification of the stream of interest, or employment of methods 160 that modify the treatment of the streams", according to [RFC7799]. 162 On-path telemetry is useful for application-aware networking 163 operations not only in data center and enterprise networks but also 164 in carrier networks which may cross multiple domains. Carrier 165 network operators have shown interest in utilizing such techniques 166 for various purposes. For example, it is critical for the operators 167 who offer high-bandwidth, latency and loss-sensitive services such as 168 video streaming and online gaming to closely monitor the relevant 169 flows in real-time as the basis for any further optimizations. 171 This framework document is intended to guide system designers 172 attempting to use the referenced techniques as well as to motivate 173 further work to enhance the telemetry ecosystem. It highlights 174 requirements and challenges, outlines vital techniques that are 175 applicable, and provides examples of how these might be applied for 176 critical use cases. 178 The document scope is discussed in Section 1.2. 180 1.1. Requirements and Challenges 182 Although on-path telemetry is beneficial, successfully applying such 183 techniques in carrier networks must consider performance, 184 deployability, and flexibility. Specifically, we need to address the 185 following practical deployment challenges: 187 o C1: On-path telemetry incurs extra packet processing which may 188 cause stress on the network data plane. The potential impact on 189 the forwarding performance creates an unfavorable "observer 190 effect". This will not only damages the fidelity of the 191 measurement but also defies the purpose of the measurement. For 192 example, the growing IOAM data per hop can negatively affect 193 service levels by increasing the serialization delay and header 194 parsing delay. 196 o C2: On-path telemetry can generate a considerable amount of data 197 which may claim too much transport bandwidth and inundate the 198 servers for data collection, storage, and analysis. Increasing 199 the data handling capacity is technically viable but expensive. 200 For example, if IOAM is applied to all the traffic, one node may 201 collect a few tens of bytes as telemetry data for each packet. 202 The whole forwarding path might accumulate a data-trace with a 203 size similar to or even exceeding that of the original packet. 204 Transporting the telemetry data alone is projected to consume 205 almost half of the network bandwidth, plus it creates significant 206 back-end data handling and storage requirements. 208 o C3: The collectible data defined currently are essential but 209 limited. As the network operation evolves to be declarative 210 (intent-based) and automated, and the trends of network 211 virtualization, wireline and wireless convergence, and packet- 212 optical integration continue, more data is needed in an on-demand 213 and interactive fashion. Flexibility and extensibility on data 214 defining, aggregation, acquisition, and filtering, must be 215 considered. 217 o C4: Applying only a single underlying on-path telemetry technique 218 may lead to a defective result. For example, packet drop can 219 cause the loss of the flow telemetry data and the packet drop 220 location and reason remains unknown if only the In-situ OAM trace 221 option is used. A comprehensive solution needs the flexibility to 222 switch between different underlying techniques and adjust the 223 configurations and parameters at runtime. Thus, system-level 224 orchestration is needed. 226 o C5: If we were to apply some on-path telemetry technique in 227 today's carrier operator networks, we must provide solutions to 228 tailor the provider's network deployment base and support an 229 incremental deployment strategy. That is, we need to support 230 established encapsulation schemes for various predominant 231 protocols such as Ethernet, IPv4, IPv6, and MPLS with backward 232 compatibility and properly handle various transport tunnels. 234 o C6: The development of simplified on-path telemetry primitives and 235 models for configuration and queries is essential. Telemetry 236 models may be utilized via an API-based telemetry service for 237 external applications, for end-to-end performance measurement and 238 application performance monitoring. The standard-based protocols 239 and methods are needed for network configuration and programming, 240 and telemetry data processing and export, to provide 241 interoperability. 243 1.2. Scope 245 Following the network telemetry framework discussed in 246 [I-D.ietf-opsawg-ntf], this document focuses on the on-path 247 telemetry, a specific class of data-plane telemetry techniques, and 248 provides a high-level framework which addresses the aforementioned 249 challenges for deployment, especially in carrier operator networks. 251 This document aims to clarify the problem space, essential 252 requirements, and summarizes best practices and general system design 253 considerations. The framework helps to analyze the current standard 254 status and identify gaps, and to motivate new standard works to 255 complete the ecosystem. This document provides some examples to show 256 some novel network telemetry applications under the framework. 258 As an informational document, it describes an open framework with a 259 few key components. The framework does not enforces any specific 260 implementation on each component, neither does it define interfaces 261 (e.g., API, protocol) between components. The choice of underlying 262 on-path telemetry techniques and other implementation details is 263 determined by application implementer. Therefore, the framework is 264 not a solution specification. It only provides a high-level overview 265 and is not necessarily a mandatory recommendation for on-path 266 telemetry applications. Implementation of the framework is 267 implementor specific and may utilize functional components and 268 techniques outlined in this document. 270 The standardization of the underlying techniques and interfaces 271 mentioned in this document is undertaken by various working groups. 272 Due to the limited scope and intended status of this document, it has 273 no overlap or conflict with those works. 275 1.3. Glossary 277 This section defines and explains the acronyms and terms used in this 278 document. 280 On-path Telemetry: Remotely acquiring performance and behavior data 281 about network flows on a per-packet basis on the packet's 282 forwarding path. The term refers to a class of data plane 283 telemetry techniques, including IOAM, PBT, EAM, and HTS. Such 284 techniques may need to mark user packets, or insert instruction or 285 metadata to the headers of user packets. 287 IFIT: In-situ Flow Information Telemetry, pronounced as "I-Fit". 288 The name of a high-level reference framework that shows how 289 network data-plane monitoring applications can address the 290 deployment challenges of the flow-oriented on-path telemetry 291 techniques. 293 IFIT Domain: A network domain in which an on-path telemetry 294 application operates. The network domain contains multiple 295 forwarding devices, such as routers and switches, that are capable 296 of IFIT-specific functions. It also contains a logically 297 centralized controller whose responsibility is to apply IFIT- 298 specific configurations and functions to IFIT-capable forwarding 299 devices, and to collect and analyze the on-path telemetry data 300 from those devices. An IFIT domain contains multiple network node 301 capable of IFIT-specific functions. We name all the entry nodes 302 to an IFIT domain head nodes and all the exit nodes end nodes. A 303 path in an IFIT domain starts from a head node and ends at an end 304 node. Usually the instruction header encapsulation or packet 305 marking, if needed, happens at the head nodes; the instruction 306 header decapsulation or packet unmarking, if needed, happens at 307 the end nodes. 309 Reflective Telemetry: The telemetry functions in a dynamic and 310 interactive fashion. A new telemetry action is provisioned as a 311 result of self-knowledge acquired through prior telemetry actions. 313 1.4. Requirements Language 315 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 316 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 317 "OPTIONAL" in this document are to be interpreted as described in BCP 318 14 [RFC2119][RFC8174] when, and only when, they appear in all 319 capitals, as shown here. 321 2. IFIT Overview 323 To address the challenges mentioned above, we present a high-level 324 framework based on multiple network operators' requirements and 325 common industry practice, which can help to build a workable and 326 efficient on-path telemetry application. We name the framework "In- 327 situ Flow Information Telemetry" (IFIT) to reflect the fact that this 328 framework is dedicated to on-path telemetry data about user and 329 application traffic flows. As a reference framework, IFIT covers a 330 class of on-path telemetry techniques and works a level higher than 331 any specific underlying technique. The framework is comprised of 332 some key functional components (Section 3). By assembling these 333 components, IFIT supports reflective telemetry that enables 334 autonomous network operations (Section 4). 336 2.1. Typical Deployment of IFIT 338 Figure 1 shows a typical deployment scenario of IFIT. 340 Application 341 +-------------------------------------+ 342 | Controller | 343 | +------------+ +-----------+ | 344 | | Configure | | Collector | | 345 | | & |<-------| & | | 346 | | Control | | Analyzer | | 347 | +-----:------+ +-----------+ | 348 | : ^ | 349 +-------:---------------------|-------+ 350 :configuration |telemetry data 351 :& action | 352 ...............:.....................|.......... 353 : : : | : 354 : +---------:---+-------------:---++---------:---+ 355 : | : | : | : | 356 V | V | V | V | 357 +------+-+ +-----+--+ +------+-+ +------+-+ 358 packets| Head | | Path | | Path | | End | 359 ==>| Node |====>| Node |==//==>| Node |====>| Node |==> 360 | | | A | | B | | | 361 +--------+ +--------+ +--------+ +--------+ 363 |<--- IFIT Domain --->| 365 Figure 1: IFIT Deployment Scenario 367 An on-path telemetry application can conduct some network data plane 368 monitoring and measurement tasks over an IFIT domain by applying one 369 or more underlying techniques. The application needs to contains 370 multiple elements, including configuring the network nodes and 371 processing the telemetry data. The application usually runs in a 372 logically centralized controller which is responsible for configuring 373 the network nodes in the IFIT domain, and collecting and analyzing 374 telemetry data. The configuration determines which underlying 375 technique is used, what telemetry data are of interest, which flows 376 and packets are concerned with, how the telemetry data are collected, 377 etc. The process can be dynamic and interactive: after the telemetry 378 data processing and analyzing, the application may instruct the 379 controller to modify the configuration of the nodes in the IFIT 380 domain, which affects the future telemetry data collection. 382 From the system-level view, it is recommended to use the standardized 383 configuration and data collection interfaces, regardless of the 384 underlying technique. However, the specification of these interfaces 385 and the implementation of the controller are out of scope for this 386 document. 388 The IFIT domain encompasses the head nodes and the end nodes. An 389 IFIT domain may cross multiple network domains. The head nodes are 390 responsible for enabling the IFIT-specific functions and the end 391 nodes are responsible for terminating them. All capable nodes in an 392 IFIT domain will be capable of executing the instructed IFIT-specific 393 function. It is important to note that any IFIT application must, 394 through configuration and policy, guarantee that any packet with 395 IFIT-specific header and metadata will not leak out of the IFIT 396 domain. The end nodes must be able to capture all packets with IFIT- 397 specific header and metadata and recover their format before 398 forwarding them out of the IFIT domain. 400 IFIT supports two basic on-path telemetry modes: passport mode (e.g., 401 IOAM trace option [I-D.ietf-ippm-ioam-data]), in which telemetry data 402 are carried in user packets and only exported at the end nodes, and 403 postcard mode (e.g., PBT [I-D.song-ippm-postcard-based-telemetry]), 404 in which each node in the IFIT domain may export telemetry data 405 through dedicated packets. An on-path telemetry application may need 406 to mix or switch between these two modes. 408 2.1.1. On-path Telemetry Models: Passport vs. Postcard 410 [passport-postcard] first uses the metaphor of passport and postcard 411 to describe how the packet trace data can be collected and exported. 412 In the passport mode, each node on the path adds the telemetry data 413 to the user packets (i.e., stamp the passport). The accumulated data 414 trace is exported at a configured end node. In the postcard mode, 415 each node directly exports the telemetry data using an independent 416 packet (i.e., send a postcard) while the user packets are intact. 418 A prominent advantage of the passport mode is that it naturally 419 retains the telemetry data correlation along the entire path. The 420 passport mode also reduces the number of data export packets. These 421 help to simplify the data collector and analyzer's work. On the 422 other hand, the passport mode requires more processing on the user 423 packets and increases the size of user packets, which can cause 424 various problems. These and some other issues are documented in 425 [I-D.song-ippm-postcard-based-telemetry]. 427 The postcard mode provides a perfect complement to the passport mode. 428 It addresses most of the issues faced by the passport mode, at a cost 429 of needing extra effort to correlate the postcard packets. 431 2.2. IFIT Architecture 433 The IFIT architecture is shown in Figure 2, which contains several 434 key components. These components aim to address the deployment 435 challenges discussed in Section 1. The detailed block diagram and 436 description for each component are given in Section 3. Here we only 437 provide a high level overview. 439 +------------------------------------+ 440 | On-demand Technique | 441 | Selection & Integration | 442 +------------------------------------+ 443 Control Plane | ^ 444 ---------------------+-------------------+------------- 445 Forwarding Plane V | 446 +-----------------+------------------+ 447 | Smart Flow, | Smart Data | 448 | Packet, & Data | Export | 449 | Selection | | 450 +-----------------+------------------| 451 | Dynamic Network Probe | 452 +------------------------------------| 454 Figure 2: IFIT Architecture 456 Based on the monitoring and measurement requirements, an application 457 needs to choose one or more underlying on-path telemetry techniques 458 and decide the policies to apply them. Depending on the forwarding- 459 plane protocol and tunneling configuration, the instruction header 460 and metadata encapsulation method, if needed, is also determined. 461 The encapsulation happens at the head nodes and the decapsulation 462 happens at the end nodes. 464 Based on the network condition and application requirement, the head 465 nodes also need to be able to choose flows and packets to enable the 466 IFIT-specific functions, and decide the set of data to be collected. 467 All the nodes who are responsible for exporting telemetry data are 468 configured with special functions to prepare the data. The IFIT- 469 specific functions can be deployed into the network nodes as dynamic 470 network probes. 472 2.3. Relationship with Network Telemetry Framework (NTF) 474 [I-D.ietf-opsawg-ntf] describes a Network Telemetry Framework (NTF). 475 One dimension used by NTF to partition network telemetry techniques 476 and systems is based on the three planes in networks plus external 477 data sources. IFIT fits in the category of forwarding-plane 478 telemetry and deals with the specific on-path technical branch of the 479 forwarding-plane telemetry. 481 According to NTF, an on-path telemetry application mainly subscribes 482 event-triggered or streaming data. The key functional components of 483 IFIT also match the components in NTF. "On-demand Technique 484 Selection and Integration" is an application layer function, matching 485 the "Data Query, Analysis, and Storage" component in NTF; "Smart 486 Flow, Packet, and Data Selection" matches the "Data Configuration and 487 Subscription" component; "Smart Data Export" matches the "Data 488 Encoding and Export" component; the other two components match the 489 "Data Generation and Processing" component. 491 3. Key Components of IFIT 493 As shown in the IFIT architecture, the key components of IFIT are as 494 follows: 496 o Smart flow, packet, and data selection policy, addressing the 497 challenge C1 described in Section 1; 499 o Smart data export, addressing the challenge C2; 501 o Dynamic network probe, addressing C3; 503 o On-demand technique selection and integration, addressing C4. 505 Note that the challenges C5 and C6 are mostly standard related, which 506 are fundamental to IFIT. We discuss the standard status and gaps in 507 Section 5. 509 In the following section, we provide a detailed description of each 510 component. 512 3.1. Smart Flow, Packet, and Data Selection 514 In most cases, it is impractical to enable the data collection for 515 all the flows and for all the packets in a flow due to the potential 516 performance and bandwidth impact. Therefore, a workable solution 517 usually need to select only a subset of flows and flow packets to 518 enable the data collection, even though this means the loss of some 519 information and accuracy. 521 In the data plane, the Access Control List (ACL) provides an ideal 522 means to determine the subset of flow(s). An application can set a 523 sample rate or probability to a flow to allow only a subset of flow 524 packets to be monitored, collect a different set of data for 525 different packets, and disable or enable data collection on any 526 specific network node. An application can further allow any node to 527 accept or deny the data collection process in full or partially. 529 Based on these flexible mechanisms, IFIT allows applications to apply 530 smart flow and data selection policies to suit the requirements. The 531 applications can dynamically change the policies at any time based on 532 the network load, processing capability, focus of interest, and any 533 other criteria. 535 3.1.1. Block Diagram 537 +----------------------------+ 538 | +----------+ +----------+ | 539 | |Flow | |Data | | 540 | |Selection | |Selection | | 541 | +----------+ +----------+ | 542 | +----------+ | 543 | |Packet | | 544 | |Selection | | 545 | +----------+ | 546 +----------------------------+ 548 Figure 3: Smart Flow, Packet, and Data Selection 550 Figure 3 shows the block diagram of this component. The flow 551 selection block defines the policies to choose target flows for 552 monitoring. Flow has different granularity. A basic flow is defined 553 by 5-tuple IP header fields. Flow can also be aggregated at 554 interface level, tunnel level, protocol level, and so on. The packet 555 selection block defines the policies to choose packets from a target 556 flow. The policy can be either a sampling interval, a sampling 557 probability, or some specific packet signature. The data selection 558 block defines the set of data to be collected. This can be changed 559 on a per-packet or per-flow basis. 561 3.1.2. Example: Sketch-guided Elephant Flow Selection 563 Network operators are usually more interested in elephant flows which 564 consume more resource and are sensitive to changes in network 565 conditions. A CountMin Sketch [CMSketch] can be used on the data 566 path of the head nodes, which identifies and reports the elephant 567 flows periodically. The controller maintains a current set of 568 elephant flows and dynamically enables the on-path telemetry for only 569 these flows. 571 3.1.3. Example: Adaptive Packet Sampling 573 Applying on-path telemetry on all packets of selected flows can still 574 be out of reach. A sample rate should be set for these flows and 575 only enable telemetry on the sampled packets. However, the head 576 nodes have no clue on the proper sampling rate. An overly high rate 577 would exhaust the network resource and even cause packet drops; An 578 overly low rate, on the contrary, would result in the loss of 579 information and inaccuracy of measurements. 581 An adaptive approach can be used based on the network conditions to 582 dynamically adjust the sampling rate. Every node gives user traffic 583 forwarding higher priority than telemetry data export. In case of 584 network congestion, the telemetry can sense some signals from the 585 data collected (e.g., deep buffer size, long delay, packet drop, and 586 data loss). The controller may use these signals to adjust the 587 packet sampling rate. In each adjustment period (i.e., RTT of the 588 feedback loop), the sampling rate is either decreased or increased in 589 response of the signals. An AIMD policy similar to the TCP flow 590 control mechanism for the rate adjustment can be used. 592 3.2. Smart Data Export 594 The flow telemetry data can catch the dynamics of the network and the 595 interactions between user traffic and network. Nevertheless, the 596 data inevitably contain redundancy. It is advisable to remove the 597 redundancy from the data in order to reduce the data transport 598 bandwidth and server processing load. 600 In addition to efficient export data encoding (e.g., IPFIX [RFC7011] 601 or protobuf [1]), nodes have several other ways to reduce the export 602 data by taking advantage of network device's capability and 603 programmability. Nodes can cache the data and send the accumulated 604 data in batch if the data is not time sensitive. Various 605 deduplication and compression techniques can be applied on the batch 606 data. 608 From the application perspective, an application may only be 609 interested in some special events which can be derived from the 610 telemetry data. For example, in case that the forwarding delay of a 611 packet exceeds a threshold, or a flow changes its forwarding path is 612 of interest, it is unnecessary to send the original raw data to the 613 data collecting and processing servers. Rather, IFIT takes advantage 614 of the in-network computing capability of network devices to process 615 the raw data and only push the event notifications to the subscribing 616 applications. 618 Such events can be expressed as policies. An policy can request data 619 export only on change, on exception, on timeout, or on threshold. 621 3.2.1. Block Diagram 623 +-------------------------------------------+ 624 | +-----------+ +-----------+ +-----------+ | 625 | |Data | |Data | |Export | | 626 | |Encoding | |Batching | |Protocol | | 627 | +-----------+ +-----------+ +-----------+ | 628 | +-----------+ +-----------+ +-----------+ | 629 | |Data | |Data | |Data | | 630 | |Compression| |Dedup. | |Filter | | 631 | +-----------+ +-----------+ +-----------+ | 632 | +-----------+ +-----------+ | 633 | |Data | |Data | | 634 | |Computing | |Aggregation| | 635 | +-----------+ +-----------+ | 636 +-------------------------------------------+ 638 Figure 4: Smart Data Export 640 Figure 4 shows the block diagram of this component. The data 641 encoding block defines the method to encode the telemetry data. The 642 data batching block defines the size of batch data buffered at the 643 device side before export. The export protocol block defines the 644 protocol used for telemetry data export. The data compression block 645 defines the algorithm to compress the raw data. The data 646 deduplication block defines the algorithm to remove the redundancy in 647 the raw data. The data filter block defines the policies to filter 648 the needed data. The data computing block defines the policies to 649 prepocess the raw data and generate some new data. The data 650 aggregation block defines the procedure to combine and synthesize the 651 data. 653 3.2.2. Example: Event-based Anomaly Monitor 655 Network operators are interested in the anomalies such as path 656 change, network congestion, and packet drop. Such anomalies are 657 hidden in raw telemetry data (e.g., path trace, timestamp). Such 658 anomalies can be described as events and programmed into the device 659 data plane. Only the triggered events are exported. For example, if 660 a new flow appears at any node, a path change event is triggered; if 661 the packet delay exceeds a predefined threshold in a node, the 662 congestion event is triggered; if a packet is dropped due to buffer 663 overflow, a packet drop event is triggered. 665 The export data reduction due to such optimization is substantial. 666 For example, given a single 5-hop 10Gbps path, assume a moderate 667 number of 1 million packets per second are monitored, and the 668 telemetry data plus the export packet overhead consume less than 30 669 bytes per hop. Without such optimization, the bandwidth consumed by 670 the telemetry data can easily exceed 1Gbps (>10% of the path 671 bandwidth), When the optimization is used, the bandwidth consumed by 672 the telemetry data is negligible. Moreover, the pre-processed 673 telemetry data greatly simplify the work of data analyzers. 675 3.3. Dynamic Network Probe 677 Due to limited data plane resource and network bandwidth, it is 678 unlikely one can monitor all the data all the time. On the other 679 hand, the data needed by applications may be arbitrary but ephemeral. 680 It is critical to meet the dynamic data requirements with limited 681 resource. 683 Fortunately, data plane programmability allows IFIT to dynamically 684 load new data probes. These on-demand probes are called Dynamic 685 Network Probes (DNP). DNP is the technique to enable probes for 686 customized data collection in different network planes. When working 687 with IOAM or PBT, DNP is loaded to the data plane through incremental 688 programming or configuration. The DNP can effectively conduct data 689 generation, processing, and aggregation. 691 DNP introduces enough flexibility and extensibility to IFIT. It can 692 implement the optimizations for export data reduction motioned in the 693 previous section. It can also generate custom data as required by 694 today and tomorrow's applications. 696 3.3.1. Block Diagram 698 +----------------------------+ 699 | +----------+ +----------+ | 700 | |ACL | |YANG | | 701 | | | |Model | | 702 | +----------+ +----------+ | 703 | +----------+ +----------+ | 704 | |Hardware | |Software | | 705 | |Function | |Function | | 706 | +----------+ +----------+ | 707 +----------------------------+ 709 Figure 5: Dynamic Network Probes 711 Figure 5 shows the block diagram of this component. The Access 712 Control List (ACL) block is available in most hardware and it defines 713 DNPs through dynamically update the ACL policies (including flow 714 filtering and action). YANG models can be dynamically deployed to 715 enable different data processing and filtering functions. Some 716 hardware allows dynamically loading hardware-based functions into the 717 forwarding path at runtime through mechanisms such as reserved 718 pipelines and function stubs. Dynamically loadable software 719 functions can be implemented in the control processors in IFIT nodes. 721 3.3.2. Examples 723 Following are some possible DNPs that can be dynamically deployed to 724 support applications. 726 On-demand Flow Sketch: A flow sketch is a compact online data 727 structure (usually a variation of multi-hashing table) for 728 approximate estimation of multiple flow properties. It can be 729 used to facilitate flow selection. The aforementioned CountMin 730 Sketch [CMSketch] is such an example. Since a sketch consumes 731 data plane resources, it should only be deployed when actually 732 needed. 734 Smart Flow Filter: The policies that choose flows and packet 735 sampling rate can change during the lifetime of an application. 737 Smart Statistics: An application may need to count flows based on 738 different flow granularity or maintain hit counters for selected 739 flow table entries. 741 Smart Data Reduction: DNP can be used to program the events that 742 conditionally trigger data export. 744 3.4. On-demand Technique Selection and Integration 746 With multiple underlying data collection and export techniques at its 747 disposal, IFIT can flexibly adapt to different network conditions and 748 different application requirements. 750 For example, depending on the types of data that are of interest, 751 IFIT may choose either IOAM or PBT to collect the data; if an 752 application needs to track down where the packets are lost, switching 753 from IOAM to PBT should be supported. 755 IFIT can further integrate multiple data plane monitoring and 756 measurement techniques together and present a comprehensive data 757 plane telemetry solution. 759 Based on the application requirements and the real-time telemetry 760 data analysis results, new configurations and actions can be 761 deployed. 763 3.4.1. Block Diagram 765 +----------------------------------------------+ 766 | +------------+ +-------------+ +---------+ | | 767 | |Application | |Configuration| |Telemetry| | 768 | |Requirements|->|& Action |<-|Data | | 769 | | | | | |Analysis | | 770 | +------------+ +-------------+ +---------+ | 771 +----------------------------------------------+ 772 | Passport: | 773 | +----------+ +----------+ +----------+ | 774 | |IOAM E2E | |IOAM Trace| |EAM | | 775 | +----------+ +----------+ +----------+ | 776 | Postcard: | 777 | +----------+ +----------+ | 778 | |PBT-M | |IOAM DEX | | 779 | +----------+ +----------+ | 780 | Hybrid: | 781 | +----------+ +----------+ | 782 | |HTS | |Multicast | | 783 | | | |Telemetry | | 784 | +----------+ +----------+ | 785 +----------------------------------------------+ 787 Figure 6: Technique Selection and Integration 789 Figure 7 shows the block diagram of this component, which lists the 790 candidate on-path telemetry techniques. IOAM E2E and Trace options 791 are described in [I-D.ietf-ippm-ioam-data]. EAM is described in 792 [I-D.zhou-ippm-enhanced-alternate-marking]. PBT-M is described in 793 [I-D.song-ippm-postcard-based-telemetry]. IOAM DEX option is 794 described in [I-D.ioamteam-ippm-ioam-direct-export]. HTS is 795 described in [I-D.mirsky-ippm-hybrid-two-step]. Multicast Telemetry 796 is described in [I-D.song-multicast-telemetry]. 798 Located in the logically centralized controller of an IFIT domain, 799 this component makes all the control and configuration dynamically to 800 the capable nodes in the domain which will affect the future 801 telemetry data. The configuration and action decisions are based on 802 the inputs from the application requirements and the realtime 803 telemetry data analysis results. Note that here the telemetry data 804 source is not limited to the data plane. The data can come form all 805 the sources mentioned in [I-D.ietf-opsawg-ntf], including external 806 data sources. 808 4. IFIT for Reflective Telemetry 810 The IFIT components can work together to support reflective 811 telemetry, as shown in Figure 8. 813 +---------------------+ 814 | | 815 +------+ Applications |<------+ 816 | | | | 817 | +---------------------+ | 818 | Technique Selection | 819 | and Integration | 820 | | 821 |Smart Flow Smart | 822 |and Data reflection-loop Data | 823 |Selection Export| 824 | | 825 | +----+----+ 826 V +---------+| 827 +----------+ Encapsulation +---------+|| 828 | Head | and Tunneling | Path ||| 829 | Node |----------------------->| Nodes ||+ 830 | | | |+ 831 +----------+ +---------+ 832 DNP DNP 834 Figure 7: IFIT-based Reflective Telemetry 836 An application may pick a suite of telemetry techniques based on its 837 requirements and apply an initial technique to the data plane. It 838 then configures the head nodes to decide the initial target flows/ 839 packets and telemetry data set, the encapsulation and tunneling 840 scheme based on the underlying network architecture, and the IFIT- 841 capable nodes to decide the initial telemetry data export policy. 842 Based on the network condition and the analysis results of the 843 telemetry data, the application can change the telemetry technique, 844 the flow/data selection policy, and the data export approach in real 845 time without breaking the normal network operation. Many of such 846 dynamic changes can be done through loading and unloading DNPs. 848 The reflective telemetry enabled by the IFIT allows numerous new 849 applications suitable for future network operation architecture. 851 4.1. Example: Intelligent Multipoint Performance Monitoring 853 [I-D.ietf-ippm-multipoint-alt-mark] describes an intelligent 854 performance management based on the network condition. The idea is 855 to split the monitoring network into clusters. The cluster partition 856 that can be applied to every type of network graph and the 857 possibility to combine clusters at different levels enable the so- 858 called Network Zooming. It allows a controller to calibrate the 859 network telemetry, so that it can start without examining in depth 860 and monitor the network as a whole. In case of necessity (packet 861 loss or too high delay), an immediate detailed analysis can be 862 reconfigured. In particular, the controller, that is aware of the 863 network topology, can set up the most suited cluster partition by 864 changing the traffic filter or activate new measurement points and 865 the problem can be localized with a step-by-step process. 867 An application on top of the controllers can manage such mechanism 868 and IFIT's architecture allows its dynamic and reflective operation. 870 4.2. Example: Intent-based Network Monitoring 872 User Intents 873 | 874 V Per-packet 875 +------------+ Telemetry 876 ACL | | Data 877 +--------+ Controller |<--------+ 878 | | | | 879 | +--+---------+ | 880 | | ^ | 881 | |DNPs |Network | 882 | | |Information| 883 | V | | 884 +------+-------------------+-----------+---+ 885 | | | 886 | V +------+ | 887 | +-------+ +------+| | 888 | | Head | IFIT Domain +------+|| | 889 | | Node | |Path ||+ | 890 | | | |Nodes |+ | 891 | +-------+ +------+ | 892 +------------------------------------------+ 894 Figure 8: Intent-based Monitoring 896 In this example, a user can express high level intents for network 897 monitoring. The controller translates an intent and configure the 898 corresponding DNPs in IFIT-capable nodes which collect necessary 899 network information. Based on the real-time information feedback, 900 the controller runs a local algorithm to determine the suspicious 901 flows. It then deploys ACLs to the head node to initiate the high 902 precision per-packet on-path telemetry for these flows. 904 5. Standard Status and Gaps 906 A complete IFIT-based solution needs standard interfaces for 907 configuration and data extraction, and standard encapsulation on 908 various transport protocols. It may also need standard API and 909 primitives for application programming and deployment. The draft 910 [I-D.brockners-opsawg-ioam-deployment] summarizes some current 911 proposals on encapsulation and data export for IOAM. These works 912 should be extended or modified to support other types of on-path 913 telemetry techniques and other transport protocols. The high-level 914 IFIT helps to develop coherent and universal standard encapsulation 915 and data export approaches. 917 5.1. Encapsulation in Transport Protocols 919 Since the introduction of IOAM, the IOAM option header encapsulation 920 schemes in various network protocols have been proposed. Similar 921 encapsulation schemes need to be extended to cover the other on-path 922 telemetry techniques. On the other hand, the encapsulation scheme 923 for some popular protocols, such as MPLS and IPv4, are noticeably 924 missing. It is important to provide the encapsulation schemes for 925 these protocols because they are still prevalent in carrier networks. 926 IFIT needs to provide solutions to apply the on-path flow telemetry 927 techniques in such networks. PBT-M 928 [I-D.song-ippm-postcard-based-telemetry] does not introduce new 929 headers to the packets so the trouble of encapsulation for a new 930 header is avoided. While there are some proposals which allow new 931 header encapsulation in MPLS packets (e.g., 932 [I-D.song-mpls-extension-header]) or in IPv4 packets (e.g., 933 [I-D.herbert-ipv4-eh]), they are still in their infancy stage and 934 require significant future work. For the meantime, in a confined 935 IFIT domain, pre-standard encapsulation approaches may be applied. 937 5.2. Tunneling Support 939 In carrier networks, it is common for user traffic to traverse 940 various tunnels for QoS, traffic engineering, or security. IFIT 941 supports both the uniform mode and the pipe mode for tunnel support 942 as described in [I-D.song-ippm-ioam-tunnel-mode]. With such 943 flexibility, the operator can either gain a true end-to-end 944 visibility or apply a hierarchical approach which isolates the 945 monitoring domain between customer and provider. 947 5.3. Deployment Automation 949 In addition, standard approaches that automates the function 950 configuration, and capability query and advertisement, either in a 951 centralized fashion or a distributed fashion, are still immature. 952 The draft [I-D.zhou-ippm-ioam-yang] provides the YANG model for IOAM 953 configuration. Similar models needs to be defined for other 954 techniques. It is also helpful to provide standards-based approaches 955 for configuration in various network environments. For example, in 956 segment routing networks, extensions to BGP or PCEP can be defined to 957 distribute SR policies carrying IFIT information, so that IFIT 958 behavior can be enabled automatically when the SR policy is applied. 959 [I-D.chen-pce-sr-policy-ifit] proposes to extend PCEP policy for IFIT 960 configuration in segment routing networks. 961 [I-D.qin-idr-sr-policy-ifit] proposes to extend BGP policy instead 962 for IFIT configuration in segment routing networks. Additional 963 capability discovery and dissemination will be needed for other types 964 of networks. 966 To realize the potential of IFIT, programming and deploying DNPs are 967 important. ForCES [RFC5810] is a standard protocol for network 968 device programming, which can be used for DNP deployment. Currently 969 some related works such as [I-D.wwx-netmod-event-yang] and 970 [I-D.bwd-netmod-eca-framework] have proposed to use YANG model to 971 define the smart policies which can be used to implement DNPs. In 972 the future, other approaches for hardware and software-based 973 functions can be development to enhance the programmability and 974 flexibility. 976 6. Summary 978 IFIT is a high-level framework for applying on-path telemetry 979 techniques, and this document has outlined how the framework may be 980 used to solve essential use cases. IFIT enables a practical data- 981 plane telemetry application based on two basic on-path traffic data 982 collection modes: passport and postcard. 984 The operation of techniques covered by IFIT differs from both active 985 OAM and passive OAM as defined in [RFC7799]. It does not generate 986 any active probe packets or passively observes unmodified user 987 packets. Instead, it modifies selected user packets to collect 988 useful information about them. Therefore, the operation can be 989 categorized as the hybrid OAM type I mode per [RFC7799], which can 990 provide more flexible and accurate network monitoring and 991 measurement. 993 IFIT addresses the key challenges for operators to deploy a complete 994 on-path telemetry solution. However, as a reference and open 995 framework, IFIT only describes the basic functions of each identified 996 component and suggests possible applications. It has no intention of 997 specifying the implementation of the components and the interfaces 998 between the components. The compliance of IFIT is by no means 999 mandatory either. Instead, this informational document aims to 1000 clarify the problem domain, and summarize the best practices and 1001 sensible system design considerations. IFIT can guide the analysis 1002 of the current standard status and gaps, and motivate new works to 1003 complete the ecosystem. IFIT enables data-plane reflective telemetry 1004 applications for advanced network operations. 1006 Having a high-level framework covering a class of related techniques 1007 also promotes a holistic approach for standard development and helps 1008 to avoid duplicated efforts and piecemeal solutions that only focus 1009 on a specific technique while omitting the compatibility and 1010 extensibility issues, which is important to a healthy ecosystem for 1011 network telemetry. 1013 7. Security Considerations 1015 In addition to the specific security issues discussed in each 1016 individual document on on-path telemetry, this document considers the 1017 overall security issues at the IFIT system level. This should serve 1018 as a guide to the on-path telemetry application developers and users. 1020 8. IANA Considerations 1022 This document includes no request to IANA. 1024 9. Contributors 1026 Other major contributors of this document include Giuseppe Fioccola, 1027 Daniel King, Zhenqiang Li, Zhenbin Li, Tianran Zhou, and James 1028 Guichard. 1030 10. Acknowledgments 1032 We thank Diego Lopez, Shwetha Bhandari, Joe Clarke, Adrian Farrel, 1033 Frank Brockners, Al Morton, Alex Clemm, Alan DeKok, and Warren Kumari 1034 for their constructive suggestions for improving this document. 1036 11. References 1038 11.1. Normative References 1040 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1041 Requirement Levels", BCP 14, RFC 2119, 1042 DOI 10.17487/RFC2119, March 1997, 1043 . 1045 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1046 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1047 May 2016, . 1049 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1050 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1051 May 2017, . 1053 11.2. Informative References 1055 [CMSketch] 1056 Cormode, G. and S. Muthukrishnan, "An improved data stream 1057 summary: the count-min sketch and its applications", 2005, 1058 . 1060 [I-D.brockners-opsawg-ioam-deployment] 1061 Brockners, F., Bhandari, S., and d. 1062 daniel.bernier@bell.ca, "In-situ OAM Deployment", draft- 1063 brockners-opsawg-ioam-deployment-00 (work in progress), 1064 October 2019. 1066 [I-D.bwd-netmod-eca-framework] 1067 Boucadair, M., WU, Q., Wang, Z., King, D., and C. Xie, 1068 "Framework for Use of ECA (Event Condition Action) in 1069 Network Self Management", draft-bwd-netmod-eca- 1070 framework-00 (work in progress), November 2019. 1072 [I-D.chen-pce-sr-policy-ifit] 1073 Chen, H., Yuan, H., Zhou, T., Li, W., and G. Fioccola, 1074 "PCEP SR Policy Extensions to Enable IFIT", draft-chen- 1075 pce-sr-policy-ifit-00 (work in progress), January 2020. 1077 [I-D.herbert-ipv4-eh] 1078 Herbert, T., "IPv4 Extension Headers and Flow Label", 1079 draft-herbert-ipv4-eh-01 (work in progress), May 2019. 1081 [I-D.ietf-ippm-ioam-data] 1082 Brockners, F., Bhandari, S., Pignataro, C., Gredler, H., 1083 Leddy, J., Youell, S., Mizrahi, T., Mozes, D., Lapukhov, 1084 P., remy@barefootnetworks.com, r., daniel.bernier@bell.ca, 1085 d., and J. Lemon, "Data Fields for In-situ OAM", draft- 1086 ietf-ippm-ioam-data-08 (work in progress), October 2019. 1088 [I-D.ietf-ippm-multipoint-alt-mark] 1089 Fioccola, G., Cociglio, M., Sapio, A., and R. Sisto, 1090 "Multipoint Alternate Marking method for passive and 1091 hybrid performance monitoring", draft-ietf-ippm- 1092 multipoint-alt-mark-06 (work in progress), February 2020. 1094 [I-D.ietf-opsawg-ntf] 1095 Song, H., Qin, F., Martinez-Julia, P., Ciavaglia, L., and 1096 A. Wang, "Network Telemetry Framework", draft-ietf-opsawg- 1097 ntf-02 (work in progress), October 2019. 1099 [I-D.ioamteam-ippm-ioam-direct-export] 1100 Song, H., Gafni, B., Zhou, T., Li, Z., Brockners, F., 1101 Bhandari, S., Sivakolundu, R., and T. Mizrahi, "In-situ 1102 OAM Direct Exporting", draft-ioamteam-ippm-ioam-direct- 1103 export-00 (work in progress), October 2019. 1105 [I-D.mirsky-ippm-hybrid-two-step] 1106 Mirsky, G., Lingqiang, W., and G. Zhui, "Hybrid Two-Step 1107 Performance Measurement Method", draft-mirsky-ippm-hybrid- 1108 two-step-04 (work in progress), October 2019. 1110 [I-D.qin-idr-sr-policy-ifit] 1111 Qin, F., Yuan, H., Zhou, T., Min, L., and G. Fioccola, 1112 "BGP SR Policy Extensions to Enable IFIT", draft-qin-idr- 1113 sr-policy-ifit-00 (work in progress), January 2020. 1115 [I-D.song-ippm-ioam-tunnel-mode] 1116 Song, H., Li, Z., Zhou, T., and Z. Wang, "In-situ OAM 1117 Processing in Tunnels", draft-song-ippm-ioam-tunnel- 1118 mode-00 (work in progress), June 2018. 1120 [I-D.song-ippm-postcard-based-telemetry] 1121 Song, H., Zhou, T., Li, Z., Shin, J., and K. Lee, 1122 "Postcard-based On-Path Flow Data Telemetry", draft-song- 1123 ippm-postcard-based-telemetry-06 (work in progress), 1124 October 2019. 1126 [I-D.song-mpls-extension-header] 1127 Song, H., Li, Z., Zhou, T., and L. Andersson, "MPLS 1128 Extension Header", draft-song-mpls-extension-header-02 1129 (work in progress), February 2019. 1131 [I-D.song-multicast-telemetry] 1132 Song, H., McBride, M., and G. Mirsky, "Requirement and 1133 Solution for Multicast Traffic Telemetry", draft-song- 1134 multicast-telemetry-02 (work in progress), February 2020. 1136 [I-D.wwx-netmod-event-yang] 1137 Wang, Z., WU, Q., Bryskin, I., Liu, X., and B. Claise, "A 1138 YANG Data model for ECA Policy Management", draft-wwx- 1139 netmod-event-yang-06 (work in progress), December 2019. 1141 [I-D.zhou-ippm-enhanced-alternate-marking] 1142 Zhou, T., Fioccola, G., Li, Z., Lee, S., and M. Cociglio, 1143 "Enhanced Alternate Marking Method", draft-zhou-ippm- 1144 enhanced-alternate-marking-04 (work in progress), October 1145 2019. 1147 [I-D.zhou-ippm-ioam-yang] 1148 Zhou, T., Guichard, J., Brockners, F., and S. Raghavan, "A 1149 YANG Data Model for In-Situ OAM", draft-zhou-ippm-ioam- 1150 yang-06 (work in progress), March 2020. 1152 [passport-postcard] 1153 Handigol, N., Heller, B., Jeyakumar, V., Mazieres, D., and 1154 N. McKeown, "Where is the debugger for my software-defined 1155 network?", 2012, 1156 . 1158 [RFC2113] Katz, D., "IP Router Alert Option", RFC 2113, 1159 DOI 10.17487/RFC2113, February 1997, 1160 . 1162 [RFC5810] Doria, A., Ed., Hadi Salim, J., Ed., Haas, R., Ed., 1163 Khosravi, H., Ed., Wang, W., Ed., Dong, L., Gopal, R., and 1164 J. Halpern, "Forwarding and Control Element Separation 1165 (ForCES) Protocol Specification", RFC 5810, 1166 DOI 10.17487/RFC5810, March 2010, 1167 . 1169 [RFC7011] Claise, B., Ed., Trammell, B., Ed., and P. Aitken, 1170 "Specification of the IP Flow Information Export (IPFIX) 1171 Protocol for the Exchange of Flow Information", STD 77, 1172 RFC 7011, DOI 10.17487/RFC7011, September 2013, 1173 . 1175 11.3. URIs 1177 [1] https://developers.google.com/protocol-buffers/ 1179 Authors' Addresses 1180 Haoyu Song (editor) 1181 Futurewei 1182 2330 Central Expressway 1183 Santa Clara 1184 USA 1186 Email: haoyu.song@futurewei.com 1188 Fengwei Qin 1189 China Mobile 1190 No. 32 Xuanwumenxi Ave., Xicheng District 1191 Beijing, 100032 1192 P.R. China 1194 Email: qinfengwei@chinamobile.com 1196 Huanan Chen 1197 China Telecom 1198 P. R. China 1200 Email: chenhuan6@chinatelecom.cn 1202 Jaehwan Jin 1203 LG U+ 1204 South Korea 1206 Email: daenamu1@lguplus.co.kr 1208 Jongyoon Shin 1209 SK Telecom 1210 South Korea 1212 Email: jongyoon.shin@sk.com