idnits 2.17.1 draft-song-opsawg-ifit-framework-10.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 39 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 869 has weird spacing: '...nd Data refl...' == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (December 31, 2019) is 1571 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '1' on line 1167 == Unused Reference: 'RFC2113' is defined on line 1155, but no explicit reference was found in the text == Outdated reference: A later version (-03) exists of draft-brockners-opsawg-ioam-deployment-00 == Outdated reference: A later version (-03) exists of draft-herbert-ipv4-eh-01 == Outdated reference: A later version (-17) exists of draft-ietf-ippm-ioam-data-08 == Outdated reference: A later version (-09) exists of draft-ietf-ippm-multipoint-alt-mark-03 == Outdated reference: A later version (-13) exists of draft-ietf-opsawg-ntf-02 == Outdated reference: A later version (-15) exists of draft-mirsky-ippm-hybrid-two-step-04 == Outdated reference: A later version (-16) exists of draft-song-ippm-postcard-based-telemetry-06 == Outdated reference: A later version (-13) exists of draft-song-mpls-extension-header-02 == Outdated reference: A later version (-07) exists of draft-song-multicast-telemetry-01 == Outdated reference: A later version (-10) exists of draft-wwx-netmod-event-yang-06 == Outdated reference: A later version (-14) exists of draft-zhou-ippm-enhanced-alternate-marking-04 == Outdated reference: A later version (-08) exists of draft-zhou-ippm-ioam-yang-04 Summary: 1 error (**), 0 flaws (~~), 16 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 OPSAWG H. Song, Ed. 3 Internet-Draft Futurewei 4 Intended status: Informational F. Qin 5 Expires: July 3, 2020 China Mobile 6 H. Chen 7 China Telecom 8 J. Jin 9 LG U+ 10 J. Shin 11 SK Telecom 12 December 31, 2019 14 In-situ Flow Information Telemetry 15 draft-song-opsawg-ifit-framework-10 17 Abstract 19 As networks increase in scale and network operations become more 20 sophisticated, traditional Operation, Administration and Maintenance 21 (OAM) methods, which include proactive and reactive techniques, 22 running in active and passive modes, become more susceptible to 23 measurement accuracy and misconfiguration errors. With the advent of 24 programmable data-plane, emerging on-path telemetry techniques 25 provide unprecedented flow insight and realtime notification of 26 network issues. 28 This document enumerates the key deployment challenges for flow- 29 oriented on-path telemetry techniques, especially in carrier operator 30 networks. To address these issues, a high-level framework, In-situ 31 Flow Information Telemetry (iFIT), is outlined. iFIT includes several 32 essential functional components that can be materialized and 33 assembled to implement a complete solution for on-path telemetry. 35 This informational document aims to clarify the problem domain, and 36 summarize the best practices and sensible system design 37 considerations. The iFIT framework helps to guide the analysis on 38 the current standard status and gaps, and motivate new works to 39 complete the ecosystem. It also helps to inspire innovative network 40 telemetry applications supporting advanced network operations. As a 41 reference and open framework, iFIT does not specify the 42 implementation of the components and the interfaces between the 43 components. The compliance with iFIT framework is not mandatory for 44 telemetry applications either. 46 Status of This Memo 48 This Internet-Draft is submitted in full conformance with the 49 provisions of BCP 78 and BCP 79. 51 Internet-Drafts are working documents of the Internet Engineering 52 Task Force (IETF). Note that other groups may also distribute 53 working documents as Internet-Drafts. The list of current Internet- 54 Drafts is at https://datatracker.ietf.org/drafts/current/. 56 Internet-Drafts are draft documents valid for a maximum of six months 57 and may be updated, replaced, or obsoleted by other documents at any 58 time. It is inappropriate to use Internet-Drafts as reference 59 material or to cite them other than as "work in progress." 61 This Internet-Draft will expire on July 3, 2020. 63 Copyright Notice 65 Copyright (c) 2019 IETF Trust and the persons identified as the 66 document authors. All rights reserved. 68 This document is subject to BCP 78 and the IETF Trust's Legal 69 Provisions Relating to IETF Documents 70 (https://trustee.ietf.org/license-info) in effect on the date of 71 publication of this document. Please review these documents 72 carefully, as they describe your rights and restrictions with respect 73 to this document. Code Components extracted from this document must 74 include Simplified BSD License text as described in Section 4.e of 75 the Trust Legal Provisions and are provided without warranty as 76 described in the Simplified BSD License. 78 Table of Contents 80 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 81 1.1. Requirements and Challenges . . . . . . . . . . . . . . . 4 82 1.2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 6 83 1.3. Glossary . . . . . . . . . . . . . . . . . . . . . . . . 6 84 1.4. Requirements Language . . . . . . . . . . . . . . . . . . 7 85 2. iFIT Framework Overview . . . . . . . . . . . . . . . . . . . 7 86 2.1. iFIT Network Architecture . . . . . . . . . . . . . . . . 8 87 2.1.1. On-path Telemetry Models: Passport vs. Postcard . . . 9 88 2.2. iFIT Framework Architecture . . . . . . . . . . . . . . . 10 89 2.3. Relationship with Network Telemetry Framework (NTF) . . . 11 90 3. Key Components of iFIT . . . . . . . . . . . . . . . . . . . 11 91 3.1. Smart Flow, Packet, and Data Selection . . . . . . . . . 11 92 3.1.1. Block Diagram . . . . . . . . . . . . . . . . . . . . 12 93 3.1.2. Example: Sketch-guided Elephant Flow Selection . . . 12 94 3.1.3. Example: Adaptive Packet Sampling . . . . . . . . . . 13 95 3.2. Smart Data Export . . . . . . . . . . . . . . . . . . . . 13 96 3.2.1. Block Diagram . . . . . . . . . . . . . . . . . . . . 14 97 3.2.2. Example: Event-based Anomaly Monitor . . . . . . . . 14 98 3.3. Dynamic Network Probe . . . . . . . . . . . . . . . . . . 15 99 3.3.1. Block Diagram . . . . . . . . . . . . . . . . . . . . 15 100 3.3.2. Examples . . . . . . . . . . . . . . . . . . . . . . 16 101 3.4. Encapsulation and Tunneling . . . . . . . . . . . . . . . 16 102 3.4.1. Block Diagram . . . . . . . . . . . . . . . . . . . . 17 103 3.5. On-demand Technique Selection and Integration . . . . . . 17 104 3.5.1. Block Diagram . . . . . . . . . . . . . . . . . . . . 18 105 4. iFIT for Reflective Telemetry . . . . . . . . . . . . . . . . 19 106 4.1. Example: Intelligent Multipoint Performance Monitoring . 20 107 4.2. Example: Intent-based Network Monitoring . . . . . . . . 20 108 5. Standard Status and Gaps . . . . . . . . . . . . . . . . . . 21 109 6. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 110 7. Security Considerations . . . . . . . . . . . . . . . . . . . 22 111 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 112 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 22 113 10. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 22 114 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 23 115 11.1. Normative References . . . . . . . . . . . . . . . . . . 23 116 11.2. Informative References . . . . . . . . . . . . . . . . . 23 117 11.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 25 118 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 25 120 1. Introduction 122 Efficient network operation increasingly relies on high quality data- 123 plane telemetry to provide the necessary visibility. Traditional 124 Operation, Administration and Maintenance (OAM) methods, which 125 include proactive and reactive techniques, running in active and 126 passive modes, become more susceptible to measurement accuracy and 127 misconfiguration errors, as networks increase in scale and network 128 operations become more sophisticated. 130 The sheer complexity of today's networks and stringent service 131 requirements require new traffic monitoring and measurement solutions 132 for a wide range of use cases with high performance and high 133 precision. Furthermore, the ability to expedite failure detection, 134 fault localization, and recovery mechanisms, particularly in the case 135 of soft failures or path degradation are expected, without causing 136 service disruption. 138 Future networks also need to be application-aware. Application-aware 139 networking is an emerging industry term and typically used to 140 describe the capacity of an intelligent network to maintain current 141 information about user and application connections that use network 142 resources and, as a result, the operator can optimize the network 143 resource usage and monitoring to ensure application and traffic 144 optimality. 146 With the advent of programmable data-plane, emerging on-path 147 telemetry techniques provide unprecedented flow insight and realtime 148 notification of network issues (e.g., jitter, increased latency, 149 packet loss, significant bit error variations, and unequal load- 150 balancing). On-path telemetry refers to the data-plane telemetry 151 techniques that directly tap and measure network traffic by embedding 152 instructions or metadata into user packets. The data provided by on- 153 path telemetry are especially useful for network operations that need 154 user SLA compliance, service path enforcement, fault diagnosis, and 155 network resource optimization. A family of on-path telemetry 156 techniques, including In-situ OAM (IOAM) [I-D.ietf-ippm-ioam-data], 157 Postcard-based Telemetry (PBT) 158 [I-D.song-ippm-postcard-based-telemetry], Enhanced Alternate Marking 159 (EAM) [I-D.zhou-ippm-enhanced-alternate-marking], and Hybrid Two 160 Steps (HTS) [I-D.mirsky-ippm-hybrid-two-step], have been proposed, 161 which can provide flow information on the entire forwarding path on a 162 per-packet basis in real time. These on-path telemetry techniques 163 are very different from the previous active and passive OAM schemes 164 in that they directly modify the user packets and can guarantee 100% 165 accuracy. These on-path telemetry techniques can be classified as 166 the OAM hybrid type I, since they involve "augmentation or 167 modification of the stream of interest, or employment of methods that 168 modify the treatment of the streams", according to [RFC7799]. 170 On-path telemetry is invaluable for application-aware networking 171 operations not only in data center and enterprise networks but also 172 in carrier networks which may cross multiple domains. Carrier 173 network operators have shown strong interest in utilizing such 174 techniques for various purposes. For example, it is vital for the 175 operators who offer bandwidth intensive, latency and loss sensitive 176 services such as video streaming and online gaming to closely monitor 177 the relevant flows in real time as the indispensable first step for 178 any further measure. 180 1.1. Requirements and Challenges 182 The potential benefits of on-path telemetry are substantial. 183 However, successfully applying such techniques in carrier networks 184 needs to consider performance, deployability, and flexibility. 185 Specifically, we need to address the following practical deployment 186 challenges: 188 o C1: On-path telemetry incurs extra packet processing which may 189 strain the network data plane. The potential impact on the 190 forwarding performance creates an unfavorable "observer effect" 191 which not only damages the fidelity of the measurement but also 192 defies the purpose of the measurement. For example, the growing 193 IOAM data per hop can negatively affect service levels by 194 increasing the serialization delay and header parsing delay. 196 o C2: On-path telemetry can generate a huge amount of data which may 197 claim too much transport bandwidth and inundate the servers for 198 data collection, storage, and analysis. Increasing the data 199 handling capacity is technically viable but expensive. For 200 example, if IOAM is applied to all the traffic, one node may 201 collect a few tens of bytes as telemetry data for each packet. 202 The whole forwarding path might accumulate a data trace with a 203 size similar to or even exceeding that of the original packet. 204 Transporting the telemetry data alone will consume almost half of 205 the network bandwidth, not to mention the back-end data handling 206 load. 208 o C3: The collectible data defined currently are essential but 209 limited. As the network operation evolves to be declarative 210 (intent-based) and automated, and the trends of network 211 virtualization, wireline and wireless convergence, and packet- 212 optical integration continue, more data will be needed in an on- 213 demand and interactive fashion. Flexibility and extensibility on 214 data defining, aggregation, acquisition, and filtering, must be 215 considered. 217 o C4: If we were to apply some on-path telemetry technique in 218 today's carrier networks, we must provide solutions to tailor the 219 provider's network deployment base and support an incremental 220 deployment strategy. That is, we need to support established 221 encapsulation schemes for various predominant protocols such as 222 Ethernet, IPv4, and MPLS with backward compatibility and properly 223 handle various transport tunnels. 225 o C5: Applying only a single underlying on-path telemetry technique 226 may lead to defective result. For example, packet drop can cause 227 the loss of the flow telemetry data and the packet drop location 228 and reason remains unknown if only the In-situ OAM trace option is 229 used. A comprehensive solution needs the flexibility to switch 230 between different underlying techniques and adjust the 231 configurations and parameters at runtime. The system level 232 orchestration is needed. 234 o C6: The development of simplified on-path telemetry primitives and 235 models for configuration and query is important and necessary. 236 These may be used by an API-based telemetry service for external 237 applications, for end-to-end performance measurement of network 238 paths and application performance monitoring. 240 1.2. Scope 242 Following the network telemetry framework discussed in 243 [I-D.ietf-opsawg-ntf], this document focuses on the on-path 244 telemetry, a specific class of data plane telemetry technique, and 245 provides a high level application framework which addresses the 246 aforementioned challenges for deployment especially in carrier 247 operator networks. 249 This document aims to clarify the problem domain, and summarize the 250 best practices and sensible system design considerations. The 251 framework helps to guide the analysis on the current standard status 252 and gaps, and motivate new works to complete the ecosystem. It also 253 helps to inspire innovative network telemetry applications supporting 254 advanced network operations. 256 As an informational document, it describes an open framework with a 257 few key components. The framework does not enforces any specific 258 implementation on each component, neither does it define interfaces 259 (e.g., API, protocol) between components. The choice of underlying 260 on-path telemetry techniques and other implementation details is 261 determined by application implementer. The compliance of the 262 reference framework is not mandatory either. 264 The standardization of the underlying techniques and interfaces is 265 undertaken by various working groups. Due to the limited scope and 266 intended status of this document, it has no overlap or conflict with 267 those works. 269 1.3. Glossary 271 This section defines and explains the acronyms and terms used in this 272 document. 274 On-path Telemetry: Remotely acquiring performance and behavior data 275 about a network flow on a per-packet basis on the packet's 276 forwarding path. The term refers to a class of data plane 277 telemetry techniques, including IOAM, PBT, EAM, and HTS. Such 278 techniques may need to mark user packets, or insert instruction or 279 metadata to the headers of user packets. 281 iFIT: In-situ Flow Information Telemetry, pronounced as "I-Fit". 283 iFIT Framework: A high-level reference framework that supports 284 network data-plane monitoring applications which apply one or more 285 of the underlying on-path telemetry techniques and materialize the 286 iFIT functional components for practical deployment. iFIT 287 framework is dedicated for flow-oriented data plane telemetry. 289 iFIT Application: A network monitoring application that conforms to 290 the iFIT framework. 292 iFIT Domain: A network domain in which an iFIT application operates. 293 The network domain contains multiple forwarding devices, such as 294 routers and switches, that are capable of iFIT-specific functions. 295 It also contains a logically centralized controller whose 296 responsibility is to apply iFIT-specific configurations and 297 functions to iFIT-capable forwarding devices, and to collect and 298 analyze the on-path telemetry data from those devices. 300 iFIT Node: A network node, usually a forwarding device, that is in 301 an iFIT domain and is capable of iFIT-specific functions. 303 iFIT Head Node: A special iFIT node. It is the entry node to an 304 iFIT domain. Usually the instruction header encapsulation, if 305 needed, happens here. 307 iFIT End Node: A special iFIT node. It is the exit node of an iFIT 308 domain. Usually the instruction header decapsulation, if needed, 309 happens here. 311 Reflective Telemetry: The telemetry functions in a dynamic and 312 interactive fashion. New telemetry action is provisioned as a 313 result of self-knowledge acquired through prior telemetry actions. 315 1.4. Requirements Language 317 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 318 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 319 "OPTIONAL" in this document are to be interpreted as described in BCP 320 14 [RFC2119][RFC8174] when, and only when, they appear in all 321 capitals, as shown here. 323 2. iFIT Framework Overview 325 To address the aforementioned challenges, we present a high-level 326 framework based on multiple network operators' requirements and 327 common industry practice, which can help to build a workable and 328 efficient on-path telemetry solution. We name the framework "In-situ 329 Flow Information Telemetry" (iFIT) to reflect the fact that this 330 framework is dedicated to on-path telemetry data about user/ 331 application traffic flows. As a reference framework for building a 332 complete solution, iFIT covers a class of on-path telemetry 333 techniques and works a level higher than any specific underlying 334 technique. The framework is built up on a few key functional 335 components (Section 3). By assembling these components, iFIT 336 supports reflective telemetry that enables autonomous network 337 operations (Section 4). 339 2.1. iFIT Network Architecture 341 The network architecture that applies iFIT is shown in Figure 1. 343 iFIT Application 344 +-------------------------------------+ 345 | Controller | 346 | +------------+ +-----------+ | 347 | | Configure | | Collector | | 348 | | & |<-------| & | | 349 | | Control | | Analyzer | | 350 | +-----:------+ +-----------+ | 351 | : ^ | 352 +-------:---------------------|-------+ 353 :configuration |telemetry data 354 :& action | 355 ...............:.....................|.......... 356 : : : | : 357 : +---------:---+-------------:---++---------:---+ 358 : | : | : | : | 359 V | V | V | V | 360 +------+-+ +-----+--+ +------+-+ +------+-+ 361 packets| iFIT | | Path | | Path | | iFIT | 362 ==>| Head |====>| Node |==//==>| Node |====>| End |==> 363 | Node | | A | | B | | Node | 364 +--------+ +--------+ +--------+ +--------+ 366 |<--- iFIT Domain --->| 368 Figure 1: iFIT Network Architecture 370 An iFIT application conducts some network data plane monitoring and 371 measurement tasks over an iFIT domain through applying one or more 372 underlying on-path telemetry techniques. The application usually 373 runs in a logically centralized controller which is responsible for 374 configuring the network nodes in the iFIT domain, and collecting and 375 analyzing telemetry data. The configuration determines which 376 underlying technique is used, what telemetry data are of interest, 377 which flows and packets are concerned with, how the telemetry data 378 are collected, etc. The process can be dynamic and interactive: 379 after the telemetry data processing and analyzing, the iFIT 380 application may instruct the controller to modify the iFIT node 381 configuration which affects the future telemetry data collection. 383 From system-level view, it is recommended to use the standardized 384 configuration and data collection interfaces, regardless of the 385 underlying technique. However, the specification of these interfaces 386 and the implementation of the controller are out of scope for this 387 document. 389 The iFIT domain is confined between the iFIT head nodes and the iFIT 390 end nodes. An iFIT domain may cross multiple network domains. The 391 iFIT head nodes are responsible for enabling the iFIT-specific 392 functions and the iFIT end nodes are responsible for annulling them. 393 All active iFIT nodes in an iFIT domain will then execute the 394 instructed iFIT-specific function. Any iFIT application must 395 guarantee that any packet with iFIT-specific header and metadata will 396 not leak out from the iFIT domain. The iFIT end nodes must be able 397 to capture all packets with iFIT-specific header and metadata and 398 recover their format before forwarding them out of the iFIT domain. 400 iFIT supports two basic on-path telemetry modes: passport mode (e.g., 401 IOAM trace option), in which telemetry data are carried in user 402 packets and only exported at the iFIT end nodes, and postcard mode 403 (e.g., PBT), in which each node in the iFIT domain may export 404 telemetry data through dedicated packets. An on-path telemetry 405 application may need to mix or switch between the two modes. 407 2.1.1. On-path Telemetry Models: Passport vs. Postcard 409 [passport-postcard] first uses the analogy of passport and postcard 410 to describe how the packet trace data can be collected and exported. 411 In the passport mode, each node on the path adds the telemetry data 412 to the user packets. The accumulated data trace is exported at a 413 configured end node. In the postcard mode, each node directly 414 exports the telemetry data using an independent packet while the user 415 packets are intact. 417 A prominent advantage of the passport mode is that it naturally 418 retains the telemetry data correlation along the entire path. The 419 passport mode also reduces the number of data export packets. These 420 help to simplify the data collector and analyzer's work. On the 421 other hand, the passport mode requires more processing on the user 422 packets and increases the size of user packets, which can cause 423 various problems. Some other issues are documented in 424 [I-D.song-ippm-postcard-based-telemetry]. 426 The postcard mode provides a perfect complement to the passport mode. 427 It addresses most of the issues faced by the passport mode, at a cost 428 of needing extra effort to correlate the postcard packets. 430 2.2. iFIT Framework Architecture 432 The iFIT framework architecture is shown in Figure 2, which contains 433 several key components. These components aim to address the 434 deployment challenges discussed in Section 1. The detailed block 435 diagram and description for each component are given in Section 3. 436 Here we only provide a high level overview. 438 +------------------------------------+ 439 | On-demand Technique | 440 | Selection & Integration | 441 +------------------------------------+ 442 Control Plane | ^ 443 ---------------------+-------------------+------------- 444 Forwarding Plane V | 445 +-----------------+------------------+ 446 | Smart Flow, | Smart Data | 447 | Packet, & Data | Export | 448 | Selection | | 449 +-----------------+------------------| 450 | Dynamic Network Probe | 451 +------------------------------------| 452 | Encapsulation & Tunneling | 453 +------------------------------------+ 455 Figure 2: iFIT Framework Architecture 457 Based on the monitoring and measurement requirements, an iFIT 458 application needs to choose one or more underlying on-path telemetry 459 techniques and decide the policies to apply them. Depending on the 460 forwarding-plane protocol and tunneling configuration, the 461 instruction header and metadata encapsulation method, if needed, is 462 also determined. The encapsulation happens at the iFIT head nodes 463 and the decapsulation happens at the iFIT end nodes. 465 Based on the network condition and application requirement, the iFIT 466 head nodes also need to be able to choose flows and packets to enable 467 the iFIT-specific functions, and decide the set of data to be 468 collected. All the iFIT nodes who are responsible for exporting 469 telemetry data are configured with special functions to prepare the 470 data. The iFIT-specific functions can be dynamically deployed into 471 the iFIT nodes as dynamic network probes. 473 2.3. Relationship with Network Telemetry Framework (NTF) 475 [I-D.ietf-opsawg-ntf] describes a Network Telemetry Framework (NTF). 476 One dimension used by NTF to partition network telemetry techniques 477 and systems is based on the three planes in networks plus external 478 data sources. iFIT framework fits in the forwarding-plane telemetry 479 category and deals with the specific on-path technical branch of the 480 forwarding-plane telemetry. 482 According to NTF, an iFIT application mainly subscribes event- 483 triggered or streaming data. The key functional components of iFIT 484 framework also match the components in NTF. On-demand Technique 485 Selection and Integration is basically an application layer function, 486 matching the Data Query, Analysis, and Storage component in NTF; 487 Smart Flow, Packet, and Data Selection matches the Data Configuration 488 and Subscription component; Smart Data Export matches the Data 489 Encoding and Export component; The other two components match the 490 Data Generation and Processing component. 492 3. Key Components of iFIT 494 As shown in the iFIT framework architecture, the key components of 495 iFIT are as follows: 497 o Smart flow, packet, and data selection policy, addressing the 498 challenge C1 described in Section 1. 500 o Smart data export, addressing the challenge C2. 502 o Dynamic network probe, addressing C3. 504 o Encapsulation and tunneling, addressing C4. 506 o On-demand technique selection and integration, addressing C5. 508 Note that this document does not directly address the challenge C6 509 which is open for future standard proposals and left as the concern 510 of application implementers. 512 Next we provide a detailed description of each component. 514 3.1. Smart Flow, Packet, and Data Selection 516 In most cases, it is impractical to enable the data collection for 517 all the flows and for all the packets in a flow due to the potential 518 performance and bandwidth impact. Therefore, a workable solution 519 usually need to select only a subset of flows and flow packets to 520 enable the data collection, even though this means the loss of some 521 information and accuracy. 523 In the data plane, the Access Control List (ACL) provides an ideal 524 means to determine the subset of flow(s). An application can set a 525 sample rate or probability to a flow to allow only a subset of flow 526 packets to be monitored, collect a different set of data for 527 different packets, and disable or enable data collection on any 528 specific network node. An application can further allow any node to 529 accept or deny the data collection process in full or partially. 531 Based on these flexible mechanisms, iFIT allows applications to apply 532 smart flow and data selection policies to suit the requirements. The 533 applications can dynamically change the policies at any time based on 534 the network load, processing capability, focus of interest, and any 535 other criteria. 537 3.1.1. Block Diagram 539 +----------------------------+ 540 | +----------+ +----------+ | 541 | |Flow | |Data | | 542 | |Selection | |Selection | | 543 | +----------+ +----------+ | 544 | +----------+ | 545 | |Packet | | 546 | |Selection | | 547 | +----------+ | 548 +----------------------------+ 550 Figure 3: Samrt Flow, Packet, and Data Selection 552 Figure 3 shows the block diagram of this component. The flow 553 selection block defines the policies to choose target flows for 554 monitoring. Flow has different granularity. A basic flow is defined 555 by 5-tuple IP header fields. Flow can also be aggregated at 556 interface level, tunnel level, protocol level, and so on. The packet 557 selection block defines the policies to choose packets from a target 558 flow. The policy can be either a sampling interval, a sampling 559 probability, or some specific packet signature. The data selection 560 block defines the set of data to be collected. This can be changed 561 on a per packet or per flow basis. 563 3.1.2. Example: Sketch-guided Elephant Flow Selection 565 Network operators are usually more interested in elephant flows which 566 consume more resource and are sensitive to changes in network 567 conditions. A CountMin Sketch [CMSketch] can be used on the data 568 path of the head nodes, which identifies and reports the elephant 569 flows periodically. The controller maintains a current set of 570 elephant flows and dynamically enables the on-path telemetry for only 571 these flows. 573 3.1.3. Example: Adaptive Packet Sampling 575 Applying on-path telemetry on all packets of selected flows can still 576 be out of reach. A sample rate should be set for these flows and 577 only enable telemetry on the sampled packets. However, the head 578 nodes have no clue on the proper sampling rate. An overly high rate 579 would exhaust the network resource and even cause packet drops; An 580 overly low rate, on the contrary, would result in the loss of 581 information and inaccuracy of measurements. 583 An adaptive approach can be used based on the network conditions to 584 dynamically adjust the sampling rate. Every node gives user traffic 585 forwarding higher priority than telemetry data export. In case of 586 network congestion, the telemetry can sense some signals from the 587 data collected (e.g., deep buffer size, long delay, packet drop, and 588 data loss). The controller may use these signals to adjust the 589 packet sampling rate. In each adjustment period (i.e., RTT of the 590 feedback loop), the sampling rate is either decreased or increased in 591 response of the signals. An AIMD policy similar to the TCP flow 592 control mechanism for the rate adjustment can be used. 594 3.2. Smart Data Export 596 The flow telemetry data can catch the dynamics of the network and the 597 interactions between user traffic and network. Nevertheless, the 598 data inevitably contain redundancy. It is advisable to remove the 599 redundancy from the data in order to reduce the data transport 600 bandwidth and server processing load. 602 In addition to efficient export data encoding (e.g., IPFIX [RFC7011] 603 or protobuf [1]), iFIT nodes have several other ways to reduce the 604 export data by taking advantage of network device's capability and 605 programmability. iFIT nodes can cache the data and send the 606 accumulated data in batch if the data is not time sensitive. Various 607 deduplication and compression techniques can be applied on the batch 608 data. 610 From the application perspective, an application may only be 611 interested in some special events which can be derived from the 612 telemetry data. For example, in case that the forwarding delay of a 613 packet exceeds a threshold, or a flow changes its forwarding path is 614 of interest, it is unnecessary to send the original raw data to the 615 data collecting and processing servers. Rather, iFIT takes advantage 616 of the in-network computing capability of network devices to process 617 the raw data and only push the event notifications to the subscribing 618 applications. 620 Such events can be expressed as policies. An policy can request data 621 export only on change, on exception, on timeout, or on threshold. 623 3.2.1. Block Diagram 625 +--------------------------------------------+ 626 | +-----------+ +-----------+ +-----------+ | 627 | |Data | |Data | |Export | | 628 | |Encoding | |Batching | |Protocol | | 629 | +-----------+ +-----------+ +-----------+ | 630 | +-----------+ +-----------+ +-----------+ | 631 | |Data | |Data | |Data | | 632 | |Compression| |Dedup. | |Filter | | 633 | +-----------+ +-----------+ +-----------+ | 634 | +-----------+ +-----------+ | 635 | |Data | |Data | | 636 | |Computing | |Aggregation| | 637 | +-----------+ +-----------+ | 638 +--------------------------------------------+ 640 Figure 4: Smart Data Export 642 Figure 4 shows the block diagram of this component. The data 643 encoding block defines the method to encode the telemetry data. The 644 data batching block defines the size of batch data buffered at the 645 device side before export. The export protocol block defines the 646 protocol used for telemetry data export. The data compression block 647 defines the algorithm to compress the raw data. The data 648 deduplication block defines the algorithm to remove the redundancy in 649 the raw data. The data filter block defines the policies to filter 650 the needed data. The data computing block defines the policies to 651 prepocess the raw data and generate some new data. The data 652 aggregation block defines the procedure to combine and synthesize the 653 data. 655 3.2.2. Example: Event-based Anomaly Monitor 657 Network operators are interested in the anomalies such as path 658 change, network congestion, and packet drop. Such anomalies are 659 hidden in raw telemetry data (e.g., path trace, timestamp). Such 660 anomalies can be described as events and programmed into the device 661 data plane. Only the triggered events are exported. For example, if 662 a new flow appears at any node, a path change event is triggered; if 663 the packet delay exceeds a predefined threshold in a node, the 664 congestion event is triggered; if a packet is dropped due to buffer 665 overflow, a packet drop event is triggered. 667 The export data reduction due to such optimization is substantial. 668 For example, given a single 5-hop 10Gbps path, assume a moderate 669 number of 1 million packets per second are monitored, and the 670 telemetry data plus the export packet overhead consume less than 30 671 bytes per hop. Without such optimization, the bandwidth consumed by 672 the telemetry data can easily exceed 1Gbps (>10% of the path 673 bandwidth), When the optimization is used, the bandwidth consumed by 674 the telemetry data is negligible. Moreover, the pre-processed 675 telemetry data greatly simplify the work of data analyzers. 677 3.3. Dynamic Network Probe 679 Due to limited data plane resource and network bandwidth, it is 680 unlikely one can monitor all the data all the time. On the other 681 hand, the data needed by applications may be arbitrary but ephemeral. 682 It is critical to meet the dynamic data requirements with limited 683 resource. 685 Fortunately, data plane programmability allows iFIT to dynamically 686 load new data probes. These on-demand probes are called Dynamic 687 Network Probes (DNP). DNP is the technique to enable probes for 688 customized data collection in different network planes. When working 689 with IOAM or PBT, DNP is loaded to the data plane through incremental 690 programming or configuration. The DNP can effectively conduct data 691 generation, processing, and aggregation. 693 DNP introduces enough flexibility and extensibility to iFIT. It can 694 implement the optimizations for export data reduction motioned in the 695 previous section. It can also generate custom data as required by 696 today and tomorrow's applications. 698 3.3.1. Block Diagram 700 +----------------------------+ 701 | +----------+ +----------+ | 702 | |ACL | |YANG | | 703 | | | |Model | | 704 | +----------+ +----------+ | 705 | +----------+ +----------+ | 706 | |Hardware | |Software | | 707 | |Function | |Function | | 708 | +----------+ +----------+ | 709 +----------------------------+ 711 Figure 5: Dynamic Network Probes 713 Figure 5 shows the block diagram of this component. The ACL block is 714 available in most hardware and it defines DNPs through dynamically 715 update the ACL policies (including flow filtering and action). YANG 716 models can be dynamically deployed to enable different data 717 processing and filtering functions. Some hardware allows dynamically 718 loading hardware-based functions into the forwarding path at runtime 719 through mechanisms such as reserved pipelines and function stubs. 720 Dynamically loadable software functions can be implemented in the 721 control processors in iFIT nodes. 723 3.3.2. Examples 725 Following are some possible DNPs that can be dynamically deployed to 726 support iFIT applications. 728 On-demand Flow Sketch: A flow sketch is a compact online data 729 structure for approximate flow statistics which can be used to 730 facilitate flow selection. The aforementioned CountMin Sketch is 731 such an example. Since a sketch consumes data plane resources, it 732 should only be deployed when needed. 734 Smart Flow Filter: The policies that choose flows and packet 735 sampling rate can change during the lifetime of an application. 737 Smart Statistics: An application may need to interactively count 738 flows based on different flow granularity or maintain hit counters 739 for selected flow table entries. 741 Smart Data Reduction: DNP can be used to program the events that 742 conditionally trigger data export. 744 3.4. Encapsulation and Tunneling 746 Since the introduction of IOAM, the IOAM option header encapsulation 747 schemes in various network protocols have been proposed. Similar 748 encapsulation schemes need to be extended to cover the other on-path 749 telemetry techniques. On the other hand, the encapsulation scheme 750 for some popular protocols, such as MPLS and IPv4, are noticeably 751 missing. It is important to provide the encapsulation schemes for 752 these protocols because they are still prevalent in carrier networks. 753 iFIT needs to provide solutions to apply the on-path flow telemetry 754 techniques in such networks. PBT-M 755 [I-D.song-ippm-postcard-based-telemetry] does not introduce new 756 headers to the packets so the trouble of encapsulation for a new 757 header is avoided. While there are some proposals which allow new 758 header encapsulation in MPLS packets (e.g., 759 [I-D.song-mpls-extension-header]) or in IPv4 packets (e.g., 760 [I-D.herbert-ipv4-eh]), they are still in their infancy stage and 761 require significant future work. For the meantime, in a confined 762 iFIT domain, pre-standard encapsulation approaches may be applied. 764 In carrier networks, it is common for user traffic to traverse 765 various tunnels for QoS, traffic engineering, or security. iFIT 766 supports both the uniform mode and the pipe mode for tunnel support 767 as described in [I-D.song-ippm-ioam-tunnel-mode]. With such 768 flexibility, the operator can either gain a true end-to-end 769 visibility or apply a hierarchical approach which isolates the 770 monitoring domain between customer and provider. 772 3.4.1. Block Diagram 774 +----------------------------+ 775 | +----------+ +----------+ | 776 | |Uniform | |Pipe | | 777 | |Tunnel | |Tunnel | | 778 | +----------+ +----------+ | 779 | +------+ +------+ +------+ | 780 | |IPv6 | |SRv6 | |MPLS | | 781 | +------+ +------+ +------+ | 782 | +------+ +------+ +------+ | 783 | |IPv4 | |Ether.| |Others| | 784 | +------+ +------+ +------+ | 785 +----------------------------+ 787 Figure 6: Tunnel Mode and Encapsulation Scheme 789 Figure 6 shows the block diagram of this component, which lists two 790 tunnel modes supported and various protocols with each needing an 791 iFIT-specific header encapsulation solution. 793 3.5. On-demand Technique Selection and Integration 795 With multiple underlying data collection and export techniques at its 796 disposal, iFIT can flexibly adapt to different network conditions and 797 different application requirements. 799 For example, depending on the types of data that are of interest, 800 iFIT may choose either IOAM or PBT to collect the data; if an 801 application needs to track down where the packets are lost, it may 802 switch from IOAM to PBT. 804 iFIT can further integrate multiple data plane monitoring and 805 measurement techniques together and present a comprehensive data 806 plane telemetry solution to network operating applications. 808 Based on the application requirements and the realtime telemetry data 809 analysis results, new configurations and actions can be deployed. 811 3.5.1. Block Diagram 813 +----------------------------------------------+ 814 | +------------+ +-------------+ +---------+ | | 815 | |Application | |Configuration| |Telemetry| | 816 | |Requirements|->|& Action |<-|Data | | 817 | | | | | |Analysis | | 818 | +------------+ +-------------+ +---------+ | 819 +----------------------------------------------+ 820 | Passport: | 821 | +----------+ +----------+ +----------+ | 822 | |IOAM E2E | |IOAM Trace| |EAM | | 823 | +----------+ +----------+ +----------+ | 824 | Postcard: | 825 | +----------+ +----------+ | 826 | |PBT-M | |IOAM DEX | | 827 | +----------+ +----------+ | 828 | Hybrid: | 829 | +----------+ +----------+ | 830 | |HTS | |Multicast | | 831 | | | |Telemetry | | 832 | +----------+ +----------+ | 833 +----------------------------------------------+ 835 Figure 7: Technique Selection and Integration 837 Figure 7 shows the block diagram of this component, which lists the 838 candidate on-path telemetry techniques. IOAM E2E and Trace options 839 are described in [I-D.ietf-ippm-ioam-data]. EAM is described in 840 [I-D.zhou-ippm-enhanced-alternate-marking]. PBT-M is described in 841 [I-D.song-ippm-postcard-based-telemetry]. IOAM DEX option is 842 described in [I-D.ioamteam-ippm-ioam-direct-export]. HTS is 843 described in [I-D.mirsky-ippm-hybrid-two-step]. Multicast Telemetry 844 is described in [I-D.song-multicast-telemetry]. 846 Located in the logically centralized controller of an iFIT domain, 847 this component makes all the control and configuration dynamically to 848 the iFIT nodes which will affect the future telemetry data. The 849 configuration and action decisions are based on the inputs from the 850 application requirements and the realtime telemetry data analysis 851 results. Note that here the telemetry data source is not limited to 852 the data plane. The data can come form all the sources mentioned in 853 [I-D.ietf-opsawg-ntf], including external data sources. 855 4. iFIT for Reflective Telemetry 857 The iFIT components can work together to support reflective 858 telemetry, as shown in Figure 8. 860 +---------------------+ 861 | | 862 +------+ iFIT Applications |<------+ 863 | | | | 864 | +---------------------+ | 865 | Technique Selection | 866 | and Integration | 867 | | 868 |Smart Flow Smart | 869 |and Data reflection-loop Data | 870 |Selection Export| 871 | | 872 | +----+----+ 873 V +---------+| 874 +----------+ Encapsulation +---------+|| 875 | iFIT | and Tunneling | iFIT ||| 876 | Head |----------------------->| ||+ 877 | Node | | Nodes |+ 878 +----------+ +---------+ 879 DNP DNP 881 Figure 8: iFIT-based Reflective Telemetry 883 An iFIT application may pick a suite of telemetry techniques based on 884 its requirements and apply an initial technique to the data plane. 885 It then configures the iFIT head nodes to decide the initial target 886 flows/packets and telemetry data set, the encapsulation and tunneling 887 scheme based on the underlying network architecture, and the iFIT- 888 capable nodes to decide the initial telemetry data export policy. 889 Based on the network condition and the analysis results of the 890 telemetry data, the iFIT application can change the telemetry 891 technique, the flow/data selection policy, and the data export 892 approach in real time without breaking the normal network operation. 893 Many of such dynamic changes can be done through loading and 894 unloading DNPs. 896 The reflective telemetry enabled by the iFIT framework allows 897 numerous new applications suitable for future network operation 898 architecture. 900 4.1. Example: Intelligent Multipoint Performance Monitoring 902 [I-D.ietf-ippm-multipoint-alt-mark] describes an intelligent 903 performance management based on the network condition. The idea is 904 to split the monitoring network into clusters. The cluster partition 905 that can be applied to every type of network graph and the 906 possibility to combine clusters at different levels enable the so- 907 called Network Zooming. It allows a controller to calibrate the 908 network telemetry, so that it can start without examining in depth 909 and monitor the network as a whole. In case of necessity (packet 910 loss or too high delay), an immediate detailed analysis can be 911 reconfigured. In particular, the controller, that is aware of the 912 network topology, can set up the most suited cluster partition by 913 changing the traffic filter or activate new measurement points and 914 the problem can be localized with a step-by-step process. 916 An iFIT application on top of the controllers can manage such 917 mechanism and iFIT's architecture allows its dynamic and reflective 918 operation. 920 4.2. Example: Intent-based Network Monitoring 922 User Intents 923 | 924 V Per-packet 925 +------------+ Telemetry 926 ACL | | Data 927 +--------+ Controller |<--------+ 928 | | | | 929 | +--+---------+ | 930 | | ^ | 931 | |DNPs |Network | 932 | | |Information| 933 | V | | 934 +------+-------------------+-----------+---+ 935 | | | 936 | V +------+ | 937 | +-------+ +------+| | 938 | | iFIT | iFIT Domain +------+|| | 939 | | Head | |iFIT ||+ | 940 | | Node | |Nodes |+ | 941 | +-------+ +------+ | 942 +------------------------------------------+ 944 Figure 9: Intent-based Monitoring 946 In this example, a user can express high level intents for network 947 monitoring. The controller translates an intent and configure the 948 corresponding DNPs in iFIT nodes which collect necessary network 949 information. Based on the real-time information feedback, the 950 controller runs a local algorithm to determine the suspicious flows. 951 It then deploys ACLs to the iFIT head node to initiate the high 952 precision per-packet on-path telemetry for these flows. 954 5. Standard Status and Gaps 956 A complete iFIT solution needs standard interfaces for configuration 957 and data extraction, and standard encapsulation on various transport 958 protocols. It may also need standard API and primitives for 959 application programming and deployment. The draft 960 [I-D.brockners-opsawg-ioam-deployment] summarizes some current 961 proposals on encapsulation and data export for IOAM. These works 962 should be extended or modified to support other types of on-path 963 telemetry techniques and other transport protocols. The high level 964 iFIT framework helps to develop coherent and universal standard 965 encapsulation and data export approaches. 967 In addition, standard approaches for function configuration, 968 capability query and advertisement, either in a centralized fashion 969 or a distributed fashion, are still immature. The draft 970 [I-D.zhou-ippm-ioam-yang] provides the YANG model for IOAM 971 configuration. Similar models needs to be defined for other 972 techniques. It is helpful to provide standard approaches for 973 distributed configuration in various network environments. 975 To realize the potential of iFIT, programming and deploying DNPs are 976 important. Currently some related works such as 977 [I-D.wwx-netmod-event-yang] and [I-D.bwd-netmod-eca-framework] have 978 proposed to use YANG model to define the smart policies which can be 979 used to implement DNPs. In the future, other approaches for hardware 980 and software-based functions can be development to enhance the 981 programmability and flexibility. 983 6. Summary 985 iFIT is a high level and open framework for applying on-path 986 telemetry techniques. Combining with algorithmic and architectural 987 schemes that fit into the framework components, iFIT enables a 988 practical telemetry solution based on two basic on-path traffic data 989 collection modes: passport and postcard. 991 The operation of iFIT differs from both active OAM and passive OAM as 992 defined in [RFC7799]. It does not generate any active probe packets 993 or passively observe unmodified user packets. Instead, it modifies 994 selected user packets to collect useful information about them. 995 Therefore, the iFIT operation can be categorized as the hybrid OAM 996 type I mode per [RFC7799], which can provide more flexible and 997 accurate network monitoring and measurement. 999 iFIT addresses the key challenges for operators to deploy a complete 1000 on-path telemetry solution. However, as a reference and open 1001 framework, iFIT only describes the basic functions of each identified 1002 component and suggests possible applications. It has no intention of 1003 specifying the implementation of the components and the interfaces 1004 between the components. The compliance of iFIT framework is by no 1005 means mandatory either. Instead, this informational document aims to 1006 clarify the problem domain, and summarize the best practices and 1007 sensible system design considerations. The iFIT framework can guide 1008 the analysis of the current standard status and gaps, and motivate 1009 new works to complete the ecosystem. It also helps to inspire 1010 innovative data-plane reflective telemetry applications supporting 1011 advanced network operations. 1013 Having a framework covering a class of related techniques also 1014 promotes a holistic approach for standard development and helps to 1015 avoid duplicated efforts and piecemeal solutions that only focus on a 1016 specific technique while omitting the compatibility and extensibility 1017 issues. To foster a healthy ecosystem for network telemetry, we 1018 consider this essential. 1020 7. Security Considerations 1022 In addition to the specific security issues discussed in each 1023 individual document on on-path telemetry, this document considers the 1024 overall security issues at the iFIT system level. This should serve 1025 as a guide to the iFIT application developers and users. 1027 8. IANA Considerations 1029 This document includes no request to IANA. 1031 9. Contributors 1033 Other major contributors of this document include Giuseppe Fioccola, 1034 Daniel King, Zhenqiang Li, Zhenbin Li, Tianran Zhou, and James 1035 Guichard. 1037 10. Acknowledgments 1039 We thank Diego Lopez, Shwetha Bhandari, Joe Clarke, Adrian Farrel, 1040 Frank Brockners, Al Morton, Alex Clemm for their constructive 1041 suggestions for improving this document. 1043 11. References 1045 11.1. Normative References 1047 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1048 Requirement Levels", BCP 14, RFC 2119, 1049 DOI 10.17487/RFC2119, March 1997, 1050 . 1052 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1053 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1054 May 2016, . 1056 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1057 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1058 May 2017, . 1060 11.2. Informative References 1062 [CMSketch] 1063 Cormode, G. and S. Muthukrishnan, "An improved data stream 1064 summary: the count-min sketch and its applications", 2005, 1065 . 1067 [I-D.brockners-opsawg-ioam-deployment] 1068 Brockners, F., Bhandari, S., and d. 1069 daniel.bernier@bell.ca, "In-situ OAM Deployment", draft- 1070 brockners-opsawg-ioam-deployment-00 (work in progress), 1071 October 2019. 1073 [I-D.bwd-netmod-eca-framework] 1074 Boucadair, M., WU, Q., Wang, Z., King, D., and C. Xie, 1075 "Framework for Use of ECA (Event Condition Action) in 1076 Network Self Management", draft-bwd-netmod-eca- 1077 framework-00 (work in progress), November 2019. 1079 [I-D.herbert-ipv4-eh] 1080 Herbert, T., "IPv4 Extension Headers and Flow Label", 1081 draft-herbert-ipv4-eh-01 (work in progress), May 2019. 1083 [I-D.ietf-ippm-ioam-data] 1084 Brockners, F., Bhandari, S., Pignataro, C., Gredler, H., 1085 Leddy, J., Youell, S., Mizrahi, T., Mozes, D., Lapukhov, 1086 P., remy@barefootnetworks.com, r., daniel.bernier@bell.ca, 1087 d., and J. Lemon, "Data Fields for In-situ OAM", draft- 1088 ietf-ippm-ioam-data-08 (work in progress), October 2019. 1090 [I-D.ietf-ippm-multipoint-alt-mark] 1091 Fioccola, G., Cociglio, M., Sapio, A., and R. Sisto, 1092 "Multipoint Alternate Marking method for passive and 1093 hybrid performance monitoring", draft-ietf-ippm- 1094 multipoint-alt-mark-03 (work in progress), November 2019. 1096 [I-D.ietf-opsawg-ntf] 1097 Song, H., Qin, F., Martinez-Julia, P., Ciavaglia, L., and 1098 A. Wang, "Network Telemetry Framework", draft-ietf-opsawg- 1099 ntf-02 (work in progress), October 2019. 1101 [I-D.ioamteam-ippm-ioam-direct-export] 1102 Song, H., Gafni, B., Zhou, T., Li, Z., Brockners, F., 1103 Bhandari, S., Sivakolundu, R., and T. Mizrahi, "In-situ 1104 OAM Direct Exporting", draft-ioamteam-ippm-ioam-direct- 1105 export-00 (work in progress), October 2019. 1107 [I-D.mirsky-ippm-hybrid-two-step] 1108 Mirsky, G., Lingqiang, W., and G. Zhui, "Hybrid Two-Step 1109 Performance Measurement Method", draft-mirsky-ippm-hybrid- 1110 two-step-04 (work in progress), October 2019. 1112 [I-D.song-ippm-ioam-tunnel-mode] 1113 Song, H., Li, Z., Zhou, T., and Z. Wang, "In-situ OAM 1114 Processing in Tunnels", draft-song-ippm-ioam-tunnel- 1115 mode-00 (work in progress), June 2018. 1117 [I-D.song-ippm-postcard-based-telemetry] 1118 Song, H., Zhou, T., Li, Z., Shin, J., and K. Lee, 1119 "Postcard-based On-Path Flow Data Telemetry", draft-song- 1120 ippm-postcard-based-telemetry-06 (work in progress), 1121 October 2019. 1123 [I-D.song-mpls-extension-header] 1124 Song, H., Li, Z., Zhou, T., and L. Andersson, "MPLS 1125 Extension Header", draft-song-mpls-extension-header-02 1126 (work in progress), February 2019. 1128 [I-D.song-multicast-telemetry] 1129 Song, H., McBride, M., and G. Mirsky, "Requirement and 1130 Solution for Multicast Traffic Telemetry", draft-song- 1131 multicast-telemetry-01 (work in progress), November 2019. 1133 [I-D.wwx-netmod-event-yang] 1134 Wang, Z., WU, Q., Bryskin, I., Liu, X., and B. Claise, "A 1135 YANG Data model for ECA Policy Management", draft-wwx- 1136 netmod-event-yang-06 (work in progress), December 2019. 1138 [I-D.zhou-ippm-enhanced-alternate-marking] 1139 Zhou, T., Fioccola, G., Li, Z., Lee, S., and M. Cociglio, 1140 "Enhanced Alternate Marking Method", draft-zhou-ippm- 1141 enhanced-alternate-marking-04 (work in progress), October 1142 2019. 1144 [I-D.zhou-ippm-ioam-yang] 1145 Zhou, T., Guichard, J., Brockners, F., and S. Raghavan, "A 1146 YANG Data Model for In-Situ OAM", draft-zhou-ippm-ioam- 1147 yang-04 (work in progress), June 2019. 1149 [passport-postcard] 1150 Handigol, N., Heller, B., Jeyakumar, V., Mazieres, D., and 1151 N. McKeown, "Where is the debugger for my software-defined 1152 network?", 2012, 1153 . 1155 [RFC2113] Katz, D., "IP Router Alert Option", RFC 2113, 1156 DOI 10.17487/RFC2113, February 1997, 1157 . 1159 [RFC7011] Claise, B., Ed., Trammell, B., Ed., and P. Aitken, 1160 "Specification of the IP Flow Information Export (IPFIX) 1161 Protocol for the Exchange of Flow Information", STD 77, 1162 RFC 7011, DOI 10.17487/RFC7011, September 2013, 1163 . 1165 11.3. URIs 1167 [1] https://developers.google.com/protocol-buffers/ 1169 Authors' Addresses 1171 Haoyu Song (editor) 1172 Futurewei 1173 2330 Central Expressway 1174 Santa Clara 1175 USA 1177 Email: haoyu.song@futurewei.com 1178 Fengwei Qin 1179 China Mobile 1180 No. 32 Xuanwumenxi Ave., Xicheng District 1181 Beijing, 100032 1182 P.R. China 1184 Email: qinfengwei@chinamobile.com 1186 Huanan Chen 1187 China Telecom 1188 P. R. China 1190 Email: chenhuan6@chinatelecom.cn 1192 Jaehwan Jin 1193 LG U+ 1194 South Korea 1196 Email: daenamu1@lguplus.co.kr 1198 Jongyoon Shin 1199 SK Telecom 1200 South Korea 1202 Email: jongyoon.shin@sk.com