idnits 2.17.1 draft-spiegel-ippm-ioam-rawexport-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 9, 2020) is 1502 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-17) exists of draft-ietf-ippm-ioam-data-09 == Outdated reference: A later version (-05) exists of draft-brockners-ippm-ioam-geneve-03 == Outdated reference: A later version (-05) exists of draft-brockners-ippm-ioam-vxlan-gpe-03 == Outdated reference: A later version (-12) exists of draft-ietf-ippm-ioam-ipv6-options-01 == Outdated reference: A later version (-13) exists of draft-ietf-sfc-ioam-nsh-02 == Outdated reference: A later version (-05) exists of draft-weis-ippm-ioam-eth-02 -- Obsolete informational reference (is this intentional?): RFC 5226 (Obsoleted by RFC 8126) Summary: 0 errors (**), 0 flaws (~~), 7 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 ippm M. Spiegel 3 Internet-Draft Intel 4 Intended status: Informational F. Brockners 5 Expires: September 10, 2020 S. Bhandari 6 R. Sivakolundu 7 Cisco 8 March 9, 2020 10 In-situ OAM raw data export with IPFIX 11 draft-spiegel-ippm-ioam-rawexport-03 13 Abstract 15 In-situ Operations, Administration, and Maintenance (IOAM) records 16 operational and telemetry information in the packet while the packet 17 traverses a path between two points in the network. This document 18 discusses how In-situ Operations, Administration, and Maintenance 19 (IOAM) information can be exported in raw, i.e. uninterpreted, format 20 from network devices to systems, such as monitoring or analytics 21 systems using IPFIX. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at https://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on September 10, 2020. 40 Copyright Notice 42 Copyright (c) 2020 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (https://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 58 1.1. Requirements . . . . . . . . . . . . . . . . . . . . . . 4 59 1.2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 5 60 2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 5 61 3. IPFIX for IOAM raw data export . . . . . . . . . . . . . . . 6 62 3.1. Key IPFIX information elements leveraged for IOAM raw 63 data export . . . . . . . . . . . . . . . . . . . . . . . 6 64 3.2. New IPFIX information elements leveraged for IOAM raw 65 data export . . . . . . . . . . . . . . . . . . . . . . . 7 66 3.2.1. ioamReportFlags . . . . . . . . . . . . . . . . . . . 7 67 3.2.2. ioamEncapsulationType . . . . . . . . . . . . . . . . 7 68 3.2.3. ioamPreallocatedTraceData . . . . . . . . . . . . . . 8 69 3.2.4. ioamIncrementalTraceData . . . . . . . . . . . . . . 9 70 3.2.5. ioamE2EData . . . . . . . . . . . . . . . . . . . . . 9 71 3.2.6. ioamPOTData . . . . . . . . . . . . . . . . . . . . . 10 72 3.2.7. ipHeaderPacketSectionWithPadding . . . . . . . . . . 10 73 3.2.8. ethernetFrameSection . . . . . . . . . . . . . . . . 11 74 4. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 12 75 4.1. Fixed Length IP Packet . . . . . . . . . . . . . . . . . 12 76 4.2. Variable Length IP Packet (length < 255) . . . . . . . . 13 77 4.3. Variable Length IP Packet (length > 255) . . . . . . . . 14 78 4.4. Variable Length ETHERNET Packet (length < 255) . . . . . 15 79 4.5. Variable Length IP Packet with Fixed Length IOAM 80 Incremental Trace Data . . . . . . . . . . . . . . . . . 16 81 4.6. Variable Length IP Packet with Variable Length IOAM 82 Incremental Trace Data . . . . . . . . . . . . . . . . . 17 83 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 84 6. Manageability Considerations . . . . . . . . . . . . . . . . 19 85 7. Security Considerations . . . . . . . . . . . . . . . . . . . 19 86 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19 87 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 19 88 9.1. Normative References . . . . . . . . . . . . . . . . . . 19 89 9.2. Informative References . . . . . . . . . . . . . . . . . 20 90 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 21 92 1. Introduction 94 In-situ Operations, Administration, and Maintenance (IOAM) records 95 operational and telemetry information in the packet while the packet 96 traverses a path between two points in the network. IOAM data fields 97 are defined in [I-D.ietf-ippm-ioam-data]. This document discusses 98 how In-situ Operations, Administration, and Maintenance (IOAM) 99 information can be exported in raw format, i.e. uninterpreted format, 100 from network devices to systems, such as monitoring or analytics 101 systems using IPFIX [RFC7011]. 103 "Raw export of IOAM data" refers to a mode of operation where a node 104 exports the IOAM data as it is received in the packet. The exporting 105 node neither interprets, aggregates nor reformats the IOAM data 106 before it is exported. Raw export of IOAM data is to support an 107 operational model where the processing and interpretation of IOAM 108 data is decoupled from the operation of encapsulating/updating/ 109 decapsulating IOAM data, which is also referred to as IOAM data-plane 110 operation. The figure below shows the separation of concerns for 111 IOAM export: Exporting IOAM data is performed by the "IOAM node" 112 which performs IOAM data-plane operation, whereas the interpretation 113 of IOAM data is performed by the IOAM data processing system. The 114 separation of concerns is to off-load interpretation, aggregation and 115 formatting of IOAM data from the node which performs data-plane 116 operations. In other words, a node which is focused on data-plane 117 operations, i.e. forwarding of packets and handling IOAM data will 118 not be tasked to also interpret the IOAM data, but can leave this 119 task to another system. Note that for scalability reasons, a single 120 IOAM node could choose to export IOAM data to several IOAM data 121 processing systems. 123 +-------------+ 124 | Monitoring/ | 125 | Analytics | 126 | system | 127 +-------------+ 128 ^ 129 | Processed/interpreted/aggregated 130 | IOAM data 131 | 132 +-------------+ 133 | IOAM data | 134 | processing | 135 | system | 136 +-------------+ 137 ^ 138 | Raw export of 139 | IOAM data 140 | 141 +------------+ 142 | | 143 ..--| IOAM node |--.. 144 | | 145 +------------+ 147 IOAM node: IOAM encapsulating, IOAM decapsulating or 148 IOAM transit node. 150 IOAM data processing system: System that receives raw IOAM data 151 and provides for formatting, aggregation and 152 interpretation of the IOAM data. 154 Monitoring/Analytics system: System that receives telemetry and 155 other operational information from a variety of 156 sources and provides for correlation and 157 interpretation of the data received. 159 Raw export of IOAM data is typically generated by network devices at 160 the edges of the network. Deployment and use-case dependent, such as 161 in cases where the operator is interested in dropped packets, raw 162 export of IOAM data may be generated by IOAM transit nodes. 164 1.1. Requirements 166 Requirements for raw export of IOAM data: 168 o Export all IOAM information contained in a packet. 170 o Export a specific IOAM data type - Incremental Trace type, 171 Preallocated Trace type, Proof of Transit type, Edge to Edge type. 173 o Support coalescing of the IOAM data from multiple packets into a 174 single raw export packet. 176 o Support export of additional parts of the packet, other than the 177 IOAM data as part of the raw export. This could be parts of the 178 packet header and/or parts of the packet payload. This additional 179 information provides context to the IOAM data (e.g. to be used for 180 flow identification) and is to enable the IOAM data processing 181 system to perform further analysis on the received data. 183 o Report the reason why IOAM data was exported. The "reason for 184 export" is to complement the IOAM data retrieved from the packet. 185 For example, if a packet was dropped by a node due to congestion, 186 it could be helpful to export the IOAM data of this dropped packet 187 along with an indication that the packet that the IOAM data 188 belongs to was dropped due to congestion. 190 1.2. Scope 192 This document discusses raw export of IOAM data using IPFIX. 194 The following is considered out of scope for this document: 196 o Protocols other than IPFIX for raw export of IOAM data. 198 o Interpretation or aggregation of IOAM data prior to exporting. 200 o Configuration of network devices so that they can determine when 201 to generate IOAM reports, and what information to include in those 202 reports. 204 o Events that trigger generation of IOAM reports. 206 o Selection of particular destinations within distributed telemetry 207 monitoring systems, to which IOAM reports will be sent. 209 o Export format for flow statistics or processed/interpreted/ 210 aggregated IOAM data. 212 2. Conventions 214 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 215 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 216 document are to be interpreted as described in [RFC2119]. 218 Abbreviations used in this document: 220 E2E: Edge to Edge 222 IOAM: In-situ Operations, Administration, and Maintenance 224 MTU: Maximum Transmit Unit 226 OAM: Operations, Administration, and Maintenance 228 POT: Proof of Transit 230 3. IPFIX for IOAM raw data export 232 IPFIX, being a generic export protocol, can export any Information 233 Elements as long as they are described in the information model. The 234 IPFIX protocol is well suited for and is defined as the protocol for 235 exporting packet samples in [RFC5476]. 237 IPFIX/PSAMP [RFC7011], [RFC5476] already define many of the 238 information elements needed for exporting sections of packets needed 239 for deriving context and raw IOAM data export. This document 240 specifies extensions of the IPFIX information model for meeting the 241 requirements in Section 1.1. 243 3.1. Key IPFIX information elements leveraged for IOAM raw data export 245 The existing IPFIX Information Elements that are required for IOAM 246 raw data export are listed here. Their details are available in 247 IANA's IPFIX registry [IANA-IPFIX]. 249 The existing IPFIX Information Elements used to carry the sections of 250 the packets including IOAM data within it are as follows: 252 313 - ipHeaderPacketSection 254 315 - dataLinkFrameSection 256 The following Information Elements will be used to provide context to 257 the ipHeaderPacketSection and dataLinkFrameSection as described in 258 [IANA-IPFIX]: 260 408 - dataLinkFrameType 262 409 - sectionOffset 264 410 - sectionExportedOctets 265 The following Information Element will be used to provide forwarding 266 status of the flow and any attached reasons. 268 89 - forwardingStatus 270 3.2. New IPFIX information elements leveraged for IOAM raw data export 272 IOAM data raw export using IPFIX requires a set of new information 273 elements which are described in this section. 275 3.2.1. ioamReportFlags 277 Description: 279 This Information Element describes properties associated with an IOAM 280 report. 282 The ioamReportFlags data type is an 8-bit field. The following bits 283 are defined here: 285 Bit 0 Dropped Association - Dropped packet of interest. 287 Bit 1 Congested Queue Association - Indicates the presence of 288 congestion on a monitored queue. 290 Bit 2 Tracked Flow Association - Matched a flow of interest. 292 Bit 3-7 Reserved 294 IANA is requested to create a new subregistry for IOAM Report Flags 295 and fill it with the initial list from the description. New 296 assignments for IOAM Encapsulation Types are administered by IANA 297 through Expert Review [RFC5226] i.e., review by one of a group of 298 experts designated by an IETF Area Director. 300 Abstract Data Type: unsigned8 302 Data Type Semantics: flags 304 ElementId: TBD1 306 Status: current 308 3.2.2. ioamEncapsulationType 310 Description: 312 This Information Element specifies the type of encapsulation to 313 interpret ioamPreallocatedTraceHeader, ioamIncrementalTraceHeader, 314 ioamE2EHeader, ioamPOTHeader. 316 The following ioamEncapsulationType values are defined here: 318 0 None : IOAM data follows format defined in 319 [I-D.ietf-ippm-ioam-data] 321 1 GRE : IOAM data follows format defined in [I-D.weis-ippm-ioam-eth] 323 2 IPv6 : IOAM data follows format defined in 324 [I-D.ietf-ippm-ioam-ipv6-options] 326 3 VXLAN-GPE : IOAM data follows format defined in 327 [I-D.brockners-ippm-ioam-vxlan-gpe] 329 4 GENEVE Option: IOAM data follows format defined in 330 [I-D.brockners-ippm-ioam-geneve] 332 5 GENEVE Next Protocol: IOAM data follows format defined in 333 [I-D.weis-ippm-ioam-eth] 335 6 NSH : IOAM data follows format defined in [I-D.ietf-sfc-ioam-nsh] 337 IANA is requested to create a new subregistry for IOAM Encapsulation 338 Types and fill it with the initial list from the description. New 339 assignments for IOAM Encapsulation Types are administered by IANA 340 through Expert Review [RFC5226] i.e., review by one of a group of 341 experts designated by an IETF Area Director. 343 Abstract Data Type: unsigned8 345 Data Type Semantics: identifier 347 ElementId: TBD2 349 Status: current 351 3.2.3. ioamPreallocatedTraceData 353 Description: 355 This Information Element carries n octets of IOAM Preallocated Trace 356 data defined in [I-D.ietf-ippm-ioam-data]. 358 The format of the data is determined by the ioamEncapsulationType 359 information element, if present. When the ioamEncapsulationType 360 information element is present and has a value other than "None", and 361 with sufficient length, this element may also report octets from 362 subsequent headers and payload. If no ioamEncapsulationType 363 information element is present, then the encapsulation type shall be 364 assumed to be "None" and this information element only contains 365 octets from the IOAM Preallocated Trace Option. 367 Abstract Data Type: octetArray 369 ElementId: TBD3 371 Status: current 373 3.2.4. ioamIncrementalTraceData 375 Description: 377 This Information Element carries n octets of IOAM Incremental Trace 378 data defined in [I-D.ietf-ippm-ioam-data]. 380 The format of the data is determined by the ioamEncapsulationType 381 information element, if present. When the ioamEncapsulationType 382 information element is present and has a value other than "None", and 383 with sufficient length, this element may also report octets from 384 subsequent headers and payload. If no ioamEncapsulationType 385 information element is present, then the encapsulation type shall be 386 assumed to be "None" and this information element only contains 387 octets from the IOAM Incremental Trace Option. 389 Abstract Data Type: octetArray 391 ElementId: TBD4 393 Status: current 395 3.2.5. ioamE2EData 397 Description: 399 This Information Element carries n octets of IOAM E2E data defined in 400 [I-D.ietf-ippm-ioam-data]. 402 The format of the data is determined by the ioamEncapsulationType 403 information element, if present. When the ioamEncapsulationType 404 information element is present and has a value other than "None", and 405 with sufficient length, this element may also report octets from 406 subsequent headers and payload. If no ioamEncapsulationType 407 information element is present, then the encapsulation type shall be 408 assumed to be "None" and this information element only contains 409 octets from the IOAM Edge-to-Edge Option. 411 Abstract Data Type: octetArray 413 ElementId: TBD5 415 Status: current 417 3.2.6. ioamPOTData 419 Description: 421 This Information Element carries n octets of IOAM POT data defined in 422 [I-D.ietf-ippm-ioam-data]. 424 The format of the data is determined by the ioamEncapsulationType 425 information element, if present. When the ioamEncapsulationType 426 information element is present and has a value other than "None", and 427 with sufficient length, this element may also report octets from 428 subsequent headers and payload. If no ioamEncapsulationType 429 information element is present, then the encapsulation type shall be 430 assumed to be "None" and this information element only contains 431 octets from the IOAM Proof of Transit Option. 433 Abstract Data Type: octetArray 435 ElementId: TBD6 437 Status: current 439 3.2.7. ipHeaderPacketSectionWithPadding 441 Description: 443 This Information Element carries a series of n octets from the IP 444 header of a sampled packet, starting sectionOffset octets into the IP 445 header. 447 However, if no sectionOffset field corresponding to this Information 448 Element is present, then a sectionOffset of zero applies, and the 449 octets MUST be from the start of the IP header. 451 With sufficient length, this element also reports octets from the IP 452 payload. However, full packet capture of arbitrary packet streams is 453 explicitly out of scope per the Security Considerations sections of 454 [RFC5477] and [RFC2804]. 456 When this Information Element has a fixed length, this MAY include 457 padding octets that are used to fill out that fixed length. 459 When this information element has a variable length, the variable 460 length MAY include up to 3 octets of padding, used to preserve 461 4-octet alignment of subequent Information Elements or subsequent 462 records within the same set. 464 In either case of fixed or variable length, the amount of populated 465 octets MAY be specified in the sectionExportedOctets field 466 corresponding to this Information Element, in which case the 467 remainder (if any) MUST be padding. If there is no 468 sectionExportedOctets field corresponding to this Information 469 Element, then all octets MUST be populated unless the total length of 470 the IP packet is less than the fixed length of this Information 471 Element, in which case the remainder MUST be padding. 473 Abstract Data Type: octetArray 475 ElementId: TBD7 477 Status: current 479 3.2.8. ethernetFrameSection 481 Description: 483 This Information Element carries a series of n octets from the IEEE 484 802.3 Ethernet frame of a sampled packet, starting after the preamble 485 and start frame delimiter (SFD), plus sectionOffset octets into the 486 frame if there is a sectionOffset field corresponding to this 487 Information Element. 489 With sufficient length, this element also reports octets from the 490 Ethernet payload. However, full packet capture of arbitrary packet 491 streams is explicitly out of scope per the Security Considerations 492 sections of [RFC5477] and [RFC2804]. 494 When this Information Element has a fixed length, this MAY include 495 padding octets that are used to fill out that fixed length. 497 When this information element has a variable length, the variable 498 length MAY include up to 3 octets of padding, used to preserve 499 4-octet alignment of subequent Information Elements or subsequent 500 records within the same set. 502 In either case of fixed or variable length, the amount of populated 503 octets MAY be specified in the sectionExportedOctets field 504 corresponding to this Information Element, in which case the 505 remainder (if any) MUST be padding. If there is no 506 sectionExportedOctets field corresponding to this Information 507 Element, then all octets MUST be populated unless the total length of 508 the Ethernet frame is less than the fixed length of this Information 509 Element, in which case the remainder MUST be padding. 511 Abstract Data Type: octetArray 513 ElementId: TBD8 515 Status: current 517 4. Examples 519 This section shows a set of examples of how IOAM information along 520 with other parts of the packet can be carried using IPFIX. 522 4.1. Fixed Length IP Packet 524 This example shows a fixed length IP packet. IOAM data is part of 525 the ipHeaderPacketSection. 527 0 1 2 3 528 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 529 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--- 530 | Version Number | Length | | 531 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 532 | Export Time (seconds) |IPFIX 533 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message 534 | Sequence Number |Header 535 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 536 | Observation Domain ID | | 537 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 538 | Set ID ( = Template ID) | Length |SetHdr 539 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 540 |ioamReportFlags| fwdingStatus | sectionExportedOctets | | 541 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 542 | ipHeaderPacketSection (start) | | 543 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1 544 | ... | | 545 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 546 | ipHeaderPacketSection (end) | | 547 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 548 |ioamReportFlags| fwdingStatus | sectionExportedOctets | | 549 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 550 | ipHeaderPacketSection (start) | | 551 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record2 552 | ... | | 553 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 554 | ipHeaderPacketSection (end) | | 555 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 557 4.2. Variable Length IP Packet (length < 255) 559 This examples shows a variable length IP packet, with length < 255 560 bytes. IOAM data is part of the ipHeaderPacketSectionWithPadding. 562 0 1 2 3 563 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 564 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 565 | Version Number | Length | | 566 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 567 | Export Time (seconds) |IPFIX 568 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message 569 | Sequence Number |Header 570 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 571 | Observation Domain ID | | 572 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 573 | Set ID ( = Template ID) | Length |SetHdr 574 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 575 |ioamReportFlags| fwdingStatus | paddingOctets | Length (< 255)| | 576 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 577 | ipHeaderPacketSectionWithPadding (start) | | 578 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1 579 | ... | | 580 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 581 | ipHeaderPacketSectionWithPadding (end) | | 582 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 583 |ioamReportFlags| fwdingStatus | paddingOctets | Length (< 255)| | 584 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 585 | ipHeaderPacketSectionWithPadding (start) | | 586 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record2 587 | ... | | 588 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 589 | ipHeaderPacketSectionWithPadding (end) | | 590 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 592 4.3. Variable Length IP Packet (length > 255) 594 This examples shows a variable length IP packet, with length > 255 595 bytes. IOAM data is part of the ipHeaderPacketSectionWithPadding. 597 0 1 2 3 598 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 599 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 600 | Version Number | Length | | 601 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 602 | Export Time (seconds) |IPFIX 603 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message 604 | Sequence Number |Header 605 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 606 | Observation Domain ID | | 607 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 608 | Set ID ( = Template ID) | Length |SetHdr 609 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 610 |ioamReportFlags| fwdingStatus | paddingOctets | | 611 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 612 | paddingOctets | 255 | Length (0 to 65535) | | 613 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 614 | ipHeaderPacketSectionWithPadding (start) |Record1 615 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 616 | ... | | 617 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 618 | ipHeaderPacketSectionWithPadding (end) | | 619 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 621 4.4. Variable Length ETHERNET Packet (length < 255) 623 This examples shows a variable length Ethernet packet, with length < 624 255 bytes. IOAM data is part of the ethernetFrameSection. 626 0 1 2 3 627 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 628 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 629 | Version Number | Length | | 630 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 631 | Export Time (seconds) |IPFIX 632 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message 633 | Sequence Number |Header 634 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 635 | Observation Domain ID | | 636 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 637 | Set ID ( = Template ID) | Length |SetHdr 638 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 639 |ioamReportFlags| fwdingStatus | paddingOctets | Length (< 255)| | 640 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 641 | ethernetFrameSection (start) | | 642 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1 643 | ... | | 644 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 645 | ethernetFrameSection (end) | | 646 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 648 4.5. Variable Length IP Packet with Fixed Length IOAM Incremental Trace 649 Data 651 This examples shows a variable length IP packet with length < 255 652 bytes and fixed length ioamIncrementalTraceData carried separately. 654 0 1 2 3 655 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 656 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 657 | Version Number | Length | | 658 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 659 | Export Time (seconds) |IPFIX 660 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message 661 | Sequence Number |Header 662 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 663 | Observation Domain ID | | 664 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 665 | Set ID ( = Template ID) | Length |SetHdr 666 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 667 | ioamIncrementalTraceData (start) | | 668 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 669 | ... | | 670 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 671 | ioamIncrementalTraceData (end) | | 672 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1 673 |ioamReportFlags| fwdingStatus |ioamEncapType | Length (< 255)| | 674 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 675 | ipHeaderPacketSectionWithPadding (start) | | 676 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 677 | ... | | 678 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 679 | ipHeaderPacketSectionWithPadding (end) | | 680 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 682 4.6. Variable Length IP Packet with Variable Length IOAM Incremental 683 Trace Data 685 This examples shows a variable length IP packet with length < 255 686 bytes and variable length ioamIncrementalTraceData with length < 255 687 bytes carried separately. 689 0 1 2 3 690 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 691 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 692 | Version Number | Length | | 693 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 694 | Export Time (seconds) |IPFIX 695 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message 696 | Sequence Number |Header 697 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 698 | Observation Domain ID | | 699 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 700 | Set ID ( = Template ID) | Length |SetHdr 701 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 702 |ioamReportFlags| fwdingStatus |ioamEncapType | Length (< 255)| | 703 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 704 | ioamIncrementalTraceData (start) | | 705 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 706 | ... | | 707 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 708 | ioamIncrementalTraceData (end) | | 709 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1 710 | paddingOctets | Length (< 255)| | 711 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 712 | ipHeaderPacketSectionWithPadding (start) | | 713 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 714 | ... | | 715 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 716 | ipHeaderPacketSectionWithPadding (end) | | 717 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 719 5. IANA Considerations 721 IANA is requested to allocate code points for the following 722 Information Elements in [IANA-IPFIX]: 724 TBD1 ioamReportFlags 726 TBD2 ioamEncapsulationType 728 TBD3 ioamPreallocatedTraceData 730 TBD4 ioamIncrementalTraceData 732 TBD5 ioamE2EData 734 TBD6 ioamPOTData 735 TBD7 ipHeaderPacketSectionWithPadding 737 TBD8 ethernetFrameSection 739 See Section 3.2 for further details. 741 IANA is requested to create subregistries for ioamReportFlags defined 742 in Section 3.2.1 and ioamEncapsulationType defined in Section 3.2.2. 744 6. Manageability Considerations 746 Manageability considerations will be addressed in a later version of 747 this document.. 749 7. Security Considerations 751 Security considerations will be addressed in a later version of this 752 document. 754 8. Acknowledgements 756 The authors would like to thank Barak Gafni, Tal Mizrahi, John Lemon, 757 and Aviv Kfir for their thoughts and comments on raw IOAM data 758 export. 760 9. References 762 9.1. Normative References 764 [I-D.ietf-ippm-ioam-data] 765 Brockners, F., Bhandari, S., Pignataro, C., Gredler, H., 766 Leddy, J., Youell, S., Mizrahi, T., Mozes, D., Lapukhov, 767 P., Chang, R., Bernier, D., and J. Lemon, "Data Fields for 768 In-situ OAM", draft-ietf-ippm-ioam-data-09 (work in 769 progress), March 2020. 771 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 772 Requirement Levels", BCP 14, RFC 2119, 773 DOI 10.17487/RFC2119, March 1997, 774 . 776 [RFC5476] Claise, B., Ed., Johnson, A., and J. Quittek, "Packet 777 Sampling (PSAMP) Protocol Specifications", RFC 5476, 778 DOI 10.17487/RFC5476, March 2009, 779 . 781 [RFC7011] Claise, B., Ed., Trammell, B., Ed., and P. Aitken, 782 "Specification of the IP Flow Information Export (IPFIX) 783 Protocol for the Exchange of Flow Information", STD 77, 784 RFC 7011, DOI 10.17487/RFC7011, September 2013, 785 . 787 9.2. Informative References 789 [I-D.brockners-ippm-ioam-geneve] 790 Brockners, F., Bhandari, S., Govindan, V., Pignataro, C., 791 Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Lapukhov, 792 P., Gafni, B., Kfir, A., and M. Spiegel, "Geneve 793 encapsulation for In-situ OAM Data", draft-brockners-ippm- 794 ioam-geneve-03 (work in progress), September 2019. 796 [I-D.brockners-ippm-ioam-vxlan-gpe] 797 Brockners, F., Bhandari, S., Govindan, V., Pignataro, C., 798 Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Kfir, A., 799 Gafni, B., Lapukhov, P., and M. Spiegel, "VXLAN-GPE 800 Encapsulation for In-situ OAM Data", draft-brockners-ippm- 801 ioam-vxlan-gpe-03 (work in progress), November 2019. 803 [I-D.ietf-ippm-ioam-ipv6-options] 804 Bhandari, S., Brockners, F., Pignataro, C., Gredler, H., 805 Leddy, J., Youell, S., Mizrahi, T., Kfir, A., Gafni, B., 806 Lapukhov, P., Spiegel, M., Krishnan, S., and R. Asati, 807 "In-situ OAM IPv6 Options", draft-ietf-ippm-ioam- 808 ipv6-options-01 (work in progress), March 2020. 810 [I-D.ietf-sfc-ioam-nsh] 811 Brockners, F. and S. Bhandari, "Network Service Header 812 (NSH) Encapsulation for In-situ OAM (IOAM) Data", draft- 813 ietf-sfc-ioam-nsh-02 (work in progress), September 2019. 815 [I-D.weis-ippm-ioam-eth] 816 Weis, B., Brockners, F., Hill, C., Bhandari, S., Govindan, 817 V., Pignataro, C., Gredler, H., Leddy, J., Youell, S., 818 Mizrahi, T., Kfir, A., Gafni, B., Lapukhov, P., and M. 819 Spiegel, "EtherType Protocol Identification of In-situ OAM 820 Data", draft-weis-ippm-ioam-eth-02 (work in progress), 821 September 2019. 823 [IANA-IPFIX] 824 "IP Flow Information Export (IPFIX) Entities", 825 . 827 [RFC2804] IAB and IESG, "IETF Policy on Wiretapping", RFC 2804, 828 DOI 10.17487/RFC2804, May 2000, 829 . 831 [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an 832 IANA Considerations Section in RFCs", RFC 5226, 833 DOI 10.17487/RFC5226, May 2008, 834 . 836 [RFC5477] Dietz, T., Claise, B., Aitken, P., Dressler, F., and G. 837 Carle, "Information Model for Packet Sampling Exports", 838 RFC 5477, DOI 10.17487/RFC5477, March 2009, 839 . 841 Authors' Addresses 843 Mickey Spiegel 844 Barefoot Networks, an Intel company 845 4750 Patrick Henry Drive 846 Santa Clara, CA 95054 847 US 849 Email: mickey.spiegel@intel.com 851 Frank Brockners 852 Cisco Systems, Inc. 853 Hansaallee 249, 3rd Floor 854 DUESSELDORF, NORDRHEIN-WESTFALEN 40549 855 Germany 857 Email: fbrockne@cisco.com 859 Shwetha Bhandari 860 Cisco Systems, Inc. 861 Cessna Business Park, Sarjapura Marathalli Outer Ring Road 862 Bangalore, KARNATAKA 560 087 863 India 865 Email: shwethab@cisco.com 866 Ramesh Sivakolundu 867 Cisco Systems, Inc. 868 170 West Tasman Dr. 869 SAN JOSE, CA 95134 870 U.S.A. 872 Email: sramesh@cisco.com