ippm M. Spiegel Internet-Draft Barefoot Networks Intended status: Informational F. Brockners Expires: January 9, 2020 S. Bhandari R. Sivakolundu Cisco July 8, 2019 In-situ OAM raw data export with IPFIX draft-spiegel-ippm-ioam-rawexport-02 Abstract In-situ Operations, Administration, and Maintenance (IOAM) records operational and telemetry information in the packet while the packet traverses a path between two points in the network. This document discusses how In-situ Operations, Administration, and Maintenance (IOAM) information can be exported in raw, i.e. uninterpreted, format from network devices to systems, such as monitoring or analytics systems using IPFIX. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on January 9, 2020. Copyright Notice Copyright (c) 2019 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents Spiegel, et al. Expires January 9, 2020 [Page 1] Internet-Draft IOAM raw data export July 2019 carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Requirements . . . . . . . . . . . . . . . . . . . . . . 4 1.2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 5 3. IPFIX for IOAM raw data export . . . . . . . . . . . . . . . 6 3.1. Key IPFIX information elements leveraged for IOAM raw data export . . . . . . . . . . . . . . . . . . . . . . . 6 3.2. New IPFIX information elements leveraged for IOAM raw data export . . . . . . . . . . . . . . . . . . . . . . . 7 3.2.1. ioamReportFlags . . . . . . . . . . . . . . . . . . . 7 3.2.2. ioamEncapsulationType . . . . . . . . . . . . . . . . 7 3.2.3. ioamPreallocatedTraceData . . . . . . . . . . . . . . 8 3.2.4. ioamIncrementalTraceData . . . . . . . . . . . . . . 9 3.2.5. ioamE2EData . . . . . . . . . . . . . . . . . . . . . 9 3.2.6. ioamPOTData . . . . . . . . . . . . . . . . . . . . . 10 3.2.7. ipHeaderPacketSectionWithPadding . . . . . . . . . . 10 3.2.8. ethernetFrameSection . . . . . . . . . . . . . . . . 11 4. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 12 4.1. Fixed Length IP Packet . . . . . . . . . . . . . . . . . 12 4.2. Variable Length IP Packet (length < 255) . . . . . . . . 13 4.3. Variable Length IP Packet (length > 255) . . . . . . . . 14 4.4. Variable Length ETHERNET Packet (length < 255) . . . . . 15 4.5. Variable Length IP Packet with Fixed Length IOAM Incremental Trace Data . . . . . . . . . . . . . . . . . 16 4.6. Variable Length IP Packet with Variable Length IOAM Incremental Trace Data . . . . . . . . . . . . . . . . . 17 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 6. Manageability Considerations . . . . . . . . . . . . . . . . 19 7. Security Considerations . . . . . . . . . . . . . . . . . . . 19 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 19 9.1. Normative References . . . . . . . . . . . . . . . . . . 19 9.2. Informative References . . . . . . . . . . . . . . . . . 20 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 21 1. Introduction In-situ Operations, Administration, and Maintenance (IOAM) records operational and telemetry information in the packet while the packet traverses a path between two points in the network. IOAM data fields Spiegel, et al. Expires January 9, 2020 [Page 2] Internet-Draft IOAM raw data export July 2019 are defined in [I-D.ietf-ippm-ioam-data]. This document discusses how In-situ Operations, Administration, and Maintenance (IOAM) information can be exported in raw format, i.e. uninterpreted format, from network devices to systems, such as monitoring or analytics systems using IPFIX [RFC7011]. "Raw export of IOAM data" refers to a mode of operation where a node exports the IOAM data as it is received in the packet. The exporting node neither interprets, aggregates nor reformats the IOAM data before it is exported. Raw export of IOAM data is to support an operational model where the processing and interpretation of IOAM data is decoupled from the operation of encapsulating/updating/ decapsulating IOAM data, which is also referred to as IOAM data-plane operation. The figure below shows the separation of concerns for IOAM export: Exporting IOAM data is performed by the "IOAM node" which performs IOAM data-plane operation, whereas the interpretation of IOAM data is performed by the IOAM data processing system. The separation of concerns is to off-load interpretation, aggregation and formatting of IOAM data from the node which performs data-plane operations. In other words, a node which is focused on data-plane operations, i.e. forwarding of packets and handling IOAM data will not be tasked to also interpret the IOAM data, but can leave this task to another system. Note that for scalability reasons, a single IOAM node could choose to export IOAM data to several IOAM data processing systems. Spiegel, et al. Expires January 9, 2020 [Page 3] Internet-Draft IOAM raw data export July 2019 +-------------+ | Monitoring/ | | Analytics | | system | +-------------+ ^ | Processed/interpreted/aggregated | IOAM data | +-------------+ | IOAM data | | processing | | system | +-------------+ ^ | Raw export of | IOAM data | +------------+ | | ..--| IOAM node |--.. | | +------------+ IOAM node: IOAM encapsulating, IOAM decapsulating or IOAM transit node. IOAM data processing system: System that receives raw IOAM data and provides for formatting, aggregation and interpretation of the IOAM data. Monitoring/Analytics system: System that receives telemetry and other operational information from a variety of sources and provides for correlation and interpretation of the data received. Raw export of IOAM data is typically generated by network devices at the edges of the network. Deployment and use-case dependent, such as in cases where the operator is interested in dropped packets, raw export of IOAM data may be generated by IOAM transit nodes. 1.1. Requirements Requirements for raw export of IOAM data: o Export all IOAM information contained in a packet. Spiegel, et al. Expires January 9, 2020 [Page 4] Internet-Draft IOAM raw data export July 2019 o Export a specific IOAM data type - Incremental Trace type, Preallocated Trace type, Proof of Transit type, Edge to Edge type. o Support coalescing of the IOAM data from multiple packets into a single raw export packet. o Support export of additional parts of the packet, other than the IOAM data as part of the raw export. This could be parts of the packet header and/or parts of the packet payload. This additional information provides context to the IOAM data (e.g. to be used for flow identification) and is to enable the IOAM data processing system to perform further analysis on the received data. o Report the reason why IOAM data was exported. The "reason for export" is to complement the IOAM data retrieved from the packet. For example, if a packet was dropped by a node due to congestion, it could be helpful to export the IOAM data of this dropped packet along with an indication that the packet that the IOAM data belongs to was dropped due to congestion. 1.2. Scope This document discusses raw export of IOAM data using IPFIX. The following is considered out of scope for this document: o Protocols other than IPFIX for raw export of IOAM data. o Interpretation or aggregation of IOAM data prior to exporting. o Configuration of network devices so that they can determine when to generate IOAM reports, and what information to include in those reports. o Events that trigger generation of IOAM reports. o Selection of particular destinations within distributed telemetry monitoring systems, to which IOAM reports will be sent. o Export format for flow statistics or processed/interpreted/ aggregated IOAM data. 2. Conventions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. Spiegel, et al. Expires January 9, 2020 [Page 5] Internet-Draft IOAM raw data export July 2019 Abbreviations used in this document: E2E: Edge to Edge IOAM: In-situ Operations, Administration, and Maintenance MTU: Maximum Transmit Unit OAM: Operations, Administration, and Maintenance POT: Proof of Transit 3. IPFIX for IOAM raw data export IPFIX, being a generic export protocol, can export any Information Elements as long as they are described in the information model. The IPFIX protocol is well suited for and is defined as the protocol for exporting packet samples in [RFC5476]. IPFIX/PSAMP [RFC7011], [RFC5476] already define many of the information elements needed for exporting sections of packets needed for deriving context and raw IOAM data export. This document specifies extensions of the IPFIX information model for meeting the requirements in Section 1.1. 3.1. Key IPFIX information elements leveraged for IOAM raw data export The existing IPFIX Information Elements that are required for IOAM raw data export are listed here. Their details are available in IANA's IPFIX registry [IANA-IPFIX]. The existing IPFIX Information Elements used to carry the sections of the packets including IOAM data within it are as follows: 313 - ipHeaderPacketSection 315 - dataLinkFrameSection The following Information Elements will be used to provide context to the ipHeaderPacketSection and dataLinkFrameSection as described in [IANA-IPFIX]: 408 - dataLinkFrameType 409 - sectionOffset 410 - sectionExportedOctets Spiegel, et al. Expires January 9, 2020 [Page 6] Internet-Draft IOAM raw data export July 2019 The following Information Element will be used to provide forwarding status of the flow and any attached reasons. 89 - forwardingStatus 3.2. New IPFIX information elements leveraged for IOAM raw data export IOAM data raw export using IPFIX requires a set of new information elements which are described in this section. 3.2.1. ioamReportFlags Description: This Information Element describes properties associated with an IOAM report. The ioamReportFlags data type is an 8-bit field. The following bits are defined here: Bit 0 Dropped Association - Dropped packet of interest. Bit 1 Congested Queue Association - Indicates the presence of congestion on a monitored queue. Bit 2 Tracked Flow Association - Matched a flow of interest. Bit 3-7 Reserved IANA is requested to create a new subregistry for IOAM Report Flags and fill it with the initial list from the description. New assignments for IOAM Encapsulation Types are administered by IANA through Expert Review [RFC5226] i.e., review by one of a group of experts designated by an IETF Area Director. Abstract Data Type: unsigned8 Data Type Semantics: flags ElementId: TBD1 Status: current 3.2.2. ioamEncapsulationType Description: Spiegel, et al. Expires January 9, 2020 [Page 7] Internet-Draft IOAM raw data export July 2019 This Information Element specifies the type of encapsulation to interpret ioamPreallocatedTraceHeader, ioamIncrementalTraceHeader, ioamE2EHeader, ioamPOTHeader. The following ioamEncapsulationType values are defined here: 0 None : IOAM data follows format defined in [I-D.ietf-ippm-ioam-data] 1 GRE : IOAM data follows format defined in [I-D.weis-ippm-ioam-eth] 2 IPv6 : IOAM data follows format defined in [I-D.ioametal-ippm-6man-ioam-ipv6-options] 3 VXLAN-GPE : IOAM data follows format defined in [I-D.brockners-ippm-ioam-vxlan-gpe] 4 GENEVE Option: IOAM data follows format defined in [I-D.brockners-ippm-ioam-geneve] 5 GENEVE Next Protocol: IOAM data follows format defined in [I-D.weis-ippm-ioam-eth] 6 NSH : IOAM data follows format defined in [I-D.ietf-sfc-ioam-nsh] IANA is requested to create a new subregistry for IOAM Encapsulation Types and fill it with the initial list from the description. New assignments for IOAM Encapsulation Types are administered by IANA through Expert Review [RFC5226] i.e., review by one of a group of experts designated by an IETF Area Director. Abstract Data Type: unsigned8 Data Type Semantics: identifier ElementId: TBD2 Status: current 3.2.3. ioamPreallocatedTraceData Description: This Information Element carries n octets of IOAM Preallocated Trace data defined in [I-D.ietf-ippm-ioam-data]. The format of the data is determined by the ioamEncapsulationType information element, if present. When the ioamEncapsulationType Spiegel, et al. Expires January 9, 2020 [Page 8] Internet-Draft IOAM raw data export July 2019 information element is present and has a value other than "None", and with sufficient length, this element may also report octets from subsequent headers and payload. If no ioamEncapsulationType information element is present, then the encapsulation type shall be assumed to be "None" and this information element only contains octets from the IOAM Preallocated Trace Option. Abstract Data Type: octetArray ElementId: TBD3 Status: current 3.2.4. ioamIncrementalTraceData Description: This Information Element carries n octets of IOAM Incremental Trace data defined in [I-D.ietf-ippm-ioam-data]. The format of the data is determined by the ioamEncapsulationType information element, if present. When the ioamEncapsulationType information element is present and has a value other than "None", and with sufficient length, this element may also report octets from subsequent headers and payload. If no ioamEncapsulationType information element is present, then the encapsulation type shall be assumed to be "None" and this information element only contains octets from the IOAM Incremental Trace Option. Abstract Data Type: octetArray ElementId: TBD4 Status: current 3.2.5. ioamE2EData Description: This Information Element carries n octets of IOAM E2E data defined in [I-D.ietf-ippm-ioam-data]. The format of the data is determined by the ioamEncapsulationType information element, if present. When the ioamEncapsulationType information element is present and has a value other than "None", and with sufficient length, this element may also report octets from subsequent headers and payload. If no ioamEncapsulationType information element is present, then the encapsulation type shall be Spiegel, et al. Expires January 9, 2020 [Page 9] Internet-Draft IOAM raw data export July 2019 assumed to be "None" and this information element only contains octets from the IOAM Edge-to-Edge Option. Abstract Data Type: octetArray ElementId: TBD5 Status: current 3.2.6. ioamPOTData Description: This Information Element carries n octets of IOAM POT data defined in [I-D.ietf-ippm-ioam-data]. The format of the data is determined by the ioamEncapsulationType information element, if present. When the ioamEncapsulationType information element is present and has a value other than "None", and with sufficient length, this element may also report octets from subsequent headers and payload. If no ioamEncapsulationType information element is present, then the encapsulation type shall be assumed to be "None" and this information element only contains octets from the IOAM Proof of Transit Option. Abstract Data Type: octetArray ElementId: TBD6 Status: current 3.2.7. ipHeaderPacketSectionWithPadding Description: This Information Element carries a series of n octets from the IP header of a sampled packet, starting sectionOffset octets into the IP header. However, if no sectionOffset field corresponding to this Information Element is present, then a sectionOffset of zero applies, and the octets MUST be from the start of the IP header. With sufficient length, this element also reports octets from the IP payload. However, full packet capture of arbitrary packet streams is explicitly out of scope per the Security Considerations sections of [RFC5477] and [RFC2804]. Spiegel, et al. Expires January 9, 2020 [Page 10] Internet-Draft IOAM raw data export July 2019 When this Information Element has a fixed length, this MAY include padding octets that are used to fill out that fixed length. When this information element has a variable length, the variable length MAY include up to 3 octets of padding, used to preserve 4-octet alignment of subequent Information Elements or subsequent records within the same set. In either case of fixed or variable length, the amount of populated octets MAY be specified in the sectionExportedOctets field corresponding to this Information Element, in which case the remainder (if any) MUST be padding. If there is no sectionExportedOctets field corresponding to this Information Element, then all octets MUST be populated unless the total length of the IP packet is less than the fixed length of this Information Element, in which case the remainder MUST be padding. Abstract Data Type: octetArray ElementId: TBD7 Status: current 3.2.8. ethernetFrameSection Description: This Information Element carries a series of n octets from the IEEE 802.3 Ethernet frame of a sampled packet, starting after the preamble and start frame delimiter (SFD), plus sectionOffset octets into the frame if there is a sectionOffset field corresponding to this Information Element. With sufficient length, this element also reports octets from the Ethernet payload. However, full packet capture of arbitrary packet streams is explicitly out of scope per the Security Considerations sections of [RFC5477] and [RFC2804]. When this Information Element has a fixed length, this MAY include padding octets that are used to fill out that fixed length. When this information element has a variable length, the variable length MAY include up to 3 octets of padding, used to preserve 4-octet alignment of subequent Information Elements or subsequent records within the same set. In either case of fixed or variable length, the amount of populated octets MAY be specified in the sectionExportedOctets field Spiegel, et al. Expires January 9, 2020 [Page 11] Internet-Draft IOAM raw data export July 2019 corresponding to this Information Element, in which case the remainder (if any) MUST be padding. If there is no sectionExportedOctets field corresponding to this Information Element, then all octets MUST be populated unless the total length of the Ethernet frame is less than the fixed length of this Information Element, in which case the remainder MUST be padding. Abstract Data Type: octetArray ElementId: TBD8 Status: current 4. Examples This section shows a set of examples of how IOAM information along with other parts of the packet can be carried using IPFIX. 4.1. Fixed Length IP Packet This example shows a fixed length IP packet. IOAM data is part of the ipHeaderPacketSection. Spiegel, et al. Expires January 9, 2020 [Page 12] Internet-Draft IOAM raw data export July 2019 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--- | Version Number | Length | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | Export Time (seconds) |IPFIX +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message | Sequence Number |Header +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | Observation Domain ID | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- | Set ID ( = Template ID) | Length |SetHdr +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- |ioamReportFlags| fwdingStatus | sectionExportedOctets | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ipHeaderPacketSection (start) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1 | ... | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ipHeaderPacketSection (end) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- |ioamReportFlags| fwdingStatus | sectionExportedOctets | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ipHeaderPacketSection (start) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record2 | ... | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ipHeaderPacketSection (end) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 4.2. Variable Length IP Packet (length < 255) This examples shows a variable length IP packet, with length < 255 bytes. IOAM data is part of the ipHeaderPacketSectionWithPadding. Spiegel, et al. Expires January 9, 2020 [Page 13] Internet-Draft IOAM raw data export July 2019 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- | Version Number | Length | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | Export Time (seconds) |IPFIX +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message | Sequence Number |Header +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | Observation Domain ID | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- | Set ID ( = Template ID) | Length |SetHdr +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- |ioamReportFlags| fwdingStatus | paddingOctets | Length (< 255)| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ipHeaderPacketSectionWithPadding (start) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1 | ... | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ipHeaderPacketSectionWithPadding (end) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- |ioamReportFlags| fwdingStatus | paddingOctets | Length (< 255)| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ipHeaderPacketSectionWithPadding (start) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record2 | ... | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ipHeaderPacketSectionWithPadding (end) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 4.3. Variable Length IP Packet (length > 255) This examples shows a variable length IP packet, with length > 255 bytes. IOAM data is part of the ipHeaderPacketSectionWithPadding. Spiegel, et al. Expires January 9, 2020 [Page 14] Internet-Draft IOAM raw data export July 2019 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- | Version Number | Length | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | Export Time (seconds) |IPFIX +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message | Sequence Number |Header +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | Observation Domain ID | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- | Set ID ( = Template ID) | Length |SetHdr +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- |ioamReportFlags| fwdingStatus | paddingOctets | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | paddingOctets | 255 | Length (0 to 65535) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ipHeaderPacketSectionWithPadding (start) |Record1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ... | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ipHeaderPacketSectionWithPadding (end) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 4.4. Variable Length ETHERNET Packet (length < 255) This examples shows a variable length Ethernet packet, with length < 255 bytes. IOAM data is part of the ethernetFrameSection. Spiegel, et al. Expires January 9, 2020 [Page 15] Internet-Draft IOAM raw data export July 2019 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- | Version Number | Length | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | Export Time (seconds) |IPFIX +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message | Sequence Number |Header +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | Observation Domain ID | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- | Set ID ( = Template ID) | Length |SetHdr +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- |ioamReportFlags| fwdingStatus | paddingOctets | Length (< 255)| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ethernetFrameSection (start) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1 | ... | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ethernetFrameSection (end) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 4.5. Variable Length IP Packet with Fixed Length IOAM Incremental Trace Data This examples shows a variable length IP packet with length < 255 bytes and fixed length ioamIncrementalTraceData carried separately. Spiegel, et al. Expires January 9, 2020 [Page 16] Internet-Draft IOAM raw data export July 2019 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- | Version Number | Length | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | Export Time (seconds) |IPFIX +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message | Sequence Number |Header +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | Observation Domain ID | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- | Set ID ( = Template ID) | Length |SetHdr +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- | ioamIncrementalTraceData (start) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ... | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ioamIncrementalTraceData (end) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1 |ioamReportFlags| fwdingStatus |ioamEncapType | Length (< 255)| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ipHeaderPacketSectionWithPadding (start) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ... | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ipHeaderPacketSectionWithPadding (end) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 4.6. Variable Length IP Packet with Variable Length IOAM Incremental Trace Data This examples shows a variable length IP packet with length < 255 bytes and variable length ioamIncrementalTraceData with length < 255 bytes carried separately. Spiegel, et al. Expires January 9, 2020 [Page 17] Internet-Draft IOAM raw data export July 2019 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- | Version Number | Length | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | Export Time (seconds) |IPFIX +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message | Sequence Number |Header +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | Observation Domain ID | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- | Set ID ( = Template ID) | Length |SetHdr +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- |ioamReportFlags| fwdingStatus |ioamEncapType | Length (< 255)| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ioamIncrementalTraceData (start) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ... | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ioamIncrementalTraceData (end) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1 | paddingOctets | Length (< 255)| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ipHeaderPacketSectionWithPadding (start) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ... | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ipHeaderPacketSectionWithPadding (end) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 5. IANA Considerations IANA is requested to allocate code points for the following Information Elements in [IANA-IPFIX]: TBD1 ioamReportFlags TBD2 ioamEncapsulationType TBD3 ioamPreallocatedTraceData TBD4 ioamIncrementalTraceData TBD5 ioamE2EData TBD6 ioamPOTData Spiegel, et al. Expires January 9, 2020 [Page 18] Internet-Draft IOAM raw data export July 2019 TBD7 ipHeaderPacketSectionWithPadding TBD8 ethernetFrameSection See Section 3.2 for further details. IANA is requested to create subregistries for ioamReportFlags defined in Section 3.2.1 and ioamEncapsulationType defined in Section 3.2.2. 6. Manageability Considerations Manageability considerations will be addressed in a later version of this document.. 7. Security Considerations Security considerations will be addressed in a later version of this document. 8. Acknowledgements The authors would like to thank Barak Gafni, Tal Mizrahi, John Lemon, and Aviv Kfir for their thoughts and comments on raw IOAM data export. 9. References 9.1. Normative References [I-D.ietf-ippm-ioam-data] Brockners, F., Bhandari, S., Pignataro, C., Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Mozes, D., Lapukhov, P., Chang, R., daniel.bernier@bell.ca, d., and J. Lemon, "Data Fields for In-situ OAM", draft-ietf-ippm-ioam- data-06 (work in progress), July 2019. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC5476] Claise, B., Ed., Johnson, A., and J. Quittek, "Packet Sampling (PSAMP) Protocol Specifications", RFC 5476, DOI 10.17487/RFC5476, March 2009, . Spiegel, et al. Expires January 9, 2020 [Page 19] Internet-Draft IOAM raw data export July 2019 [RFC7011] Claise, B., Ed., Trammell, B., Ed., and P. Aitken, "Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information", STD 77, RFC 7011, DOI 10.17487/RFC7011, September 2013, . 9.2. Informative References [I-D.brockners-ippm-ioam-geneve] Brockners, F., Bhandari, S., Govindan, V., Pignataro, C., Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Lapukhov, P., Gafni, B., Kfir, A., and M. Spiegel, "Geneve encapsulation for In-situ OAM Data", draft-brockners-ippm- ioam-geneve-02 (work in progress), March 2019. [I-D.brockners-ippm-ioam-vxlan-gpe] Brockners, F., Bhandari, S., Govindan, V., Pignataro, C., Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Kfir, A., Gafni, B., Lapukhov, P., and M. Spiegel, "VXLAN-GPE Encapsulation for In-situ OAM Data", draft-brockners-ippm- ioam-vxlan-gpe-02 (work in progress), July 2019. [I-D.ietf-sfc-ioam-nsh] Brockners, F., Bhandari, S., Govindan, V., Pignataro, C., Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Mozes, D., Lapukhov, P., and R. Chang, "Network Service Header (NSH) Encapsulation for In-situ OAM (IOAM) Data", draft- ietf-sfc-ioam-nsh-01 (work in progress), March 2019. [I-D.ioametal-ippm-6man-ioam-ipv6-options] Bhandari, S., Brockners, F., Pignataro, C., Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Kfir, A., Gafni, B., Lapukhov, P., Spiegel, M., Krishnan, S., and R. Asati, "In-situ OAM IPv6 Options", draft-ioametal-ippm-6man-ioam- ipv6-options-02 (work in progress), March 2019. [I-D.weis-ippm-ioam-eth] Weis, B., Brockners, F., Hill, C., Bhandari, S., Govindan, V., Pignataro, C., Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Kfir, A., Gafni, B., Lapukhov, P., and M. Spiegel, "EtherType Protocol Identification of In-situ OAM Data", draft-weis-ippm-ioam-eth-01 (work in progress), March 2019. [IANA-IPFIX] "IP Flow Information Export (IPFIX) Entities", . Spiegel, et al. Expires January 9, 2020 [Page 20] Internet-Draft IOAM raw data export July 2019 [RFC2804] IAB and IESG, "IETF Policy on Wiretapping", RFC 2804, DOI 10.17487/RFC2804, May 2000, . [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", RFC 5226, DOI 10.17487/RFC5226, May 2008, . [RFC5477] Dietz, T., Claise, B., Aitken, P., Dressler, F., and G. Carle, "Information Model for Packet Sampling Exports", RFC 5477, DOI 10.17487/RFC5477, March 2009, . Authors' Addresses Mickey Spiegel Barefoot Networks 4750 Patrick Henry Drive Santa Clara, CA 95054 US Email: mspiegel@barefootnetworks.com Frank Brockners Cisco Systems, Inc. Hansaallee 249, 3rd Floor DUESSELDORF, NORDRHEIN-WESTFALEN 40549 Germany Email: fbrockne@cisco.com Shwetha Bhandari Cisco Systems, Inc. Cessna Business Park, Sarjapura Marathalli Outer Ring Road Bangalore, KARNATAKA 560 087 India Email: shwethab@cisco.com Spiegel, et al. Expires January 9, 2020 [Page 21] Internet-Draft IOAM raw data export July 2019 Ramesh Sivakolundu Cisco Systems, Inc. 170 West Tasman Dr. SAN JOSE, CA 95134 U.S.A. Email: sramesh@cisco.com Spiegel, et al. Expires January 9, 2020 [Page 22]