Reducing redundancy in IPFIX and PSAMP reports Internet-Draft E. Boschi draft-boschi-ipfix-reducing-redundancy-01.txt Hitachi Europe Expires: September 7, 2006 L. Mark Fraunhofer FOKUS March 2006 Reducing redundancy in IPFIX and PSAMP reports draft-boschi-ipfix-reducing-redundancy-01.txt Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on September 7, 2006. Copyright Notice Copyright (C) The Internet Society (2006). Boschi, Mark Expires September 2006 [Page 1] Reducing redundancy in IPFIX and PSAMP reports Abstract This document describes a bandwidth saving methodology for exporting flow or packet information using the IP Flow Information Export (IPFIX) protocol. Being the PSAMP protocol based on IPFIX, these considerations are valid for PSAMP exports as well. The main idea is to separate the export of information common to several flows (or packets) and the specific flow (or packet) information, using two different records. The association between the records is kept using unique Identifiers. Table of Contents 1. Introduction·········································2 2. Terminology··········································3 3. Open issues··········································3 4. Techniques for bandwidth saving information export···3 4.1 Single and multiple associations···················4 4.2 Export of Per-Packet Information···················5 4.3 Using Scopes·······································6 4.4 CommonPropertiesID Management······················7 5. Examples·············································7 5.1 Single commonPropertiesIDs·························7 5.2 Multiple commonPropertiesIDs······················10 5.3 Per-Packet Information Export·····················12 6. Export and evaluation considerations················14 7. IANA Considerations·································15 8. Security Considerations·····························15 9. Normative References································15 10. Author's Addresses··································16 11. Intellectual Property Statement·····················16 12. Copyright Statement·································16 13. Disclaimer··········································16 1. Introduction In [IPFIX-PROTO] the IPFIX working group has defined a protocol to export IP Flow information. The main purpose of the protocol is to exchange information about IP traffic Flows in combination with related measurement data. The export of that information is done via Flow Records. In this scope a Flow is defined by a set of key attributes (e.g. source and destination IP address, source and destination port, etc.). When exporting Flow Records sharing a set of common attributes, these attributes have to be repeated for each Data Record even though the values remain the same. The elimination of redundant data from the export stream can result in a significant reduction of the transferred data. This draft proposes to use Option Data Records to export common properties, whose value remains constant, and to send these properties only once. Measured properties vary over time and are sent multiple times in regular Data Records. The linkage of the Flow Records with the common properties exported via option data records is done via a Common Properties Identifier. The proposed method can be applied also to export per packet information (e.g. for Passive One-Way Delay measurements or Boschi, Mark Expires September 2006 [Page 2] Reducing redundancy in IPFIX and PSAMP reports PSAMP exports). In this case a Flow is a collection of packets that share a set of common attributes. The proposed method does not need any changes to the IPFIX protocol. 2. Terminology Collecting Process The collecting process receives records of flow or packet information. The data is stored for later processing (by the calculation process) Exporting Process The exporting processes send flow and packet records to the collecting processes. The records are generated by the measurement process. Filtering Filtering selects a subset of packets by applying deterministic functions on parts of the packet content like header fields or parts of the payload. A filtering process needs to process the packet (look at packet header and/or payload) in order to make the selection decision. Measurement Device A measurement device has access to at least one observation point. It is hosting at least one measurement process and one export process. Metering/Measurement Process The measurement process generates records of packet and flow information. Packets passing the observation point are captured, time stamped, filtered and classified. The measurement process calculates the packet Ids. Passive One-Way-Delay Measurement Abbreviated: POWD Measurement CommonPropertiesID An identifier of a set of common properties that is unique within an Observation Domain. This ID can be used to link to information reported in separate records. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT","SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 3. Open issues [1] This draft proposes a new IE, commonPropertiesID; its definition should be included in [IPFIX-INFO] and is currently specified in the "IANA considerations" of this draft. 4. Techniques for bandwidth saving information export The IPFIX protocol is template based. Templates define how data should be exported, describing data fields together with their type and meaning. IPFIX specifies two types of records to export data: Data Records and Option Data Records. The layout of these records is specified by Template Records and Option Template Records. Boschi, Mark Expires September 2006 [Page 3] Reducing redundancy in IPFIX and PSAMP reports To export information more efficiently we group different flow records depending on their common properties. We define Common Properties as a collection of attributes shared by a set of different Flow Records. Specific Properties are the remaining flow or packet attributes, not shared with the same set of flows. Since a packet can be considered as a special kind of flow, these considerations remain valid for packets too. We split the information export into two parts, using two different records. Common Properties are exported via Option Data Records and are sent only once. These properties represent values common to several flows. The associated Specific Properties are subsequently sent using Data Records. The association is kept using a particular index that uniquely identifies the Common Properties, the commonPropertiesID. Figure 1 shows the relation between Template and Data Sets for Common and Specific Properties. The Common Properties Option Template defines the common flow attributes (e.g. IP source and destination address) and the commonPropertiesID. The commonPropertiesID is a unique identifier for a set of attributes; this field allows the linkage between those attributes and Flow Data Records. Subsequent Option Data Records of this template export the Common Properties values. The assignment of Flow Records to common attributes could be alternatively provided by the templateID, as explained in Section 4.3. In this case the commonPropertiesID can be omitted. The format for the information related to single flows or packets is defined in the Specific Properties template. This information is flow/packet specific and normally not shared between many flows/packets. +---------------------+ +---------------------+ | Option Template Set | | Template Set | | | | | Description of | Common Properties | | Specific Properties | exported data +----------+----------+ +----------+----------+ ...........|............................|......................... | | +----------v----------+ +----------v----------+ Exported data | Option Data Set | | Data Set | with references | <------+ | through common | Common Properties | | Specific Properties | properties or +---------------------+ +---------------------+ template IDs Figure 1: Template Set and Data Set dependencies The Common Properties option data records SHOULD be sent prior to the corresponding Specific Properties data records. 4.1 Single and multiple associations Several Flow Records often share a set of common properties. Repeating the information about these common properties for every record introduces a huge amount of redundancy. Let's consider a set of properties "A", e.g. common sourceAddressA and sourcePortA and another set of properties "B", e.g. destinationAddressB and destinationPortB. Figure 2 shows how this information is repeated with classical IPFIX export in several records. Boschi, Mark Expires September 2006 [Page 4] Reducing redundancy in IPFIX and PSAMP reports +--------------------+---------------------------------+ | properties A | | +--------------------+---------------------------------+ +--------------------+-----------------+---------------+ | properties A | properties B | | +--------------------+-----------------+---------------+ Figure 2: Common and Specific Properties exported in the same record In Figure 3 we separate Common Properties from Specific flow (or packet) properties. In order to maintain the relation between these sets of properties we introduce indices (idxA and idxB) for the Common Properties that are unique for all Common Property entries. The purpose of the indices is to serve as "primary key" that identifies rows of the Common Properties. More details about these indices will be given in section 4.3. The rows are then referenced by the Specific Properties by using the appropriate value for the Common Properties identifier. A Flow Record can refer to one or more Common Properties sets, achieving thus, in the latter case, a better export efficiency. Note that in this case potential collisions between different properties must be avoided. The Exporting Process must make sure that if two or more sets of Common Properties are referenced by the same Specific Properties, there are no ambiguities or conflicts on the values of the common attributes. Common Properties Specific Properties +--------------+-----+ +------+----------------------+ | properties A |idxA <--------> idxA | | +--------------+-----+ +------+----------------------+ | properties B |idxB < > idxA | idxB | | +--------------+-----+ +------+------+---------------+ Figure 3: Common and Specific Properties exported separately 4.2 Export of Per-Packet Information A particular case is the export of per packet information (e.g. for Passive One-Way Delay measurements or PSAMP exports). In this case a single packet could be considered a special case of a flow and consequently per-packet information could be exported using Flow Records. Doing this though would introduce additional overhead, since packets belonging to the same flow share common attributes (e.g. source address, destination address, etc). Exporting these Common Properties on a per-packet basis, would be redundant. Figure 4 depicts three packets belonging to flow A and one packet belonging to flow B, respectively. It shows export records containing packet information plus flow information (source and destination address). Undoubtedly, flow information (or Common Properties, as defined in this draft) introduces a huge amount of redundancy, as it is repeated for every packet in every record. Boschi, Mark Expires September 2006 [Page 5] Reducing redundancy in IPFIX and PSAMP reports One-packet flows +-------------------+----------------------+ | flow properties A | | +-------------------+----------------------+ | flow properties A | | +-------------------+----------------------+ | flow properties B | | +-------------------+----------------------+ | flow properties A | | +-------------------+----------------------+ Figure 4: Flow and packet information represented in one-packet flows In Figure 5 we separate Common Properties from Specific Properties, i.e. flow from packet information. In order to maintain the relation between Specific (Packet) Properties and Common (Flow) Properties we introduce indices (idxA and idxB), much in the same way as explained in the previous sections. The linkage of one packet and flow B (flow properties B, idxB) is explicitly drawn. Specific (Packet) Properties +------+--------------------+ Common (Flow) Properties > idxA | | +------------------+-----+ +------+--------------------+ |flow properties A |idxA < > idxA | | +------------------+-----+ +------+--------------------+ |flow properties B |idxB <-------> idxB | | +------------------+-----+ +------+--------------------+ > idxB | | +------+--------------------+ Figure 5: Flow information and packet information 4.3 Using Scopes Common Properties are sent via IPFIX option records. IPFIX option records contain one or more scope fields. The Flow Properties record can contain the commonPropertiesID and/or the TemplateID as scope fields. There are three options: 1) Use commonPropertiesID as scope The Common Properties are valid for all data records containing that commonPropertiesID. This solution limits the number of different flows that can be exported at the same time in the same observation domain to 2**32 (using 32 bits commonPropertiesIDs) 2) Use commonPropertiesID and templateID as scope The Common Properties are valid only for data records referring to the template specified by the templateID and containing that commonPropertiesID. This allows the export up to 2**32 flows per template. The solution is to be chosen when the number of flows to be exported is expected to be very high (and beyond the limit posed by solution 1) 3) Use templateID as scope Boschi, Mark Expires September 2006 [Page 6] Reducing redundancy in IPFIX and PSAMP reports The common properties are valid for all data records of the specified template. In this case commonPropertiesIDs are not needed but the Exporting Process requires a templateID per flow. In the general case this solution is not scalable but can be suitable for certain applications (e.g. flow aggregation). 4.4 CommonPropertiesID Management The management of commonPropertiesIDs is very similar to the management of templateIDs described in [IPFIX-PROTO]. The Exporting Process maintains the commonPropertiesIDs for the exporter's Observation Domains. Like templateIDs, a commonPropertiesID MUST be unique per Observation Domain (source identifier in the IPFIX header). Different Observation Domains from the same exporter may use the same commonPropertiesID value to refer to different sets of Common Properties. There are no constraints regarding the order of the commonPropertiesID allocation. When limiting the scope to special templates, the commonPropertiesIDs have to be unique per Observation Domain and Template. Using 32 bit commonPropertiesIDs allows the export of 2**32 active flows in parallel. CommonPropertiesIDs have a certain lifetime inside which they cannot be reused. After that time a commonPropertiesID can be assigned to another set of Common Properties. CommonPropertiesID whose lifetime has longer expired SHOULD be preferred. The lifetime MUST be configurable. The collecting process associates a lifetime with each commonPropertiesID. The lifetime MUST be configurable. The mapping of Data Records to Common Properties uses the most recent Common Properties definition associated to the specified commonPropertiesID. If there is no flow definition associated with that commonPropertiesID or the lifetime of the flow definition has expired, no mapping is possible. In this case the collecting process SHOULD log an error. When IPFIX uses an unreliable transport protocol to export the Option Data Records containing the Common Properties and the commonPropertiesIDs these records MUST be re-sent at regular intervals, whose frequency MUST be configurable. When using a connection oriented transport protocol the Common Properties have to be re-sent after a connection re- establishment before the corresponding Specific Properties Data Records. 5. Examples 5.1 Single commonPropertiesIDs In this section we show how flow information can be exported efficiently using the method described in this draft. Let's suppose we have to periodically export data about two IPv6 Flows. In this example we report the following information: Boschi, Mark Expires September 2006 [Page 7] Reducing redundancy in IPFIX and PSAMP reports Flow| dstIPv6Address | dst- |nPkts|nBytes | | Port | | ---------------------------------------------------------------- A |5F05:2000:80AD:5800:0058:0800:2023:1D71| 80 | 30 | 6000 | | | | A |5F05:2000:80AD:5800:0058:0800:2023:1D71| 80 | 50 | 9500 | | | | B |5F05:2000:80AD:5800:0058:00AA:00B7:AF2B| 1932 | 60 | 8000 | | | | A |5F05:2000:80AD:5800:0058:0800:2023:1D71| 80 | 40 | 6500 | | | | A |5F05:2000:80AD:5800:0058:0800:2023:1D71| 80 | 60 | 9500 | | | | B |5F05:2000:80AD:5800:0058:00AA:00B7:AF2B| 1932 | 54 | 7600 The Common Properties in this case are the destination IPv6 address and the destination port. We first define an Option Template that contains the following Information Elements: - the destination IPv6 address, destinationIPv6Address [IPFIX-INFO], with a type of 28 and a length of 16 octets - the destination port, destinationTransportPort [IPFIX-INFO] with a type of 11, and a length of 2 octets - the Common Properties ID, with a type [TO BE ASSIGNED, in this Draft referenced to as XX] and a length of 4 octets. Figure 6 shows the Option template defining the Common Properties with commonPropertiesID as scope: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 3 | Length = 24 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 257 | Field Count = 3 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field count = 1 |0| commonPropertiesID = XX | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length = 4 |0| destinationIPv6Address = 28| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 16 |0|destinationTransportPort = 11| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 2 | (Padding) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 6: Common Properties Option Template The Specific Properties Template consists of the information not contained in the Option Templates, i.e. flow specific information, in this case the number of packets and the number of bytes to be reported. Additionally, this Template contains the commonPropertiesID. In Data Records, the value of this field will contain one of the unique indices of the Option Records exported before. It contains the following Information Elements (see also Figure 7): - commonPropertiesID with a length of 4 octets - the number of packets of the Flow: inPacketDeltaCount in [IPFIX-INFO], with a length of 4 octets Boschi, Mark Expires September 2006 [Page 8] Reducing redundancy in IPFIX and PSAMP reports - the number of octets of the Flow: inOctetDeltaCount in [IPFIX-INFO], with a length of 4 octets 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 2 | Length = 20 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 258 | Field Count = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| commonPropertiesID = XX | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| inPacketDeltaCount = 2 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| inOctetDeltaCount = 1 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 7: Specific Properties Template Considering the data shown at the beginning of this example, the following two Option Data Records will be exported: Common- | dstAddress | dst- PropertiesID | | Port -------------+-----------------------------------------+------- 1 | 5F05:2000:80AD:5800:0058:0800:2023:1D71 | 80 | | 2 | 5F05:2000:80AD:5800:0058:00AA:00B7:AF2B | 1932 The Option Data Records reporting the Common Properties will look like: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 257 | Length = 52 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 5F05:2000: ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... 80AD:5800: ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... 0058:0800: ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... 2023:1D71 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 80 | (Padding) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 5F05:2000: ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... 80AD:5800: ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... 0058:00AA: ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... 00B7:AF2B | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1932 | (Padding) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 8: Option Data Records reporting Common Properties Boschi, Mark Expires September 2006 [Page 9] Reducing redundancy in IPFIX and PSAMP reports The Data Records will in turn be: commonPropertiesID | inPacketDeltaCount | inOctetDeltaCount --------------------------------------------------------------- 1 | 30 | 6000 1 | 50 | 9500 2 | 60 | 8000 1 | 40 | 6500 1 | 60 | 9500 2 | 54 | 7600 The Figure below shows the first Data Record listed in the table: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 258 | Length = 12 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 30 | 6000 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 9: Data Record reporting Specific Properties 5.2 Multiple commonPropertiesIDs In case a flow shares two set of Common Properties, an additional overhead reduction can be obtained. Let's suppose we have to export the following flow information: Flow | srcAddr | srcPort | dstAddr | dstPort | nPackets | nBytes ---------------------------------------------------------------- A |10.0.0.1 | 1932 |10.0.1.2 | 80 | 30 | 6000 B |10.0.0.3 | 2032 |10.0.1.2 | 80 | 50 | 9500 Fehler! Verweisquelle konnte nicht gefunden werden. shows the Option Templates, containing the Common Properties together with the commonPropertiesID, which is the scope. In the first Common Properties Option Template we export the following Information Elements: - the source IPv4 Address, sourceIPv4Address [IPFIX-INFO], with a type of 8 and a length of 4 octets - the source Port, sourceTransportPort [IPFIX-INFO], with a type of 7 and a length of 2 octets - the Common Properties ID, commonPropertiesId with a type [TO BE ASSIGNED, in this Draft referenced to as XX] and a length of 4 octets. The second Option Template contains the following Information Elements: - the destination IPv4 Address, destinationIPv4Address [IPFIX-INFO], with a type of 12 and a length of 4 octets Boschi, Mark Expires September 2006 [Page 10] Reducing redundancy in IPFIX and PSAMP reports - the destination port, destinationTransportPort [IPFIX-INFO] with a type of 11, and a length of 2 octets - the Common Properties ID, with a type [TO BE ASSIGNED, in this Draft referenced to as XX] and a length of 4 octets. The Flow identifier, which is represented by the commonPropertiesId Information Element [NOTE: to be included in IPFIX-INFO], is used in both cases as the Scope Field. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 3 | Length = 24 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 256 | Field Count = 3 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field count = 1 |0| commonPropertiesID = XX | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length = 4 |0| sourceIPv4Address = 8 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 4 |0| transportSourcePort = 7 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 2 | (Padding) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 3 | Length = 24 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 257 | Field Count = 3 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field count = 1 |0| commonPropertiesID = XX | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length = 4 |0| destinationIPv4Address = 12| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 4 |0|transportDestinationPort = 11| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 2 | (Padding) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 10: Example Common Properties Template Considering the values given at the beginning of this section we will export the following Option Data Records: commonPropertiesID | sourceAddress | sourcePort --------------------+-----------------+------------- 1 | 10.0.0.1 | 1932 2 | 10.0.0.3 | 2032 and commonPropertiesID | dstAddress | dstPort --------------------+---------------+----------- 3 | 10.0.1.2 | 80 The Specific Properties Template consists of the information not contained in the Option Templates, i.e. flow specific information. Additionally, this Template contains the two commonPropertiesIDs. In Data Records, the values of these fields Boschi, Mark Expires September 2006 [Page 11] Reducing redundancy in IPFIX and PSAMP reports will contain one of the unique indices specified in the Option Records exported before. Figure 11 displays the Template including the commonPropertiesIDs plus the Specific Properties. In this example we export the following Information Elements: - commonPropertiesID for the source fields with a length of 4 octets - commonPropertiesID for the destination fields with a length of 4 octets - the number of packets of the Flow: inPacketDeltaCount in [IPFIX-INFO], with a length of 4 octets - the number of octets of the Flow: inOctetDeltaCount in [IPFIX-INFO], with a length of 4 octets 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 2 | Length = 24 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 259 | Field Count = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| commonPropertiesID = XX | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| commonPropertiesID = XX | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| inPacketDeltaCount = 2 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| inOctetDeltaCount = 1 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 11: Example Specific Properties Template Considering the values given at the beginning of this section, the Data Records of the two flows will look like: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 256 | Length = 28 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 3 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 30 | 6000 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 3 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 50 | 9500 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 12: Specific Properties 5.3 Per-Packet Information Export Boschi, Mark Expires September 2006 [Page 12] Reducing redundancy in IPFIX and PSAMP reports To demonstrate how to use IPFIX efficiently to export per-packet information, this section proposes how to use the IPFIX protocol for exporting Flow information (Common Properties) and per-packet information (Specific Properties, in this case related to a long-lived flow) for OWD computation. In order to acquire a One-Way path delay information, two measurement points with synchronized clocks must exist, one at each end of the path under examination. Both measurement points will capture packets, assign them timestamps and generate an identifier for a packet passing that point. Each measurement point will export its measurement data to a Collecting Process where the data are correlated based on the packet identifiers and timestamps and then the delay is calculated as a difference of two timestamps of a packet pair. The Templates that would be needed for exporting measurement data of this kind are illustrated in the figures below. Figure 13 shows the Option Template containing the information concerning Flows using the commonPropertiesID as scope. In the Flow Properties Template (the Common Porperties Template) we export the following Information Elements: - the source IPv4 Address, sourceIPv4Address [IPFIX-INFO], with a type of 8 and a length of 4 octets - the destination IPv4 Address, destinationIPv4Address [IPFIX-INFO], with a type of 12 and a length of 4 octets - the Class of Service field, ClassOfServiceIPv4 [IPFIX- INFO], with a type of 5 and a length of 1 octet - source and destination ports, sourceTransportPort and destinationTransportPort [IPFIX-INFO], with a type of 7 and 11 respectively, and a length of 2 octets each The Flow identifier, which is represented by the commonPropertiesID Information Element, is used as the Scope Field. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 3 | Length = 40 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 256 | Field Count = 7 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field count = 1 |0| commonPropertiesID = XX | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length = 4 |0| sourceIPv4Address = 8 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 4 |0| destinationIPv4Address = 12 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 4 |0| classOfServiceIPv4 = 5 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 1 |0| protocolIdentifier = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 1 |0| transportSourcePort = 7 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 2 |0|transportDestinationPort = 11| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 2 | (Padding) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 13: Example Flow Properties Template Boschi, Mark Expires September 2006 [Page 13] Reducing redundancy in IPFIX and PSAMP reports For passive One-Way-Delay measurement, the Packet Properties Template, or Specific Properties Template, consists of at least Timestamp and Packet ID. Additionally, this template contains a flow identifier field. In packet Records, the value of this field will contain one of the unique indices of the Flow Records exported before. Figure 14 displays the template with the packet properties. In this example we export the following Information Elements: - commonPropertiesID with a length of 4 octets - packetTimestamp, packetID, and packetLength. Since packetTimestamp, packetID, and packetLength are not (yet) IETF-defined information elements, we export them as enterprise-specific IEs. The three IEs have respectively a type of 220, 221, and 222 and a length of 8, 4, and 4 octets. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 2 | Length = 36 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 257 | Field Count = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| commonPropertiesID = XX | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1| packetTimestamp = 220 | Field Length = 8 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Enterprise number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1| packetID = 221 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Enterprise number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1| packetLength = 222 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Enterprise number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 14: Example Packet Properties Template The delay is derived by a calculation step: at the collection point packet records of two measurement points are gathered and correlated by means of the packet ID. The resulting delay data records are exported in a similar manner as the packet data have been. Especially, the linkage between delay data and flow information is represented with the Flow identifier commonPropertiesID fields. The OWD properties contain the Packet Pair ID (which is the packet ID matching that of the two contributing packet records), a timestamp (which is the timestamp of the packet passing the reference monitor point) in order to reconstruct a time series, the calculated delay value, and finally the flow identifier (commonPropertiesID). 6. Export and evaluation considerations The main advantage of the methods proposed in this draft is the reduced amount of measurement data that has to be transferred from the Exporter to the Collector. In addition there is less storage capacity needed at the Collector side. Boschi, Mark Expires September 2006 [Page 14] Reducing redundancy in IPFIX and PSAMP reports On the other hand there is some extra processing power needed on the Exporter side to manage Common Properties information and to assign them to Flow Records. The Collector has to process records of two templates instead of just one but has to read and write less data. Additional effort is needed when post processing the measurement data, because now the correlation of Flow Records with Common Properties information is needed. In the example in section 5.3 using IPFIX to export the measurement data for each received packet 30 bytes have to be transferred (sourceAddressV4=4, destinationAddressV4=4, classOfServiceV4=1, protocolIdentifier=1, sourceTransportPort=2, destionationTransportPort=2, packetTimestamp=8, packetID=4, packetLength=4). Without considering the IPFIX protocol overhead a flow of 1000 packets produces 28000 bytes of measurement data. Using the proposed optimization each packet produces an export of only 20 bytes (packetTimestamp=8, packetID=4, packetLength=4, commonPropertiesID=4). The export of the flow information produces 16 bytes (sourceAddressV4=4, destinationAddressV4=4, classOfServiceV4=1, protocolIdentifier=1, sourceTransportPort=2, destionationTransportPort=2, commonPropertiesID =4). For a flow of 1000 packet this sums up to 16016 bytes. This is a decrease of more than 40 percent. Note that IPFIX allows the reduction of the size of exported Information Elements. This can be applied also to the commonPropertiesID and would result in a further reduction of bytes. 7. IANA Considerations IANA should provide an ElementId for the commonPropertiesID Information Element. commonPropertiesId Description An identifier of a set of common properties that is unique within an Observation Domain. This ID can be used to link to information reported in separate records Abstract Data Type: unsigned32 Data Type Semantics: identifier ElementId: TBA Status: proposed 8. Security Considerations For the proposed use of the IPFIX protocol for bandwidth-saving export the security considerations as for the IPFIX protocol apply. 9. Normative References [IPFIX-PROTO] Benoit Claise et Al.: IPFIX Protocol Specification, IETF draft work in progress , September 2005 [IPFIX-INFO] J. Quittek, S.Bryant, B.Claise, J. Meyer: Information Model for IP Flow Information Export Internet-draft work in progress , September 2005 Boschi, Mark Expires September 2006 [Page 15] Reducing redundancy in IPFIX and PSAMP reports [PSAMP-PROTO] Benoit Claise: PSAMP Protocol Specification, Internet Draft , October 2005 10. Author's Addresses Elisa Boschi Hitachi Europe SAS Immeuble Le Theleme 1503 Route des Dolines 06560 Valbonne, France Phone: +33 4 89874180 Email: elisa.boschi@hitachi-eu.com Lutz Mark Fraunhofer Institute for Open Communication Systems Kaiserin-Augusta-Allee 31 10589 Berlin Germany Phone: +49-30-34 63 7306 Fax: +49-30-34 53 8306 Email: mark@fokus.fraunhofer.de 11. Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. 12. Copyright Statement Copyright (C) The Internet Society (2006). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. 13. Disclaimer This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Boschi, Mark Expires September 2006 [Page 16]