Operation of the IP Flow Information Export (IPFIX) Protocol on IPFIX Mediators
Cisco Systems, Inc.
De Kleetlaan 6a b11831 DiegemBelgium+32 2 704 5622bclaise@cisco.com
NTT Information Sharing Platform Laboratories
3-9-11 Midori-choMusashino-shi, Tokyo 180-8585Japan+81 422 59 3978akoba@nttv6.net
Swiss Federal Institute of Technology Zurich
Gloriastrasse 358092 ZurichSwitzerland+41 44 632 70 13trammell@tik.ee.ethz.ch
Operations
IPFIX Working GroupThis document specifies the operation of the IP Flow Information
Export (IPFIX) protocol specific to IPFIX Mediators, including Template
and Observation Point management, timing considerations, and other
Mediator-specific concerns.The IPFIX architectural components in consist of
IPFIX Devices and IPFIX Collectors communicating using the IPFIX protocol
, which specifies how to
export IP Flow information. This protocol is designed to export information
about IP traffic Flows and related measurement data, where a Flow is defined
by a set of key attributes (e.g. source and destination IP address, source
and destination port, etc.).However, thanks to its Template mechanism, the IPFIX protocol can export
any type of information, as long as the relevant Information Element is
specified in the IPFIX Information Model , registered with
IANA, or specified as an enterprise-specific Information Element. The
specifications in the IPFIX protocol have not been defined in the
context of an IPFIX Mediator receiving, aggregating, correlating,
anonymizing, etc... Flow Records from one or more Exporters. Indeed,
the IPFIX protocol must be adapted for Intermediate Processes, as defined in
the IPFIX Mediation Reference Model as specified in Figure A of , which is based on the IPFIX Mediation Problem Statement
.This document specifies the IP Flow Information Export (IPFIX) protocol
in the context of the implementation and deployment of IPFIX Mediators. The
use of the IPFIX protocol within an IPFIX Mediator -- a device which contains both
a Collecting Process and an Exporting Process -- has an impact on the
technical details of the usage of the protocol. An overview of the technical
problem is covered in section 6 of : loss of
original Exporter information, loss of base time information, transport
sessions management, loss of Options Template Information, Template Id
management, considerations for network considerations for aggregation.The specifications in this document are based on the IPFIX protocol
specifications but
adapted according to the IPFIX Mediation Framework .The IPFIX Protocol
provides network administrators with access to IP Flow information.The architecture for the export of measured IP Flow information out of
an IPFIX Exporting Process to a Collecting Process is defined in the IPFIX
Architecture , per the requirements defined in the
IPFIX Requirement doc, .The IPFIX Architecture specifies how IPFIX
Data Records and Templates are carried via a congestion-aware transport
protocol from IPFIX Exporting Processes to IPFIX Collecting Processes.IPFIX has a formal description of IPFIX Information Elements, their
name, type and additional semantic information, as specified in the IPFIX
Information Model .
The IPFIX Information Element
registry is maintained by IANA. New
Information Element definitions can be added to this registry subject
to an Expert Review [RFC5226], with additional process considerations
described in ; that document also provides
guidelines for authors and reviewers of new Information Element
definitions. The inline export of the Information Element type
information is specified in .The IPFIX Applicability Statement describes
what type of applications can use the IPFIX protocol and how they can use
the information provided. It furthermore shows how the IPFIX framework
relates to other architectures and frameworks.The "IPFIX Mediation: Problem Statement"
provides an overview of the applicability of IPFIX Mediators, and defines
requirements for IPFIX Mediators in general terms. This document is of use
largely to define the problems to be solved through the deployment of
IPFIX Mediators, and to provide scope to the role of IPFIX Mediators within an
IPFIX collection infrastructure.The "IPFIX Mediation: Framework" , which
details the IPFIX Mediation reference model and the components of an
IPFIX Mediator, provides more architectural details of the arrangement of
Intermediate Processes within an IPFIX Mediator.Documents specifying the operations of specific
Intermediate Processes cover the operation of these Processes within the
IPFIX Mediator framework, and comply with the specifications given in this
document; they may additionally specify the operation of the process
independently, outside the context of an IPFIX Mediator, when this is
appropriate. The details of specific Intermediate Processes, when these
have additional export specifications (e.g., metadata about the
intermediate processing conveyed through IPFIX Options Templates), are
each treated in their own document. As of today, these documents are:"IP Flow Anonymization Support", , which describes
Anonymization techniques for IP flow data and the export of Anonymized
data using the IPFIX protocol."Flow Selection Techniques" , which describes the
process of selecting a subset of Flows from all Flows observed at an
Observation Point, the flow selection motivations, and some specific
flow selection techniques."Exporting Aggregated Flow Data using IP Flow Information Export"
which describes Aggregated Flow
export within the framework of IPFIX Mediators and defines an
interoperable, implementation-independent method for Aggregated Flow
export.This document specifies the IP Flow Information Export (IPFIX) protocol
specific to Mediation, i.e. the specifications that all Intermediate
Processes type must comply to. Some extra specifications might be required
per Intermediate Process type (In which case, the Intermediate Process
specific document would cover those).The specification in this document applies to the IPFIX protocol
specifications . All
specifications from
apply unless specified otherwise in this document.As the Packet Sampling (PSAMP) protocol specifications are
based on the IPFIX protocol specifications, the specifications in this
document are also valid for the PSAMP protocol. Therefore, the method
specified by this document also applies to PSAMP.The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
.IPFIX-specific terms, such as Observation Domain, Flow, Flow Key,
Metering Process, Exporting Process, Exporter, IPFIX Device, Collecting
Process, Collector, Template, IPFIX Message, Message Header, Template
Record, Data Record, Options Template Record, Set, Data Set, Information
Element, Scope and Transport Session, used in this document are defined in . The PSAMP-specific terms used
in this document, such as Filtering and Sampling, are defined in .IPFIX Mediation terms related to aggregation, such as the Interval,
Aggregated Flow, and Aggregated Function are defined in .The IPFIX Mediation-specific terminology used in this document is defined
in "IPFIX Mediation: Problem Statement" , and reused
in "IPFIX Mediation: Framework" . However, since
both of those documents are an informational RFCs, the definitions have been
reproduced here along with additional definitions.Similarly, since is an experimental RFC, the
Anonymization Record, Anonymized Data Record, and Intermediate Anonymization
Process terms, specified in , are also reproduced
here.In this document, as in , ,
, and , the first
letter of each IPFIX-specific and PSAMP-specific term is capitalized along
with the IPFIX Mediation-specific term defined here.In this document, we call a stream of records carrying flow- or packet-based information a "record stream". The records may be encoded as IPFIX Data Records or any other format.The Transport Session is
specified in . In SCTP,
the Transport Session Information is the SCTP association. In TCP and UDP,
the Transport Session Information corresponds to a 5-tuple {Exporter IP
address, Collector IP address, Exporter transport port, Collector transport
port, transport protocol}.An Original Exporter is the source from
which a Mediator receives its record stream. For simple IPFIX mediation
without protocol conversion, this is an IPFIX Device that hosts the
Observation Points where the metered IP packets are observed.An Observation Point on a
Metering Process associated with the Original Exporter. In the case of the
Intermediate Aggregation Process on an IPFIX Mediator, the Original
Observation Point can be composed of, but not limited to, a (set of)
specific Exporter(s), a (set of) specific interface(s) on an Exporter, a
(set of) line card(s) on an Exporter, or any combinations of these.IPFIX Mediation is the manipulation and
conversion of a record stream for subsequent export using the IPFIX
protocol.A mapping from Template Records and/or
Options Template Records received by an IPFIX Mediator to Template Records and/or
Options Template Records sent by that IPFIX Mediator. Each entry in a
Template Mapping is scoped by incoming or outgoing Transport Session and
Observation Domain, as with Templates and Options Templates in the IPFIX
Protocol.A record that defines the properties
of the anonymization applied to a single Information Element within a
single Template or Options Template, as in .A Data Record within a Data Set
containing at least one Information Element with Anonymized values. The
Information Element(s) within the Template or Options Template describing
this Data Record SHOULD have a corresponding Anonymization Record, as in
.The following terms are used in this document to describe the
architectural entities used by IPFIX Mediation.An Intermediate Process takes a
record stream as its input from Collecting Processes, Metering Processes,
IPFIX File Readers, other Intermediate Processes, or other record sources;
performs some transformations on this stream, based upon the content of
each record, states maintained across multiple records, or other data
sources; and passes the transformed record stream as its output to
Exporting Processes, IPFIX File Writers, or other Intermediate Processes,
in order to perform IPFIX Mediation. Typically, an Intermediate Process is
hosted by an IPFIX Mediator. Alternatively, an Intermediate Process may be
hosted by an Original Exporter.An IPFIX Mediator is an IPFIX Device that
provides IPFIX Mediation by receiving a record stream from some data
sources, hosting one or more Intermediate Processes to transform that
stream, and exporting the transformed record stream into IPFIX Messages
via an Exporting Process. In the common case, an IPFIX Mediator receives a
record stream from a Collecting Process, but it could also receive a
record stream from data sources not encoded using IPFIX, e.g., in the case
of conversion from the NetFlow V9 protocol to
IPFIX protocol.Specific Intermediate Processes are described below. (as in ): An Intermediate Conversion Process is an Intermediate
Process that transforms non-IPFIX into IPFIX or manages the relation among
Templates and states of incoming/outgoing transport sessions in the case of
transport protocol conversion (e.g., from UDP to SCTP). (as in ): an Intermediate Process (IAP) as in
[RFC6183] that aggregates records, based upon a set of Flow Keys or
functions applied to fields from the record. (as in ): An Intermediate Correlation Process is an Intermediate
Process that adds information to records, noting correlations among them,
or generates new records with correlated data from multiple records (e.g.,
the production of bidirectional flow records from unidirectional flow
records). (as in ): An intermediate process that takes Data Records and
transforms them into Anonymized Data Records. (as in ): An Intermediate Selection Process is an Intermediate
Process that selects records from a sequence based upon criteria-evaluated
record values and passes only those records that match the criteria (e.g.,
Filtering only records from a given network to a given Collector). (as in : An Intermediate Flow
Selection Process is an Intermediate Process as in [RFC6183] that takes
Flow Records as its input and selects a subset of this set as its output.
Intermediate Flow Selection Process is a more general concept than
Intermediate Selection Process as defined in [RFC6183]. While an
Intermediate Selection Process selects Flow Records from a sequence based
upon criteria-evaluated Flow record values and passes only those Flow
Records that match the criteria, an Intermediate Flow Selection Process
selects Flow Records using selection criteria applicable to a larger set of
Flow characteristics and information.Note: for more information on the difference between Intermediate Flow
Selection Process and Intermediate Selection Process, see Section 4 in
.The format of the IPFIX Message Header as exported by an IPFIX Mediator is
shown in . This is identical to the format defined
for IPFIX in , though Export Time and Observation
Domain ID may be handled differently at certain Mediators, as noted below.The header fields as exported by an IPFIX Mediator are describe below.Version of IPFIX to which this Message conforms.
The value of this field is 0x000a for the current version, incrementing
by one the version used in the NetFlow services export version 9 .Total length of the IPFIX Message, measured in
octets, including Message Header and Set(s).Time at which the IPFIX Message Header
leaves the IPFIX Mediator, expressed in seconds since the UNIX epoch of
1 January 1970 at 00:00 UTC, encoded as an unsigned 32-bit integer.
However, in the specific case of an IPFIX Mediator
containing an Intermediate Conversion Process, the IPFIX Mediator MAY
use the export time received from the incoming Transport Session. Incremental sequence counter modulo 2^32
of all IPFIX Data Records sent in a the current stream from the current
Observation Domain by the Exporting Process. Each SCTP Stream counts
sequence numbers separately, while all messages in a TCP connection or UDP
transport session are considered to be part of the same stream. This value
SHOULD be used by the Collecting Process to identify whether any IPFIX
Data Records have been missed. Template and Options Template Records do
not increase the Sequence Number. A 32-bit identifier of the
Observation Domain that is locally unique to the Exporting Process. The
Exporting Process uses the Observation Domain ID to uniquely identify to
the Collecting Process the Observation Domain that metered the Flows. It
is RECOMMENDED that this identifier also be unique per IPFIX Device.
Collecting Processes SHOULD use the Transport Session and the
Observation Domain ID field to separate different export streams
originating from the same Exporter. The Observation Domain ID
SHOULD be 0 when no specific Observation Domain ID is relevant for
the entire IPFIX Message, for example, when exporting the
Exporting Process Statistics, or in case of a hierarchy of
Collectors when aggregated Data Records are exported. See
for special considerations for
Observation Domain management while passing unmodified templates through an IPFIX
Mediator, and for guidelines for preservation of
original Observation Domain information at an IPFIX Mediator.The following specifications, copied over from have some implications in
this document: "Template Withdrawals MAY appear interleaved with Template Sets,
Options Template Sets, and Data Sets within an IPFIX Message. In this
case, the Templates and Template Withdrawals shall be taken to take
effect in the order in which they appear in the IPFIX Message."If an IPFIX Mediator receives an IPFIX Message composed of Template
Withdrawals and Template Sets, and if the IPFIX Mediator forwards this IPFIX
Message, it MUST not modify the Set order. If an IPFIX Mediator receives
IPFIX Messages composed of Template Withdrawals and Template Sets, and if
the IPFIX Mediator forwards these IPFIX Messages, it MUST not modify the
IPFIX Message order. Note that the Template Mapping (see section 4.1) is
the authoritative source of information on the IPFIX Mediator to decide
whether the entire IPFIX Messages can be forwarded as such.How an IPFIX Mediator handles the Templates it receives from the Original
Exporter depends entirely on the nature of the Intermediate Process running
on that IPFIX Mediator. There are two cases here:IPFIX Mediators that pass substantially the same Data Records from the
Original Exporter downstream (e.g., an Intermediate Selection Process),
pass unmodified Templates as described in ; this section describes a Template Mapping
required to make this work in the general case, and the correlation between
the received and generated IPFIX Message Withdrawals.IPFIX Mediators that export Data Records which are substantially changed
from the Data Records received from the Original Exporter follow the
guidelines in instead: in this case, the
IPFIX Mediator generates new (Options) Template Records as a result of the
Intermediate Process, and no Template Mapping is required.Subsequent subsections deal with specific issues in Template management
that may occur at IPFIX Mediators.For some Intermediate Processes, the IPFIX Mediator doesn't modify the
(Options) Template Record(s) content. A typical example is an Intermediate
Flow Selection Process acting as distributor, which collects Flow Records
from one or more Exporters, and based on the Information Elements content,
redirects the Flow Records to the appropriate Collector. This example is a
typical case of a single network operation center managing multiple
universities: an unique IPFIX Collector collects all Flow Records for the
common infrastructure, but might be re-exporting specific university Flow
Records to the responsible system administrator. As specified in ,
the Template IDs are unique per Exporter, per Transport Session, and per
Observation Domain. As there is no guarantee that, for similar Template
Records, the Template IDs received on the incoming Transport Session and
exported to the outgoing Transport Session would be same, the IPFIX
Mediator MUST maintain a Template Mapping composed of related received and
exported (Options) Template Records:for each received (Options) Template Record: Template Record
Information Elements, Template ID, Observation Domain Id, and
Transport Session Information, metadata scoped to the Template (*)for each exported (Options) Template Record: Template Record
Information Elements, Template ID, Collector, Observation Domain Id,
and Transport Session Information metadata scoped to the Template (*)(*) The "metadata scoped to the Template" encompasses the metadata,
that are scoped to the Template, and that help to determine the semantics
of the Template Record. Note that these metadata are typically sent in
Data Records described by an Options Template. A example is the
flowKeyIndicator: An IPFIX Mediator could potentially received two
different Template IDs, from the same Exporter, with the same Information
Elements, but with a different set of Flow Keys (indicated by the
flowKeyIndicator in an Options Template Record). Another example is the
combination of anonymizationFlags and anonymizationTechnique ). This metadata information must be present in the
Template Mapping, to stress that the two Template Record semantics are
different. If an IPFIX Mediator receives an IPFIX Withdrawal Message for a
(Options) Template Record that is not used anymore in any other Template
Mappings, the IPFIX Mediator SHOULD export the appropriate IPFIX
Withdrawal Message(s) on the outgoing Transport Session, and remove the
corresponding entry in the Template Mapping.If a (Options) Template Record is not used anymore in an outgoing
Transport Session, it MUST be withdrawn with an IPFIX Template Withdrawal
Message on that specific outgoing Transport Session, and its entry MUST be
removed from the Template Mapping.If an incoming or outgoing Transport Session is gracefully shutdown or
reset, the (Options) Template Records corresponding to that Transport
Session MUST be removed from the Template Mapping.For example, displays an example
of an Intermediate Flow Selection Process, re-distributing Data Records to
Collectors on the basis of customer networks, i.e. the Route Distinguisher
(RD). In this example, the Template Record received from the Exporter #1
is reused towards Collector #1, Collector #2, and Collector #3, for the
customer #1, customer #2, and customer #3, respectively. In this
example, the outgoing Template Records exported to the different
Collectors are identical. As a reminder that the Template ID uniqueness
is local to the Transport Session and Observation Domain that generated
the Template ID, a mix of Template ID 256 and 257 has been used. shows the Template Mapping for the system shown in .The Template Mapping corresponding to Figure 3 is displayed in Figure 4:Alternatively, the Template Mapping may be optimized as in Figure 5:Note that all examples use Transport Sessions based on the SCTP
protocol, as simplified use cases. However, the transport protocol would be
important in situations such as an Intermediate Conversion Process doing
transport protocol conversion.In the situation where Original Exporters each export an (Options)
Template to a single IPFIX Mediator, and the (Options) Template Record
contains the same Information Elements but in different order, should the
IPFIX Mediator maintain a Template Mapping with a single Export Template
Record (see Figure 6) or should the IPFIX Mediator maintain multiple independent
Template Records (see Figure 7) before re-exporting to the Collector? The answer depends whether the order of the Information Elements
implies some specific semantic. One of the guiding principles in IPFIX
protocol specifications is that the semantic meaning of one
Information Element doesn't depend on the value of any other
Information Element. However, there is one noticeable exception, as
mentioned in :"Multiple Scope Fields MAY be present in the Options Template Record,
in which case, the composite scope is the combination of the scopes.
For example, if the two scopes are meteringProcessId and templateId,
the combined scope is this Template for this Metering Process. If a
different order of Scope Fields would result in a Record having a
different semantic meaning, then the order of Scope Fields MUST be
preserved by the Exporting Process. For example, in the context of
PSAMP [RFC5476], if the first scope defines the filtering function,
while the second scope defines the sampling function, the order of
the scope is important. Applying the sampling function first,
followed by the filtering function, would lead to potentially
different Data Records than applying the filtering function first,
followed by the sampling function."If an IPFIX Mediator receives, from multiple Exporters, Template
Records with identical Information Elements, but ordered differently, it
SHOULD consider those Template Records as identical, subject to metadata
information in the associated Options Template (for example, the Flow Key
Options Template. See Section 10.2).If an IPFIX Mediator receives, from multiple Exporters, Options
Template Records with identical and ordered Information Elements in the
Scope fields, and with identical Information Elements, but ordered
differently, in the non Scope fields, it SHOULD consider those Template
Records as identical.If an IPFIX Mediator receives, from multiple Exporters, Options
Template Records with identical Information Elements in the scope, but
ordered differently, it MUST consider those Template Records as
semantically different.For other intermediate processes, the IPFIX Mediator generates new
(Options) Template Records as a result of the Intermediate Process.In these cases, the IPFIX Mediator doesn't need to maintain a Template
Mapping, as it generates its own series of (Options) Template Records.
However, the following special case might still require a Template Mapping,
i.e. a situation where the IPFIX Mediator, typically containing an
Intermediate Conversion Process, Intermediate Aggregation Process, or
Intermediate Anonymization Process in case of black-marker Anonymization
, generates new (Options) Template Records based on
what it receives from the Exporter(s), and based on the Intermediate
Process function. In such a case, it's important to keep the correlation
between the received (Options) Template Records and derived (Options)
Template Records in the Template Mapping. These Template Mappings would be
kept as in , except that the exported
Template would not be identical to the received Template.Depending on application requirements, Mediators which do not generate
new Records SHOULD re-export values for unknown Information Elements, for
which the Mediator does not have information about Information Element
data type and semantics. However, as there may be presence or ordering
dependencies among the unknown Information Elements, the Mediator MUST
NOT omit fields from such re-exported Records, or re-order any fields
within the Records.Mediators which generate new Records, as in , SHOULD NOT use values of Information Elements
they do not understand. If they do pass such values, they MUST NOT pass
values of unknown Information Elements unless all such values are passed
on in the original order in which they were received.In any case, Mediators handling unknown Information Elements SHOULD
log this fact, as it is likely that mediation of records containing
unknown values will have unintended consequences.Depending on the use case, the Collector in an Exporter - IPFIX Mediator -
Collector structure (for example tiered Mediators) may need to receive information
about the Original Observation Point(s), otherwise it may wrongly conclude that the
IPFIX Device exporting the Flow Records, i.e. the IPFIX Mediator, directly
observed the packets that generated the Flow Records. Two new Information
Elements are introduced to address this use case:
originalExporterIPv4Address and originalExporterIPv6Address. Practically,
the Original Exporters will not be exporting these Information Elements.
Therefore, the Intermediate Process SHOULD report the Original Observation
Point(s) to the best of its knowledge. Note that the Configuration Data
Model for IPFIX and PSAMP may report the Original
Exporter information out of band.In the IPFIX Mediator, the Observation Point(s) may be represented
by:A single Original Exporter (represented by the
originalExporterIPv4Address or originalExporterIPv6Address Information
Elements)A list of Original Exporters (represented by a list of
originalExporterIPv4Address or originalExporterIPv6Address Information
Elements).Any combination or list of Information Elements representing
Observation Points. For example:
A list of Original Exporter interface(s) (represented by the
originalExporterIPv4Address or originalExporterIPv6Address, the
ingressInterface and/or egressInterface Information Elements,
respectively)A list of Original Exporter line card (represented by the
originalExporterIPv4Address or originalExporterIPv6Address, the
lineCardId Information Elements, respectively)Some Information Elements characterizing the Observation Point may be
added. For example, the flowDirection Information Element specifies the
direction of the observation, and, as such, characterizes the Observation
Point.Any combination of the above representations is possible. An example
of an Original Observation Point for an Intermediate Aggregation Process
is displayed in Figure 8.A Mediator MAY export such complex Original Observation Point information,
depending on application requirements. If such information is exported, the
Mediator MUST use to do so, as described below.The most generic way to export the Original Observation Point is to use a
subTemplateMultiList, with the semantic "exactlyOneOf". Taking the previous
example, the encoding in Figure 9 can be used.The Original Observation Point is modeled with the Data Records
corresponding to either Template Record 1, Template Record 2, or Template
Record 3 but not more than one of these ("exactlyOneOf" semantic). This
implies that the Flow was observed at exactly one of the Observation Points
reported.When an IPFIX Mediator receives Flow Records containing the Original
Observation Point Information Element, i.e. originalExporterIPv4Address or
originalExporterIPv6Address, the IPFIX Mediator SHOULD NOT modify its
value(s) when composing new Flow Records in the general case. Known
exceptions include anonymization per section 7.2.4
and an Intermediate Correlation Process rewriting addresses across NAT. In
other words, the Original Observation Point should not be replaced with the
IPFIX Mediator Observation Point. The daisy chain of (Exporter, Observation
Point) representing the path the Flow Records took from the Exporter to the
top Collector in the Exporter - IPFIX Mediator(s) - Collector structure model
is out of the scope of this specification.originalExporterIPv4AddressThe IPv4 address used by the Exporting
Process on an Original Exporter, as seen by the Collecting Process on
an IPFIX Mediator. Used to provide information about the Original
Observation Points to a downstream Collector.ipv4AddressTBD1originalExporterIPv6AddressThe IPv6 address used by the Exporting
Process on an Original Exporter, as seen by the Collecting Process on
an IPFIX Mediator. Used to provide information about the Original
Observation Points to a downstream Collector.ipv6AddressTBD2The Observation Domain ID of any IPFIX Message containing
Flow Records relevant to no particular Observation Domain, or to multiple
Observation Domains, MUST have an Observation Domain ID of 0, as in
above, and section 3.1 of .IPFIX Mediators that do not change (Options) Template Records MUST
maintain a Template Mapping, as detailed in , to ensure that the combination of
Observation Domain IDs and Template IDs do not collide on export.For IPFIX Mediators that export New (Options) Template Records, as in
, there are two options for Observation
Domain ID management. The first and simplest of these is to completely
decouple exported Observation Domain IDs from received Observation Domain
IDs; the IPFIX Mediator, in this case, comprises its own set of
Observation Domain(s) independent of the Observation Domain(s) of the
Original Exporters.The second option is to provide or maintain a Template Mapping for
received (Options) Template Records and exported inferred (Options)
Template Records, along with the appropriate Observation Domain IDs per
Transport Session, which ensures that the combination of Observation
Domain IDs and Template IDs do not collide on export.In some cases where the IPFIX Message Header can't contain a consistent
Observation Domain for the entire IPFIX Message, but the Flow Records
exported from the IPFIX Mediator should anyway contain the Observation
Domain of the Original Exporter, the (Options) Template Record must
contain the originalObservationDomainId Information Element, specified in
Section 6.1. When an IPFIX Mediator receives Flow Records containing the
originalObservationDomainId Information Element, the IPFIX Mediator MUST NOT
modify its value(s) when composing new Flow Records with the
originalObservationDomainId Information Element.originalObservationDomainIdThe Observation Domain ID reported by the
Exporting Process on an Original Exporter, as seen by the Collecting
Process on an IPFIX Mediator. Used to provide information about the
Original Observation Domain to a downstream Collector.unsigned32identifierTBD3The IPFIX Message Header "Export Time" field is the time in seconds
since 0000 UTC Jan 1, 1970, at which the IPFIX Message leaves the IPFIX
Mediator. However, in the specific case of an IPFIX Mediator containing
an Intermediate Conversion Process, the IPFIX Mediator MAY use the
export time received from the incoming Transport Session.It is RECOMMENDED that IPFIX Mediators handle time using absolute
timestamps (e.g. flowStartSeconds, flowStartMilliseconds,
flowStartNanoseconds), which are specified relative to the UNIX epoch
(00:00 UTC 1 Jan 1970), where possible, rather than relative timestamps
(e.g. flowStartSysUpTime, flowStartDeltaMicroseconds), which are specified
relative to protocol structures such as system initialization or message
export time.The latter are difficult to manage for two reasons. First, they require
constant translation, as the system initialization time of an intermediate
system and the export time of an intermediate message will change across
mediation operations. Further, relative timestamps introduce range
problems. For example, when using the flowStartDeltaMicroseconds and
flowEndDeltaMicroseconds Information Elements , the Data Record must be exported within
a maximum of 71 minutes after its creation. Otherwise, the 32-bit counter
would not be sufficient to contain the flow start time offset. Those time
constraints might be incompatible with some of the application
requirements of some Intermediate Processes.Intermediate Processes MUST NOT assume that received records appear in
flowStartTime, flowEndTime, or observationTime order. An Intermediate
Process processing timing information (e.g., an Intermediate Aggregation
Process) MAY ignore records that are significantly out of order, in order
to meet application-specific state and latency requirements, but SHOULD
report that records were dropped.When an Intermediate Process aggregates information from different Flow
Records, the timestamps on exported records SHOULD be the minimum of the
start times and the maximum of the end times in the general case. However,
if the Flow Records do not overlap, i.e. if there is a time gap between the
times in the Flow Records, then the report may be inaccurate. The IPFIX
Mediator is only reporting what it knows, on the basis of the information
made available to it - and there may not have been any data to observe
during the gap. Then again, if there is an overlap in timestamps, there's
the potential of double-accounting: different Observation Points may have
observed the same traffic simultaneously. The specification of the precise
rules for applying Flow Record timestamps at IPFIX Mediators for all the
different situations is out of the scope of this document.Note that provides additional
specifications for handling of timestamps at an Intermediate Aggregation
Process.SCTP using the PR-SCTP extension specified in
MUST be implemented by all compliant IPFIX Mediator
implementations. TCP MAY also be implemented by
IPFIX Mediator compliant implementations. UDP MAY
also be implemented by compliant IPFIX Mediator implementations.
Transport-specific considerations for IPFIX Exporters as specified in
sections 8.3, 8.4, 9.1, 9.2, and 10 of apply to IPFIX Mediators as
well.SCTP SHOULD be used in deployments where IPFIX Mediators and
Collectors are communicating over links that are susceptible to congestion.
SCTP is capable of providing any required degree of reliability. TCP MAY
be used in deployments where IPFIX Mediators and Collectors communicate over
links that are susceptible to congestion, but SCTP is preferred due to
its ability to limit back pressure on Exporters and its message versus
stream orientation. UDP MAY be used, although it is not a congestion-aware
protocol. However, in this case, the IPFIX traffic between IPFIX Mediator and
Collector MUST run in an environment where IPFIX traffic has been
provisioned for and/or separated from non-IPFIX traffic, whether
physically or virtually.Any Collecting Process compliant with can receive IPFIX Messages from
an IPFIX Mediator. If the IPFIX Mediator uses IPFIX
Structured Data to export Original Exporter Information as in , the Collecting Process MUST support .IPFIX provides Options Templates for the reporting the reliability of
processes within the IPFIX Architecture. As each Mediator includes at least
one IPFIX Exporting Process, they MAY use the Exporting Process Reliability
Statistics Options Template, as specified in .Analogous to the Metering Process Reliability Statistics Options
Template, also specified in , Mediators MAY implement the
Intermediate Process Reliability Statistics Options Template, specified in
Section 10.1.The Flow Keys Options Template, as specified in , may require special
handling at an IPFIX Mediator as described in Section 10.2.In addition, each Intermediate Process may have its own specific
reporting requirements (e.g. Anonymization Records as in , or the Aggregation Counter Distribution Options
Template as in ); these SHOULD be
implemented as necessary, as described in the specification for each
Intermediate Process.The Intermediate Process Statistics Options Template specifies the
structure of a Data Record for reporting Intermediate Process
statistics. It SHOULD contain the following Information Elements; the
intermediateProcessId Information Element is defined in , and the ignoredFlowRecordTotalCount Information
Element is defined in : IEDescriptionobservationDomainId [scope]
An identifier of the Observation Domain (of messages exported by
this Mediator), locally unique to the Intermediate Process, to
which this statistics record applies.
---------------------------------- intermediateProcessId [scope]
An identifier for the Intermediate Process to which this
statistics record applies.
---------------------------------- ignoredFlowRecordTotalCount
The total number of Data Records received but not processed by
the Intermediate Process.
---------------------------------- time first record ignored
The timestamp of the first record that was ignored by the
Intermediate Process. For Data Records containing timestamp
ranges, this SHOULD be taken from the start timestamp of the
range; for data records containing no timing information, this
SHOULD be taken from the Export Time in the message header of
the containing IPFIX Message. For this timestamp, any of the
following timestamp can be used: observationTimeSeconds,
observationTimeMilliseconds, observationTimeMicroseconds, or
observationTimeNanoseconds.
---------------------------------- time last record ignored
The timestamp of the last record that was ignored by the
Intermediate Process. For Data Records containing timestamp
ranges, this SHOULD be taken from the end timestamp of the
range; for data records containing no timing information, this
SHOULD be taken from the Export Time in the message header of
the containing IPFIX Message. For this timestamp, any of the
following timestamp can be used: observationTimeSeconds,
observationTimeMilliseconds, observationTimeMicroseconds, or
observationTimeNanoseconds.
The Flow Keys Options Template specifies the structure of a Data Record
for reporting the Flow Keys of reported Flows. A Flow Keys Data Record
extends a particular Template Record that is referenced by its templateId
identifier. The Template Record is extended by specifying which of the
Information Elements contained in the corresponding Data Records describe
Flow properties that serve as Flow Keys of the reported Flow. This Options
Template is defined in section 4.4 of , and SHOULD be used by
Mediators for export as defined there.When an Intermediate Process exports Data Records containing different
Flow Keys from those received from the Original Exporter, and the Original
Exporter sent a Flow Keys Options record to the IPFIX Mediator, the IPFIX
Mediator MUST export a Flow Keys Options record defining the new set of
Flow Keys.intermediateProcessIdAn identifier of an Intermediate Process that is unique per IPFIX Device. Typically, this Information Element is used for limiting the scope of other Information Elements. Note that process identifiers may be assigned dynamically; ie., an Intermediate Process may be re-started with a different ID.unsigned32identifierTBD4ignoredFlowRecordTotalCountThe total number of received Data Records that the Intermediate Process did not process since the (re-)initialization of the Intermediate Process; includes only Data Records not examined or otherwise handled by the Intermediate Process due to resource constraints, not Data Records which were examined or otherwise handled by the Intermediate Process but which merely do not contribute to any exported Data Record due to the operations performed by the Intermediate Process.unsigned64totalCounterTBD5In general, using IPFIX Mediators to combine information from multiple
Original Exporters requires a consistent configuration of the Metering
Processes behind these Original Exporters. The details of this consistency
are specific to each Intermediate Process. Consistency of configuration
should be verified out of band, with the MIB modules ( and ) or with the Configuration
Data Model for IPFIX and PSAMP .As they act as both IPFIX Collecting Processes and Exporting Processes,
the Security Considerations for the IPFIX Protocol also apply to IPFIX Mediators. The
Security Considerations for IPFIX Files also apply
to IPFIX Mediators that write IPFIX Files or use them for internal storage.
However, there are a few specific considerations that IPFIX Mediator
implementations must also take into account.By design, IPFIX Mediators are "men-in-the-middle": they intercede in the
communication between an Original Exporter (or another upstream IPFIX Mediator)
and a downstream Collecting Process. This has two important implications for
the level of confidentiality provided across an IPFIX Mediator, and the
ability to protect data integrity and Original Exporter authenticity across
an IPFIX Mediator. These are addressed in more detail in the Security
Considerations for IPFIX Mediators in .Note that, while IPFIX Mediators can use the exporterCertificate and
collectorCertificate Information Elements defined in as described in section 9.3 of
to export information about X.509 identities in upstream TLS-protected
Transport Sessions, this mechanism cannot be used to provide true end-to-end
assertions about a chain of IPFIX Mediators: any IPFIX Mediator in the chain can
simply falsify the information about upstream Transport Sessions. In
situations where information about the chain of mediation is important, it
must be determined out of band.This document specifies new IPFIX Information Elements,
originalExporterIPv4Address in ,
originalExporterIPv6Address in ,
originalObservationDomainId in ,
intermediateProcessId in , and
ignoredFlowRecordTotalCount in , to be added to the
IPFIX Information Element
registry. [IANA NOTE: please add the five Information Elements as
specified in the references subsections, change TBD1, TBD2, TBD3, TBD4,
and TBD5 in this document to reflect the assigned identifiers, put the Status
as current, insert THISRFC into the Requester entry, insert 0 for the Revision,
and use the current date for Date.]We would like to thank the IPFIX contributors, specifically Paul Aitken
(THE ultimate IPFIX document reviewer) and Andrew Feren for their thorough
reviews, and Rahul Patel for his feedback and comments. This work is
materially supported by the European Union Seventh Framework Programme under
grant agreement 257315 (DEMONS).IP Flow Information Export Information Elements (http://www.iana.org/assignments/ipfix/ipfix.xml)IEEE 1003.1-2008 - IEEE Standard for Information Technology - Portable Operating System Interface