Subject: [ipfix] current enterprise IE approach is insufficient

From: David Moore <[email-masked]>
Date: Thu, 22 Sep 2005 08:20:34 -0700

In BOF and small group discussions at FloCon this week, several current
developers of flow meters/exporters had concerns with the usefulness of
IPFIX for their data.  Developers of analysis/aggregation/reporting
tools also had similar concerns about their ability to write generic
processing tools.  While several concerns were raised, this one seemed
to be the greatest problem for both adoption and functionality.

As a short summary: The current information model approach only provides
information elements in the templates which combine *both* data type
(i.e. section 3 of info draft) and specific semantics.  And there is no
current mechanism to provide the data type for a given information
element (e.g. a vendor specific one).  Allowing this information to be
provided either in the data-stream via templates/records or via some
automatable external lookup (e.g. XDS).  The current situation makes
writing lots of analysis tools difficult.

Personal, real life example of the pain:

For caida network telescope data about DoS backscatter, one natural
representation of the data would be in flows which contain 3 IP
addresses as part of the key.  We call these 3 addresses: Attack IP,
Target IP and Responding IP.  None of these 3 addresses have semantic
meaning which natural maps to IPFIX's sourceIPv4Address or
sourceIPv4Address.  However all five have the same data type,
ipv4Address.

So the current procedure for me to provide this data in an IPFIX format
would be to create 3 new IE (information elements) identifiers within my
enterpriseID space.  At first glance, this seems like there is
extensibility supported in the protocol for me to do what I want.
However, I will not be able to use any existing analysis tools on this
data, since they will not recognize these IEs.  As a specific example, I
would be unable to use nafscii to convert my records to a text format
without either adding code to nafscii or explicitly adding my IEs to
nafscii config files in a tool-specific manner.  But nafscii (and a
large set of other tools) doesn't need to know the semantics of these
fields to display them.  Many tools only need to know the field's data
type to work on them.  However, the IPFIX working group hasn't provided
any mechanism for people to provide the data type of an IE in a
standardized automatable way.

Having to do some sort of tool-specific configuration (or code
modification) on every tool just to tell it that X is an ipv4Address and
Y is a totalCounter in an unsigned64 is a huge amount of work for users
and developers of vendor specific IEs.

Two possible approaches to get around this would be to have some
mechanism for including the data type infomation in the stream, perhaps
via some sort of record which allows mapping <enterpriseID,IE> pairs to
data type information, or to have some external mechanism for tools to
be able to obtain the information about IEs for a given enterpriseID.
One potential advantage of the external approach is that there would
then be plenty of space to include all of the additional information
that one would want, such as the name, description, references, etc.
However, including the information inline, allows more rapid
processessing of the data, and allows the flow stream (or file) to be
partially self-documenting (the point of templates in general).

Solving this problem seems very appropriate for the ipfix wg.

-- david

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Jeroen Massar <[email-masked]>
Date: Thu, 22 Sep 2005 18:47:06 +0200

David Moore wrote:
<SNIP>
> (i.e. section 3 of info draft) and specific semantics.

This is also an idea that popped into my head. In SNMP one knows what
type a certain field is, we could also define this for IPFIX records. Eg
 IPv4Address, IPv6Address, Counter, Gauge, etc, just look at SNMP.
Basically this will turn IPFIX into a push-model SNMP which has it's own
set of (dis)advantages.

Greets,
 Jeroen

---

From: Nevil Brownlee <[email-masked]>
Date: Mon, 26 Sep 2005 09:13:01 +1200

> Solving this problem seems very appropriate for the ipfix wg.

I think David summarised this problem very clearly.  IPFIX has tended
to focus on minimising the overhead for its info elements, that's
why their type is included in the semantics for each one.  The notion
was that when a collector meets an IE it doesn't know, it does know its
length, so it can preserve it in archived files, etc.

Having a way to create generalised, (probably) application-specific
IE, e.g. three IPv4 addresses, in a way that allows 'generalised' collecting
and processing packages to know their type so as to display them sounds useful 
to me.

Right now the IPFIX WG is trying hard to get its four current drafts submitted
to IESG, so that the protocol and information model (at last) have an agreed
standard form.  That will complete the current charter work items.

From there, there are certainly quite a few potential work items the group
could take on.  However, for that to happen, we'd need to modify/extend
the WG charter.  The most effective way to do that is for interested groups
to develop ways to extend IPFIX (within/buiilding on its current framework)
to achieve whetever it is they need, then publish their proposal for that
as individual IDs.  That way we have a clearly-defined, active community,
who can be seen to be working to get their proposal standardised.

So now, David has put forward the case for 'generic, type-specified
information elements' in IPFIX.  The mailing list is here to support
discussion and development for extensions like that ...

Cheers, Nevil

-----------------------------------------------------------------------
   Nevil Brownlee                 Computer Science Department | ITSS
   Phone: +64 9 373 7599 x88941           The University of Auckland
   FAX: +64 9 373 7021      Private Bag 92019, Auckland, New Zealand

-------------------------------------------------
This mail sent through University of Auckland
http://www.auckland.ac.nz/