Re: [Sipping] Alternate CLF syntax proposal

Theo Zourzouvillys <theo@crazygreek.co.uk> Sun, 29 March 2009 10:25 UTC

MIME-Version: 1.0
In-Reply-To: <c164605b0903260836p45a8bd8eg9ad50f670ef82302@mail.gmail.com>
References: <49CAE21C.5060309@nostrum.com> <c164605b0903260836p45a8bd8eg9ad50f670ef82302@mail.gmail.com>
Date: Sun, 29 Mar 2009 11:26:39 +0100
Message-ID: <167dfb9b0903290326t519e6df8naacdb396a259f8f2@mail.gmail.com>
From: Theo Zourzouvillys <theo@crazygreek.co.uk>
To: Jason Fischl <jason.fischl@gmail.com>
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7bit
Cc: sipping WG <sipping@ietf.org>, Adam Roach <adam@nostrum.com>, draft-gurbani-sipping-clf@tools.ietf.org
Subject: Re: [Sipping] Alternate CLF syntax proposal
Precedence: list

On Thu, Mar 26, 2009 at 4:36 PM, Jason Fischl <jason.fischl@gmail.com> wrote:
> I quite like this scheme. It is very simple to generate and parse and will
> be blazingly fast. It would also be very simple to create a utility to
> generate "human-readable" versions from it which would address that
> particular concern.

I also really like Adam's binary proposal and would much prefer we
progress down a binary path instead of plain text.  My reasoning is
that the point of standardising such a format would be for analysis or
monitoring/diagnostics, which would inherently be processed by
machines, not humans.  The only advantage of a text format is for tail
-f'ing on a live server;  such a thing has no reason to be
standardised short of the operator in question knowing which fields
are which in the same way i do when tailing an access.log.  This is
commonly done by something such as:

 LogFormat "%h %l %u %t \"%r\" %>s %b" common

and can easily be changed to include other stuff on a
per-server/implementation basis, i.e:

 LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-Agent}i\" VLOG=%{VLOG}e" vhost

However, the format which would be processed by collectors, stored,
analysed, and possibly post-processed or merged with others would
really want to be binary.  We currently use our own format internally
for transmitting between exporters and collectors and it would be nice
to see a standardised format so we can perhaps build some open source
analysis tools that would work with multiple vendors or a collector
that provides nice network statistics.

> Comments about the format:
> - add a version number to the header
> - set aside a range of tags for vendor attributes.

yup, and in addition, i would like to see an exporter identifier.

> A few miscellaneous comments about contents:
> - for sent messages: local ip:port used to send the message + transport used
> + destination ip:port
> - for received messages: remote ip:port message was received from + local
> ip:port + transport used
> - it may be a good idea to have a predefined tag which can be used as an
> index into another diagnostic file. e.g. pointer into a PCAP file or vendor
> diagnostic log.
> - Is there anything we want to jam in here to help debug TCP/TLS issues or
> is that out of scope?

.. infact, come to think of it, IPFIX (RFC5101) seems perfect for all
of this - It would even allow encoding lower level data should you
want to such as ICMP type/code, source/dst IP, interfaces, or TCP/TLS
packet, SYN/ACK/RST events etc.

all we'd need to do is define a bunch of IEs in a document, and the
rest would "just work" (tm).

 ~ Theo

Sent from: Bicester Oxfordshire United Kingdom.

[Sipping] Alternate CLF syntax proposal Adam Roach
Re: [Sipping] Alternate CLF syntax proposal Vijay K. Gurbani
Re: [Sipping] Alternate CLF syntax proposal Jason Fischl
Re: [Sipping] Alternate CLF syntax proposal Hadriel Kaplan
Re: [Sipping] Alternate CLF syntax proposal Hadriel Kaplan
Re: [Sipping] Alternate CLF syntax proposal Hadriel Kaplan
Re: [Sipping] Alternate CLF syntax proposal Theo Zourzouvillys
Re: [Sipping] Alternate CLF syntax proposal Jiri Kuthan
Re: [Sipping] Alternate CLF syntax proposal Theo Zourzouvillys
Re: [Sipping] Alternate CLF syntax proposal Cullen Jennings
Re: [Sipping] Alternate CLF syntax proposal Daryl Malas
Re: [Sipping] Alternate CLF syntax proposal Hadriel Kaplan
Re: [Sipping] Alternate CLF syntax proposal Hadriel Kaplan
Re: [Sipping] Alternate CLF syntax proposal Theo Zourzouvillys
Re: [Sipping] Alternate CLF syntax proposal Hisham Khartabil
Re: [Sipping] Alternate CLF syntax proposal Vijay Gurbani
Re: [Sipping] Alternate CLF syntax proposal Dale Worley
Re: [Sipping] Alternate CLF syntax proposal Adam Roach
Re: [Sipping] Alternate CLF syntax proposal Adam Roach