IPFIX File Format for SIPCLF
draft-niccolini-sipclf-ipfix-03

Abstract

This draft defines an log file format conforming to the information model defined in the SIP-CLF problem statement based upon the IPFIX File Format. It details the creation and intepretation of these files, and provides examples based on common SIP situations.

Status of this Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

This Internet-Draft will expire on January 7, 2011.

Copyright Notice

Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

1.  Introduction
2.  Information Elements for SIPCLF
3.  IPFIX File Format for SIPCLF
    3.1.  Example
4.  Security Considerations
5.  IANA Considerations
6.  Acknowledgments
7.  Open Issues
8.  References
    8.1.  Normative References
    8.2.  Informative References
§  Authors' Addresses

1.  Introduction

The SIPCLF WG is chartered to produce a format suitable for logging at any SIP element. The charter explicitly addresses the need to search, merge and summarize log records from one or more possibly diverse elements as well as the need to correlate messages from multiple elements. An additional consideration to be taken into account when defining the file format is the extensibility of SIP. SIPCLF is additionally chartered to identify the fields to appear in a log record and provide one or more formats for encoding those fields. As the IPFIX File format (Trammell, B., Boschi, E., Mark, L., Zseby, T., and A. Wagner, “Specification of the IP Flow Information Export (IPFIX) File Format,” October 2009.) [RFC5655] meets these requirements, SIPCLF is presently considering an IPFIX File-based binary logging format, in addition to an ASCII text-based format. This document describes the IPFIX format.

Specifically, this document defines new information elements for the SIPCLF IPFIX file format (based on the data model detailed in [I‑D.gurbani‑sipclf‑problem‑statement] (Gurbani, V., Burger, E., Anjali, T., Abdelnur, H., and O. Festor, “The Common Log Format (CLF) for the Session Initiation Protocol (SIP),” January 2010.)), proposes common IPFIX Templates for this data model, and presents examples of an IPFIX File logging format for SIPCLF. For this purpose examples were taken from [RFC3665] (Johnston, A., Donovan, S., Sparks, R., Cunningham, C., and K. Summers, “Session Initiation Protocol (SIP) Basic Call Flow Examples,” December 2003.).

2.  Information Elements for SIPCLF

This section formally defines information elements for SIPCLF based on (and extending) the data model initially proposed in [I‑D.gurbani‑sipclf‑problem‑statement] (Gurbani, V., Burger, E., Anjali, T., Abdelnur, H., and O. Festor, “The Common Log Format (CLF) for the Session Initiation Protocol (SIP),” January 2010.).

The information appearing in a SIPCLF log record is largely already defined in other documents, e.g., in [RFC3261] (Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, “SIP: Session Initiation Protocol,” June 2002.).

The information elements are shown in the table below; new Information Elements necessary for SIPCLF that are not yet registered with IANA are provisionally located within the number space assigned to Private Enterprise Number (PEN), here registered within the private enterprise number 35566, which belongs to one of the authors of this draft. It is intended that these Information Elements be registered within the IANA number space, should this draft become a Proposed Standard; however, it is necessary to define real Information Elements in order to provide examples for experimentation.

The sipMethod IE encodes the supported methods from [RFC3261] (Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, “SIP: Session Initiation Protocol,” June 2002.) as integers, by the order in which they appear in the IANA SIP Parameters Methods registry at http://www.iana.org/assignments/sip-parameters:

3.  IPFIX File Format for SIPCLF

The IPFIX File format is defined in [RFC5655] (Trammell, B., Boschi, E., Mark, L., Zseby, T., and A. Wagner, “Specification of the IP Flow Information Export (IPFIX) File Format,” October 2009.) as a serialized set of IPFIX Messages containing Data Records organized in Sets defined by Templates; these are in turn defined in the IPFIX Protocol specification (Claise, B., “Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information,” January 2008.) [RFC5101]. The file format is designed to facilitate interoperability, flexibility, and reusability among a wide variety of storage, processing, and analysis tools. In defining an IPFIX File format for SIPCLF, we seek to leverage this work. All that is necessary to define, then, are any Information Elements not yet defined in the IPFIX Information Model, as in the previous section; and a set of recommended Templates for representing the records required by SIPCLF. This is done in the examples below.

3.1.  Example

This section presents several views of an example IPFIX-based SIPCLF log, in order to increase understanding of what IPFIX would mean for SIPCLF. We present both binary and textual forms. The tools to generate this section are based upon the open-source ripfix (Trammell, B., “ripfix: IPFIX for Ruby,” .) [ripfix] implementation of IPFIX, maintined by one of the authors of this draft.

The information model in textual IE specification format (Trammell, B., “A Lightweight Textual Format for IPFIX Information Models and Templates,” April 2010.) [I‑D.trammell‑ipfix‑text‑iespec] is as follows; IE numbers appear in parentheses, IPFIX types in angle brackets, and sizes ("v" for variable-length) in square brackets:

sipAuthUsername(35566/401)<string>[v]
sipMethod(35566/402)<unsigned8>[1]
sipRequestURI(35566/403)<string>[v]
sipFromURI(35566/404)<string>[v]
sipFromTag(35566/405)<string>[v]
sipToURI(35566/406)<string>[v]
sipToTag(35566/407)<string>[v]
sipCallId(35566/408)<string>[v]
sipSequenceNumber(35566/409)<unsigned32>[4]
sipContactURI(35566/410)<string>[v]
sipPaiURI(35566/411)<string>[v]
sipResponseStatus(35566/412)<unsigned16>[2]
sipServerTransaction(35566/413)<string>[v]
sipClientTransaction(35566/414)<string>[v]
sipSessionId(35566/415)<string>[v]

In accordance wth version -02 of [I‑D.gurbani‑sipclf‑problem‑statement] (Gurbani, V., Burger, E., Anjali, T., Abdelnur, H., and O. Festor, “The Common Log Format (CLF) for the Session Initiation Protocol (SIP),” January 2010.), when a UAC generates a request, the SIPCLF record generated follows the template (given in textual IE specification format; IE numbers and sizes are given for convenience):

observationTimeSeconds(322)[4]
sourceIPv4Address(8)[4]
destinationIPv4Address(12)[4]
sourceTransportPort(7)[2]
destinationTransportPort(11)[2]
sipSequenceNumber(35566/409)[4]
sipMethod(35566/402)[1]
sipRequestURI(35566/403)[v]
sipClientTransaction(35566/414)[v]
sipToURI(35566/406)[v]
sipToTag(35566/407)[v]
sipFromURI(35566/404)[v]
sipFromTag(35566/405)[v]
sipCallId(35566/408)[v]

If, instead, the request is sent via IPv6, the addresses are replaced as follows:

observationTimeSeconds(322)[4]
sourceIPv6Address(27)[16]
destinationIPv6Address(28)[16]
sourceTransportPort(7)[2]
destinationTransportPort(11)[2]
sipSequenceNumber(35566/409)[4]
sipMethod(35566/402)[1]
sipRequestURI(35566/403)[v]
sipClientTransaction(35566/414)[v]
sipToURI(35566/406)[v]
sipToTag(35566/407)[v]
sipFromURI(35566/404)[v]
sipFromTag(35566/405)[v]
sipCallId(35566/408)[v]

Similarly, when a UAC receives a response, the SIPCLF record that is subsequently logged follows the template:

observationTimeSeconds(322)[4]
sourceIPv4Address(8)[4]
destinationIPv4Address(12)[4]
sourceTransportPort(7)[2]
destinationTransportPort(11)[2]
sipSequenceNumber(35566/409)[4]
sipResponseStatus(35566/412)[2]
sipMethod(35566/402)[1]
sipServerTransaction(35566/413)[v]
sipToURI(35566/406)[v]
sipToTag(35566/407)[v]

and the IPv6 response template is modified similarly to the request template:

observationTimeSeconds(322)[4]
sourceIPv6Address(27)[16]
destinationIPv6Address(28)[16]
sourceTransportPort(7)[2]
destinationTransportPort(11)[2]
sipSequenceNumber(35566/409)[4]
sipResponseStatus(35566/412)[2]
sipMethod(35566/402)[1]
sipServerTransaction(35566/413)[v]
sipToURI(35566/406)[v]
sipToTag(35566/407)[v]

Please note that the information model here defined currently uses only mandatory fields, additional optional fields will be added as this draft develops.

Other examples are also possible following what detailed in in [I‑D.gurbani‑sipclf‑problem‑statement] (Gurbani, V., Burger, E., Anjali, T., Abdelnur, H., and O. Festor, “The Common Log Format (CLF) for the Session Initiation Protocol (SIP),” January 2010.), and will elaborated further as this document is extended and when the reference data model stabilizes.

Now that we have defined an information model and templates using them, we can define a request log record and a response log record, and generate example IPFIX output using our ripfix-based Assuming that our IPv4 request template has a template ID of 1234, we can create the request log record with the following input to our tool, which accepts colon-separated Information Element, value pairs (the _ipfix_tid "special" IE selects a pre-exported template):

_ipfix_tid: 1234
observationTimeSeconds: 2010-04-01 01:10:12 +0000
sipMethod: 5 # INVITE
sipSequenceNumber: 1
sipRequestURI: sip:bob@biloxi.example.com
sourceIPv4Address: 192.168.0.3
sourceTransportPort: 37920
destinationIPv4Address: 192.168.0.2
destinationTransportPort: 5060
sipClientTransaction: ABC
sipToURI: sip:bob@biloxi.example.com
sipToTag:
sipFromURI: sip:alice@atlanta.example.com
sipFromTag: 9fxced76sl
sipCallId: 3848276298220188511@atlanta.example.com

and the response record a follows:

_ipfix_tid: 4321
observationTimeSeconds: 2010-04-01 01:10:14 +0000
sipMethod: 5 # INVITE
sipSequenceNumber: 1
sourceIPv4Address: 192.168.0.2
sourceTransportPort: 5060
destinationIPv4Address: 192.168.0.3
destinationTransportPort: 37920
sipResponseStatus: 180
sipServerTransaction: 123
sipToURI: sip:bob@biloxi.example.com
sipToTag: 8321234356

Bringing it all together: request and response records, templates, and IPFIX File format framing, we have the following file in Base64 format:

AAoBp0wzRUMAAAAAAADd1QACAKwE0gAOAUIABAAIAAQADAAEAAcAAgALAAKB
mQAEAACK7oGSAAEAAIrugZP//wAAiu6Bnv//AACK7oGW//8AAIrugZf//wAA
iu6BlP//AACK7oGV//8AAIrugZj//wAAiu4Q4QALAUIABAAIAAQADAAEAAcA
AgALAAKBmQAEAACK7oGcAAIAAIrugZIAAQAAiu6Bnf//AACK7oGW//8AAIru
gZf//wAAiu4E0gCmS7PydMCoAAPAqAAClCATxAAAAAEFGnNpcDpib2JAYmls
b3hpLmV4YW1wbGUuY29tA0FCQxpzaXA6Ym9iQGJpbG94aS5leGFtcGxlLmNv
bQEAHXNpcDphbGljZUBhdGxhbnRhLmV4YW1wbGUuY29tCjlmeGNlZDc2c2wn
Mzg0ODI3NjI5ODIyMDE4ODUxMUBhdGxhbnRhLmV4YW1wbGUuY29tEOEARUuz
8nbAqAACwKgAAxPElCAAAAABALQFAzEyMxpzaXA6Ym9iQGJpbG94aS5leGFt
cGxlLmNvbQo4MzIxMjM0MzU2

This is not a particularly useful representation for explaining the example, but does demonstrate the binary nature of the format. Here is a debug hex dump of the same file, annotated below each line or block to show the interesting structures:

0000: 00 0a 01 a7 4c 33 45 43 00 00 00 00 00 00 dd d5  ....L3EC........
     [ IPFIX message header, length 423              ]
0010: 00 02 00 ac                                      ....
     [ Template set header, set length 172 ]
                  04 d2 00 0e 01 42 00 04 00 08 00 04      .....B......
0020: 00 0c 00 04 00 07 00 02 00 0b 00 02 81 99 00 04  ................
0030: 00 00 8a ee 81 92 00 01 00 00 8a ee 81 93 ff ff  ................
0040: 00 00 8a ee 81 9e ff ff 00 00 8a ee 81 96 ff ff  ................
0050: 00 00 8a ee 81 97 ff ff 00 00 8a ee 81 94 ff ff  ................
0060: 00 00 8a ee 81 95 ff ff 00 00 8a ee 81 98 ff ff  ................
0070: 00 00 8a ee                                      ....
     [ Template 1234 (0x04d2), 14 elements ]
                  10 e1 00 0b 01 42 00 04 00 08 00 04      .....B......
0080: 00 0c 00 04 00 07 00 02 00 0b 00 02 81 99 00 04  ................
0090: 00 00 8a ee 81 9c 00 02 00 00 8a ee 81 92 00 01  ................
00a0: 00 00 8a ee 81 9d ff ff 00 00 8a ee 81 96 ff ff  ................
00b0: 00 00 8a ee 81 97 ff ff 00 00 8a ee
     [ Template 4321 (0x10e1), 11 elements ]
                                          04 d2 00 a6  ................
     [ Data set header, ID 1234 (0x04d2), length 166 ]
00c0: 4b b3 f2 74 c0 a8 00 03 c0 a8 00 02 94 20 13 c4  K..t......... ..
00d0: 00 00 00 01 05 1a 73 69 70 3a 62 6f 62 40 62 69  ......sip:bob@bi
00e0: 6c 6f 78 69 2e 65 78 61 6d 70 6c 65 2e 63 6f 6d  loxi.example.com
00f0: 03 41 42 43 1a 73 69 70 3a 62 6f 62 40 62 69 6c  .ABC.sip:bob@bil
0100: 6f 78 69 2e 65 78 61 6d 70 6c 65 2e 63 6f 6d 01  oxi.example.com.
0110: 00 1d 73 69 70 3a 61 6c 69 63 65 40 61 74 6c 61  ..sip:alice@atla
0120: 6e 74 61 2e 65 78 61 6d 70 6c 65 2e 63 6f 6d 0a  nta.example.com.
0130: 39 66 78 63 65 64 37 36 73 6c 27 33 38 34 38 32  9fxced76sl'38482
0140: 37 36 32 39 38 32 32 30 31 38 38 35 31 31 40 61  76298220188511@a
0150: 74 6c 61 6e 74 61 2e 65 78 61 6d 70 6c 65 2e 63  tlanta.example.c
0160: 6f 6d                                            om
     [ First record ]
            10 e1 00 45                                  ...E
     [ Data set header, ID 4321 (0x10e1), length 69 ]
                        4b b3 f2 76 c0 a8 00 02 c0 a8        K..v......
0170: 00 03 13 c4 94 20 00 00 00 01 00 b4 05 03 31 32  ..... ........12
0180: 33 1a 73 69 70 3a 62 6f 62 40 62 69 6c 6f 78 69  3.sip:bob@biloxi
0190: 2e 65 78 61 6d 70 6c 65 2e 63 6f 6d 0a 38 33 32  .example.com.832
01a0: 31 32 33 34 33 35 36                             1234356
     [ Second record ]

Notice here that although in this example, more than a third of the size of the file is templates, that templates need only appear once per log file; in real applications, the overhead of the templates is amortized over thousands to millions of records.

To demonstrate convertability between text and binary representations, here we show the output of the rfdump tool provided with ripfix, which prints templates and records in human-readable form (note here that the observation domain ID of the message exported by our tool is 56789):

----- template 56789/1234 -----
observationTimeSeconds(322)<dateTimeSeconds>[4]
sourceIPv4Address(8)<ipv4Address>[4]
destinationIPv4Address(12)<ipv4Address>[4]
sourceTransportPort(7)<unsigned16>[2]
destinationTransportPort(11)<unsigned16>[2]
sipSequenceNumber(35566/409)<unsigned32>[4]
sipMethod(35566/402)<unsigned8>[1]
sipRequestURI(35566/403)<string>[v]
sipClientTransaction(35566/414)<string>[v]
sipToURI(35566/406)<string>[v]
sipToTag(35566/407)<string>[v]
sipFromURI(35566/404)<string>[v]
sipFromTag(35566/405)<string>[v]
sipCallId(35566/408)<string>[v]
----- template 56789/4321 -----
observationTimeSeconds(322)<dateTimeSeconds>[4]
sourceIPv4Address(8)<ipv4Address>[4]
destinationIPv4Address(12)<ipv4Address>[4]
sourceTransportPort(7)<unsigned16>[2]
destinationTransportPort(11)<unsigned16>[2]
sipSequenceNumber(35566/409)<unsigned32>[4]
sipResponseStatus(35566/412)<unsigned16>[2]
sipMethod(35566/402)<unsigned8>[1]
sipServerTransaction(35566/413)<string>[v]
sipToURI(35566/406)<string>[v]
sipToTag(35566/407)<string>[v]
----- record 56789/1234 -----
observationTimeSeconds => 2010-04-01 03:10:12 +0200
sourceIPv4Address => 192.168.0.3
destinationIPv4Address => 192.168.0.2
sourceTransportPort => 37920
destinationTransportPort => 5060
sipSequenceNumber => 1
sipMethod => 5
sipRequestURI => sip:bob@biloxi.example.com
sipClientTransaction => ABC
sipToURI => sip:bob@biloxi.example.com
sipToTag =>
sipFromURI => sip:alice@atlanta.example.com
sipFromTag => 9fxced76sl
sipCallId => 3848276298220188511@atlanta.example.com
----- record 56789/4321 -----
observationTimeSeconds => 2010-04-01 03:10:14 +0200
sourceIPv4Address => 192.168.0.2
destinationIPv4Address => 192.168.0.3
sourceTransportPort => 5060
destinationTransportPort => 37920
sipSequenceNumber => 1
sipResponseStatus => 180
sipMethod => 5
sipServerTransaction => 123
sipToURI => sip:bob@biloxi.example.com
sipToTag => 8321234356

4.  Security Considerations

[TODO]

5.  IANA Considerations

[TODO: add new SIP IEs to, create new SIP Method registry, or add numbering to existing registry.]

6.  Acknowledgments

Cullen Jennings has provided insightful discussions, specific comments and much needed corrections. Nico d'Heureuse helped with the RFC 3665 examples.

7.  Open Issues

• Need an example showing how to store a realistically large log record sets to highlight the low overhead of the chosen approach
• Need an example showing cross-conversion with indexed ASCII format
• Need a analysis of space efficiency

8.  References

8.1. Normative References

8.2. Informative References

Authors' Addresses