idnits 2.17.1 draft-ietf-ipfix-file-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** The document seems to lack a License Notice according IETF Trust Provisions of 28 Dec 2009, Section 6.b.i or Provisions of 12 Sep 2009 Section 6.b -- however, there's a paragraph with a matching beginning. Boilerplate error? (You're using the IETF Trust Provisions' Section 6.b License Notice from 12 Feb 2009 rather than one of the newer Notices. See https://trustee.ietf.org/license-info/.) Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 3 characters in excess of 72. -- The document has examples using IPv4 documentation addresses according to RFC6890, but does not use any IPv6 documentation addresses. Maybe there should be IPv6 examples, too? Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 1701 has weird spacing: '...entType indic...' == Line 1706 has weird spacing: '...content conta...' == Line 1716 has weird spacing: '...orithms is a ...' == Line 1720 has weird spacing: '...entInfo is th...' == Line 1725 has weird spacing: '...ficates is a ...' == (10 more instances...) -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (August 6, 2009) is 5376 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Obsolete normative reference: RFC 5101 (Obsoleted by RFC 7011) ** Obsolete normative reference: RFC 5102 (Obsoleted by RFC 7012) ** Downref: Normative reference to an Informational RFC: RFC 1321 ** Downref: Normative reference to an Informational RFC: RFC 1952 ** Obsolete normative reference: RFC 3852 (Obsoleted by RFC 5652) ** Downref: Normative reference to an Informational RFC: RFC 4810 -- Obsolete informational reference (is this intentional?): RFC 3851 (Obsoleted by RFC 5751) -- Obsolete informational reference (is this intentional?): RFC 4288 (Obsoleted by RFC 6838) Summary: 8 errors (**), 0 flaws (~~), 7 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 IPFIX Working Group B. Trammell 3 Internet-Draft E. Boschi 4 Intended status: Standards Track Hitachi Europe 5 Expires: February 7, 2010 L. Mark 6 Fraunhofer IFAM 7 T. Zseby 8 Fraunhofer FOKUS 9 A. Wagner 10 ETH Zurich 11 August 6, 2009 13 Specification of the IPFIX File Format 14 draft-ietf-ipfix-file-05.txt 16 Status of this Memo 18 This Internet-Draft is submitted to IETF in full conformance with the 19 provisions of BCP 78 and BCP 79. 21 Internet-Drafts are working documents of the Internet Engineering 22 Task Force (IETF), its areas, and its working groups. Note that 23 other groups may also distribute working documents as Internet- 24 Drafts. 26 Internet-Drafts are draft documents valid for a maximum of six months 27 and may be updated, replaced, or obsoleted by other documents at any 28 time. It is inappropriate to use Internet-Drafts as reference 29 material or to cite them other than as "work in progress." 31 The list of current Internet-Drafts can be accessed at 32 http://www.ietf.org/ietf/1id-abstracts.txt. 34 The list of Internet-Draft Shadow Directories can be accessed at 35 http://www.ietf.org/shadow.html. 37 This Internet-Draft will expire on February 7, 2010. 39 Copyright Notice 41 Copyright (c) 2009 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents in effect on the date of 46 publication of this document (http://trustee.ietf.org/license-info). 47 Please review these documents carefully, as they describe your rights 48 and restrictions with respect to this document. 50 Abstract 52 This document describes a file format for the storage of flow data 53 based upon the IPFIX Protocol. It proposes a set of requirements for 54 flat-file, binary flow data file formats, then specifies the IPFIX 55 File format to meet these requirements based upon IPFIX Messages. 56 This IPFIX File format is designed to facilitate interoperability and 57 reusability among a wide variety of flow storage, processing, and 58 analysis tools. 60 Table of Contents 62 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5 63 1.1. IPFIX Documents Overview . . . . . . . . . . . . . . . . . 5 64 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6 65 3. Design Overview . . . . . . . . . . . . . . . . . . . . . . . 7 66 4. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 8 67 5. Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 10 68 5.1. Record Format Flexibility . . . . . . . . . . . . . . . . 11 69 5.2. Self Description . . . . . . . . . . . . . . . . . . . . . 11 70 5.3. Data Compression . . . . . . . . . . . . . . . . . . . . . 12 71 5.4. Indexing and Searching . . . . . . . . . . . . . . . . . . 12 72 5.5. Error Recovery . . . . . . . . . . . . . . . . . . . . . . 13 73 5.6. Authentication, Confidentiality, and Integrity . . . . . . 13 74 5.7. Anonymization and Obfuscation . . . . . . . . . . . . . . 14 75 5.8. Session Auditability and Replayability . . . . . . . . . . 14 76 5.9. Performance Characteristics . . . . . . . . . . . . . . . 15 77 6. Applicability . . . . . . . . . . . . . . . . . . . . . . . . 15 78 6.1. Storage of IPFIX-collected Flow Data . . . . . . . . . . . 15 79 6.2. Storage of NetFlow V9-collected Flow Data . . . . . . . . 15 80 6.3. Testing IPFIX Collecting Processes . . . . . . . . . . . . 16 81 6.4. IPFIX Device Diagnostics . . . . . . . . . . . . . . . . . 16 82 7. Detailed File Format Specification . . . . . . . . . . . . . . 17 83 7.1. File Reader Specification . . . . . . . . . . . . . . . . 17 84 7.2. File Writer Specification . . . . . . . . . . . . . . . . 18 85 7.3. Specific File Writer Use Cases . . . . . . . . . . . . . . 19 86 7.3.1. Collocating a File Writer with a Collecting Process . 19 87 7.3.2. Collocating a File Writer with a Metering Process . . 20 88 7.3.3. Using IPFIX Files for Archival Storage . . . . . . . . 21 89 7.3.4. Using IPFIX Files as Documents . . . . . . . . . . . . 21 90 7.3.5. Using IPFIX Files for Testing . . . . . . . . . . . . 22 91 7.3.6. Writing IPFIX Files for Device Diagnostics . . . . . . 22 92 7.3.7. IPFIX File Manipulation . . . . . . . . . . . . . . . 22 93 7.4. Media type of IPFIX Files . . . . . . . . . . . . . . . . 23 94 8. File Format Metadata Specification . . . . . . . . . . . . . . 23 95 8.1. Recommended Options Templates for IPFIX Files . . . . . . 23 96 8.1.1. Message Checksum Options Template . . . . . . . . . . 23 97 8.1.2. File Time Window Options Template . . . . . . . . . . 24 98 8.1.3. Export Session Details Options Template . . . . . . . 25 99 8.1.4. Message Details Options Template . . . . . . . . . . . 27 100 8.2. Recommended Information Elements for IPFIX Files . . . . . 29 101 8.2.1. collectionTimeMilliseconds . . . . . . . . . . . . . . 30 102 8.2.2. collectorCertificate . . . . . . . . . . . . . . . . . 30 103 8.2.3. exporterCertificate . . . . . . . . . . . . . . . . . 30 104 8.2.4. exportSctpStreamId . . . . . . . . . . . . . . . . . . 31 105 8.2.5. maxExportSeconds . . . . . . . . . . . . . . . . . . . 31 106 8.2.6. maxFlowEndMicroseconds . . . . . . . . . . . . . . . . 31 107 8.2.7. maxFlowEndMilliseconds . . . . . . . . . . . . . . . . 32 108 8.2.8. maxFlowEndNanoseconds . . . . . . . . . . . . . . . . 32 109 8.2.9. maxFlowEndSeconds . . . . . . . . . . . . . . . . . . 32 110 8.2.10. messageMD5Checksum . . . . . . . . . . . . . . . . . . 33 111 8.2.11. messageScope . . . . . . . . . . . . . . . . . . . . . 33 112 8.2.12. minExportSeconds . . . . . . . . . . . . . . . . . . . 34 113 8.2.13. minFlowStartMicroseconds . . . . . . . . . . . . . . . 34 114 8.2.14. minFlowStartMilliseconds . . . . . . . . . . . . . . . 34 115 8.2.15. minFlowStartNanoseconds . . . . . . . . . . . . . . . 35 116 8.2.16. minFlowStartSeconds . . . . . . . . . . . . . . . . . 35 117 8.2.17. opaqueOctets . . . . . . . . . . . . . . . . . . . . . 36 118 8.2.18. sessionScope . . . . . . . . . . . . . . . . . . . . . 36 119 9. Signing and Encryption of IPFIX Files . . . . . . . . . . . . 37 120 9.1. CMS Detached Signatures . . . . . . . . . . . . . . . . . 37 121 9.1.1. ContentInfo . . . . . . . . . . . . . . . . . . . . . 38 122 9.1.2. SignedData . . . . . . . . . . . . . . . . . . . . . . 39 123 9.1.3. SignerInfo . . . . . . . . . . . . . . . . . . . . . . 39 124 9.1.4. EncapsulatedContentInfo . . . . . . . . . . . . . . . 40 125 9.2. Encryption Error Resilience . . . . . . . . . . . . . . . 40 126 10. Compression of IPFIX Files . . . . . . . . . . . . . . . . . . 40 127 10.1. Supported Compression Formats . . . . . . . . . . . . . . 41 128 10.2. Compression Recognition at the File Reader . . . . . . . . 41 129 10.3. Compression Error Resilience . . . . . . . . . . . . . . . 41 130 11. Recommended File Integration Strategies . . . . . . . . . . . 42 131 11.1. Encapsulation of Non-IPFIX Data in IPFIX Files . . . . . . 43 132 11.2. Encapsulation of IPFIX Files within Other File Formats . . 43 133 12. Security Considerations . . . . . . . . . . . . . . . . . . . 43 134 12.1. Relationship between IPFIX File and Transport 135 Encryption . . . . . . . . . . . . . . . . . . . . . . . . 44 136 12.2. End-to-End Assertions for IPFIX Files . . . . . . . . . . 44 137 12.3. Recommendations for Strength of Cryptography for IPFIX 138 Files . . . . . . . . . . . . . . . . . . . . . . . . . . 45 139 13. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 45 140 14. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 48 141 15. References . . . . . . . . . . . . . . . . . . . . . . . . . . 48 142 15.1. Normative References . . . . . . . . . . . . . . . . . . . 48 143 15.2. Informative References . . . . . . . . . . . . . . . . . . 49 144 Appendix A. Example IPFIX File . . . . . . . . . . . . . . . . . 50 145 A.1. Example Options Templates . . . . . . . . . . . . . . . . 52 146 A.2. Example Supplemental Options Data . . . . . . . . . . . . 54 147 A.3. Example Message Checksum . . . . . . . . . . . . . . . . . 56 148 A.4. File Example Data Set . . . . . . . . . . . . . . . . . . 57 149 A.5. Complete File Example . . . . . . . . . . . . . . . . . . 57 150 Appendix B. Applicability of IPFIX Files to NetFlow V9 flow 151 storage . . . . . . . . . . . . . . . . . . . . . . . 59 152 B.1. Comparing NetFlow V9 to IPFIX . . . . . . . . . . . . . . 59 153 B.1.1. Message Header Format . . . . . . . . . . . . . . . . 59 154 B.1.2. Set Header Format . . . . . . . . . . . . . . . . . . 60 155 B.1.3. Template Format . . . . . . . . . . . . . . . . . . . 61 156 B.1.4. Information Model . . . . . . . . . . . . . . . . . . 61 157 B.1.5. Template Management . . . . . . . . . . . . . . . . . 61 158 B.1.6. Transport . . . . . . . . . . . . . . . . . . . . . . 61 159 B.2. A Method for Transforming NetFlow V9 messages to IPFIX . . 62 160 B.3. NetFlow V9 Transformation Example . . . . . . . . . . . . 63 161 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 65 163 1. Introduction 165 This document specifies a file format based upon IPFIX, designed to 166 facilitate interoperability and reusability among a wide variety of 167 flow storage, processing, and analysis tools. It begins with an 168 overview of the IPFIX File format, and a quick summary of how IPFIX 169 Files work in Section 3. The detailed specification of the IPFIX 170 File format appears in Section 7; this section includes general 171 specifications for IPFIX File Readers and IPFIX File Writers and 172 specific recommendations for common situations in which they are 173 used. The format makes use of the IPFIX Options mechanism for 174 additional file metadata, in order to avoid requiring any protocol 175 extensions, and to minimize the effort required to adapt IPFIX 176 implementations to use the file format; a detailed definition of the 177 Options Templates used for storage metadata appears in Section 8. 178 Appendix A contains a detailed example IPFIX File. 180 An advantage of file-based storage is that files can be readily 181 encapsulated within each other and other data storage and 182 transmission formats. The IPFIX File Format leverages this to 183 provide encryption, described in Section 9 and compression, described 184 in Section 10. Section 11 provides specific recommendations for 185 integration of IPFIX File data with other formats. 187 The IPFIX File format was designed to be applicable to a wide variety 188 of flow storage situations; the motivation behind its creation is 189 described in Section 4. The document outlines of the set of 190 requirements the format is designed to meet in Section 5, and 191 explores the applicability of such a format to various specific 192 application areas in Section 6. These sections are intended to give 193 background on the development of IPFIX Files. 195 1.1. IPFIX Documents Overview 197 "Specification of the IPFIX Protocol for the Exchange of IP Traffic 198 Flow Information" [RFC5101] and its associated documents define the 199 IPFIX Protocol, which provides network engineers and administrators 200 with access to IP traffic flow information. 202 "Architecture for IP Flow Information Export" [RFC5470] defines the 203 architecture for the export of measured IP flow information out of an 204 IPFIX Exporting Process to an IPFIX Collecting Process, and the basic 205 terminology used to describe the elements of this architecture, per 206 the requirements defined in "Requirements for IP Flow Information 207 Export" [RFC3917]. [RFC5101] then covers the details of the method 208 for transporting IPFIX Data Records and Templates via a congestion- 209 aware transport protocol from an IPFIX Exporting Process to an IPFIX 210 Collecting Process. 212 "Information Model for IP Flow Information Export" [RFC5102] 213 describes the Information Elements used by IPFIX, including details 214 on Information Element naming, numbering, and data type encoding. 216 "IPFIX Applicability" [RFC5472] describes the various applications of 217 the IPFIX protocol and their use of information exported via IPFIX, 218 and relates the IPFIX architecture to other measurement architectures 219 and frameworks. 221 In addition, "Exporting Type Information for IPFIX Information 222 Elements" [RFC5610] specifies a method for encoding Information Model 223 properties within an IPFIX Message stream. 225 This document references [RFC5101] and the architecture document for 226 terminology, defines IPFIX File Writer and IPFIX File Reader in terms 227 of the IPFIX Exporting Processes and IPFIX Collecting Process 228 definitions from [RFC5101], and extends the IPFIX Information Model 229 defined in [RFC5102] to provide new Information Elements for IPFIX 230 File metadata. It uses the method described in "Exporting Type 231 Information for IPFIX Information Elements" [RFC5610] document to 232 support the self-description of IPFIX Files containing enterprise- 233 specific Information Elements. 235 2. Terminology 237 This section defines terminology related to the IPFIX File Format. 238 In addition, terms used in this document that are defined in the 239 Terminology section of [RFC5101] are to be interpreted as defined 240 there. 242 IPFIX File: An IPFIX File is a serialized stream of IPFIX Messages; 243 this stream may be stored on a filesystem or transported using any 244 technique customarily used for files. Any IPFIX Message stream 245 that would be considered valid when transported one or more of the 246 specified IPFIX transports (SCTP, TCP, or UDP) as defined in 247 [RFC5101] is considered an IPFIX File. However, this document 248 extends that definition with recommendations on the construction 249 of IPFIX Files that meet the requirements identified in Section 5. 251 IPFIX File Reader: An IPFIX File Reader is a process which reads 252 IPFIX Files from a filesystem. An IPFIX File Reader operates as 253 an IPFIX Collecting Process as specified in [RFC5101], except as 254 modified by this document. 256 IPFIX File Writer: An IPFIX File Writer is a process which writes 257 IPFIX Files to a filesystem. An IPFIX File Writer operates as an 258 IPFIX Exporting Process as specified in [RFC5101], except as 259 modified by this document. 261 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 262 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 263 document are to be interpreted as described in [RFC2119]. 265 3. Design Overview 267 An IPFIX File is simply a data stream containing one or more IPFIX 268 Messages serialized to some filesystem. Though any set of valid 269 IPFIX Messages can be serialized into an IPFIX File, the 270 specification includes guidelines designed to ease storage and 271 retrieval of flow data using the IPFIX File format. 273 IPFIX Files contain only IPFIX Messages; any file metadata such as 274 checksums or export session details are stored using Options within 275 the IPFIX Message. This design is completely compatible with the 276 IPFIX protocol on the wire. A schematic of a typical IPFIX File is 277 shown below: 279 +=======================================+ 280 | IPFIX File | 281 | +===================================+ | 282 | | IPFIX Message | | 283 | | +-------------------------------+ | | 284 | | | IPFIX Message Header | | | 285 | | +-------------------------------+ | | 286 | | +-------------------------------+ | | 287 | | | Options Template Set | | | 288 | | | Options Template Record | | | 289 | | | . . . | | | 290 | | +-------------------------------+ | | 291 | | +-------------------------------+ | | 292 | | | Template Set | | | 293 | | | Template Record | | | 294 | | | . . . | | | 295 | | +-------------------------------+ | | 296 | +===================================+ | 297 | | IPFIX Message | | 298 | | +-------------------------------+ | | 299 | | | IPFIX Message Header | | | 300 | | +-------------------------------+ | | 301 | | +-------------------------------+ | | 302 | | | Data Set | | | 303 | | | Data Record | | | 304 | | | . . . | | | 305 | | +-------------------------------+ | | 306 | | +-------------------------------+ | | 307 | | | Data Set | | | 308 | | | Data Record | | | 309 | | | . . . | | | 310 | | +-------------------------------+ | | 311 | | . . . | | 312 | +===================================+ | 313 | . . . | 314 +=======================================+ 316 Figure 1: Typical File Structure 318 4. Motivation 320 There is a wide variety of applications for the file-based storage of 321 IP flow data, across a continuum of time scales. Tools used in the 322 analysis of flow data and creation of analysis products often use 323 files as a convenient unit of work, with an ephemeral lifetime. A 324 set of flows relevant to a security investigation may be stored in a 325 file for the duration of that investigation, and further exchanged 326 among incident handlers via email or within an external incident 327 handling workflow application. Sets of flow data relevant to 328 Internet measurement research may be published as files, much as 329 libpcap packet trace files are, to provide common datasets for the 330 repeatability of research efforts; these files would have lifetimes 331 measured in months or years. Operational flow measurement systems 332 also have a need for long-term, archival storage of flow data, either 333 as a primary flow data repository, or as a backing tier for online 334 storage in a relational database management system (RDBMS). 336 The variety of applications of flow data, and the variety of 337 presently deployed storage approaches, indicates the need for a 338 standard approach to flow storage with applicability across the 339 continuum of time scales over which flow data is stored. A storage 340 format based around flat files would best address the variety of 341 storage requirements. While much work has been done on structured 342 storage via RDBMS, relational database systems are not a good basis 343 for format standardization owing to the fact that their internal data 344 structures are generally private to a single implementation and 345 subject to change for internal reasons. Also, there are a wide 346 variety of operations available on flat files, and external tools and 347 standards can be leveraged to meet file-based flow storage 348 requirements. Further, flow data is often not very semantically 349 complicated, and is managed in very high volume; therefore, an RDBMS- 350 based flow storage system would not benefit much from the advantages 351 of relational database technology. 353 The simplest way to create a new file format is simply to serialize 354 some internal data model to disk, with either textual or binary 355 representation of data elements, and some framing strategy for 356 delimiting fields and records. "Ad-hoc" file formats such as this 357 have several important disadvantages. They impose the semantics of 358 the data model from which they are derived on the file format, and as 359 such, they are difficult to extend, describe, and standardize. 361 Indeed, one de facto standard for the storage of flow data is one of 362 these ad-hoc formats. A common method of storing data collected via 363 Cisco NetFlow is to serialize a stream of raw NetFlow datagrams into 364 files. These NetFlow PDU files consist of a collection of header- 365 prefixed blocks (corresponding to the datagrams as received on the 366 wire) containing fixed-length binary flow records. NetFlow V5, V7, 367 and V8 data may be mixed within a given file, as the header on each 368 datagram defines the NetFlow version of the records following. While 369 this NetFlow PDU file format has all the disadvantages of an ad-hoc 370 format, and is not extensible to data models other than that defined 371 by Cisco NetFlow, it is at least reasonably well-understood due to 372 its ubiquity. 374 Over the past decade XML markup has emerged as a new "universal" 375 representation format for structured data. It is intended to be 376 human-readable; indeed, that is one reason for its rapid adoption. 377 However XML has limited usefulness for representing network flow 378 data. Network flow data has a simple, repetitive, non-hierarchical 379 structure that does not benefit much from XML. An XML representation 380 of flow data would be an essentially flat list of the attributes and 381 their values for each flow record. 383 The XML approach to data encoding is very heavyweight when compared 384 to binary flow encoding. XML's use of start- and end-tags, and 385 plain-text encoding of the actual values, leads to significant 386 inefficiency in encoding size. Typical network traffic datasets can 387 contain millions or billions of flows per hour of traffic 388 represented. Any increase in storage size per record can have 389 dramatic impact on flow data storage and transfer sizes. While data 390 compression algorithms can partially remove the redundancy introduced 391 by XML encoding, they introduce additional overhead of their own. 393 A further problem is that XML processing tools require a full XML 394 parser. XML parsers are fully general and therefore complex, 395 resource-intensive and relatively slow, introducing significant 396 processing time overhead for large network-flow datasets. In 397 contrast, parsers for typical binary flow data encodings are simply 398 structured, since they only need to parse a very small header and 399 then have complete knowledge of all following fields for the 400 particular flow. These can then be read in a very efficient linear 401 fashion. 403 This leads us to propose the IPFIX Message format as the basis for a 404 new flow data file format. The IPFIX working group, in defining the 405 IPFIX Protocol, has already defined an information model and data 406 formatting rules for representation of flow data. Especially at 407 shorter time scales, when a file is a unit of data interchange, the 408 filesystem may be viewed as simply another IPFIX Message transport 409 between processes. This format is especially well suited to 410 representing flow data, as it was designed specifically for flow data 411 export; it is easily extensible unlike ad-hoc serialization, and 412 compact unlike XML. In addition, IPFIX is an IETF standard for the 413 export and collection of flow data; using a common format for storage 414 and analysis at the collection side allows implementors to use 415 substantially the same information model and data formatting 416 implementation for transport as well as storage. 418 5. Requirements 420 In this section, we outline a proposed set of requirements 422 [SAINT2007] for any persistent storage format for flow data. First 423 and foremost, a flow data file format should support storage across 424 the continuum of time scales important to flow storage applications. 425 Each of the requirements enumerated in the sections below is broadly 426 applicable to flow storage applications, though each may be more 427 important at certain time scales. For each, we first identify the 428 requirement, then explain how the IPFIX Message format addresses it, 429 or briefly outline the changes that must be made in order for an 430 IPFIX-based file format to meet the requirement. 432 5.1. Record Format Flexibility 434 Due to the wide variety of flow attributes collected by different 435 network flow attribute measurement systems, the ideal flow storage 436 format will not impose a single data model or a specific record type 437 on the flows it stores. The file format must be flexible and 438 extensible; that is, it must support the definition of multiple 439 record types within the file itself, and must be able to support new 440 field types for data within the records in a graceful way. 442 IPFIX provides record format flexibility through the use of Templates 443 to describe each Data Record, through the use of an IANA Registry to 444 define its Information Elements, and through the use of enterprise- 445 specific Information Elements. 447 5.2. Self Description 449 Archived data may be read at a time in the future where any external 450 reference to the meaning of the data may be lost. The ideal flow 451 storage format should be self-describing; that is, a process reading 452 flow data from storage should be able to properly interpret the 453 stored flows without reference to anything other than standard 454 sources (e.g., the standards document describing the file format) and 455 the stored flow data itself. 457 The IPFIX Message format is partially self-describing; that is, IPFIX 458 Templates containing only IANA-assigned Information Elements can be 459 completely interpreted according to the IPFIX Information Model 460 without additional external data. 462 However, Templates containing private information elements lack 463 detailed type and semantic information; a Collecting Process 464 receiving Data Records described by a Template containing enterprise- 465 specific Information Elements it does not understand can only treat 466 the data contained within those Information Elements as octet arrays. 467 To be fully self-describing, enterprise-specific Information Elements 468 must be additionally described via IPFIX Options according to the 469 Information Element Type Options Template defined in "Exporting Type 470 Information for IPFIX Information Elements" [RFC5610]. 472 5.3. Data Compression 474 Regardless of the representation format, flow data describing traffic 475 on real networks tends to be highly compressible. Compression tends 476 to improve the scalability of flow collection systems, by reducing 477 the disk storage and I/O bandwidth requirement for a given workload. 478 The ideal flow storage format should support applications which wish 479 to leverage this fact by supporting compression of stored data. 481 The IPFIX Message format has no support for data compression, as the 482 IPFIX protocol was designed for speed and simplicity of export. Of 483 course, any flat file is readily compressible using a wide variety of 484 external data compression tools, formats, and algorithms; therefore, 485 this requirement can be met via enapsulation in one of these formats. 486 Section 10 specifies an encapsulation based on bzip2 or gzip, to 487 maximize interoperability. 489 A few simple optimizations can be made by File Writers to increase 490 the integrity and usability of compressed IPFIX data; these are 491 outlined in Section 10.3. 493 5.4. Indexing and Searching 495 Binary, record stream oriented file formats natively support only one 496 form of searching, sequential scan in file order. By choosing the 497 order of records in a file carefully (e.g., by flow end time), a file 498 can be indexed by a single key. 500 Beyond this, properly addressing indexing is an application-specific 501 problem, as it inherently involves tradeoffs between storage 502 complexity and retrieval speed, and requirements vary widely based on 503 time scales and the types of queries used from site to site. 504 However, a generic standard flow storage format may provide limited 505 direct support for indexing and searching. 507 The ideal flow storage format will support a limited table of 508 contents facility noting that the records in a file contain data 509 relating only to certain keys or values of keys, in order to keep 510 multi-file search implementations from having to scan a file for data 511 it does not contain. 513 The IPFIX Message format has no direct support for indexing. 514 However, the technique described in "Reducing Redundancy in IPFIX and 515 PSAMP Reports" [RFC5473] can be used to describe the contents of a 516 file in a limited way. Additionally, as flow data is often sorted 517 and divided by time, the start and end time of the flows in a file 518 may be declared using the File Time Window Options Template defined 519 in Section 8.1.2. 521 5.5. Error Recovery 523 When storing flow data for archival purposes, it is important to 524 ensure that hardware or software faults do not introduce errors into 525 the data over time. The ideal flow storage format will support the 526 detection and correction of encoding-level errors in the data. 528 Note that more advanced error correction is best handled at a layer 529 below that addressed by this document. Error correction is a topic 530 well addressed by the storage industry in general (e.g. by RAID and 531 other technologies), and by specifying a flow storage format based 532 upon files, we can leverage these features to meet this requirement. 534 However, the ideal flow storage format will be resilient against 535 errors, providing an internal facility for the detection of errors 536 and the ability to isolate errors to as few data records as possible. 538 Note that this requirement interacts with the choice of data 539 compression or encryption algorithm. For example, the use of block 540 compression algorithms can serve to isolate errors to a single 541 compression block, unlike stream compressors, which may fail to 542 resynchronize after a single bit error, invalidating the entire 543 message stream. 545 The IPFIX Message format does not support data integrity assurance. 546 It is assumed that advanced error correction will be provided 547 externally. Compression and encryption, if used, provide some 548 allowance for detection, if not correction, of errors. For simple 549 error detection support in the absence of compression or encryption, 550 checksums may be attached to messages via IPFIX Options according to 551 the Message Checksum Options Template defined in Section 8.1.1. 553 5.6. Authentication, Confidentiality, and Integrity 555 Archival storage of flow data may also require assurance that no 556 unauthorized entity can read or modify the stored data. Cryptography 557 can be applied to this problem to ensure integrity and 558 confidentiality by signing and encryption. 560 As with error correction, this problem has been addressed well at a 561 layer below that addressed by this document. We can leverage the 562 fact that existing cryptographic technologies work quite well on data 563 stored in files to meet this requirement. 565 Beyond support for the use of TLS for transport over TCP or DTLS for 566 transport over SCTP or UDP, both of which provide transient 567 authentication and confidentiality, the IPFIX protocol does not 568 support this requirement directly. The IETF has specified the 569 Cryptographic Message Syntax (CMS) [RFC3852] for creating detached 570 signatures for integrity and authentication; Section 9 specifies a 571 CMS-based method for signing IPFIX Files. Confidentiality protection 572 is assumed to be met by methods external to this specification, 573 leveraging one of the many such technologies for encrypting files to 574 meet specific application and process requirements; however, notes on 575 improving archival integrity of encrypted IPFIX Files are given in 576 Section 9.2. 578 5.7. Anonymization and Obfuscation 580 To ensure the privacy of individuals and organizations at the 581 endpoints of communications represented by flow records, it is often 582 necessary to obfuscate or anonymize stored and exported flow data. 583 The ideal flow storage format will provide for a notation that a 584 given information element on a given record type represents 585 anonymized, rather than real, data. 587 The IPFIX Protocol presently has no support for anonymization 588 notation. It should be noted that anonymization is one of the 589 requirements given for IPFIX in [RFC3917]. The decision to qualify 590 this requirement with 'MAY' and not 'MUST' in the requirements 591 document, and its subsequent lack of specification in the current 592 version of the IPFIX protocol, is due to the fact that anonymization 593 algorithms are still an open area of research, and that there 594 currently exist no standardized methods for anonymization. 596 No support is presently defined in [RFC5101] or this IPFIX-based File 597 Format for anonymization, as anonymization notation is an area of 598 open work for the IPFIX working group. 600 5.8. Session Auditability and Replayability 602 Certain use cases for archival flow storage require the storage of 603 collection infrastructure details alongside the data itself. These 604 details include information about how and when data was received, and 605 where it was received from, and are useful for auditing as well as 606 for the replaying received data for testing purposes. 608 The IPFIX Protocol contains no direct support for auditability and 609 replayability, though the IPFIX Information Model does define various 610 Information Elements required to represent collection infrastructure 611 details. These details may be stored in IPFIX Files using the Export 612 Session Details Options Template defined in Section 8.1.3 and the 613 Message Details Options Template defined in Section 8.1.4. 615 5.9. Performance Characteristics 617 The ideal standard flow storage format will not have a significant 618 negative impact on the performance of the application generating or 619 processing flow data stored in the format. This is a non-functional 620 requirement, but it is important to note that a standard that implies 621 a significant performance penalty is unlikely to be widely 622 implemented and adopted. 624 An examination of the IPFIX Protocol would seem to suggest that 625 implementations of it are not particularly prone to slowness; indeed, 626 a template-based data representation is more easily subject to 627 optimization for common cases than representations that embed 628 structural information directly in the data stream (e.g. XML). 629 However, a full analysis of the impact of using IPFIX Messages as a 630 basis for flow data storage on read/write performance will require 631 more implementation experience and performance measurement. 633 6. Applicability 635 This section describes the specific applicability of IPFIX Files to 636 various use cases. IPFIX Files are particularly useful in a flow 637 collection and processing infrastructure using IPFIX for flow export. 638 We explore the applicability and provide guidelines for using IPFIX 639 files for the storage of flow data collected by IPFIX Collecting 640 Processes and NetFlow V9 collectors, the testing of IPFIX Collecting 641 Processes, and diagnostics of IPFIX Devices. 643 6.1. Storage of IPFIX-collected Flow Data 645 IPFIX Files can naturally be used to store flow data collected by an 646 IPFIX Collecting Process; indeed, this was one of the primary initial 647 motivations behind the file format described within this document. 648 Using IPFIX Files as such provides a single, standard, well- 649 understood encoding to be used for flow data on disk and on the wire, 650 and allows IPFIX implementations to leverage substantially the same 651 code for flow export and flow storage. In addition, the storage of 652 single Transport Sessions in IPFIX Files is particularly important 653 for network measurement research, allowing repeatability of 654 experiments by providing a format for the storage and exchange of 655 IPFIX flow trace data much as the libpcap format is used for 656 experiments on packet trace data. 658 6.2. Storage of NetFlow V9-collected Flow Data 660 Although the IPFIX protocol is based on the Cisco Netflow Services, 661 Version 9 (NetFlow V9) protocol [RFC3954], the two have diverged 662 since work began on IPFIX. However, since the NetFlow V9 information 663 model is a compatible subset of the IPFIX information model, it is 664 possible to use IPFIX files to store collected NetFlow V9 flow data. 665 This approach may be particularly useful in multi-vendor, multi- 666 protocol collection infrastructures using both NetFlow V9 and IPFIX 667 to export flow data. 669 The applicability of IPFIX Files to this use case is outlined in 670 Appendix B. 672 6.3. Testing IPFIX Collecting Processes 674 IPFIX Files can be used to store IPFIX Messages for the testing of 675 IPFIX Collecting Processes. A variety of test cases may be stored in 676 IPFIX Files. First, IPFIX data collected in real network 677 environments and stored in an IPFIX File can be used as input to 678 check the behavior of new or extended implementations of IPFIX 679 Collectors. Furthermore, IPFIX Files can be used to validate the 680 operation of a given IPFIX Collecting Process in a new environment, 681 i.e., to test with recorded IPFIX data from the target network before 682 installing the Collecting Process in the network. 684 The IPFIX File format can also be used to store artificial, non- 685 compliant reference messages for specific Collecting Process test 686 cases. Examples for such test cases are sets of IPFIX records with 687 undefined Information Elements, Data Records described by missing 688 Templates, or incorrectly framed Messages or Data Sets. 689 Representative error handling test cases are defined in "IPFIX 690 Testing" [RFC5471]. 692 Furthermore, fast replay of IPFIX Messages stored in a file can be 693 used for stress/load tests (e.g., high rate of incoming Data Records, 694 large Templates with high Information Element counts), as described 695 in "IPFIX Testing" [RFC5471]. The provisioning and use of a set of 696 reference files for testing simplifies the performance of tests and 697 increases the comparability of test results. 699 6.4. IPFIX Device Diagnostics 701 As an IPFIX File can be used to store any collection of flows, the 702 format may also be used for dumping and storing various types of flow 703 data for IPFIX Device diagnostics (e.g., the open flow cache of a 704 Metering Process or the flow backlog of an Exporting or Collecting 705 Process at the time of a process reset or crash). File-based storage 706 is preferable to remote transmission in such error-recovery 707 situations. 709 7. Detailed File Format Specification 711 Any valid serialized IPFIX Message stream MUST be accepted by a File 712 Reader as a valid IPFIX File. In this way, the filesystem is simply 713 treated as another IPFIX transport alongside SCTP, TCP, and UDP, 714 albeit a potentially high-latency transport, as the File Reader and 715 File Writer do not necessarily run at the same time. 717 This section specifies the detailed actions of File Readers and File 718 Writers in handling IPFIX Files, and further specifies actions of 719 File Writers in specific use cases. Unless otherwise specified 720 herein, IPFIX File Writers MUST behave as IPFIX Exporting Processes, 721 and IPFIX File Readers MUST behave as IPFIX Collecting Processes, 722 where appropriate. 724 7.1. File Reader Specification 726 An IPFIX File Reader MUST act as an IPFIX Collecting Process as 727 specified in [RFC5101], except as modified by this document. 729 An IPFIX File Reader MUST accept as valid any serialized IPFIX 730 Message stream that would be considered valid by one or more of the 731 other defined IPFIX transport layers. Practically, this means that 732 the union of IPFIX Template management features supported by SCTP, 733 TCP, and UDP MUST be supported in IPFIX Files. File Readers MUST: 735 o accept IPFIX Messages containing Template Sets, Options Template 736 Sets, and Data Sets within the same message, as with IPFIX over 737 TCP or UDP; 739 o accept Template Sets that define Templates already defined within 740 the File, as may occur with retransmission of Templates when using 741 IPFIX over UDP as described in section 10.3.6 of [RFC5101]; 743 o resolve any conflict between a resent definition and a previous 744 definition by assuming that the new Template replaces the old, as 745 consistent with Template expiration and ID reuse when using UDP at 746 the IPFIX transport protocol; and 748 o accept Template Withdrawals as described in section 8 of 749 [RFC5101], provided that the Template to be withdrawn is defined, 750 as is the case with IPFIX over TCP and SCTP. 752 Considering the filesystem-as-transport view, in the general case an 753 IPFIX File SHOULD be treated as containing a single Transport Session 754 as defined by [RFC5101]. However, some applications may benefit from 755 the ability to treat a collection of IPFIX Files as a single 756 Transport Session; see especially Section 7.3.3 below. A File Reader 757 MAY be configurable to treat a collection of Files as a single 758 Transport Session. However, a File Reader MUST NOT treat a single 759 IPFIX File as containing multiple Transport Sessions. 761 If an IPFIX File uses the technique described in "Reducing Redundancy 762 in IPFIX and PSAMP Reports" [RFC5473] AND all of the non-Options 763 Templates in the File contain the commonPropertiesId Information 764 Element, a File Reader MAY assume the set of commonPropertiesId 765 definitions provides a complete table of contents for the File for 766 searching purposes. 768 7.2. File Writer Specification 770 An IPFIX File Writer MUST act as an IPFIX Exporting Process as 771 specified in [RFC5101], except as modified by this document. This 772 section contains specifications for IPFIX File Writers in all 773 situations; specifications and recommendations for specific File 774 Writer use cases are found in below. 776 File Writers SHOULD store the Templates and Options required to 777 decode the data within the File in the File itself, unless modified 778 by the requirements of a specific use case in a subsection of 779 Section 7.3. In this way, a single IPFIX File generally contains a 780 single notional Transport Session as defined by [RFC5101]. 782 File Writers SHOULD emit each Template Set or Options Template Set to 783 appear in the File before any Data Set described by the Templates 784 within that Set, to ensure the File Reader can decode every Data Set 785 without waiting to process subsequent Templates or Options Templates. 787 File Writers SHOULD emit Data Records described by Options Templates 788 to appear in the File before any Data Records which depend on the 789 scopes defined by those options. 791 File Writers SHOULD use Template Withdrawals to withdraw Templates if 792 Template IDs need to be reused. Template Withdrawals SHOULD NOT be 793 used unless necessary to reuse Template IDs. 795 File Writers SHOULD write IPFIX Messages within an IPFIX File in 796 ascending Export Time order. 798 File Writers MAY write Data Records to an IPFIX File in any order. 799 However, File Writers that write flow records to an IPFIX File in 800 flowStartTime or flowEndTime order SHOULD be consistent in this 801 ordering within each File. 803 7.3. Specific File Writer Use Cases 805 The specifications in this section apply to specific situations. 806 Each section below extends or modifies the base File Writer 807 specification in Section 7.2. Considerations for collocation of a 808 File Writer with IPFIX Collecting Processes and Metering Processes 809 are given, as are specific guidelines for using IPFIX Files for 810 archival storage, or as documents. Also covered are the use of IPFIX 811 Files in the testing and diagnostics of IPFIX Devices. 813 7.3.1. Collocating a File Writer with a Collecting Process 815 When collocating a File Writer with an IPFIX Collecting Process for 816 archival storage of collected data in IPFIX Files as described in 817 Section 6.1, the following recommendations may improve the usefulness 818 of the stored data. 820 The simplest way for a File Writer to store the data collected in a 821 single Transport Session is to simply write the incoming IPFIX 822 Messages to an IPFIX File as they are collected. This approach has 823 several drawbacks. First, if the original Exporting Process did not 824 conform to the recommendations in Section 7.2 with respect to 825 Template and Data Record ordering, the written file can be difficult 826 to use later; in this case, File Writers MAY reorder records as 827 received in order to ensure that Templates appear before the Data 828 Records they describe. 830 A File Writer collocated with a Collecting Process that starts 831 writing data from a running Transport Session SHOULD write all the 832 Templates currently active within that Transport Session before 833 writing any Data Records described by them. 835 Also, the resulting IPFIX Files will lack information about the IPFIX 836 Transport Session used to export them, such as the network addresses 837 of the Exporting and Collecting Processes and the protocols used to 838 transport them. In this case, if information about the Transport 839 Session is required, the File Writer SHOULD store a single IPFIX 840 Transport Session in an IPFIX File and SHOULD record information 841 about the Transport Session using the Export Session Details Options 842 Template described in Section 8.1.3. 844 Additional per-Message information MAY be recorded by the File Writer 845 using the Message Details Options Template described in 846 Section 8.1.4. Per-Message information includes the time at which 847 each IPFIX Message was received at the Collecting Process, and can be 848 used to resend IPFIX Messages while keeping the original measurement 849 plane traffic profile. 851 When collocating a File Writer with a Collecting Process, the Export 852 Time of each Message SHOULD be the Export Time of the Message 853 received by the Collecting Process containing the first Data Record 854 in the Message. Note that File Writers storing IPFIX data collected 855 from an IPFIX Collecting Process using SCTP as the transport protocol 856 SHOULD interleave messages from multiple streams in order to preserve 857 Export Time order, and SHOULD reorder the written messages as 858 necessary to ensure that each Template Set or Options Template Set 859 appears in the File before any Data Set described by the Templates 860 within that Set. Template reordering MUST preserve the sequence of 861 Template Sets with Template Withdrawals in order to ensure 862 consistency of Templates. 864 Note that when adding additional information to IPFIX Messages 865 received from Collecting Processes (e.g. Message Checksum Options, 866 Message Detail Options), the File Writer SHOULD extend the length of 867 the Message for the additional data if possible; otherwise, the 868 Message SHOULD be split into two approximately equal-size Messages 869 aligned on a Data Set or Template Set boundary from the original 870 Message if possible; otherwise, the Message SHOULD be split into 871 approximately two equal size Messages aligned on a Data Record 872 boundary. Note that, since the MSS or MTU of most network links 873 (1500-9000 for common Ethernets) is smaller than the maximum IPFIX 874 Message size (65536) within an IPFIX File, it is expected that 875 message length extension will suffice in most circumstances. 877 A File Writer collocated with a Collecting Process SHOULD NOT sign a 878 File as specified in Section 9.1 unless the Transport Session over 879 which the data was exported was protected via TLS or DTLS, and the 880 Collecting Process positively identified the Exporting Process by its 881 certificate. See Section 12.2 for more information on this issue. 883 7.3.2. Collocating a File Writer with a Metering Process 885 Note that File Writers may also be collocated directly with IPFIX 886 Metering Processes, for writing measured information directly to disk 887 without intermediate IPFIX Exporting or Collecting Processes. This 888 arrangement may be particularly useful when providing data to an 889 analysis environment with an IPFIX File based workflow, when testing 890 Metering Processes during development, or when the authentication of 891 a Metering Process is important. 893 When collocating a File Writer with a Metering Process, note that 894 Information Elements associated with Exporting or Collecting 895 Processes are meaningless, and SHOULD NOT appear in the Export 896 Session Details Options Template described in Section 8.1.3 or the 897 Message Details Options Template described in Section 8.1.4. 899 When collocating a File Writer with an Metering Process, the Export 900 Time of each Message SHOULD be the time at which the first Data 901 Record in the Message was received from the Metering Process. 903 Note that collocating a File Writer with a Metering Process is the 904 only way to provide positive authentication of a Metering Process 905 through signatures as in Section 9.1. See Section 12.2 for more 906 information on this issue. 908 7.3.3. Using IPFIX Files for Archival Storage 910 While in the general case File Writers should store one Transport 911 Session per IPFIX File, some applications storing large collections 912 of data over long periods of time may benefit from the ability to 913 treat a collection of IPFIX Files as a single Transport Session. A 914 File Writer MAY be configurable to write data from a single Transport 915 Session into multiple IPFIX Files; however, File Writers supporting 916 such a configuration option MUST provide a configuration option to 917 support one-file-per-session behavior for interoperability purposes. 919 File Writers using IPFIX Files for archival storage SHOULD support 920 compression as in Section 10. 922 7.3.4. Using IPFIX Files as Documents 924 When IPFIX Files are used as documents, to store a set of flows 925 relevant to query, investigation, or other common context, or for the 926 publication of traffic datasets relevant to network research, each 927 File MUST be readable as a single Transport Session, self-contained 928 and making no reference to metadata stored in separate Files, in 929 order to ensure interoperability. 931 When writing Files to be used as documents, File Writers MAY emit the 932 special Data Records described by Options Templates before any other 933 Data Records in the File in the following order to ease the 934 inspection and use of documents by File Readers: 936 o Time Window records described by the File Time Window Options 937 Template as defined in Section 8.1.2 below; followed by 939 o Information Element Type Records as described in "Exporting Type 940 Information for IPFIX Information Elements" [RFC5610]; followed by 942 o commonPropertiesId definitions as described in "Reducing 943 Redundancy in IPFIX and PSAMP Reports" [RFC5473]; followed by 945 o Export Session details records described by the Export Session 946 Details Options Template as defined in Section 8.1.3 below. 948 The Export Time of each Message within a File used as a document 949 SHOULD be the time at which the Message was written by the File 950 Writer. 952 If an IPFIX File used as a document uses the technique described in 953 "Reducing Redundancy in IPFIX and PSAMP Reports" [RFC5473] AND all of 954 the non-Options Templates in the File contain the commonPropertiesId 955 Information Element, a File Reader MAY assume the set of 956 commonPropertiesId definitions provides a complete table of contents 957 for the File for searching purposes. 959 7.3.5. Using IPFIX Files for Testing 961 IPFIX Files can be used for testing IPFIX Collecting Processes in two 962 ways. First, IPFIX Files can be used to store specific flow data for 963 regression and stress testing of Collectors; there are no special 964 considerations for IPFIX Files used in this way. 966 Second, IPFIX Files are useful for storing reference messages which 967 do not comply to the IPFIX Protocol in order to test the error 968 handling and recovery behavior of Collectors. Of course, IPFIX Files 969 intended to be used in this application necessarily MAY violate any 970 of the specifications in this document or in [RFC5101], and such 971 Files MUST NOT be transmitted to Collecting Processes or given as 972 input to File Readers not under test. 974 Note that an extremely simple IPFIX Exporting Process may be crafted 975 for testing purposes by simply reading an IPFIX File and transmitting 976 it directly to a Collecting Process. Similarly, an extremely simple 977 Collecting Process may be crafted for testing purposes by simply 978 accepting connections and/or IPFIX Messages from Exporting Processes 979 and writing the session's message stream to an IPFIX File. 981 7.3.6. Writing IPFIX Files for Device Diagnostics 983 IPFIX Files can be used in the debugging of devices which use flow 984 data as internal state, as a common format for the representation of 985 flow tables. In such situations, the opaqueOctets information 986 element can be used to store additional non-IPFIX encoded, non-flow 987 information (e.g., stack backtraces, process state, etc.) within the 988 IPFIX File as in Section 11.1; the IPFIX flow table information could 989 also be embedded in a larger proprietary diagnostic format using 990 delimiters as in Section 11.2 992 7.3.7. IPFIX File Manipulation 994 For many applications, it may prove useful for implementations to 995 provide functionality for the manipulation of IPFIX Files; for 996 example, to select data from a File, to change the Templates used 997 within a File, or to split or join data in Files. Any such utility 998 should take special care to ensure that its output remains a valid 999 IPFIX File, specifically with respect to Templates and Options, which 1000 are scoped to Transport Sessions. 1002 Any operation which splits one File into multiple Files SHOULD write 1003 all necessary Templates and Options to each resulting File, and 1004 ensure that written Options are valid for each resulting File (e.g., 1005 the Time Window Options Template in Section 8.1.2). Any operation 1006 which joins multiple Files into a single File should do the same, 1007 additionally ensuring that Template IDs do not collide, through the 1008 use of different Observation Domain IDs or Template ID rewrition. 1009 Combining operations may also want to ensure any desired ordering of 1010 flow records is maintained. 1012 7.4. Media type of IPFIX Files 1014 The media type for IPFIX Files is application/ipfix. The 1015 registration information [RFC4288] for this media type is given in 1016 the IANA Considerations section. 1018 8. File Format Metadata Specification 1020 This section defines the Options Templates used for IPFIX File 1021 metadata, and the Information Elements they require. 1023 8.1. Recommended Options Templates for IPFIX Files 1025 The following Options Templates allow IPFIX Message streams to meet 1026 the requirements outlined above without extension to the message 1027 format or protocol. They are defined in terms of existing 1028 Information Elements defined in [RFC5102], the Information Elements 1029 defined in "Exporting Type Information for IPFIX Information 1030 Elements" [RFC5610], as well as Information Elements defined in 1031 Section 8.2. IPFIX File Readers and Writers SHOULD support these 1032 Options Templates as defined below. 1034 In addition, IPFIX File Readers and Writers SHOULD support the 1035 Options Templates defined in "Exporting Type Information for IPFIX 1036 Information Elements" [RFC5610] in order to support self-description 1037 of enterprise-specific Information Elements. 1039 8.1.1. Message Checksum Options Template 1041 The Message Checksum Options Template specifies the structure of a 1042 Data Record for attaching an MD5 message checksum to an IPFIX 1043 Message. An MD5 message checksum as described MAY be used if data 1044 integrity is important to the application but file signing is not 1045 available or desired. The described Data Record MUST appear only 1046 once per IPFIX Message, but MAY appear anywhere within the Message. 1048 This Options Template SHOULD contain the following Information 1049 Elements: 1051 +--------------------+----------------------------------------------+ 1052 | IE | Description | 1053 +--------------------+----------------------------------------------+ 1054 | messageScope | A marker denoting this Option applies to the | 1055 | [scope] | whole IPFIX Message; content is ignored. | 1056 | | This Information Element MUST be defined as | 1057 | | a Scope Field. | 1058 | messageMD5Checksum | The MD5 checksum of the containing IPFIX | 1059 | | Message. | 1060 +--------------------+----------------------------------------------+ 1062 8.1.2. File Time Window Options Template 1064 The File Time Window Options Template specifies the structure of a 1065 Data Record for attaching a time window to an IPFIX File; this Data 1066 Record is referred to as a time window record. A time window record 1067 defines the earliest flow start time and the latest flow end time of 1068 the flow records within a File. One and only one time window record 1069 MAY appear within an IPFIX File if the time window information is 1070 available; a File Writer MUST NOT write more than one time window 1071 record to an IPFIX File. A File Writer that writes a time window 1072 record to a File MUST NOT write any Flow with a start time before the 1073 beginning of the window or an end time after the end of the window to 1074 that File. 1076 This Options Template SHOULD contain the following Information 1077 Elements: 1079 +---------------+---------------------------------------------------+ 1080 | IE | Description | 1081 +---------------+---------------------------------------------------+ 1082 | sessionScope | A marker denoting this Option applies to the | 1083 | [scope] | whole IPFIX Transport Session (i.e., the IPFIX | 1084 | | File in the common case); content is ignored. | 1085 | | This Information Element MUST be defined as a | 1086 | | Scope Field. | 1087 | minFlowStart* | Exactly one of minFlowStartSeconds, | 1088 | | minFlowStartMilliseconds, | 1089 | | minFlowStartMicroseconds, or | 1090 | | minFlowStartNanoseconds; SHOULD match the | 1091 | | precision of the accompanying maxFlowEnd* | 1092 | | Information Element. The start time of the | 1093 | | earliest flow in the Transport Session (i.e., | 1094 | | File). | 1095 | maxFlowEnd* | Exactly one of maxFlowEndSeconds, | 1096 | | maxFlowEndMilliseconds, maxFlowEndMicroseconds, | 1097 | | or maxFlowEndNanoseconds; SHOULD match the | 1098 | | precision of the accompanying minFlowStart* | 1099 | | Information Element. The end time of the latest | 1100 | | flow in the Transport Session (i.e., File). | 1101 +---------------+---------------------------------------------------+ 1103 8.1.3. Export Session Details Options Template 1105 The Export Session Details Options Template specifies the structure 1106 of a Data Record for recording the details of an IPFIX Transport 1107 Session in an IPFIX File. It is intended for use in storing a single 1108 complete IPFIX Transport Session in a single IPFIX File. The 1109 described Data Record SHOULD appear only once in a given IPFIX File. 1111 This Options Template SHOULD contain at least the following 1112 Information Elements, subject to applicability as noted on each 1113 Information Element: 1115 +----------------------------+--------------------------------------+ 1116 | IE | Description | 1117 +----------------------------+--------------------------------------+ 1118 | sessionScope [scope] | A marker denoting this Option | 1119 | | applies to the whole IPFIX Transport | 1120 | | Session (i.e., the IPFIX File in the | 1121 | | common case); content is ignored. | 1122 | | This Information Element MUST be | 1123 | | defined as a Scope Field. | 1124 | exporterIPv4Address | IPv4 address of the IPFIX Exporting | 1125 | | Process from which the Messages in | 1126 | | this Transport Session were | 1127 | | received. Present only for | 1128 | | Exporting Processes with an IPv4 | 1129 | | interface. For multi-homed SCTP | 1130 | | associations, this SHOULD be the | 1131 | | primary path endpoint address of the | 1132 | | Exporting Process. | 1133 | exporterIPv6Address | IPv6 address of the IPFIX Exporting | 1134 | | Process from which the Messages in | 1135 | | this Transport Session were | 1136 | | received. Present only for | 1137 | | Exporting Processes with an IPv6 | 1138 | | interface. For multi-homed SCTP | 1139 | | associations, this SHOULD be the | 1140 | | primary path endpoint address of the | 1141 | | Exporting Process. | 1142 | exporterTransportPort | The source port from which the | 1143 | | Messages in this Transport Session | 1144 | | were received. | 1145 | exporterCertificate | The certificate used by the IPFIX | 1146 | | Exporting Process from which the | 1147 | | Messages in this Transport Session | 1148 | | were received. Present only for | 1149 | | Transport Sessions protected by TLS | 1150 | | or DTLS. | 1151 | collectorIPv4Address | IPv4 address of the IPFIX Collecting | 1152 | | Process which received the Messages | 1153 | | in this Transport Session. Present | 1154 | | only for Collecting Processes with | 1155 | | an IPv4 interface. For multi-homed | 1156 | | SCTP associations, this SHOULD be | 1157 | | the primary path endpoint address of | 1158 | | the Collecting Process. | 1159 | collectorIPv6Address | IPv6 address of the IPFIX Collecting | 1160 | | Process which received the Messages | 1161 | | in this Transport Session. Present | 1162 | | only for Collecting Processes with | 1163 | | an IPv6 interface. For multi-homed | 1164 | | SCTP associations, this SHOULD be | 1165 | | the primary path endpoint address of | 1166 | | the Collecting Process. | 1167 | collectorTransportPort | The destination port on which the | 1168 | | Messages in this Transport Session | 1169 | | were received. | 1170 | collectorTransportProtocol | The IP Protocol Identifier of the | 1171 | | transport protocol used to transport | 1172 | | Messages within this Transport | 1173 | | Session. | 1174 | collectorProtocolVersion | The version of the export protocol | 1175 | | used to transport Messages within | 1176 | | this Transport Session. Applicable | 1177 | | only in mixed NetFlow V9-IPFIX | 1178 | | collection environments when storing | 1179 | | NetFlow V9 data in IPFIX Messages, | 1180 | | as in Appendix B | 1181 | collectorCertificate | The certificate used by the IPFIX | 1182 | | Collecting Process which received | 1183 | | the Messages in this Transport | 1184 | | Session. Present only for Transport | 1185 | | Sessions protected by TLS or DTLS. | 1186 | minExportSeconds | The Export Time of the first Message | 1187 | | in the Transport Session. | 1188 | maxExportSeconds | The Export Time of the last Message | 1189 | | in the Transport Session. | 1190 +----------------------------+--------------------------------------+ 1192 8.1.4. Message Details Options Template 1194 The Message Details Options Template specifies the structure of a 1195 Data Record for attaching additional export details to an IPFIX 1196 Message. These details include the time at which a message was 1197 received and information about the export and collection 1198 infrastructure used to transport the Message. This Options Template 1199 also allows the storage of the export session metadata provided the 1200 Export Session Details Options Template, for storing information from 1201 multiple Transport Sessions in the same IPFIX File. 1203 This Options Template SHOULD contain at least the following 1204 Information Elements, subject to applicability as noted for each 1205 Information Element. Note that when used in conjunction with the 1206 Export Session Details Options Template, when storing a single 1207 complete IPFIX Transport Session in an IPFIX File, this Options 1208 Template SHOULD contain only the messageScope and 1209 collectionTimeMilliseconds Information Elements, and the 1210 exportSctpStreamId Information Element for Messages transported via 1211 SCTP. 1213 +----------------------------+--------------------------------------+ 1214 | IE | Description | 1215 +----------------------------+--------------------------------------+ 1216 | messageScope [scope] | A marker denoting this Option | 1217 | | applies to the whole IPFIX message; | 1218 | | content is ignored. This | 1219 | | Information Element MUST be defined | 1220 | | as a Scope Field. | 1221 | collectionTimeMilliseconds | The absolute time at which this | 1222 | | Message was received by the IPFIX | 1223 | | Collecting Process. | 1224 | exporterIPv4Address | IPv4 address of the IPFIX Exporting | 1225 | | Process from which this Message was | 1226 | | received. Present only for | 1227 | | Exporting Processes with an IPv4 | 1228 | | interface, and if this information | 1229 | | is not available via the Export | 1230 | | Session Details Options Template. | 1231 | | For multi-homed SCTP associations, | 1232 | | this SHOULD be the primary path | 1233 | | endpoint address of the Exporting | 1234 | | Process. | 1235 | exporterIPv6Address | IPv6 address of the IPFIX Exporting | 1236 | | Process this Message was received. | 1237 | | Present only for Exporting Processes | 1238 | | with an IPv6 interface, and if this | 1239 | | information is not available via the | 1240 | | Export Session Details Options | 1241 | | Template. For multi-homed SCTP | 1242 | | associations, this SHOULD be the | 1243 | | primary path endpoint address of the | 1244 | | Exporting Process. | 1245 | exporterTransportPort | The source port from which this | 1246 | | Message received. Present only if | 1247 | | this information is not available | 1248 | | via the Export Session Details | 1249 | | Options Template. | 1250 | exporterCertificate | The certificate used by the IPFIX | 1251 | | Exporting Process from which this | 1252 | | Message was received. Present only | 1253 | | for Transport Sessions protected by | 1254 | | TLS or DTLS. | 1255 | collectorIPv4Address | IPv4 address of the IPFIX Collecting | 1256 | | Process which received this Message. | 1257 | | Present only for Collecting | 1258 | | Processes with an IPv4 interface, | 1259 | | and if this information is not | 1260 | | available via the Export Session | 1261 | | Details Options Template. For | 1262 | | multi-homed SCTP associations, this | 1263 | | SHOULD be the primary path endpoint | 1264 | | address of the Collecting Process. | 1265 | collectorIPv6Address | IPv6 address of the IPFIX Collecting | 1266 | | Process which received this Message. | 1267 | | Present only for Collecting | 1268 | | Processes with an IPv6 interface, | 1269 | | and if this information is not | 1270 | | available via the Export Session | 1271 | | Details Options Template. For | 1272 | | multi-homed SCTP associations, this | 1273 | | SHOULD be the primary path endpoint | 1274 | | address of the Collecting Process. | 1275 | collectorTransportPort | The destination port on which this | 1276 | | Message was received. Present only | 1277 | | if this information is not available | 1278 | | via the Export Session Details | 1279 | | Options Template. | 1280 | collectorTransportProtocol | The IP Protocol Identifier of the | 1281 | | transport protocol used to transport | 1282 | | this Message. Present only if this | 1283 | | information is not available via the | 1284 | | Export Session Details Options | 1285 | | Template. | 1286 | collectorProtocolVersion | The version of the export protocol | 1287 | | used to transport this Message. | 1288 | | Present only if necessary and if | 1289 | | this information is not available | 1290 | | via the Export Session Details | 1291 | | Options Template. | 1292 | collectorCertificate | The certificate used by the IPFIX | 1293 | | Collecting Process which received | 1294 | | this Message. Present only for | 1295 | | Transport Sessions protected by TLS | 1296 | | or DTLS. | 1297 | exportSctpStreamId | The SCTP stream used to transport | 1298 | | this Message. Present only if the | 1299 | | Message was transported via SCTP. | 1300 +----------------------------+--------------------------------------+ 1302 8.2. Recommended Information Elements for IPFIX Files 1304 The following Information Elements are used by the Options Templates 1305 in Section 8.1 to allow IPFIX Message streams to meet the 1306 requirements outlined above without extension of the protocol. IPFIX 1307 File Readers and Writers SHOULD support these Information Elements as 1308 defined below. 1310 In addition, IPFIX File Readers and Writers SHOULD support the 1311 Information Elements defined in "Exporting Type Information for IPFIX 1312 Information Elements" [RFC5610] in order to support full self- 1313 description of Information Elements. 1315 8.2.1. collectionTimeMilliseconds 1317 Description: The absolute timestamp at which the data within the 1318 scope containing this Information Element was received by a 1319 Collecting Process. This Information Element SHOULD be bound to 1320 its containing IPFIX Message via IPFIX Options and the 1321 messageScope Information Element, as defined below. 1323 Abstract Data Type: dateTimeMilliseconds 1325 ElementId: TBD1 1327 Status: current 1329 8.2.2. collectorCertificate 1331 Description: The full X.509 certificate, encoded in ASN.1 DER 1332 format, used by the collector when IPFIX Messages were transmitted 1333 using TLS or DTLS. This Information Element SHOULD be bound to 1334 its containing IPFIX Transport Session via an options record and 1335 the sessionScope Information Element, or to its containing IPFIX 1336 Message via an options record and the messageScope Information 1337 Element. 1339 Abstract Data Type: octetArray 1341 ElementId: TBD17 1343 Status: current 1345 8.2.3. exporterCertificate 1347 Description: The full X.509 certificate, encoded in ASN.1 DER 1348 format, used by the collector when IPFIX Messages were transmitted 1349 using TLS or DTLS. This Information Element SHOULD be bound to 1350 its containing IPFIX Transport Session via an options record and 1351 the sessionScope Information Element, or to its containing IPFIX 1352 Message via an options record and the messageScope Information 1353 Element. 1355 Abstract Data Type: octetArray 1357 ElementId: TBD18 1358 Status: current 1360 8.2.4. exportSctpStreamId 1362 Description: The value of the SCTP Stream Identifier used by the 1363 Exporting Process for exporting IPFIX Message data. This is 1364 carried in the Stream Identifier field of the header of the SCTP 1365 DATA chunk containing the IPFIX Message(s). 1367 Abstract Data Type: unsigned16 1369 Data Type Semantics: identifier 1371 ElementId: TBD2 1373 Status: current 1375 8.2.5. maxExportSeconds 1377 Description: The absolute Export Time of the latest IPFIX Message 1378 within the scope containing this Information Element. This 1379 Information Element SHOULD be bound to its containing IPFIX 1380 Transport Session via IPFIX Options and the sessionScope 1381 Information Element. 1383 Abstract Data Type: dateTimeSeconds 1385 ElementId: TBD3 1387 Status: current 1389 Units: seconds 1391 8.2.6. maxFlowEndMicroseconds 1393 Description: The latest absolute timestamp of the last packet 1394 within any Flow within the scope containing this Information 1395 Element, rounded up to the microsecond if necessary. This 1396 Information Element SHOULD be bound to its containing IPFIX 1397 Transport Session via IPFIX Options and the sessionScope 1398 Information Element. This Information Element SHOULD be used only 1399 in Transport Sessions containing Flow Records with microsecond- 1400 precision (or better) timestamp Information Elements. 1402 Abstract Data Type: dateTimeMicroseconds 1403 ElementId: TBD11 1405 Status: current 1407 Units: microseconds 1409 8.2.7. maxFlowEndMilliseconds 1411 Description: The latest absolute timestamp of the last packet 1412 within any Flow within the scope containing this Information 1413 Element, rounded up to the millisecond if necessary. This 1414 Information Element SHOULD be bound to its containing IPFIX 1415 Transport Session via IPFIX Options and the sessionScope 1416 Information Element. This Information Element SHOULD be used only 1417 in Transport Sessions containing Flow Records with millisecond- 1418 precision (or better) timestamp Information Elements. 1420 Abstract Data Type: dateTimeMilliseconds 1422 ElementId: TBD12 1424 Status: current 1426 Units: milliseconds 1428 8.2.8. maxFlowEndNanoseconds 1430 Description: The latest absolute timestamp of the last packet 1431 within any Flow within the scope containing this Information 1432 Element. This Information Element SHOULD be bound to its 1433 containing IPFIX Transport Session via IPFIX Options and the 1434 sessionScope Information Element. This Information Element SHOULD 1435 be used only in Transport Sessions containing Flow Records with 1436 nanosecond-precision timestamp Information Elements. 1438 Abstract Data Type: dateTimeNanoseconds 1440 ElementId: TBD13 1442 Status: current 1444 Units: nanoseconds 1446 8.2.9. maxFlowEndSeconds 1447 Description: The latest absolute timestamp of the last packet 1448 within any Flow within the scope containing this Information 1449 Element, rounded up to the second if necessary. This Information 1450 Element SHOULD be bound to its containing IPFIX Transport Session 1451 via IPFIX Options and the sessionScope Information Element. 1453 Abstract Data Type: dateTimeSeconds 1455 ElementId: TBD4 1457 Status: current 1459 Units: seconds 1461 8.2.10. messageMD5Checksum 1463 Description: The MD5 checksum of the IPFIX Message containing this 1464 record. This Information Element SHOULD be bound to its 1465 containing IPFIX Message via an options record and the 1466 messageScope Information Element, as defined below, and SHOULD 1467 appear only once in a given IPFIX Message. To calculate the value 1468 of this Information Element, first buffer the containing IPFIX 1469 Message, setting the value of this Information Element to all 1470 zeroes. Then calculate the MD5 checksum of the resulting buffer 1471 as defined in [RFC1321], place the resulting value in this 1472 Information Element, and export the buffered message. This 1473 Information Element is intended as a simple checksum only; 1474 therefore collision resistance and algorithm agility are not 1475 required, and MD5 is an appropriate message digest. 1477 Abstract Data Type: octetArray (16 bytes) 1479 ElementId: TBD5 1481 Status: current 1483 Reference: RFC 1321, The MD5 Message-Digest Algorithm [RFC1321] 1485 8.2.11. messageScope 1487 Description: The presence of this Information Element as scope in 1488 an Options Template signifies that the options described by the 1489 Template apply to the IPFIX Message that contains them. It is 1490 defined for general purpose message scoping of options, and 1491 proposed specifically to allow the attachment a checksum to a 1492 message via IPFIX Options. The value of this Information Element 1493 MUST be written as 0 by the File Writer or Exporting Process. The 1494 value of this Information Element MUST be ignored by the File 1495 Reader or the Collecting Process. 1497 Abstract Data Type: octet 1499 ElementId: TBD6 1501 Status: current 1503 8.2.12. minExportSeconds 1505 Description: The absolute Export Time of the earliest IPFIX Message 1506 within the scope containing this Information Element. This 1507 Information Element SHOULD be bound to its containing IPFIX 1508 Transport Session via an options record and the sessionScope 1509 Information Element. 1511 Abstract Data Type: dateTimeSeconds 1513 ElementId: TBD7 1515 Status: current 1517 Units: seconds 1519 8.2.13. minFlowStartMicroseconds 1521 Description: The earliest absolute timestamp of the first packet 1522 within any Flow within the scope containing this Information 1523 Element, rounded down to the microsecond if necessary. This 1524 Information Element SHOULD be bound to its containing IPFIX 1525 Transport Session via an options record and the sessionScope 1526 Information Element. This Information Element SHOULD be used only 1527 in Transport Sessions containing Flow Records with microsecond- 1528 precision (or better) timestamp Information Elements. 1530 Abstract Data Type: dateTimeMicroseconds 1532 ElementId: TBD14 1534 Status: current 1536 Units: microseconds 1538 8.2.14. minFlowStartMilliseconds 1539 Description: The earliest absolute timestamp of the first packet 1540 within any Flow within the scope containing this Information 1541 Element, rounded down to the millisecond if necessary. This 1542 Information Element SHOULD be bound to its containing IPFIX 1543 Transport Session via an options record and the sessionScope 1544 Information Element. This Information Element SHOULD be used only 1545 in Transport Sessions containing Flow Records with millisecond- 1546 precision (or better) timestamp Information Elements. 1548 Abstract Data Type: dateTimeMilliseconds 1550 ElementId: TBD15 1552 Status: current 1554 Units: milliseconds 1556 8.2.15. minFlowStartNanoseconds 1558 Description: The earliest absolute timestamp of the first packet 1559 within any Flow within the scope containing this Information 1560 Element. This Information Element SHOULD be bound to its 1561 containing IPFIX Transport Session via an options record and the 1562 sessionScope Information Element. This Information Element SHOULD 1563 be used only in Transport Sessions containing Flow Records with 1564 nanosecond-precision timestamp Information Elements. 1566 Abstract Data Type: dateTimeNanoseconds 1568 ElementId: TBD16 1570 Status: current 1572 Units: nanoseconds 1574 8.2.16. minFlowStartSeconds 1576 Description: The earliest absolute timestamp of the first packet 1577 within any Flow within the scope containing this Information 1578 Element, rounded down to the second if necessary. This 1579 Information Element SHOULD be bound to its containing IPFIX 1580 Transport Session via an options record and the sessionScope 1581 Information Element. 1583 Abstract Data Type: dateTimeSeconds 1584 ElementId: TBD8 1586 Status: current 1588 Units: seconds 1590 8.2.17. opaqueOctets 1592 Description: This Information Element is used to encapsulate non- 1593 IPFIX data into an IPFIX Message stream, for the purpose of 1594 allowing a non-IPFIX data processor to store a data stream inline 1595 within an IPFIX File. A Collecting Process or File Writer MUST 1596 NOT try to interpret this binary data. This Information Element 1597 differs from paddingOctets as its contents are meaningful in some 1598 non-IPFIX context, while the contents of paddingOctets MUST be 1599 0x00 and are intended only for Information Element alignment. 1601 Abstract Data Type: octet 1603 ElementId: TBD9 1605 Status: current 1607 8.2.18. sessionScope 1609 Description: The presence of this Information Element as scope in 1610 an Options Template signifies that the options described by the 1611 Template apply to the IPFIX Transport Session that contains them. 1612 Note that as all options are implicitly scoped to Transport 1613 Session and Observation Domain, this Information Element is 1614 equivalent to a "null" scope. It is defined for general purpose 1615 session scoping of options, and proposed specifically to allow the 1616 attachment of time window to an IPFIX File via IPFIX Options. The 1617 value of this Information Element MUST be written as 0 by the File 1618 Writer or Exporting Process. The value of this Information 1619 Element MUST be ignored by the File Reader or the Collecting 1620 Process. 1622 Abstract Data Type: octet 1624 ElementId: TBD10 1626 Status: current 1628 9. Signing and Encryption of IPFIX Files 1630 In order to ensure the integrity of IPFIX Files and the identity of 1631 IPFIX File Writers, File Writers and File Readers SHOULD provide for 1632 an interoperable and easily-implemented method for signing IPFIX 1633 Files, and verifying those signatures. This section specifies method 1634 via CMS detached signatures. 1636 Note that while CMS specifies an encapsulation format which can be 1637 used for encryption as well as signing, no method is specified for 1638 encapsulation for confidentiality protection. It is assumed that 1639 application-specific or process-specific requirements outweigh the 1640 need for interoperability for encrypted files. 1642 9.1. CMS Detached Signatures 1644 The Cryptographic Message Syntax (CMS) [RFC3852] defines an 1645 encapsulation syntax for data protection, used to digitally sign, 1646 authenticate, or encrypt arbitrary message content. CMS can also be 1647 used to create detached signatures, in which the signature is stored 1648 in a separate file. This arrangement maximizes interoperability, as 1649 File Readers which are not aware of CMS detached signatures and have 1650 no requirement for them can simply ignore them; the content of the 1651 IPFIX File itself is unchanged by the signature. 1653 The detached signature file for an IPFIX File SHOULD be stored, 1654 transported, or otherwise made available (e.g., by FTP or HTTP) 1655 alongside the signed IPFIX File, with the same filename as the IPFIX 1656 File, except that the file extension ".p7s" is added to the end, 1657 conforming to the naming convention in [RFC3851]. 1659 Within the detached signature, the CMS ContentInfo type MUST always 1660 be present, and it MUST encapsulate the CMS SignedData content type, 1661 which in turn MUST NOT encapsulate the signed IPFIX File content. 1662 The CMS detached signature is summarized as follows: 1664 ContentInfo { 1665 contentType id-signedData, -- (1.2.840.113549.1.7.2) 1666 content SignedData 1667 } 1669 SignedData { 1670 version CMSVersion, -- Always set to 3 1671 digestAlgorithms DigestAlgorithmIdentifiers, 1672 encapContentInfo EncapsulatedContentInfo, 1673 certificates CertificateSet, -- File Writer certificate(s) 1674 crls CertificateRevocationLists, -- Optional 1675 signerInfos SET OF SignerInfo -- Only one signer 1676 } 1678 SignerInfo { 1679 version CMSVersion, -- Always set to 3 1680 sid SignerIdentifier, 1681 digestAlgorithm DigestAlgorithmIdentifier, 1682 signedAttrs SignedAttributes, 1683 signatureAlgorithm SignatureAlgorithmIdentifier, 1684 signature SignatureValue, 1685 unsignedAttrs UnsignedAttributes 1686 } 1688 EncapsulatedContentInfo { 1689 eContentType id-ct-anyContentType, -- (1.2.840.113549.1.9.16.1.0) 1690 eContent OCTET STRING -- Always absent 1691 } 1693 The details of the contents of each CMS encapsulation are detailed in 1694 the subsections below. 1696 9.1.1. ContentInfo 1698 [RFC3852] requires the outer-most encapsulation to be ContentInfo; 1699 the fields of ContentInfo are as follows: 1701 contentType indicates the type of the associated content. For the 1702 detached signature file, the encapsulated type is always 1703 SignedData, so the id-signedData (1.2.840.113549.1.7.2) object 1704 identifier MUST be present in this field. 1706 content contains a SignedData content, detailed in Section 9.1.2. 1708 9.1.2. SignedData 1710 The SignedData content type contains the signature of the IPFIX File 1711 and information to aid in validation; the fields of SignedData are as 1712 follows: 1714 version MUST be 3. 1716 digestAlgorithms is a collection of one-way hash function 1717 identifiers. It MUST contain the identifier used by the File 1718 Writer to generate the digital signature. 1720 encapContentInfo is the signed content, including a content type 1721 identifier. Since a detached signature is being created, it does 1722 not encapsulate the IPFIX File. The EncapsulatedContentInfo is 1723 detailed in Section 9.1.4. 1725 certificates is a collection of certificates. It SHOULD include the 1726 X.509 certificate needed to validate the digital signature file. 1727 Certification Authority (CA) and File Writer certificates MUST 1728 conform to the certificate profile specified in [RFC5280]. 1730 crls is an optional collection of certificate revocation lists 1731 (CRLs). It SHOULD NOT contain any CRLs; any CRLs which are 1732 present MUST conform to the certificate profile specified in 1733 [RFC5280]. 1735 signerInfos is a collection of per-signer information; this 1736 identifies the File Writer. More than one SignerInfo MAY appear 1737 to facilitate transitions between keys or algorithms. The 1738 SignerInfo type is detailed in Section 9.1.3. 1740 9.1.3. SignerInfo 1742 The SignerInfo type identifies the File Writer; the fields of 1743 SignerInfo are as follows: 1745 version MUST be 3. 1747 sid identifies the File Writer's public key. This identifier MUST 1748 match the value included in the subjectKeyIdentifier certificate 1749 extension on the File Writer's X.509 certificate. 1751 digestAlgorithm identifies the one-way hash function and associated 1752 parameters used to generate the signature. 1754 signedAttrs is an optional set of attributes that are signed along 1755 with the content. 1757 digestAlgorithm identifies the digital signature algorithm and 1758 associated parameters used to generate the signature. 1760 signature is the digital signature of the associated file. 1762 unsignedAttrs is an optional of attributes that are not signed. 1764 9.1.4. EncapsulatedContentInfo 1766 The EncapsulatedContentInfo structure contains a content type 1767 identifier. Since a detached signature is being created, it does not 1768 encapsulate the Internet-Draft. The fields of 1769 EncapsulatedContentInfo are as follows: 1771 eContentType is an object identifier that uniquely specifies the 1772 content type. The content type associated with the plain text 1773 file MUST be id-ct-anyContentType (1.2.840.113549.1.9.16.1.0). 1775 eContent is an optional field containing the signed content. Since 1776 this is a detached signature, eContent MUST be absent. 1778 9.2. Encryption Error Resilience 1780 File-level encryption has error resilience issues similar to file- 1781 level compression. Single bit errors in the encrypted data stream 1782 can result in larger errors in the decrypted stream, depending on the 1783 encryption scheme used. 1785 In applications (e.g. archival storage) in which error resilience is 1786 very important, File Writers SHOULD use an encryption scheme which 1787 can resynchronize after bit errors. An common example is a block 1788 cipher in CBC (Cipher Block Chaining) mode. In this case, File 1789 Writers MAY also use the Message Checksum Options Template to attach 1790 a checksum to each IPFIX Message in the IPFIX File, in order to 1791 support the recognition of errors in the decrypted data. 1793 10. Compression of IPFIX Files 1795 Network traffic measurement data is also generally highly 1796 compressible. IPFIX Templates tend to increase the information 1797 content per record by not requiring the export of irrelevant or non- 1798 present fields in records, and the technique described in [RFC5473] 1799 also reduces the export of redundant information. However, even with 1800 these techniques, generalized compression can decrease storage 1801 requirements significantly; therefore, IPFIX File Writers and File 1802 Readers SHOULD support compression as described in this section. 1804 10.1. Supported Compression Formats 1806 IPFIX files support two compression encapsulation formats: bzip2 1807 [bzip2] and gzip [RFC1952]. bzip2 provides better compression than 1808 gzip and, as a block compression algorithm, better error recovery 1809 characteristics, at the expense of slower compression. gzip is 1810 potentially a better choice when compression time is an issue. These 1811 two algorithms and encapsulation formats were chosen for ubiquity and 1812 ease of implementation. 1814 IPFIX File Readers and Writers supporting compression MUST support 1815 bzip2, and SHOULD support gzip. 1817 10.2. Compression Recognition at the File Reader 1819 bzip2, gzip, and uncompressed IPFIX Files have distinct magic 1820 numbers. IPFIX File Readers SHOULD use these magic numbers to 1821 determine what compression, if any, is in use for an IPFIX File, and 1822 invoke the proper decompression. bzip2 files are identified by the 1823 initial three-octet string 0x42, 0x5A, 0x68 ("BZh"). gzip files are 1824 identified by the initial two-octet string 0x1F, 0x8B. IPFIX Files 1825 are identified by the initial two-octet string 0x00, 0x0A; these are 1826 the version bytes of the first IPFIX Message header in the File. 1828 10.3. Compression Error Resilience 1830 Note that, since any file may be compressed and decompressed with a 1831 variety of widely available tools implementing a variety of 1832 compression standards (both specified and de facto), compression of 1833 IPFIX File data can be accomplished externally. However, compression 1834 at the file level is not particularly resilient to errors; in the 1835 worst case, a single bit error in a stream-compressed file may result 1836 in the loss of the entire file. 1838 Since block compression algorithms that support the identification 1839 and isolation of blocks containing errors limit the impact of errors 1840 on the recoverability of compressed data, the use of bzip2 in 1841 applications where error resilience is important is RECOMMENDED. 1843 Since the block boundary of a block-compressed IPFIX File may fall in 1844 the middle of an IPFIX Message, resynchronization of an IPFIX Message 1845 stream by a File Reader after a compression error requires some care. 1846 The beginning of an IPFIX Message may be identified by its header 1847 signature (the Version field of the Message Header, 0x00 0x0A, 1848 followed by a 16-bit Message Length), but simply searching for the 1849 first occurance of the Version field is insufficient, since these two 1850 bytes may occur in valid IPFIX Template or Data Sets. 1852 Therefore, we specify the following algorithm for File Readers to 1853 resynchronize an IPFIX Message Stream after skipping a compressed 1854 block containing errors: 1856 1. Search after the error for the first occurrence of the octet 1857 string 0x00, 0x0A (the IPFIX Message Header Version field.) 1859 2. Treat this field as the beginning of a candidate IPFIX Message. 1860 Read the two bytes following the Version field as a Message 1861 Length, and seek to that offset from the beginning of the 1862 candidate IPFIX Message. 1864 3. If the first two octets after the candidate IPFIX Message are 1865 0x00, 0x0A (i.e., the IPFIX Message Header Version field of the 1866 next message in the stream), or if end-of-file is reached 1867 precisely at the end of the candidate IPFIX Message, presume that 1868 the candidate IPFIX Message is valid, and begin reading the IPFIX 1869 File from the start of the candidate IPFIX Message. 1871 4. If not, or if the seek reaches end-of-file or another block 1872 containing errors before finding the end of the candidate 1873 message, go back to step 1, starting the search two bytes from 1874 the start of the candidate IPFIX Message. 1876 The algorithm above will improperly identify a non-message as a 1877 message approximately 1 in 2^32 times, assuming random IPFIX data. 1878 It may be expanded to consider multiple candidate IPFIX Messages in 1879 order to increase reliability. 1881 In applications (e.g. archival storage) in which error resilience is 1882 very important, File Writers SHOULD use block compression algorithms, 1883 and MAY attempt to align IPFIX Messages within compression blocks to 1884 ease resynchronization after errors, if such is supported by the 1885 chosen block compressor. File Readers SHOULD use the 1886 resynchronization algorithm above to minimize data loss due to 1887 compression errors. 1889 11. Recommended File Integration Strategies 1891 This section describes methods for integrating IPFIX File data with 1892 other file formats. 1894 11.1. Encapsulation of Non-IPFIX Data in IPFIX Files 1896 At times it may be useful to export or store non-IPFIX data inline in 1897 an IPFIX File or Message stream. To do this cleanly, this data must 1898 be encapsulated into IPFIX Messages so that an IPFIX File Reader or 1899 Collecting Process can handle it without any need to interpret it. 1900 At the same time, this data must not be changed during transmission 1901 or storage. The opaqueOctets Information Element as defined in 1902 Section 8.2.17 is provided for this encapsulation. 1904 Processing the encapsulated non-IPFIX data is left to a separate 1905 processing mechanisms that can identify encapsulated non-IPFIX data 1906 in an IPFIX message stream, but need not have any other IPFIX 1907 handling capability, except the ability to skip over all IPFIX 1908 messages that do not encapsulate non-IPFIX data. 1910 The Message Checksum Options Template, described in Section 8.1.1 may 1911 be used as a uniform mechanism to identify errors within encapsulated 1912 data. 1914 Note that this mechanism can only encapsulate data objects up to 1915 65,515 octets in length. If the space available in one IPFIX Message 1916 is not enough for the amount of data to be encapsulated, then the 1917 data must be broken into smaller segments that are encapsulated into 1918 consecutive IPFIX Messages. Any additional structuring or semantics 1919 of the raw data is outside the scope of IPFIX and must be implemented 1920 within the encapsulated binary data itself. Furthermore, the raw 1921 encapsulated data cannot be assumed by an IPFIX File Reader to have 1922 any specific format. 1924 11.2. Encapsulation of IPFIX Files within Other File Formats 1926 Consequently, it may also be useful to reverse the encapsulation, 1927 that is, to export or store IPFIX data inline within a non-IPFIX file 1928 or data stream. This makes sense when the other file format is not 1929 compatible with the encapsulation described above in Section 11.1. 1930 Generally speaking, the encapsulation here will be specific to the 1931 format of the containing file. For example, IPFIX Files may be 1932 embedded in XML elements using hex or Base64 encoding, or in raw 1933 binary files using start and end delimiters or some form of run- 1934 length encoding. As there are as many potential encapsulations here 1935 as there are potential file formats, the specifics of each are out of 1936 scope for this specification. 1938 12. Security Considerations 1940 The security considerations section of [RFC5101], on which the IPFIX 1941 File format is based, is largely concerned with the proper 1942 application of TLS and DTLS to ensure confidentiality and integrity 1943 when exporting IPFIX Messages. By analogy, this document specifies 1944 the use of CMS [RFC3852] detached signatures to provide equivalent 1945 integrity protection to TLS and DTLS in Section 9.1. However, aside 1946 from merely applying CMS for signatures, there are several security 1947 issues which much be considered in certain circumstances; these are 1948 covered in the subsections below. 1950 12.1. Relationship between IPFIX File and Transport Encryption 1952 The underlying protocol used to exchange the information that will be 1953 stored using the format proposed in this document must as well apply 1954 appropriate procedures to guarantee the integrity and confidentiality 1955 of the exported information. Such issues are addressed in [RFC5101]. 1956 Specifically, IPFIX Files which store data taken from an IPFIX 1957 Collecting Process using TLS or DTLS for transport security SHOULD be 1958 signed as in Section 9.1 and SHOULD be encrypted out of band; storage 1959 of such flow data without encryption may present a potential breach 1960 of confidentiality. Conversely, flow data considered sensitive 1961 enough to require encryption in storage that is later transmitted 1962 using IPFIX SHOULD be transmitted using TLS or DTLS for transport 1963 security. 1965 12.2. End-to-End Assertions for IPFIX Files 1967 Note that while both TLS and CMS provide the ability to sign an IPFIX 1968 Transport Session or an IPFIX File, there exists no method for 1969 protecting data integrity end-to-end in the case in which a 1970 Collecting Process is collocated with a File Writer. The channel 1971 between the Exporting Process to Collecting Process using IPFIX is 1972 signed by the Exporting Process key and protected via TLS and DTLS, 1973 while the File is signed by the File Writer key and protected via 1974 CMS. The identity of the Exporting Process is not asserted in the 1975 file, and the records may be modified between the Collecting Process 1976 and the File Writer. 1978 There are two potential ways to address this issue. The first is by 1979 fiat, and is appropriate only when the application allows the 1980 Collecting Process to File Writer channel to be trusted. In this 1981 case, the File Writer's signature is an implicit assertion that the 1982 channel to the Exporting Process was protected, that the Exporting 1983 Process's signature was verified, and that the data was not changed 1984 after collection. For this to work, a File Writer collocated with a 1985 Collecting Process SHOULD NOT sign a File as specified in Section 9.1 1986 unless the Transport Session over which the data was exported was 1987 protected via TLS or DTLS, and the Collecting Process positively 1988 identified the Exporting Process by its certificate. The File Writer 1989 SHOULD include the Exporting Process and Collecting Process 1990 certificates within the File using the Export Session Detail Options 1991 Template in Section 8.1.3 or the Message Detail Options Template in 1992 Section 8.1.4 to allow for later verification. 1994 In situations in which the Collecting Process and/or File Writer 1995 cannot be trusted, end-to-end integrity can then be approximated by 1996 collocating the File Writer with the Metering Process, and removing 1997 the IPFIX Protocol completely from the chain. In this case, the File 1998 Writer's signature is an implicit assertion that the Metering Process 1999 is identified and is not tampering with the information as observed 2000 on the wire. 2002 Verification of these trust relationships is out of scope for this 2003 document, and should be considered on a per-implementation basis. 2005 12.3. Recommendations for Strength of Cryptography for IPFIX Files 2007 Note that when encrypting files for archival storage, the 2008 cryptographic strength is dependent on the length of time over which 2009 archival data is expected to be kept. Long term storage may require 2010 re-application of cryptographic protection, periodically resigning 2011 and reencrypting files with stronger keys. In this case, it is 2012 recommended that the existing signed and/or encypted data be 2013 encapsulated within newer, stronger protection. See [RFC4810] for a 2014 discussion of this issue. 2016 13. IANA Considerations 2018 This document specifies the creation of several new IPFIX Information 2019 Elements in the IPFIX Information Element registry located at 2020 http://www.iana.org/assignments/ipfix, as defined in Section 8.2 2021 above. IANA has assigned the following Information Element numbers 2022 for their respective Information Elements as specified below: 2024 o Information Element number TBD1 for the collectionTimeMilliseconds 2025 Information Element. 2027 o Information Element number TBD17 for the collectorCertificate 2028 Information Element. 2030 o Information Element number TBD18 for the exporterCertificate 2031 Information Element. 2033 o Information Element number TBD2 for the exportSctpStreamId 2034 Information Element. 2036 o Information Element number TBD3 for the maxExportSeconds 2037 Information Element. 2039 o Information Element number TBD11 for the maxFlowEndMicroseconds 2040 Information Element. 2042 o Information Element number TBD12 for the maxFlowEndMilliseconds 2043 Information Element. 2045 o Information Element number TBD13 for the maxFlowEndNanoseconds 2046 Information Element. 2048 o Information Element number TBD4 for the maxFlowEndSeconds 2049 Information Element. 2051 o Information Element number TBD5 for the messageMD5Checksum 2052 Information Element. 2054 o Information Element number TBD6 for the messageScope Information 2055 Element. 2057 o Information Element number TBD7 for the minExportSeconds 2058 Information Element. 2060 o Information Element number TBD14 for the minFlowStartMicroseconds 2061 Information Element. 2063 o Information Element number TBD15 for the minFlowStartMilliseconds 2064 Information Element. 2066 o Information Element number TBD16 for the minFlowStartNanoseconds 2067 Information Element. 2069 o Information Element number TBD8 for the minFlowStartSeconds 2070 Information Element. 2072 o Information Element number TBD9 for the opaqueOctets Information 2073 Element. 2075 o Information Element number TBD10 for the sessionScope Information 2076 Element. 2078 [NOTE for IANA: The text TBDn should be replaced with the respective 2079 assigned Information Element numbers where they appear in this 2080 document.] 2082 IANA has created the media type application/ipfix for IPFIX data, as 2083 described by the following registration information: 2085 Type name: application 2087 Subtype name: ipfix 2089 Required parameters: none 2091 Optional parameters: none 2093 Encoding considerations: IPFIX files are binary, and therefore must 2094 be encoded in non-binary contexts. 2096 Security considerations: See the Security Considerations section 2097 (Section 12) of this document, and the Security Considerations 2098 section of [RFC5101]. 2100 Interoperability considerations: See the Detailed Specification 2101 section (Section 7) of this document. The format is designed to 2102 be broadly interoperable, as any valid stream of IPFIX Messages 2103 over any transport specified in [RFC5101] MUST be recognizable as 2104 a valid IPFIX File. 2106 Published specification: This document, especially Section 7, and 2107 [RFC5101]. 2109 Applications that use this media type: Various IPFIX 2110 implementations (see [RFC5153]) support the construction of IPFIX 2111 File Readers and Writers. 2113 Additional information: none 2115 Magic number(s): None, although the first two bytes of any IPFIX 2116 file are the first two bytes of a message header, the Version 2117 field, which as of [RFC5101] are always 10 in network byte order: 2118 0x00, 0x0A. 2120 File extension(s): .ipfix 2122 Macintosh file type code(s): none 2124 For further information: Authors of this document, IETF IPFIX 2125 Working Group 2127 Intended usage: LIMITED USE 2129 Restrictions on usage: none 2130 Change controller: Authors of this document, IETF IPFIX Working 2131 Group 2133 14. Acknowledgements 2135 Thanks to Maurizio Molina, Tom Kosnar, and Andreas Kind for technical 2136 assistance with the requirements for a standard flow storage format. 2137 Thanks to Benoit Claise, Paul Aitken, Andrew Johnson, Gerhard Muenz, 2138 and Nevil Brownlee for their reviews and feedback. Thanks to Pasi 2139 Eronen for pointing out [RFC5485], and Russ Housley for writing it; 2140 it specifies a detached signature format, from which Section 9.1 is 2141 largely drawn. Thanks to the PRISM project for its support of this 2142 work. 2144 15. References 2146 15.1. Normative References 2148 [RFC5101] Claise, B., "Specification of the IP Flow Information 2149 Export (IPFIX) Protocol for the Exchange of IP Traffic 2150 Flow Information", RFC 5101, January 2008. 2152 [RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J. 2153 Meyer, "Information Model for IP Flow Information Export", 2154 RFC 5102, January 2008. 2156 [RFC5610] Boschi, E., Trammell, B., Mark, L., and T. Zseby, 2157 "Exporting Type Information for IP Flow Information Export 2158 (IPFIX) Information Elements", RFC 5610, July 2009. 2160 [RFC1321] Rivest, R., "The MD5 Message-Digest Algorithm", RFC 1321, 2161 April 1992. 2163 [RFC1952] Deutsch, P., Gailly, J-L., Adler, M., Deutsch, L., and G. 2164 Randers-Pehrson, "GZIP file format specification version 2165 4.3", RFC 1952, May 1996. 2167 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2168 Requirement Levels", BCP 14, RFC 2119, March 1997. 2170 [RFC3852] Housley, R., "Cryptographic Message Syntax (CMS)", 2171 RFC 3852, July 2004. 2173 [RFC4810] Wallace, C., Pordesch, U., and R. Brandner, "Long-Term 2174 Archive Service Requirements", RFC 4810, March 2007. 2176 [RFC5280] Cooper, D., Santesson, S., Farrell, S., Boeyen, S., 2177 Housley, R., and W. Polk, "Internet X.509 Public Key 2178 Infrastructure Certificate and Certificate Revocation List 2179 (CRL) Profile", RFC 5280, May 2008. 2181 [bzip2] Seward, J., "bzip2 (http://www.bzip.org/)", March 2008. 2183 15.2. Informative References 2185 [RFC3917] Quittek, J., Zseby, T., Claise, B., and S. Zander, 2186 "Requirements for IP Flow Information Export (IPFIX)", 2187 RFC 3917, October 2004. 2189 [RFC3954] Claise, B., "Cisco Systems NetFlow Services Export Version 2190 9", RFC 3954, October 2004. 2192 [RFC5153] Boschi, E., Mark, L., Quittek, J., Stiemerling, M., and P. 2193 Aitken, "IP Flow Information Export (IPFIX) Implementation 2194 Guidelines", RFC 5153, April 2008. 2196 [RFC5470] Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek, 2197 "Architecture for IP Flow Information Export", RFC 5470, 2198 March 2009. 2200 [RFC5471] Schmoll, C., Aitken, P., and B. Claise, "Guidelines for IP 2201 Flow Information Export (IPFIX) Testing", RFC 5471, 2202 March 2009. 2204 [RFC5472] Zseby, T., Boschi, E., Brownlee, N., and B. Claise, "IP 2205 Flow Information Export (IPFIX) Applicability", RFC 5472, 2206 March 2009. 2208 [RFC5473] Boschi, E., Mark, L., and B. Claise, "Reducing Redundancy 2209 in IP Flow Information Export (IPFIX) and Packet Sampling 2210 (PSAMP) Reports", RFC 5473, March 2009. 2212 [SAINT2007] 2213 Trammell, B., Boschi, E., Mark, L., and T. Zseby, 2214 "Requirements for a standardized flow storage solution", 2215 in Proceedings of the SAINT 2007 workshop on Internet 2216 Measurement Technology, Hiroshima, Japan, January 2007. 2218 [RFC3851] Ramsdell, B., "Secure/Multipurpose Internet Mail 2219 Extensions (S/MIME) Version 3.1 Message Specification", 2220 RFC 3851, July 2004. 2222 [RFC4288] Freed, N. and J. Klensin, "Media Type Specifications and 2223 Registration Procedures", BCP 13, RFC 4288, December 2005. 2225 [RFC5485] Housley, R., "Digital Signatures on Internet-Draft 2226 Documents", RFC 5485, March 2009. 2228 Appendix A. Example IPFIX File 2230 In this section we will explore an example IPFIX File which 2231 demonstrates the various features of the IPFIX File format. This 2232 File contains flow records described by a single Template. This File 2233 also contains a File Time Window record to note the start and end 2234 time of the data, and an Export Session Details record to record 2235 collection infrastructure information. Each Message within this File 2236 also contains a Message Checksum record, as this File may be 2237 externally encrypted and/or stored as an archive. The structure of 2238 this File is shown in Figure 2. 2240 +=================================================+ 2241 | IPFIX Message seq. 0 | 2242 | +---------------------------------------------+ | 2243 | | Template Set (id 2) 1 rec | | 2244 | | Data Tmpl. id 256 | | 2245 | +---------------------------------------------+ | 2246 | | Options Template Set (id 3) 3 recs | | 2247 | | File Time Window Opt. Tmpl. id 257 | | 2248 | | Message Checksum Opt. Tmpl. id 259 | | 2249 | | Export Session Details Opt. Tmpl. id 258 | | 2250 | +---------------------------------------------+ | 2251 | | Data Set (id 259) [Message Checksum] 1 rec | | 2252 | +---------------------------------------------+ | 2253 +=================================================+ 2254 | IPFIX Message seq. 1 | 2255 | +---------------------------------------------+ | 2256 | | Data Set (id 257) [File Time Window] 1 rec | | 2257 | +---------------------------------------------+ | 2258 | | Data Set (id 258) [Export Session] 1 rec | | 2259 | +---------------------------------------------+ | 2260 | | Data Set (id 259) [Message Checksum] 1 rec | | 2261 | +---------------------------------------------+ | 2262 +=================================================+ 2263 | IPFIX Message seq. 4 | 2264 | +---------------------------------------------+ | 2265 | | Data Set (id 256) 50 recs | | 2266 | | contains flow data | | 2267 | +---------------------------------------------+ | 2268 | | Data Set (id 259) [Message Checksum] 1 rec | | 2269 | +---------------------------------------------+ | 2270 +=================================================+ 2271 | IPFIX Message seq. 55 | 2272 | . . . | 2274 Figure 2: File Example Structure 2276 The Template describing the data records contains a flow start 2277 timestamp, an IPv4 5-tuple, and packet and octet total counts. The 2278 Template Set defining this is as shown in Figure 3 below: 2280 1 2 3 2281 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2282 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2283 | Set ID = 2 | Length = 40 | 2284 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2285 | Template ID = 256 | Field Count = 8 | 2286 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2287 |0| flowStartSeconds = 150 | Field Length = 4 | 2288 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2289 |0| sourceIPv4Address = 8 | Field Length = 4 | 2290 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2291 |0| dest.IPv4Address = 12 | Field Length = 4 | 2292 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2293 |0| sourceTransportPort = 7 | Field Length = 2 | 2294 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2295 |0| dest.TransportPort = 11 | Field Length = 2 | 2296 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2297 |0| protocolIdentifier = 4 | Field Length = 1 | 2298 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2299 |0| octetTotalCount = 85 | Field Length = 4 | 2300 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2301 |0| packetTotalCount = 86 | Field Length = 4 | 2302 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2304 Figure 3: File Example Data Template 2306 A.1. Example Options Templates 2308 This is followed by an Options Template Set containing the Options 2309 Templates required to read the File: the File Time Window Options 2310 Template defined in Section 8.1.2 above, the Export Session Details 2311 Options Template defined in Section 8.1.3 above, and the Message 2312 Checksum Options Template defined in Section 8.1.1 above. This 2313 Options Template Set is shown in Figure 4 and Figure 5 below: 2315 1 2 3 2316 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2317 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2318 | Set ID = 3 | Length = 80 | 2319 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2320 | Template ID = 257 | Field Count = 3 | 2321 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2322 | Scope Field Count = 1 |0| sessionScope = TBD10 | 2323 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2324 | Field Length = 1 |0| minFlowStartSeconds = TBD8 | 2325 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2326 | Field Length = 4 |0| maxFlowEndSeconds = TBD4 | 2327 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2328 | Field Length = 4 | Template ID = 259 | 2329 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2330 | Field Count = 2 | Scope Field Count = 1 | 2331 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2332 |0| messageScope = TBD6 | Field Length = 1 | 2333 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2334 |0| messageMD5Checksum = TBD5 | Field Length = 16 | 2335 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2337 Figure 4: File Example Options Templates (Time Window and Checksum) 2338 1 2 3 2339 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2340 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2341 | Template ID = 258 | Field Count = 9 | 2342 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2343 | Scope Field Count = 1 |0| sessionScope = TBD10 | 2344 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2345 | Field Length = 1 |0| exporterIPv4Address = 130 | 2346 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2347 | Field Length = 4 |0| collectorIPv4Address = 211 | 2348 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2349 | Field Length = 4 |0| exporterTransportPort = 217 | 2350 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2351 | Field Length = 2 |0| col.TransportPort = 216 | 2352 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2353 | Field Length = 2 |0| col.TransportProtocol = 215 | 2354 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2355 | Field Length = 1 |0| col.ProtocolVersion = 214 | 2356 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2357 | Field Length = 1 |0| minExportSeconds = TBD7 | 2358 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2359 | Field Length = 4 |0| maxExportSeconds = TBD3 | 2360 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2361 | Field Length = 4 | set padding (2 octets) | 2362 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2364 Figure 5: File Example Options Templates, Continued (Session Details) 2366 A.2. Example Supplemental Options Data 2368 Following the Templates required to decode the File is the 2369 supplemental IPFIX Options information used to describe the File's 2370 contents and type information. First comes the File Time Window 2371 record; it notes that the File contains data from 9 October 2007 2372 between 00:01:13 and 23:56:27 UTC, and appears as in Figure 6: 2374 1 2 3 2375 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2376 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2377 | Set ID = 257 | Length = 13 | 2378 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2379 | sessionScope | minFlowStartSeconds 2380 | 0 | 2007-10-09 00:01:13 UTC . . . 2381 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2382 | maxFlowEndSeconds 2383 . . . | 2007-10-09 23:56:27 UTC . . . 2384 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2385 | 2386 . . . | 2387 +-+-+-+-+-+-+-+-+ 2389 Figure 6: File Example Time Window 2391 This is followed by information about how the data in the File was 2392 collected, in the Export Session Details record. This record notes 2393 that the session stored in this File was sent via SCTP from an 2394 exporter at 192.0.2.30 port 32769 to an collector at 192.0.2.40 port 2395 4739, and contains messages exported between 00:01:57 and 23:57:12 2396 UTC on 9 October 2007; it is represented in its Data Set as in 2397 Figure 7: 2399 1 2 3 2400 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2401 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2402 | Set ID = 258 | Length = 27 | 2403 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2404 | sessionScope | exporterIPv4Address 2405 | 0 | 192.0.2.30 . . . 2406 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2407 | collectorIPv4Address 2408 . . . | 192.0.2.31 . . . 2409 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2410 | exporterTransportPort | cTPort 2411 . . . | 32769 | 4739 . . . 2412 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2413 | cTProtocol | cPVersion | 2414 . . . | 132 | 10 | . . . 2415 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2416 minExportSeconds | 2417 . . . 2007-10-09 00:01:57 UTC | . . . 2418 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2419 maxExportSeconds | 2420 . . . 2007-10-09 23:57:12 UTC | 2421 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2423 Figure 7: File Example Export Session Details 2425 A.3. Example Message Checksum 2427 Each IPFIX Message within the File is completed with a Message 2428 Checksum record; the structure of this record within its Data Set is 2429 as in Figure 8: 2431 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2432 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2433 | Set ID = 259 | Length = 24 | 2434 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2435 | messageScope | | 2436 | 0 | | 2437 +-+-+-+-+-+-+-+-+ | 2438 | messageMD5Checksum | 2439 | (16 byte MD5 checksum of options message) | 2440 | | 2441 | | 2442 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2443 | | set padding (3 octets) | 2444 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2446 Figure 8: File Example Message Checksum 2448 A.4. File Example Data Set 2450 After the Templates and supplemental Options information comes the 2451 data itself. The first record of an example Data Set is shown with 2452 its message and set headers in Figure 9: 2454 1 2 3 2455 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2456 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2457 | Version = 10 | Length = 1296 | 2458 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2459 | Export Time = 2007-10-09 00:01:57 UTC | 2460 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2461 | Sequence Number = 4 | 2462 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2463 | Observation Domain ID = 1 | 2464 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2465 | Set ID = 256 | Length = 1254 | 2466 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2467 | flowStartSeconds | 2468 | 2007-10-09 00:01:13 UTC | 2469 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2470 | sourceIPv4Address | 2471 | 192.0.2.2 | 2472 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2473 | destinationIPv4Address | 2474 | 192.0.2.3 | 2475 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2476 | sourceTransportPort | destinationTransportPort | 2477 | 32770 | 80 | 2478 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2479 | protocolId | totalOctetCount 2480 | 6 | 18000 . . . 2481 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2482 | totalPacketCount 2483 . . . | 65 . . . 2484 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2485 | (49 more records) 2486 . . . | 2487 +-+-+-+-+-+-+-+-+ 2489 Figure 9: File Example Data Set 2491 A.5. Complete File Example 2493 Bringing together the examples above and adding message headers as 2494 appropriate, a hex dump of the first 317 bytes of the example Gile 2495 constructed above would appear as in the annotated Figure 10 below. 2497 [EDITOR'S NOTE: In this figure, xx refers to unassigned IANA IE 2498 numbers as in the IANA Considerations section above; cs refers to 2499 message checksum bytes that depend on the rest of the message 2500 contents. These will have to be replaced if we keep this example 2501 once the IE numbers are assigned.] 2503 0:|00 0A 00 A0 47 0A B6 E5 00 00 00 00 00 00 00 01 2504 [^ first message header (length 160 bytes) --> 2505 16:|00 02 00 28 01 00 00 08 00 96 00 04 00 08 00 04 2506 [^ data template set --> 2507 32: 00 0C 00 04 00 07 00 02 00 0B 00 02 00 04 00 01 2509 48: 00 55 00 04 00 56 00 04|00 03 00 50 01 01 00 03 2510 [^ opt template set --> 2511 64: 00 01 xx xx 00 01 xx xx 00 04 xx xx 00 04 01 03 2513 80: 00 02 00 01 xx xx 00 01 xx xx 00 10 01 02 00 09 2515 96: 00 01 xx xx 00 01 00 82 00 04 00 D3 00 04 00 D9 2517 112: 00 02 00 D8 00 02 00 D7 00 01 00 D0 00 01 xx xx 2519 128: 00 04 xx xx 00 04 00 00|01 03 00 18 00 cs cs cs 2520 [^ checksum record --> 2521 144: cs cs cs cs cs cs cs cs cs cs cs cs cs 00 00 00 2523 176:|00 0A 00 50 47 0A B6 E5 00 00 00 01 00 00 00 01 2524 [^ second message header (length 80 bytes) --> 2525 192:|01 01 00 0E 00 47 0A B6 B9 47 0C 07 1B 00|01 02 2526 [^ time window rec -> [ session detail rec ^ --> 2527 208: 00 1C 00 C0 00 02 1E 0C 00 02 1F 80 01 12 83 84 2529 224: 0A 47 0A B6 E5 47 0C 07 48 00|01 03 00 18 00 cs 2530 [ message checksum rec ^ --> 2531 240: cs cs cs cs cs cs cs cs cs cs cs cs cs cs cs 00 2533 256:|00 0A 05 10 47 0A B6 E5 00 00 00 06 00 00 00 01 2534 [^ third message header (length 1296 bytes) --> 2535 272:|01 00 04 E6|47 0A B6 B9 C0 00 02 02 C0 00 02 03 2536 [^ set hdr ][^ first data rec --> 2537 288: 80 02 00 50 06 00 00 46 50 00 00 00 41 2539 Figure 10: File Example Hex Dump 2541 Appendix B. Applicability of IPFIX Files to NetFlow V9 flow storage 2543 As the IPFIX Message format is nearly a superset of the NetFlow V9 2544 packet format, IPFIX Files can be used for store NetFlow V9 data 2545 relatively easily. This section describes a method for doing so. 2546 The differences between the two protocols are outlined in 2547 Appendix B.1 below. A simple, lightweight, message-for-message 2548 translation method for transforming V9 Packets into IPFIX Messages 2549 for storage within IPFIX Files is described in Appendix B.2. An 2550 example of this translation method is given in Appendix B.3. 2552 B.1. Comparing NetFlow V9 to IPFIX 2554 With a few caveats, the IPFIX Protocol is a superset of the NetFlow 2555 V9 protocol, having evolved from it largely through a process of 2556 feature addition to bring it into compliance with the IPFIX 2557 Requirements and the needs of stakeholders within the IPFIX Working 2558 Group. This appendix outlines the differences between the two 2559 protocols. It is informative only, and presented as an exploration 2560 of the two protocols to motivate the usage of IPFIX Files to store 2561 V9-collected flow data. 2563 B.1.1. Message Header Format 2565 Both NetFlow V9 and IPFIX use streams of messages prefixed by a 2566 message header, though the message header differs significantly 2567 between the two. Note that in NetFlow V9 terminology, these messages 2568 are called packets, and messages must be delimited by datagram 2569 boundaries. IPFIX does not have this constraint. The header formats 2570 are detailed below: 2572 0 1 2 3 2573 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2574 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2575 | Version Number | Count | 2576 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2577 | sysUpTime | 2578 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2579 | UNIX Secs | 2580 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2581 | Sequence Number | 2582 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2583 | Source ID | 2584 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2586 Figure 11: NetFlow V9 Packet Header Format 2588 0 1 2 3 2589 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2590 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2591 | Version Number | Length | 2592 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2593 | Export Time | 2594 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2595 | Sequence Number | 2596 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2597 | Observation Domain ID | 2598 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2600 Figure 12: IPFIX Message Header Format 2602 Version Number: The IPFIX Version Number MUST be 10, while the 2603 NetFlow V9 Version Number MUST be 9. 2605 Length vs. Count: The Count field in the NetFlow V9 packet header 2606 counts records in the message (including Data and Template 2607 Records), while the Length field in the IPFIX Message Header 2608 counts octets in the message. Note that this implies that NetFlow 2609 V9 collectors must rely on datagram boundaries or some other 2610 external delimeter; or otherwise must completely consume a message 2611 before finding its end. 2613 System Uptime: System uptime in milliseconds is exported in the 2614 NetFlow V9 packet header. This field is not present in the IPFIX 2615 Message Header, and must be exported using an IPFIX Option if 2616 required. 2618 Export Time: Aside from being called UNIX Secs in the NetFlow V9 2619 packet header specification, the export time in seconds since 1 2620 January 1970 at 0000 UTC appears in both NetFlow V9 and IPFIX 2621 message headers. 2623 Sequence Number: The NetFlow V9 Sequence Number counts packets, 2624 while the IPFIX Sequence Number counts records in Data Sets. Both 2625 are scoped to Observation Domain. 2627 Observation Domain ID: Similarly, the NetFlow V9 sourceID has 2628 become the IPFIX Observation Domain ID. 2630 B.1.2. Set Header Format 2632 Set headers are identical between NetFlow V9 and IPFIX; that is, each 2633 Set (FlowSet in NetFlow V9 terminology) is prefixed by a 4-byte set 2634 header containing the Set ID and the length of the set in octets. 2636 Note that the special Set IDs are different between IPFIX and NetFlow 2637 V9. IPFIX Template Sets are identified by Set ID 2, while NetFlow V9 2638 Template FlowSets are identified by Set ID 0. Similarly, IPFIX 2639 Options Template Sets are identified by Set ID 3, while NetFlow V9 2640 Options Template FlowSets are identified by Set ID 1. 2642 Both protocols reserve Set IDs 0-255, and use Set IDs 256-65535 for 2643 Data Sets (or FlowSets, in NetFlow V9 terminology). 2645 B.1.3. Template Format 2647 Template FlowSets in NetFlow V9 support a subset of functionality of 2648 those in IPFIX. Specifically, NetFlow V9 does not have any support 2649 for vendor-specific Information Elements as IPFIX does, so there is 2650 no enterprise bit or facility for associating a private enterprise 2651 number with an information element. NetFlow V9 also does not support 2652 variable-length fields. 2654 Options Template FlowSets in NetFlow V9 are similar to Options 2655 Template Sets in IPFIX in the same way. 2657 B.1.4. Information Model 2659 The NetFlow V9 field type definitions are a compatible subset of, and 2660 have evolved in concert with, the IPFIX Information Model. IPFIX 2661 Information Element identifiers in the range 1-127 are defined by the 2662 IPFIX Information Model [RFC5102] to be compatible with the 2663 corresponding NetFlow V9 field types. 2665 B.1.5. Template Management 2667 NetFlow V9 has no concept of a Transport Session as in IPFIX, as 2668 NetFlow V9 was designed with a connectionless transport in mind. 2669 Template IDs are therefore scoped to an Exporting Process lifetime 2670 (i.e., an Exporting Process instance between restarts). There is no 2671 facility in NetFlow V9 as in IPFIX for Template withdrawal or 2672 Template ID reuse. Template retransmission at the Exporter works as 2673 in UDP-based IPFIX Exporting Processes. 2675 B.1.6. Transport 2677 In practice, though NetFlow V9 is designed to be transport- 2678 independent, it is transported only over UDP. There is no facility 2679 as in IPFIX for full connection-oriented transport without datagram 2680 boundaries, due to the use of a record count field as opposed to a 2681 message length field in the packet header. There is no support in 2682 NetFlow V9 for transport layer security via TLS or DTLS. 2684 B.2. A Method for Transforming NetFlow V9 messages to IPFIX 2686 This appendix describes a method for transforming NetFlow V9 Packets 2687 into IPFIX Messages, which can be used to store NetFlow V9 data in 2688 IPFIX Files. A process transforming NetFlow V9 Packets into IPFIX 2689 Messages must handle the fact that NetFlow V9 Packets and IPFIX 2690 Messages are framed differently, that sequence numbering works 2691 differently, and that the NetFlow V9 field type definitions are only 2692 compatible with the IPFIX Information Model below Information Element 2693 identifier 128. 2695 For each incoming NetFlow V9 packet, the transformation process must: 2697 1. Verify that the Version field of the packet header is 9. 2699 2. Verify that the Sequence Number field of the packet header is 2700 valid. 2702 3. Scan the packet to: 2704 1. verify that it contains no Templates with field types outside 2705 the range 1-127; 2707 2. verify that it contains no FlowSets with Set IDs between 2 2708 and 255 inclusive; 2710 3. verify that it contains the number of records in FlowSets, 2711 Template FlowSets, and Options Template FlowSets declared in 2712 the Count field of the packet header; and 2714 4. count the number of records in Data FlowSets for calculating 2715 the IPFIX Sequence Number. 2717 4. Calculate a Sequence Number for each IPFIX Observation Domain by 2718 storing the last Sequence Number sent for each Observation Domain 2719 plus the count of records in Data FlowSets in the previous step 2720 to be sent as the Sequence Number for the IPFIX Message following 2721 this one within that Observation Domain. 2723 5. Generate a new IPFIX Message Header with: 2725 1. a Version field of 10; 2727 2. a Length field with the number of octets in the IPFIX 2728 Message, generally available by subtracting 4 from the length 2729 of the NetFlow V9 packet as returned from the transport layer 2730 (accounting for the difference in message header lengths); 2732 3. the Sequence Number calculated for this message by the 2733 Sequence Number calculation step; and 2735 4. Export Time and Observation Domain ID taken from the UNIX 2736 secs and Source ID fields of the NetFlow V9 packet header, 2737 respectively. 2739 6. Copy each FlowSet from the Netflow V9 packet to the IPFIX Message 2740 after the header. Replace Set ID 0 with Set ID 2 for Template 2741 Sets, and Set ID 1 with Set ID 3 for Options Template Sets. 2743 Note that this process loses system uptime information; if such 2744 information is required, the transformation process will have to 2745 export that information using IPFIX Options. This may require a more 2746 sophisticated transformation process structure. 2748 B.3. NetFlow V9 Transformation Example 2750 The following two figures show a single NetFlow V9 packet with 2751 templates and the corresponding IPFIX Message, exporting a single 2752 flow record representing 60,303 octets sent from 192.0.2.2 to 2753 192.0.2.3. This would be the 3rd packet exported in Observation 2754 Domain 33 from the NetFlow V9 exporter, containing records starting 2755 with the 12th record (packet and record sequence numbers count from 2756 0). 2758 The ** symbol in the IPFIX example shows those fields that required 2759 modification from the NetFlow V9 packet by the transformation 2760 process. 2762 1 2 3 2763 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2764 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2765 | Version = 9 | Count = 2 | 2766 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2767 | Uptime = 3750405 ms (1:02:30.405) | 2768 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2769 | Export Time = 1171557627 epoch sec (2007-02-15 16:40:27) | 2770 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2771 | Sequence Number = 2 | 2772 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2773 | Observation Domain ID = 33 | 2774 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2775 | Set ID = 0 | Set Length = 20 | 2776 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2777 | Template ID = 256 | Field Count = 3 | 2778 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2779 | IPV4_SRC_ADDR = 8 | Field Length = 4 | 2780 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2781 | IPV4_DST_ADDR = 12 | Field Length = 4 | 2782 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2783 | IN_BYTES = 1 | Field Length = 4 | 2784 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2785 | Set ID = 256 | Set Length = 16 | 2786 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2787 | IPV4_SRC_ADDR | 2788 | 192.0.2.2 | 2789 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2790 | IPV4_DST_ADDR | 2791 | 192.0.2.3 | 2792 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2793 | IN_BYTES | 2794 | 60303 | 2795 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2797 Figure 13: Example NetFlow V9 Packet 2798 1 2 3 2799 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2800 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2801 | ** Version = 10 | ** Length = 52 | 2802 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2803 | Export Time = 1171557627 epoch sec (2007-02-15 16:40:27) | 2804 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2805 | ** Sequence Number = 11 | 2806 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2807 | Observation Domain ID = 33 | 2808 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2809 | ** Set ID = 2 | Set Length = 20 | 2810 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2811 | Template ID = 256 | Field Count = 3 | 2812 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2813 |0| sourceIPv4Address = 8 | Field Length = 4 | 2814 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2815 |0| destinationIPv4Address = 12 | Field Length = 4 | 2816 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2817 |0| octetDeltaCount = 1 | Field Length = 4 | 2818 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2819 | Set ID = 256 | Set Length = 16 | 2820 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2821 | sourceIPv4Address | 2822 | 192.0.2.2 | 2823 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2824 | destinationIPv4Address | 2825 | 192.0.2.3 | 2826 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2827 | octetDeltaCount | 2828 | 60303 | 2829 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2831 Figure 14: Corresponding Example IPFIX Message 2833 Authors' Addresses 2835 Brian Trammell 2836 Hitachi Europe 2837 c/o ETH Zurich 2838 Gloriastrasse 35 2839 8092 Zurich 2840 Switzerland 2842 Phone: +41 44 632 70 13 2843 Email: brian.trammell@hitachi-eu.com 2844 Elisa Boschi 2845 Hitachi Europe 2846 c/o ETH Zurich 2847 Gloriastrasse 35 2848 8092 Zurich 2849 Switzerland 2851 Phone: +41 44 632 70 57 2852 Email: elisa.boschi@hitachi-eu.com 2854 Lutz Mark 2855 Fraunhofer IFAM 2856 Wiener Str. 12 2857 28359 Bremen 2858 Germany 2860 Phone: +49 421 2246206 2861 Email: lutz.mark@ifam.fraunhofer.de 2863 Tanja Zseby 2864 Fraunhofer Institute for Open Communication Systems 2865 Kaiserin-Augusta-Allee 31 2866 10589 Berlin 2867 Germany 2869 Phone: +49 30 3463 7153 2870 Email: tanja.zseby@fokus.fraunhofer.de 2872 Arno Wagner 2873 ETH Zurich 2874 Gloriastrasse 35 2875 8092 Zurich 2876 Switzerland 2878 Phone: +41 44 632 70 04 2879 Email: arno@wagner.name