idnits 2.17.1 draft-trammell-ipfix-a8n-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 1 character in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 5, 2010) is 5043 days in the past. Is this intentional? Checking references for intended status: Experimental ---------------------------------------------------------------------------- == Missing Reference: 'TODO' is mentioned on line 442, but not defined == Unused Reference: 'RFC5102' is defined on line 455, but no explicit reference was found in the text == Unused Reference: 'RFC5610' is defined on line 459, but no explicit reference was found in the text == Unused Reference: 'RFC3330' is defined on line 467, but no explicit reference was found in the text == Unused Reference: 'RFC5103' is defined on line 472, but no explicit reference was found in the text == Unused Reference: 'RFC5470' is defined on line 480, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-ipfix-mediators-problem-statement' is defined on line 490, but no explicit reference was found in the text == Unused Reference: 'RFC5153' is defined on line 497, but no explicit reference was found in the text ** Obsolete normative reference: RFC 5101 (Obsoleted by RFC 7011) ** Obsolete normative reference: RFC 5102 (Obsoleted by RFC 7012) ** Obsolete normative reference: RFC 3330 (Obsoleted by RFC 5735) == Outdated reference: A later version (-09) exists of draft-ietf-ipfix-mediators-framework-06 Summary: 4 errors (**), 0 flaws (~~), 10 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 IPFIX Working Group B. Trammell 3 Internet-Draft E. Boschi 4 Intended status: Experimental ETH Zurich 5 Expires: January 6, 2011 A. Wagner 6 Consecom AG 7 July 5, 2010 9 Exporting Aggregated Flow Data using the IP Flow Information Export 10 (IPFIX) Protocol 11 draft-trammell-ipfix-a8n-00.txt 13 Abstract 15 This document describes the export of aggregated Flow information 16 using IPFIX. An aggregated Flow is essentially an IPFIX Flow with an 17 externally imposed time interval, generally representing packets from 18 one or more original Flows. The document describes aggregated Flow 19 export within the framework of IPFIX Mediators, defines an 20 interoperable, implementation-independent method for aggregated Flow 21 export, covering time interval export, counter distribution and 22 export, and counting of original Flows. 24 Status of this Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at http://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on January 6, 2011. 41 Copyright Notice 43 Copyright (c) 2010 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (http://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 59 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 60 3. Requirements for Aggregation Support in IPFIX . . . . . . . . 4 61 4. Aggregation of IP Flows . . . . . . . . . . . . . . . . . . . 5 62 4.1. A general model for IP Flow Aggregation . . . . . . . . . 5 63 4.2. Aggregating and Distributing Counters . . . . . . . . . . 5 64 4.3. Counting Original Flows . . . . . . . . . . . . . . . . . 6 65 5. Aggregation in the IPFIX Architecture . . . . . . . . . . . . 7 66 6. Export of Aggregated IP Flows using IPFIX . . . . . . . . . . 8 67 6.1. Flow Count Export . . . . . . . . . . . . . . . . . . . . 8 68 6.1.1. originalFlowsPresent Information Element . . . . . . . 9 69 6.1.2. originalFlowsInitiated InformationElement . . . . . . 9 70 6.1.3. originalFlowsCompleted InformationElement . . . . . . 9 71 6.1.4. originalFlows InformationElement . . . . . . . . . . . 9 72 6.2. Aggregate Counter Distibution Export . . . . . . . . . . . 10 73 6.3. Time Interval Export . . . . . . . . . . . . . . . . . . . 10 74 7. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 75 8. Security Considerations . . . . . . . . . . . . . . . . . . . 10 76 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 77 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 11 78 10.1. Normative References . . . . . . . . . . . . . . . . . . . 11 79 10.2. Informative References . . . . . . . . . . . . . . . . . . 11 80 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 12 82 1. Introduction 84 The aggregation of packet data into flows serves a variety of 85 different purposes, as noted in [RFC3917] and [RFC5472]. Aggregation 86 beyond the flow level, into records representing multiple flows, is a 87 common analysis and data reduction technique as well. Often these 88 aggregation operations result in a time series of keys and counters, 89 which is useful for gaining insight into trends and anomalies. 90 Aggregation may also has added benefits for privacy: when data about 91 specific transactions and hosts are aggregated together, it has an 92 anonymising effect on the data. 94 Aggregation can be applied in parallel with storage of original 95 packet and flow data. In certain situations, aggregation can be 96 performed at mediator, with aggregated data can be used for initial 97 analysis (e.g., to provide a reduced data stream for analysis at a 98 central point, with original flow data stored in a higher-access- 99 latency manner (e.g. tape storage at the mediator at a remote site). 100 Other applications may benefit from a reverse application: online 101 analysis of original flows, with long-term storage of aggregated 102 data. Aggregation can also be used to provide a lower-volume, lower- 103 privacy-risk source of data for reporting or data exchange across 104 organizational boundaries. 106 While IPFIX can be used to export [RFC5101] and store [RFC5655] 107 aggregated data, there exists as yet no common terminology or 108 aggregation metadata representation for this purpose, no set of 109 requirements to define what is meant by aggregation, and no facility 110 for counting original Flows from which Aggregated Flows are derived. 111 This document seeks to remedy this situation, defining a common basis 112 for the application of IPFIX to the handling of aggregated data, with 113 applicability to large-scale network data processing, archiving, and 114 inter-organization exchange. 116 2. Terminology 118 Terms used in this document that are defined in the Terminology 119 section of the IPFIX Protocol [RFC5101] document are to be 120 interpreted as defined there. 122 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 123 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 124 document are to be interpreted as described in RFC 2119 [RFC2119]. 126 In addition, this document defines the following terms 127 Aggregated Flow: A Flow, as defined by [RFC5101], derived from a 128 set of zero or more original Flows within a defined time interval. 129 The two primary differences between a Flow and an Aggregated Flow 130 are (1) that the time interval of a Flow is generally derived from 131 information about the timing of the packets comprising the Flow, 132 while the time interval of an Aggregated Flow are generally 133 externally imposed; and (2) that an Aggregated Flow may represent 134 zero packets (i.e., an assertion that no packets were seen for a 135 given Flow Key in a given time interval). 137 (Intermediate) Aggregation Function: A mapping from a set of zero 138 or more original Flows into a set of aggregated Flow, that 139 separates the original Flows into a set of one or more given time 140 intervals. 142 (Intermediate) Aggregation Process: An Intermediate Process, as in 143 [I-D.ietf-ipfix-mediators-framework], hosting an Intermediate 144 Aggregation Function. 146 [TODO: may want to define a more precise terminology for aggregated 147 flow time interval, "contribution"/"presence" of an original Flow 148 within an aggregated Flow...] 150 3. Requirements for Aggregation Support in IPFIX 152 In defining a terminology, additional information elements, and 153 metadata export methods for Aggregated Flow export using IPFIX, we 154 have sought to meet the following requirements. 156 First, a specification of Aggregated Flow export must seek to be as 157 interoperable as possible. Export of Aggregated Flows using the 158 techniques described in this document will result in Flow data which 159 can be collected by Collecting Processes and read by File Readers 160 which do not provide any special support for Aggregated Flow export. 162 Second, a specification of Aggregated Flow export must seek to be as 163 implementation-independent as the IPFIX protocol itself. In 164 Section 5, we specify the flow aggregation process as an intermediate 165 process within the IPFIX Mediator framework 166 [I-D.ietf-ipfix-mediators-framework], and specify a variety of 167 different architectural arrangements for flow aggregation; these are 168 meant to be descriptive as opposed to proscriptive. In metadata 169 export, we seek to define properties of the set of exported 170 Aggregated Flows, as opposed to the properties of the specific 171 algorithms used to aggregate these Flows. Specifically out of scope 172 for this effort are any definition of a language for proscribing 173 aggregation records, or the configuration parameters of Aggregation 174 Processes. 176 From the definition of presented in Section 2, an Aggregated Flow is 177 a Flow as in [RFC5101], with a restricted defintion as to the packets 178 making up the Flow. Practically speaking, Aggregated Flows are 179 derived from original Flows, as opposed to a raw packet stream. Key 180 to this definition of Aggregated Flow is how timing affects the 181 process of aggregation, as for the most part flow aggregation takes 182 place within some set of (usually regular) time intervals. Any 183 specification for Aggregated Flow export must account for the special 184 role time intervals play in aggregation, and the many-to-many 185 relationship between Aggregated Flows and original Flows which this 186 implies. 188 4. Aggregation of IP Flows 190 [TODO: frontmatter: restate definition: a flow is just a flow. Here 191 we talk about Aggregation Process actions.] 193 4.1. A general model for IP Flow Aggregation 195 [TODO: introduce as little of an Aggregation Process as necessary to 196 be able to talk about things practically. Define in simple terms the 197 role of externally imposed time intervals on aggregation. ] 199 4.2. Aggregating and Distributing Counters 201 In general, counters in Aggregated Flows are treated the same as in 202 any Flow: on a per-Information Element basis, the counters are 203 calculated as if they were derived from the set of packets in the 204 original flow. For the most part, when aggregating original Flows 205 into Aggregated Flows, this is simply done by summation. 207 However, this raises a complication when aggregating original Flows 208 for which original packet data is not available: often, when imposing 209 an external time interval on an original Flow, the original Flow will 210 incompletely cover one or more time intervals, and apply to one or 211 more Aggregated Flows; in this case, the Aggregation Process must 212 distribute the counters in the original Flows across the multiple 213 Aggregated Flows. There are several methods for doing this, listed 214 here in increasing order of complexity and accuracy: 216 End Interval: The counters for an original Flow are added to the 217 counters of the appropriate Aggregated Flow containing the end 218 time of the original Flow. 220 Start Interval: The counters for an original Flow are added to the 221 counters of the appropriate Aggregated Flow containing the start 222 time of the original Flow. 224 Mid Interval: The counters for an original Flow are added to the 225 counters of a single appropriate Aggregated Flow containing some 226 timestamp between start and end time of the original Flow. 228 Simple Uniform Distribution: Each counter for an original Flow is 229 divided by the number of time intervals the original Flow covers 230 (i.e., of appropriate Aggregated Flows sharing the same Flow Key), 231 and this number is added to each corresponding counter in each 232 Aggregated Flow. 234 Proportional Uniform Distribution: Each counter for an original 235 Flow is divided by the number of time _units_ the original Flow 236 covers, to derive a mean count rate. This mean count rate is then 237 multiplied by the number of time units in the intersection of the 238 duration of the original Flow and the time interval of each 239 Aggregated Flow. This is like simple uniform distribtion, but 240 accounts for the fractional portions of a time interval covered by 241 an original Flow in the first and last time interval. 243 Simulated Process: Each counter of the original Flow is distributed 244 among the intervals of the Aggregated Flows according to some 245 function the Aggregation Process uses based upon properties of 246 Flows presumed to be like the original Flow. For example, bulk 247 transfer flows might follow a more or less proportional uniform 248 distribtion, while interactive processes are far more bursty. 250 Direct: The Aggregating Process has access to the original packet 251 timings from the packets making up the original Flow, and uses 252 these to distribute or recalculate the counters. 254 A method for exporting the distribution of counters across multiple 255 Aggregated Flows is detailed in Section 6.2. In any case, counters 256 MUST be distributed across the multiple Aggregated Flows in such a 257 way that the total count is preserved; this property allows data to 258 be aggregated and re-aggregated without any loss of original count 259 information. To avoid confusion in interpretation of the aggregated 260 data, all the counters for a set of given original Flows SHOULD be 261 distributed via the same method. 263 4.3. Counting Original Flows 265 When aggregating multiple original Flows into an Aggregated Flow, it 266 is often useful to know how many original Flows are present in the 267 Aggregated Flow. This document introduces four new information 268 elements in Section 6.1 to export these counters. 270 There are two possible ways to count original Flows, which we call 271 here conservative and non-conservative. Conservative flow counting 272 has the property that each original Flow contributes exactly one to 273 the total flow count within a set of aggregated Flows. In other 274 words, conservative flow counters are distributed just as any other 275 counter, except each original Flow is assumed to have a flow count of 276 one. When a count for an original Flow must be distributed across a 277 set of Aggregated Flows, and a distribution method is used which does 278 not account for that original Flow completely within a single 279 Aggregated Flow, conservative flow counting requires a fractional 280 representation. 282 By contrast, non-conservative flow counting is used to count how many 283 flows are represented in an Aggregated Flow. Flow counters are not 284 distributed in this case. An original Flow which is present within N 285 Aggregated Flows would add N to the total non-conservative flow 286 count, one to each Aggregated Flow. 288 5. Aggregation in the IPFIX Architecture 290 [TODO: describe these diagrams] 292 +==========================================+ 293 | Exporting Process | 294 +==========================================+ 295 | | 296 | (Aggregated Flow Export) | 297 V | 298 +=============================+ | 299 | Mediator | | 300 +=============================+ | 301 | | 302 | (Aggregating Mediator) | 303 V V 304 +==========================================+ 305 | Collecting Process | 306 +==========================================+ 307 | 308 | (Aggregation for Storage) 309 V 310 +--------------------+ 311 | IPFIX File Storage | 312 +--------------------+ 314 Figure 1: Potential Aggregation Locations 316 packets --+ +- IPFIX Messages -+ 317 | | | 318 V V V 319 +==================+ +====================+ +=============+ 320 | Metering Process | | Collecting Process | | File Reader | 321 +==================+ +====================+ +=============+ 322 | original | Flows | 323 V V V 324 +=========================================================+ 325 | Intermediate Aggregation Process (IAP) | 326 +=========================================================+ 327 | Aggregated Aggregated | 328 | Flows Flows | 329 V V 330 +===================+ +=============+ 331 | Exporting Process | | File Writer | 332 +===================+ +=============+ 333 | | 334 +------------> IPFIX Messages <----------+ 336 Figure 2: Data flows through the aggregation process 338 +----+ +-----+ +----+ 339 pkts -> | MP |->| IAP |->| EP |-> Export of Aggregated Flows 340 +----+ +-----+ +----+ 341 +----+ +-----+ +----+ 342 IPFIX -> | CP |->| IAP |->| EP |-> Aggregating Mediator 343 +----+ +-----+ +----+ 344 +----+ +-----+ +----+ 345 IPFIX -> | CP |->| IAP |->| FW |-> Aggregation for storage in IPFIX Files 346 +----+ +-----+ +----+ 347 +----+ +-----+ +----+ 348 IPFIX -> | FR |->| IAP |->| FW |-> Aggregation for analysis 349 File +----+ +-----+ +----+ (aggregating File manipulator) 351 Figure 3: Possible aggregation arrangements in the IPFIX architecture 353 6. Export of Aggregated IP Flows using IPFIX 355 [TODO: frontmatter: here we talk about export specifics.] 357 6.1. Flow Count Export 359 [TODO: describe these four IEs. for now, see Section 4.3] 361 6.1.1. originalFlowsPresent Information Element 363 Description: The non-conservative count of original Flows 364 containing packets from which a this Aggregated Flow was 365 aggregated. 367 Abstract Data Type: unsigned64 369 ElementId: TBD1 371 Status: Proposed 373 6.1.2. originalFlowsInitiated InformationElement 375 Description: The conservative count of original Flows whose first 376 packet is represented within this Aggregated Flow. 378 Abstract Data Type: unsigned64 380 ElementId: TBD2 382 Status: Proposed 384 6.1.3. originalFlowsCompleted InformationElement 386 Description: The conservative count of original Flows whose last 387 packet is represented within this Aggregated Flow. 389 Abstract Data Type: unsigned64 391 ElementId: TBD3 393 Status: Proposed 395 6.1.4. originalFlows InformationElement 397 Description: The conservative count of original Flows represented 398 within this Aggregated Flow; may be distributed via any of the 399 methods described in Section 4.2 401 Abstract Data Type: float64 403 ElementId: TBD4 405 Status: Proposed 407 6.2. Aggregate Counter Distibution Export 409 [TODO: options template and IE for representing the distribution 410 methods in Section 4.2] 412 6.3. Time Interval Export 414 Since an Aggregated Flow is simply a Flow, the existing timestamp 415 Information Elements in the IPFIX Information Model (e.g., 416 flowStartMilliseconds, flowEndNanoseconds) are sufficient to specify 417 the time interval for aggregation. Therefore, this document 418 specifies no new aggregation-specific Information Elements for 419 exporting time interval information. 421 Each Aggregated Flow SHOULD contain both an interval start and 422 interval end timestamp. If an exporter of Aggregated Flows omits the 423 interval end timestamp from each Aggregated Flow, the time interval 424 for Aggregated Flows within an Observation Domain and Transport 425 Session MUST be regular and constant, and SHOULD be exported using 426 the Options Record described in this section. However, note that 427 this approach might lead to interoperability problems when exporting 428 Aggregated Flows to non-aggregation-aware Collecting Processes and 429 downstream analysis tasks; therefore, an Exporting Process capable of 430 exporting only interval start timestamps MUST provide a configuration 431 option to export interval end timestamps as well. 433 [TODO: options template and IE for binding a time interval to a 434 session.] 436 7. Examples 438 [TODO] 440 8. Security Considerations 442 [TODO] 444 9. IANA Considerations 446 [TODO: add all IEs defined in Section 6.] 448 10. References 449 10.1. Normative References 451 [RFC5101] Claise, B., "Specification of the IP Flow Information 452 Export (IPFIX) Protocol for the Exchange of IP Traffic 453 Flow Information", RFC 5101, January 2008. 455 [RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J. 456 Meyer, "Information Model for IP Flow Information Export", 457 RFC 5102, January 2008. 459 [RFC5610] Boschi, E., Trammell, B., Mark, L., and T. Zseby, 460 "Exporting Type Information for IP Flow Information Export 461 (IPFIX) Information Elements", RFC 5610, July 2009. 463 [RFC5655] Trammell, B., Boschi, E., Mark, L., Zseby, T., and A. 464 Wagner, "Specification of the IP Flow Information Export 465 (IPFIX) File Format", RFC 5655, October 2009. 467 [RFC3330] IANA, "Special-Use IPv4 Addresses", RFC 3330, 468 September 2002. 470 10.2. Informative References 472 [RFC5103] Trammell, B. and E. Boschi, "Bidirectional Flow Export 473 Using IP Flow Information Export (IPFIX)", RFC 5103, 474 January 2008. 476 [RFC5472] Zseby, T., Boschi, E., Brownlee, N., and B. Claise, "IP 477 Flow Information Export (IPFIX) Applicability", RFC 5472, 478 March 2009. 480 [RFC5470] Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek, 481 "Architecture for IP Flow Information Export", RFC 5470, 482 March 2009. 484 [I-D.ietf-ipfix-mediators-framework] 485 Kobayashi, A., Claise, B., Muenz, G., and K. Ishibashi, 486 "IPFIX Mediation: Framework", 487 draft-ietf-ipfix-mediators-framework-06 (work in 488 progress), April 2010. 490 [I-D.ietf-ipfix-mediators-problem-statement] 491 Kobayashi, A., Claise, B., Nishida, H., Sommer, C., 492 Dressler, F., and E. Stephan, "IPFIX Mediation: Problem 493 Statement", 494 draft-ietf-ipfix-mediators-problem-statement-09 (work in 495 progress), March 2010. 497 [RFC5153] Boschi, E., Mark, L., Quittek, J., Stiemerling, M., and P. 498 Aitken, "IP Flow Information Export (IPFIX) Implementation 499 Guidelines", RFC 5153, April 2008. 501 [RFC3917] Quittek, J., Zseby, T., Claise, B., and S. Zander, 502 "Requirements for IP Flow Information Export (IPFIX)", 503 RFC 3917, October 2004. 505 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 506 Requirement Levels", BCP 14, RFC 2119, March 1997. 508 Authors' Addresses 510 Brian Trammell 511 Swiss Federal Institute of Technology Zurich 512 Gloriastrasse 35 513 8092 Zurich 514 Switzerland 516 Phone: +41 44 632 70 13 517 Email: trammell@tik.ee.ethz.ch 519 Elisa Boschi 520 Swiss Federal Institute of Technology Zurich 521 Gloriastrasse 35 522 8092 Zurich 523 Switzerland 525 Email: boschie@tik.ee.ethz.ch 527 Arno Wagner 528 Consecom AG 529 Bellariastrasse 11 530 8002 Zurich 531 Switzerland 533 Email: arno@wagner.name