idnits 2.17.1 draft-ietf-ipfix-implementation-guidelines-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 20. -- Found old boilerplate from RFC 3978, Section 5.5, updated by RFC 4748 on line 1626. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 1637. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1644. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1650. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (November 17, 2007) is 6005 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC2309' is defined on line 1520, but no explicit reference was found in the text == Outdated reference: A later version (-02) exists of draft-tuexen-dtls-for-sctp-01 == Outdated reference: A later version (-11) exists of draft-ietf-tsvwg-udp-guidelines-03 -- Obsolete informational reference (is this intentional?): RFC 1305 (Obsoleted by RFC 5905) -- Obsolete informational reference (is this intentional?): RFC 2309 (Obsoleted by RFC 7567) -- Obsolete informational reference (is this intentional?): RFC 3280 (Obsoleted by RFC 5280) -- Obsolete informational reference (is this intentional?): RFC 4346 (Obsoleted by RFC 5246) -- Obsolete informational reference (is this intentional?): RFC 4347 (Obsoleted by RFC 6347) -- Obsolete informational reference (is this intentional?): RFC 4960 (Obsoleted by RFC 9260) Summary: 1 error (**), 0 flaws (~~), 4 warnings (==), 13 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 IPFIX Working Group E. Boschi 2 Internet-Draft Hitachi Europe 3 Intended status: Informational L. Mark 4 Expires: May 20, 2008 Fraunhofer FOKUS 5 J. Quittek 6 M. Stiemerling 7 NEC Europe 8 P. Aitken 9 Cisco Systems, Inc. 10 November 17, 2007 12 IPFIX Implementation Guidelines 13 draft-ietf-ipfix-implementation-guidelines-08.txt 15 Status of this Memo 17 By submitting this Internet-Draft, each author represents that any 18 applicable patent or other IPR claims of which he or she is aware 19 have been or will be disclosed, and any of which he or she becomes 20 aware will be disclosed, in accordance with Section 6 of BCP 79. 22 Internet-Drafts are working documents of the Internet Engineering 23 Task Force (IETF), its areas, and its working groups. Note that 24 other groups may also distribute working documents as Internet- 25 Drafts. 27 Internet-Drafts are draft documents valid for a maximum of six months 28 and may be updated, replaced, or obsoleted by other documents at any 29 time. It is inappropriate to use Internet-Drafts as reference 30 material or to cite them other than as "work in progress." 32 The list of current Internet-Drafts can be accessed at 33 http://www.ietf.org/ietf/1id-abstracts.txt. 35 The list of Internet-Draft Shadow Directories can be accessed at 36 http://www.ietf.org/shadow.html. 38 This Internet-Draft will expire on May 20, 2008. 40 Copyright Notice 42 Copyright (C) The IETF Trust (2007). 44 Abstract 46 The IP Flow Information eXport (IPFIX) protocol defines how IP Flow 47 information can be exported from routers, measurement probes or other 48 devices. This document provides guidelines for the implementation 49 and use of the IPFIX protocol. Several sets of guidelines address 50 template management, transport-specific issues, implementation of 51 exporting and collecting processes and IPFIX implementation on 52 middleboxes (such as firewalls, network address translators, tunnel 53 endpoints, packet classifiers, etc.). 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 58 1.1. IPFIX Documents Overview . . . . . . . . . . . . . . . . . 4 59 1.2. Overview of the IPFIX Protocol . . . . . . . . . . . . . . 4 60 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 61 3. Template Management Guidelines . . . . . . . . . . . . . . . . 5 62 3.1. Template Management . . . . . . . . . . . . . . . . . . . 5 63 3.2. Template Records versus Option Template Records . . . . . 6 64 3.3. Using Scopes . . . . . . . . . . . . . . . . . . . . . . . 7 65 3.4. Multiple Information Elements of the same type . . . . . . 7 66 3.5. Selecting Message Size . . . . . . . . . . . . . . . . . . 7 67 4. Exporting Process Guidelines . . . . . . . . . . . . . . . . . 8 68 4.1. Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 69 4.2. Information Element Coding . . . . . . . . . . . . . . . . 8 70 4.3. Using Counters . . . . . . . . . . . . . . . . . . . . . . 8 71 4.4. Padding . . . . . . . . . . . . . . . . . . . . . . . . . 9 72 4.4.1. Alignment of Information Elements within a Data 73 Record . . . . . . . . . . . . . . . . . . . . . . . . 9 74 4.4.2. Alignment of Information Elements specifiers 75 within a Template Record . . . . . . . . . . . . . . . 10 76 4.4.3. Alignment of Records within a Set . . . . . . . . . . 10 77 4.4.4. Alignment of Sets within an IPFIX Message . . . . . . 10 78 4.5. Time Issues . . . . . . . . . . . . . . . . . . . . . . . 10 79 4.6. IPFIX Message Header Export Time and Data Record Time . . 11 80 4.7. Devices Without an Absolute Clock . . . . . . . . . . . . 12 81 5. Collecting Process Guidelines . . . . . . . . . . . . . . . . 12 82 5.1. Information Element (de)coding . . . . . . . . . . . . . . 12 83 5.2. Reduced-size Encoding of Information Elements . . . . . . 12 84 5.3. Template Management . . . . . . . . . . . . . . . . . . . 13 85 6. Transport-Specific Guidelines . . . . . . . . . . . . . . . . 13 86 6.1. SCTP . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 87 6.2. UDP . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 88 6.3. TCP . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 89 7. Guidelines for implementation on Middleboxes . . . . . . . . . 19 90 7.1. Traffic Flow Scenarios at Middleboxes . . . . . . . . . . 20 91 7.2. Location of the Observation Point . . . . . . . . . . . . 22 92 7.3. Reporting Flow-related Middlebox Internals . . . . . . . . 23 93 7.3.1. Packet Dropping Middleboxes . . . . . . . . . . . . . 24 94 7.3.2. Middleboxes Changing the DSCP . . . . . . . . . . . . 24 95 7.3.3. Middleboxes Changing IP Addresses and Port Numbers . . 25 96 8. Security Guidelines . . . . . . . . . . . . . . . . . . . . . 26 97 8.1. Introduction to TLS and DTLS for IPFIX implementers . . . 26 98 8.2. X.509-based Identity Verification for IPFIX over TLS 99 or DTLS . . . . . . . . . . . . . . . . . . . . . . . . . 26 100 8.3. Implementing IPFIX over TLS over TCP . . . . . . . . . . . 27 101 8.4. Implementing IPFIX over DTLS over UDP . . . . . . . . . . 27 102 8.5. Implementing IPFIX over DTLS over SCTP . . . . . . . . . . 28 103 9. Extending the Information Model . . . . . . . . . . . . . . . 28 104 9.1. Adding new IETF specified Information Elements . . . . . . 29 105 9.2. Adding enterprise-specific Information Elements . . . . . 29 106 10. Common Implementation Mistakes . . . . . . . . . . . . . . . . 29 107 10.1. IPFIX and Netflow version 9 . . . . . . . . . . . . . . . 29 108 10.2. Padding of the Data Set . . . . . . . . . . . . . . . . . 30 109 10.3. Field ID Numbers . . . . . . . . . . . . . . . . . . . . . 31 110 10.4. Template ID Numbers . . . . . . . . . . . . . . . . . . . 31 111 11. Security Considerations . . . . . . . . . . . . . . . . . . . 31 112 12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 32 113 13. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 32 114 14. References . . . . . . . . . . . . . . . . . . . . . . . . . . 32 115 14.1. Normative References . . . . . . . . . . . . . . . . . . . 32 116 14.2. Informative References . . . . . . . . . . . . . . . . . . 33 117 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 34 118 Intellectual Property and Copyright Statements . . . . . . . . . . 36 120 1. Introduction 122 The IPFIX protocol defines how IP Flow information can be exported 123 from routers, measurement probes or other devices. In this document, 124 we provide guidelines for its implementation. 126 The guidelines are split into five main sets. These sets address 127 implementation aspects for Template management, Exporting Process, 128 Collecting Process, transport, and implementation on middleboxes. 130 Finally, this document contains a list of common mistakes about 131 issues that had been misinterpreted in the first IPFIX 132 implementations and created (and still might create) interoperability 133 problems. 135 1.1. IPFIX Documents Overview 137 The IPFIX Protocol [I-D.ietf-ipfix-protocol] provides network 138 administrators with access to IP flow information. The architecture 139 for the export of measured IP flow information out of an IPFIX 140 exporting process to a collecting process is defined in the IPFIX 141 Architecture [I-D.ietf-ipfix-architecture], per the requirements 142 defined in RFC 3917 [RFC3917]. 144 The IPFIX Architecture [I-D.ietf-ipfix-architecture] specifies how 145 IPFIX data record and templates are carried via a congestion-aware 146 transport protocol from IPFIX exporting processes to IPFIX collecting 147 process. 149 IPFIX has a formal description of IPFIX information elements, their 150 name, type and additional semantic information, as specified in the 151 IPFIX Information Model [I-D.ietf-ipfix-info]. 153 Finally the IPFIX Applicability Statement [I-D.ietf-ipfix-as] 154 describes what type of applications can use the IPFIX protocol and 155 how they can use the information provided. It furthermore shows how 156 the IPFIX framework relates to other architectures and frameworks. 158 1.2. Overview of the IPFIX Protocol 160 In the IPFIX protocol, { type, length, value } tuples are expressed 161 in templates containing { type, length } pairs, specifying which { 162 value } fields are present in data records conforming to the 163 template, giving great flexibility as to what data is transmitted. 165 Since templates are sent very infrequently compared with data 166 records, this results in significant bandwidth savings. 168 Different data records may be transmitted simply by sending new 169 templates specifying the { type, length } pairs for the new data 170 format. See [I-D.ietf-ipfix-protocol] for more information. 172 The IPFIX Information Model [I-D.ietf-ipfix-info] defines a large 173 number of standard Information Elements which provide the necessary { 174 type } information for templates. 176 The use of standard elements enables interoperability among different 177 vendors' implementations. The list of standard elements may be 178 extended in future through the process defined in Section 9 below. 179 Additionally, non-standard enterprise-specific elements may be 180 defined for private use. 182 2. Terminology 184 The terminology used in this document is fully aligned with the 185 terminology defined in [I-D.ietf-ipfix-protocol]. Therefore, the 186 terms defined in the IPFIX terminology are capitalized in this 187 document, as in other IPFIX drafts ([I-D.ietf-ipfix-protocol], 188 [I-D.ietf-ipfix-info], [I-D.ietf-ipfix-architecture]). 190 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 191 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 192 document are to be interpreted as described in RFC 2119 [RFC2119]. 194 This document is Informational. It does not specify a protocol and 195 does not use RFC 2119 keywords [RFC2119] such as "MUST" and "SHOULD", 196 except in quotations and restatements from the IPFIX standards 197 documents. The normative specification of the protocol is given in 198 the IPFIX Protocol [I-D.ietf-ipfix-protocol] and Information Model 199 [I-D.ietf-ipfix-info] documents. 201 3. Template Management Guidelines 203 3.1. Template Management 205 The Exporting Process should always endeavour to send Template 206 Records before the related Data Records. However, since the Template 207 Record may not arrive before the corresponding Data Records, the 208 Collecting Process MAY store Data Records with an unknown Template ID 209 pending the arrival of the corresponding Template (cf. section 9 of 210 [I-D.ietf-ipfix-protocol]). If no Template becomes available, the 211 event should be logged and the Transport Session reset (unless UDP is 212 used), which will cause the Templates to be resent. The amount of 213 time the Collecting Process waits for a Template before resetting 214 should be configurable. We recommend a default of 30 minutes. Note 215 that when using UDP as the transport protocol, this delay should be 216 bound, when possible, by the Template Retransmit and the Template 217 Expiry times (cf. Section 6.2). 219 The Exporting Process must be able to resend active Templates, in 220 case of SCTP association restart, UDP template refresh, or TCP 221 connection restart. 223 The Exporting Process is responsible for the management of Template 224 IDs. Should insufficient Template IDs be available, the Exporting 225 Process must send a Template Withdraw Message in order to free up the 226 allocation of unused Template IDs. Note that UDP doesn't use the 227 Template Withdraw message and the Template lifetime on the Collecting 228 Process relies on timeout. 230 3.2. Template Records versus Option Template Records 232 [I-D.ietf-ipfix-protocol] defines and specifies the use of Templates 233 and Options Templates. Templates define the layout of Data Records, 234 which represent flow data. Options Templates additionally specify 235 scope information elements, which can be used to define scoped Data 236 Records. Scoped Data Records generally export control plane data 237 (such as metadata about processes in the IPFIX collection 238 architecture) or data otherwise applicable to multiple flow Data 239 Records (such as common properties as in 240 [I-D.ietf-ipfix-reducing-redundancy]). 242 Aside from section 4 of [I-D.ietf-ipfix-protocol], which defines 243 specific Options Templates to use for reporting Metering Process and 244 Exporting Process statistics and configuration information, the 245 choice to use Options Templates is left up to the implementer. 246 Indeed, there is a trade-off between bandwidth efficiency and 247 complexity in the use of Options Templates and scoped Data Records. 249 For example, control plane information about an Observation Point 250 could be exported with every Flow Record measured at that Observation 251 Point, or in a single Data Record described by an Options Template, 252 scoped to the Observation Point identifier. In the former case, 253 simplicity of decoding the data is gained in exchange for redundant 254 export of the same data with every applicable Flow Record. The 255 latter case is more bandwidth efficient, but at the expense of 256 requiring the Collecting Process to maintain the relationship between 257 each applicable Flow Record and the Observation Point. 259 A generalized method of using Options Templates to increase bandwidth 260 efficiency is fully described in 261 [I-D.ietf-ipfix-reducing-redundancy]. 263 3.3. Using Scopes 265 The root scope for all IPFIX Messages is the Observation Domain, 266 which appears in the Message Header. In other words, all Data 267 Records within a message implicitly belong to the Observation Domain. 268 All Data Records described by Options Templates (and only those) must 269 be restricted to an additional scope within the Observation Domain, 270 as defined by the scope Information Elements in the Options Template 271 Record. 273 In IPFIX any Information Element can be used for scope. However, 274 Information Elements such as counters, timestamps, padding elements, 275 Flow properties like timeout, flow end reason, duration, or Min/Max 276 Flow properties [I-D.ietf-ipfix-info] may not be appropriate. 278 Note that it is sometimes necessary to export information about 279 entities that exist outside any Observation Domain, or within 280 multiple Observation Domains (e.g. information about Metering 281 Processes scoped to meteringProcessID). Such information SHOULD be 282 exported in an IPFIX Message with Observation Domain ID 0 (cf. 283 [I-D.ietf-ipfix-protocol], Section 3.1). 285 3.4. Multiple Information Elements of the same type 287 Exporting Process and Collecting Process MUST support the use of 288 multiple Information Elements of the same type in a single Template 289 [I-D.ietf-ipfix-protocol]. This was first required by Packet 290 Sampling (PSAMP) for the export of multiple Selector IDs. Note that 291 the IPFIX Protocol recommends that Metering Processes SHOULD use 292 packet treatment order when exporting multiple identical Information 293 Elements in the same record ([I-D.ietf-ipfix-protocol] Section 8). 294 This implies that ordering is important, and changes to the order of 295 multiple identical Information Elements could cause information loss. 296 Therefore, we strongly recommend preservation of the order of 297 multiple Information Elements of the same type by Exporting and 298 Collecting Processes for correct processing and storage. 300 3.5. Selecting Message Size 302 The IPFIX Protocol in section 10.3.3 defines the maximum message size 303 for IPFIX Messages transported over UDP to be constrained by the path 304 MTU, or if the path MTU is not available, 512 bytes which is the 305 minimum datagram size all IP implementations must support (Cf. also 306 Section 8.4). However, no maximum message size is imposed on other 307 transport protocols, beyond the 65535-byte limit imposed by the 16- 308 bit Message Length field in the IPFIX Message Header. 310 An IPFIX Exporting Process operating over SCTP or TCP may export 311 IPFIX Messages up to this 64kB limit, and an IPFIX Collecting Process 312 must accept any message up to that size. 314 4. Exporting Process Guidelines 316 4.1. Sets 318 A Set is identified by a Set ID [I-D.ietf-ipfix-protocol]. A Set ID 319 has an integral data type and its value is in the range of 0 - 65535. 320 The Set ID values of 0 and 1 are not used for historical reasons 321 [RFC3954]. A value of 2 identifies a Template Set. A value of 3 322 identifies an Options Template Set. Values from 4 to 255 are reserved 323 for future use. Values above 255 are used for Data Sets. In this 324 case the SetID corresponds to the TemplateID of the used Template. 326 A Data Set received with an unknown Set ID may be stored pending the 327 arrival of the corresponding Template (cf. section 9 of 328 [I-D.ietf-ipfix-protocol]). If no Template becomes available, the 329 event should be logged and the Transport Session reset (unless UDP is 330 used), which will cause the Templates to be resent. The amount of 331 time the Collecting Process waits for a Template before resetting 332 should be configurable. We recommend a default of 30 minutes. Note 333 that when using UDP as the transport protocol, this delay should be 334 bound, when possible, by the Template Retransmit and the Template 335 Expiry times (cf. Section 6.2). 337 The arrival of a Set with a reserved Set ID should be logged. The 338 collector must ignore the unknown Set. 340 4.2. Information Element Coding 342 [I-D.ietf-ipfix-architecture] does not specify which entities are 343 responsible for the encoding and decoding of Information Elements 344 transferred via IPFIX. An IPFIX device can do the encoding either 345 within the Metering Process or within the Exporting Process. The 346 decoding of the Information Elements can be done by the Collecting 347 Process or by the data processing application. 349 If an IPFIX node simply relays IPFIX Records (like a proxy) then no 350 decoding or encoding of Information Elements is needed. In this case 351 the Exporting Process may export unknown Information Elements, i.e. 352 Information Elements with an unknown Information Element number. 354 4.3. Using Counters 356 IPFIX offers both Delta and Total counters (e.g. octetDeltaCount, 357 octetTotalCount). If information about a flow is only ever exported 358 once, then it's not important whether Delta or Total counters are 359 used. However, if further information about additional packets in a 360 flow is exported after the first export then either: 362 o the metering system must reset its counters to zero after the 363 first export and report the new counter values using delta 364 counters. 366 Or 368 o the metering system must carefully maintain its counters and 369 report the running total using total counters. 371 At first, reporting the running total may seem to be the obvious 372 choice, but requires that the system accurately maintains information 373 about the flow over a long time without any loss or error, because 374 when reported to a Collecting Process, the previous total values will 375 be replaced with the new information. 377 Delta counters offer some advantages: information about flows don't 378 have to be permanently maintained, and any loss of information has 379 only a small impact on the total stored at the Collecting Process. 380 Finally, deltas may be exported in fewer bytes than total counters 381 using the IPFIX "Reduced Size Encoding" scheme 382 [I-D.ietf-ipfix-protocol]. 384 Note that delta counters have an origin of zero, and that a 385 Collecting Process receiving delta counters for a flow that is new to 386 the Collecting Process must assume the deltas are from zero. 388 4.4. Padding 390 The IPFIX Information Model defines an Information Element for 391 padding called paddingOctets [I-D.ietf-ipfix-info]. It is of type 392 octetArray and the IPFIX protocol allows encoding it as a fixed- 393 length array as well as a variable length array. 395 The padding Information Element can be used to align Information 396 Elements within Data Records, Records within Sets, and Sets within 397 IPFIX messages, as described below. 399 4.4.1. Alignment of Information Elements within a Data Record 401 The padding Information Element gives flexible means for aligning 402 Information Elements within a Data Record. Aligning within a Data 403 Record can be useful, because internal data structures can be easily 404 converted into Flow Records at the Exporter and vice versa at the 405 Collecting Process. 407 Alignment of Information Elements within a Data Record is achieved by 408 inserting an instances of the paddingOctets Information Element with 409 appropriate length before each unaligned Information Element. This 410 insertion is explicitly specified within the Template Record or 411 Option template record, respectively, that corresponds to the Data 412 Record. 414 4.4.2. Alignment of Information Elements specifiers within a Template 415 Record 417 There is no means for aligning Information Element specifiers within 418 Template Records. However, there is limited need for such a method, 419 as Information Element specifiers are always 32-bit aligned, and 32- 420 bit alignment is generally sufficient. 422 4.4.3. Alignment of Records within a Set 424 There is no means for aligning Template Records within a Set. 425 However, there is limited need for such a method, as Information 426 Element specifiers are always 32-bit aligned, and 32-bit alignment is 427 generally sufficient. 429 Data Records can be aligned within a Set by appending instances of 430 the paddingOctets Information Element at the end of the Record. 431 Since all Data Records within a Set have the same structure and size, 432 aligning one Data Record implies aligning all the Data Records within 433 a single Set. 435 4.4.4. Alignment of Sets within an IPFIX Message 437 If Records are already aligned within a Set by using paddingOctets 438 Information Elements, then this alignment will already be achieved. 439 But for aligning Sets within an IPFIX message, padding Information 440 Elements can be used at the end of the Set so that the subsequent Set 441 starts at an aligned boundary. This padding mechanism is described 442 in section 3.3.1 of [I-D.ietf-ipfix-protocol] and can be applied even 443 if the records within sets are not aligned. However, it should be 444 noted that this method is limited by the constraint that "the padding 445 length MUST be shorter than any allowable Record in the Set", to 446 prevent the padding from being misinterpreted as an additional Data 447 Record. 449 4.5. Time Issues 451 IPFIX messages contain the export time in the message header. In 452 addition there are a series of information elements defined to 453 transfer time values. [I-D.ietf-ipfix-info] defines four abstract 454 data types to transfer time values in second, millisecond, 455 microsecond and nanosecond resolution. 457 The accuracy and precision of these values depends on the accuracy 458 and the precision of the Metering Process clock. The accuracy and 459 precision of the Exporting Process clock, and the synchronisation of 460 the Metering Process and Exporting Process clocks, is also important 461 when using the delta timestamp Information Elements. To ensure 462 accuracy the clocks should be synchronised to a UTC time source. 463 Normally it would be sufficient to derive the time from a remote time 464 server using the Network Time Protocol (NTP) [RFC1305]. IPFIX 465 Devices operating with time values of microsecond or nanosecond 466 resolution need direct access to a time source, for example to a GPS 467 (Global Positioning System) unit. 469 The most important consideration in selecting timestamp Information 470 Elements is to use a precision appropriate for the timestamps as 471 received from the Metering Process. Specifically, an Exporting 472 Process should not export timestamp Information Elements of higher 473 precision than the timestamps used by the Metering Process (e.g. 474 millisecond-precision flows should not be exported with 475 flowStartMicroseconds and flowEndMicroseconds). 477 4.6. IPFIX Message Header Export Time and Data Record Time 479 Section 5 of [I-D.ietf-ipfix-protocol] defines a method for optimized 480 export of time-related Information Elements based upon the Export 481 Time field of the IPFIX Message header. The architectural separation 482 of the Metering Process and Exporting Process in 483 [I-D.ietf-ipfix-architecture] raises some difficulties with this 484 method, of which implementers should be aware. 486 Since the Metering Process has no information about the export time 487 of the IPFIX Message (that is, when the message leaves the Exporting 488 Process), it cannot properly use the delta time Information Elements; 489 it must store absolute timestamps and transmit these to the Exporting 490 Process. The Exporting Process must then convert these to delta 491 timestamps once the Export Time is known. This increases the 492 processing burden on the Exporting Process. Note also that the 493 absolute timestamps require more storage than their delta timestamp 494 counterparts. However, this method can result in reduced export 495 bandwidth. 497 Alternatively, the Exporting Process may simply export absolute 498 timestamp Information Elements. This simplifies the Exporting 499 Process' job and reduces processing burden, but increases export 500 bandwidth requirements. 502 4.7. Devices Without an Absolute Clock 504 Exporting just relative times in a device without an absolute clock 505 is often not sufficient. For instance, observed traffic could be 506 retained in the device's cache for some time before being exported 507 (e.g., if the exporter runs once per minute), or stuck in an Inter 508 Process Communication (IPC) queue, or stuck in the export stack, or 509 delayed in the network between the exporter and collector. 511 For these reasons it can be difficult for the Collecting Process to 512 convert the relative times exported using the flowStartSysUpTime and 513 flowEndSysUpTime Information Elements to absolute times with any sort 514 of accuracy without knowing the systemInitTimeMilliseconds. The 515 sending of the flowStartSysUpTime and flowEndSysUpTime Information 516 Elements without also sending the systemInitTimeMilliseconds 517 Information Element is not recommended. 519 5. Collecting Process Guidelines 521 5.1. Information Element (de)coding 523 Section 9 of [I-D.ietf-ipfix-protocol] specifies: "The Collecting 524 Process MUST note the Information Element identifier of any 525 Information Element that it does not understand and MAY discard that 526 Information Element from the Flow Record". The Collecting Process 527 may accept Templates with Information Elements of unknown types. In 528 this case the value received for these Information Elements should be 529 decoded as an octet array. 531 Alternatively, the Collecting Process may ignore Templates and 532 subsequent Data Sets that contain Information Elements of unknown 533 types. 535 It is recommended that Collecting Processes provide means to flexibly 536 add types of new Information Elements to their knowledge base. An 537 example is a configuration file that is read by the Collecting 538 Process and that contains a list of Information Element identifiers 539 and the corresponding types. Particularly for adding enterprise- 540 specific Information Elements, such a feature can be very useful. 542 5.2. Reduced-size Encoding of Information Elements 544 Since a Collector may receive data from the same device and 545 Observation Domain in two templates using different reduced size 546 encodings, it is recommended that the data be stored using full size 547 encoding, to ensure that the values can be stored or even aggregated 548 together. 550 5.3. Template Management 552 Template IDs are generated dynamically by the Exporting Process. 553 They are unique per Transport Session and Observation Domain. 555 Therefore, for each Transport Session, the Collecting Process has to 556 maintain a list of Observation Domains. For each Observation Domain 557 the Collecting Process has to maintain a list of current Template IDs 558 in order to decode subsequent Data Records. 560 Note that a restart of the Transport Session may lead to a Template 561 ID renumbering. 563 6. Transport-Specific Guidelines 565 IPFIX can use SCTP, TCP, or UDP as a transport protocol. IPFIX 566 implementations MUST support SCTP with partial reliability extensions 567 (PR-SCTP), and MAY support TCP and/or UDP (Cf. 568 [I-D.ietf-ipfix-protocol], Section 10.1). In the IPFIX documents the 569 terms SCTP and PR-SCTP are often used interchangeably to mean SCTP 570 with partial reliability extensions. 572 6.1. SCTP 574 PR-SCTP is the preferred transport protocol for IPFIX because it is 575 congestion-aware, reducing total bandwidth usage in the case of 576 congestion, but with a simpler state machine than TCP. This saves 577 resources on lightweight probes and router line cards. 579 SCTP as specified in RFC 4960 [RFC4960] with the PR-SCTP extension 580 defined in RFC 3758 [RFC3758] provides several features not available 581 in TCP or UDP. The two of these most universally applicable to IPFIX 582 implementations, and that IPFIX implementors need to know about, are 583 multiple streams and per-message partial reliability. 585 An SCTP association may contain multiple streams. Streams are useful 586 for avoiding head-of-line blocking, thereby minimising end to end 587 delay from the Exporting Process to the Collecting Process. Example 588 applications for this feature would be using one SCTP stream per 589 Observation Domain, one stream per type of data (or Template ID), or 590 one stream for flow data and one for metadata. 592 An Exporting Process may request any number of streams, and may send 593 IPFIX Messages containing any type of Set (Data Set, Template Set, 594 etc.) on any stream. A Collecting Process MUST be able to process 595 any Message received on any stream. 597 Stream negotiation is a feature of the SCTP protocol. Note however, 598 that the IPFIX protocol doesn't provide any mechanism for the 599 Exporter to convey any information about which streams are in use to 600 the Collector. Therefore, stream configuration must be done out of 601 band. 603 One extra advantage of the PR-SCTP association is its ability to send 604 messages with different levels of reliability, selected according to 605 the application. For example, billing or security applications might 606 require reliable delivery of all their IPFIX Messages, while capacity 607 planning applications might be more tolerant of message loss. SCTP 608 allows IPFIX Messages for all these applications to be transported 609 over the same association. 611 IPFIX Messages may be sent with full or partial reliability, on a 612 per-message basis. Fully reliable delivery guarantees that the IPFIX 613 Message will be received at the Collecting Process or that that SCTP 614 Association will be reset, as with TCP. Partially reliable delivery 615 does not guarantee the receipt of the IPFIX Message at the collecting 616 process. This feature may be used to allow Messages to be dropped 617 during network congestion, i.e. while observing a Denial of Service 618 attack. 620 RFC 3758 [RFC3758] defines the concept of a Partial Reliablity 621 policy, which specifies the interface used to control partially 622 reliable delivery. It also defines a single example Partial 623 Reliability policy called "timed reliability", which uses a single 624 parameter, lifetime. The lifetime is specified per message in 625 milliseconds, and after it expires no further attempt will be made to 626 transmit the message. Longer lifetimes specify more retransmission 627 attempts per message and therefore higher reliability; however, it 628 should be noted that the absolute reliability provided by a given 629 lifetime is highly dependent on network conditions, so an Exporting 630 Process using the timed reliability service should provide a 631 mechanism for configuring the lifetime of exported IPFIX Messages. 632 Another possible partial reliability policy could be limited 633 retransmission which guarantees a specified number of retransmissions 634 for each message. It is up to the implementer to decide which 635 Partial Reliability policy is most appropriate for its application. 637 There is an additional service provided by SCTP and useful in 638 conjunction with PR-SCTP: unordered delivery. This also works on a 639 per-message basis by declaring that a given message should be 640 delivered to the receiver as soon as it is queued rather than kept in 641 sequence; however, it should be noted that unless explicitly 642 requested by the sender even messages sent partially reliably will 643 still be delivered in order. Unordered delivery should not be used 644 when the order of IPFIX Messages may matter: e.g., a Template or 645 Options Template. Unordered delivery should not be used when Total 646 counters are used, as reordering could result in the counter value 647 decreasing at the Collecting Process, and even being left with a 648 stale value if the last message processed is stale. 650 By convention, when the IPFIX documents state a requirement for 651 reliable delivery (as, for example, the IPFIX Protocol document does 652 for Template Sets, Options Template Sets, and Template Withdrawal 653 Messages), an IPFIX Exporting Process must not use partially reliable 654 delivery for those Messages. By default, and explicitly if the IPFIX 655 documents call for "partially reliable" or "unreliable" delivery, an 656 IPFIX Exporting Process may use partially reliable delivery if the 657 other requirements of the application allow. 659 The Collecting Process may check whether IPFIX Messages are lost by 660 checking the Sequence Number in the IPFIX header. The Collecting 661 Process should use the Sequence Number in the IPFIX Message header to 662 determine whether any messages are lost when sent with partial 663 reliability. Sequence numbers should be tracked independently for 664 each stream. 666 The following may be done to mitigate message loss: 668 o Increase the SCTP buffer size on the Exporter. 670 o Increase the bandwidth available for communicating the exported 671 Data Records. 673 o Use sampling, filtering, or aggregation in the Metering Process to 674 reduce the amount of exported data (cf. [I-D.ietf-ipfix-protocol] 675 section 10.4.2.3). 677 o If partial reliability is used, switch to fully reliable delivery 678 on the Exporting Process or increase the level of partial 679 reliability (e.g., when using timed reliability, by specifying a 680 longer lifetime for exported IPFIX Messages). 682 If the SCTP association is brought down because the IFPIX Messages 683 can't be exported reliably, the options are: 685 o Increase the SCTP buffer size on the Exporter. 687 o Increase the bandwidth available for communicating the exported 688 Data Records. 690 o Use sampling, filtering, or aggregation in the Metering Process to 691 reduce the amount of exported data. 693 Note that Templates must not be resent when using SCTP, without an 694 intervening Template Withdrawal or SCTP association reset. Note also 695 that since Template Sets and Template Withdrawal Messages may be sent 696 on any SCTP stream, a Template Withdrawal Message may withdraw a 697 template sent on a different stream, and a Template Set may reuse a 698 Template ID withdrawn by a Template Withdrawal Message sent on a 699 different stream. Therefore, an Exporting Process sending Template 700 Withdrawal Messages should ensure to the extent possible that the 701 Template Withdrawal Messages and subsequent Template Sets reusing the 702 withdrawn Template IDs are received and processed at the Collecting 703 Process in proper order. The Exporting Process can achieve this by 704 one of two possible methods: 1. by sending a Template Withdrawal 705 Message reliably, in order, and on the same stream as the subsequent 706 Template Set reusing its ID; or 2. by waiting an appropriate amount 707 of time (on the scale of one minute) after sending a Template 708 Withdrawal Message before attempting to reuse the withdrawn Template 709 ID. 711 6.2. UDP 713 UDP is useful in simple systems where an SCTP stack is not available, 714 and where there is insufficient memory for TCP buffering. 716 However, UDP is not a reliable transport protocol, and IPFIX messages 717 sent over UDP might be lost as with partially-reliable SCTP streams. 718 UDP is not the recommended protocol for IPFIX and is intended for use 719 in cases in which IPFIX is replacing an existing NetFlow 720 infrastructure, with the following properties: 722 o A dedicated network, 724 o within a single administrative domain, 726 o where SCTP is not available due to implementation constraints, 728 o and the collector is as topologically close as possible to the 729 exporter. 731 Note that because UDP itself provides no congestion control 732 mechanisms, it is recommended to use UDP transport only on managed 733 networks, where the network path has been explicitly provisioned for 734 IPFIX traffic through traffic engineering mechanisms, such as rate 735 limiting or capacity reservations. 737 An important example of an explicitly provisioned managed network for 738 IPFIX is use of IPFIX to replace a functioning NetFlow implementation 739 on a dedicated network. In this situation, the dedicated network 740 should be provisioned in accordance with the NetFlow deployment 741 experience that flow export traffic generated by monitoring an 742 interface will amount to 2-5% of the monitored interface's bandwidth. 744 As recommended in [I-D.ietf-tsvwg-udp-guidelines] an application 745 should not send UDP messages that result in IP packets that exceed 746 the MTU of the path to the destination and should enable UDP 747 checksums (see sections 3.2 and 3.4 of 748 [I-D.ietf-tsvwg-udp-guidelines] respectively). 750 Since IPFIX assumes reliable transport of templates over SCTP, this 751 necessitates some changes for IPFIX template management over UDP. 752 Templates sent from the Exporting Process to the Collecting Process 753 over UDP MUST be resent at regular time intervals ; these intervals 754 MUST be configurable (see Section 10.3 of [I-D.ietf-ipfix-protocol]). 756 We recommend a default Template resend time of 10 minutes, 757 configurable between 1 minute and 1 day. 759 Note that this could become an interoperability problem, e.g. if an 760 Exporter re-sends Templates once per day, while a Collector expires 761 Templates hourly, then they may both be IPFIX-compatible, but not be 762 interoperable. 764 Retransmission time intervals that are too short waste bandwidth on 765 unnecessary template retransmissions. On the other hand, time 766 intervals that are too long introduce additional costs or risk of 767 data loss by potentially requiring the Collector to cache more data 768 without having the Templates available to decode it. 770 To increase reliability and limit the amount of potentially lost data 771 the Exporting Process may resend additional templates using a packet- 772 based schedule. In this case Templates are resent depending on the 773 number of data packets sent. Similarly to the time interval, 774 resending a Template every few packets introduces additional overhead 775 while resending after a large amount of packets have already been 776 sent means high costs due to the data caching and potential data 777 loss. 779 We recommend a default Template resend interval of 20 packets, 780 configurable between 1 and 1000 packets. 782 Note that a sufficiently small resend time or packet interval may 783 cause a system to become stuck, continually re-sending templates. 784 e.g., if the resend packet interval is 2 (i.e., templates are to be 785 sent in every other packet) but more than two packets are required to 786 send all the templates, then the resend interval will have expired by 787 the time the templates have been sent, and templates will be sent 788 continuously - possibly preventing any data from being sent at all. 790 Therefore the Template resend intervals should be considered from the 791 last data packet, and should not be tied to specific sequence 792 numbers. 794 The Collecting Process should use the Sequence Number in the IPFIX 795 Message header to determine whether any messages are lost. 797 The following may be done to mitigate message loss: 799 o Move the Collector topologically closer to the Exporter. 801 o Increase the bandwidth of the links through which the Data Records 802 are exported. 804 o Use sampling, filtering, or aggregation in the Metering Process to 805 reduce the amount of exported data. 807 o Increase the buffer size at the Collector and/or the Exporter. 809 Before using a Template for the first time, the Exporter may send it 810 in several different IPFIX Messages spaced out over a period of 811 packets in order to increase the likelihood that the Collector has 812 received the Template. 814 Template Withdraw messages MUST NOT be sent over UDP (Section 10.3.6 815 of [I-D.ietf-ipfix-protocol]). The Exporter must rely on expiration 816 at the Collector to expire old Templates or to reuse Template Ids. 818 We recommend that the collector implements a template expiry of three 819 times the Exporter refresh rate. 821 However, since the IPFIX protocol doesn't provide any mechanism for 822 the Exporter to convey any information about the Template expiry time 823 to the Collector, configuration must be done out of band. 825 If no out of band configuration is made, we recommend to initially 826 set a template expiry time at the Collector of 60 minutes. The 827 Collecting Process may estimate each Exporting Process's resend time 828 and adapt the expiry time for the corresponding Templates 829 accordingly. 831 6.3. TCP 833 TCP can be used as a transport protocol for IPFIX if one of the 834 endpoints has no support for SCTP, but a reliable transport is needed 835 and/or the network between the exporter and the collector has not 836 explicitly been provisioned for the IPFIX traffic. TCP is one of the 837 core protocols of the Internet, and is widely supported. 839 The Exporting Process may re-send Templates (per UDP, above), but 840 it's not required to do so, per section 10.4.2.2 of 841 [I-D.ietf-ipfix-protocol]: 843 "A Collecting Process MUST record all Template and Options Template 844 Records for the duration of the connection, as an Exporting Process 845 is not required to re-export Template Records." 847 If the available bandwidth between exporter and collector is not 848 sufficient or the metering process generates more data records than 849 the collector is capable of processing, then TCP congestion control 850 may cause the exporter to block. Options in this case are: 852 o Increase the TCP buffer size on the Exporter. 854 o Increase the bandwidth of the links through which the Data Records 855 are exported. 857 o Use sampling, filtering, or aggregation in the Metering Process to 858 reduce the amount of exported data. 860 7. Guidelines for implementation on Middleboxes 862 The term middlebox is defined in RFC 3234 [RFC3234] as: 864 "A middlebox is defined as any intermediary device performing 865 functions other than the normal, standard functions of an IP router 866 on the datagram path between a source host and destination host." 868 The list of middleboxes discussed in RFC 3234 contains: 870 1. Network Address Translation (NAT), 872 2. NAT-Protocol Translation (NAT-PT), 874 3. SOCKS gateway, 876 4. IP tunnel endpoints, 878 5. packet classifiers, markers, schedulers, 880 6. transport relay, 882 7. TCP performance enhancing proxies, 884 8. load balancers that divert/munge packets, 885 9. IP firewalls, 887 10. application firewalls, 889 11. application-level gateways, 891 12. gatekeepers / session control boxes, 893 13. transcoders, 895 14. proxies, 897 15. caches, 899 16. modified DNS servers, 901 17. content and applications distribution boxes, 903 18. load balancers that divert/munge URLs, 905 19. application-level interceptors, 907 20. application-level multicast, 909 21. involuntary packet redirection, 911 22. anonymizers. 913 It is likely that since the publication of RFC 3234 new kinds of 914 middleboxes have been added. 916 While the IPFIX specifications [I-D.ietf-ipfix-protocol] based the 917 requirements on the export protocol only (as the IPFIX name implies), 918 these sections cover the guidelines for the implementation of the 919 Metering Process by recommending which Information Elements to export 920 for the different middlebox considerations. 922 7.1. Traffic Flow Scenarios at Middleboxes 924 Middleboxes may delay, re-order, drop, or multiply packets; they may 925 change packet header fields and change the payload. All these 926 actions have an impact on traffic flow properties. In general, a 927 middlebox transforms a uni-directional original traffic flow T that 928 arrives at the middlebox into a transformed traffic flow T' that 929 leaves the middlebox. 931 +-----------+ 932 T ---->| middlebox |----> T' 933 +-----------+ 935 Figure 1: Uni-directional traffic flow traversing a middlebox 937 Note that in an extreme case, T' may be an empty traffic flow (a flow 938 with no packets), for example, if the middlebox is a firewall and 939 blocks the flow. 941 In case of a middlebox performing a multicast function, a single 942 original traffic flow may be transformed into more than one 943 transformed traffic flow. 945 +------> T' 946 | 947 +---------+-+ 948 T ---->| middlebox |----> T'' 949 +---------+-+ 950 | 951 +------> T''' 953 Figure 2: Uni-directional traffic flow traversing a middlebox with 954 multicast function 956 For bi-directional traffic flows we identify flows on different sides 957 of the middlebox: say T_l on the left side and T_r on the right side. 959 +-----------+ 960 T_l <--->| middlebox |<---> T_r 961 +-----------+ 963 Figure 3: Bi-directional unicast traffic flow traversing a middlebox 965 In case of a NAT T_l might be a traffic flow in a private address 966 realm and T_r the translated traffic flow in the public address 967 realm. If the middlebox is a NAT-PT, then T_l may be an IPv4 traffic 968 flow and T_r the translated IPv6 traffic flow. 970 At tunnel endpoints, flows are multiplexed or de-multiplexed. In 971 general, tunnel endpoints can deal with bi-directional traffic flows. 973 +------> T_r1 974 v 975 +---------+-+ 976 T_l <--->| middlebox |<---> T_r2 977 +---------+-+ 978 ^ 979 +------> T_r3 981 Figure 4: Multiple Data Reduction 983 An example is a traffic flow T_l of a tunnel and flows T_rx that are 984 multiplexed into or de-multiplexed out of a tunnel. According to the 985 IPFIX definition of traffic flows in [I-D.ietf-ipfix-protocol] T and 986 T' or T_l and T_rx, respectively, are different flows in general. 988 However, from an application point of view, they might be considered 989 as closely related or even as the same flow, for example if the 990 payloads they carry are identical. 992 7.2. Location of the Observation Point 994 Middleboxes might be integrated with other devices. An example is a 995 router with a NAT or a firewall at a line card. If an IPFIX 996 Observation Point is located at the line card, then the properties of 997 measured traffic flows may depend on the side of the integrated 998 middlebox at which packets were captured for traffic flow 999 measurement. 1001 Consequently, an Exporting Process reporting traffic Flows measured 1002 at a device that hosts one or more middleboxes should clearly 1003 indicate to Collecting Processes the location of the used observation 1004 point(s) with respect to the middlebox(es). This can be done by 1005 using Options with Observation Point as Scope and elements like for 1006 instance lineCardID or samplerID. Otherwise, processing the measured 1007 flow data could lead to wrong results. 1009 At the first glance, choosing an Observation Point that covers the 1010 entire middlebox looks like an attractive choice. But this leads to 1011 ambiguities for all kinds of middleboxes. Within the middlebox 1012 properties of packets are modified and it should be clear at a 1013 Collecting Process whether packets were observed and metered before 1014 or after modification. For example, it must be clear whether a 1015 reported source IP address was observed before or after a NAT changed 1016 it or whether a reported packet count was measured before or after a 1017 firewall dropped packets. For this reason, [I-D.ietf-ipfix-info] 1018 provides Information Elements with prefix "post" for Flow properties 1019 that are changed within a middlebox. 1021 If an Observation Point is located inside a middlebox, the middlebox 1022 must have well defined and well separated internal functions, for 1023 example a combined NAT and firewall, and the Observation Point should 1024 be located on a boundary between middlebox functions rather than 1025 within one of the functions. 1027 7.3. Reporting Flow-related Middlebox Internals 1029 While this document recommends IPFIX implementations using 1030 Observation Points outside of middlebox functions, there are a few 1031 special cases where reporting flow-related internals of a middlebox 1032 is of interest. 1034 For many applications that use traffic measurement results it is 1035 desirable to get more information than can be derived from just 1036 observing packets on one side of a middlebox. If, for example, 1037 packets are dropped by the middlebox acting as a firewall, NAT or 1038 traffic shaper, then information about how many observed packets are 1039 dropped may be of high interest. 1041 This section gives recommendations on middlebox internal information 1042 that may be reported if the IPFIX Observation Point is co-located 1043 with one or more middleboxes. Since the internal information to be 1044 reported depends on the kind of middlebox, it is discussed per kind. 1046 The recommendations cover middleboxes that act per packet and that do 1047 not modify the application level payload of the packet (except by 1048 dropping the entire packet) and that do not insert additional packets 1049 into an application level or transport level traffic stream. 1051 Covered are the packet level middleboxes of kind 1 - 6, 8 - 10, 21, 1052 and 22 (according to the enumeration given at the beginning of 1053 Section Section 7). Not covered are 7 and 11 - 20. TCP performance 1054 enhancing proxies (7) are not covered because they may add ACK 1055 packets to a TCP connection. 1057 Still, if possible, IPFIX implementations co-located with uncovered 1058 middleboxes (i.e. of type 7 or 11 - 20) should follow the 1059 recommendations given in this section if they can be applied in a way 1060 that reflects the intention of these recommendations. 1062 7.3.1. Packet Dropping Middleboxes 1064 If an IPFIX observation point is co-located with one or more 1065 middleboxes that potentially drop packets, then the corresponding 1066 IPFIX Exporting Process should be able to report the number of 1067 packets that were dropped per reported flow. 1069 Concerned kinds of middleboxes are NAT (1), NAT-PT (2), SOCKS gateway 1070 (3), packet schedulers (5), IP firewalls (9) and application level 1071 firewalls (10). 1073 7.3.2. Middleboxes Changing the DSCP 1075 If an IPFIX observation point is co-located with one or more 1076 middleboxes that potentially modify the DiffServ Code Point (DSCP, 1077 see [RFC2474]) in the IP header, then the corresponding IPFIX 1078 Exporting Process should be able to report both the observed incoming 1079 DSCP value and also the DSCP value on the 'other' side of the 1080 middlebox (if this is a constant value for the particular traffic 1081 flow). Related Information Elements specified in 1082 [I-D.ietf-ipfix-info] are: postIpClassOfService. 1084 Note that the current IPFIX information model only contains 1085 Information Elements supporting the case that at the Observation 1086 Point packets are observed before the DSCP is changed. Here, the 1087 relevant Information Elements are ipClassOfService and 1088 postIpClassOfService, where the latter one reports the value of the 1089 IP TOS field after the DSCP has been changed. This value is not 1090 observed at the observation point. For the other case where packets 1091 are observed after the DSCP was changed, there is no Information 1092 Element in the current information model to report the value that the 1093 TOS field had before it was changed. If reporting this value is 1094 required, then a "pre" Information Element is required, such as, for 1095 example, preIpClassOfService, or preDiffServCodePoint. Such 1096 Information Elements can be specified as enterprise-specific ones or 1097 a request for adding them to the IPFIX information model can be sent 1098 to IANA, once IANA has taken over the maintenance of the IPFIX 1099 Information Element identifier list. 1101 Note also that a classifier may change the same DSCP value of packets 1102 from the same flow to different values depending on the packet or 1103 other conditions. Also it is possible that packets of a single uni- 1104 directional arriving flow contain packets with different DSCP values 1105 that are all set to the same value by the middlebox. In both cases 1106 there is a constant value for the DSCP field in the IP packets header 1107 to be observed on one side of the middlebox, but on the other side 1108 the value may vary. In such a case reliable reporting of the DSCP 1109 value on the 'other' side of the middlebox is not possible by just 1110 reporting a single value. According to the IPFIX information model 1111 [I-D.ietf-ipfix-info], the first value observed for the DSCP is 1112 reported by the IPFIX protocol in that case. 1114 This recommendation applies to packet markers (5). 1116 7.3.3. Middleboxes Changing IP Addresses and Port Numbers 1118 If an IPFIX Observation Point is co-located with one or more 1119 middleboxes that potentially modify the: 1121 o IP version field, 1123 o IP source address header field, 1125 o IP destination header field, 1127 o Source transport port number, 1129 o Destination transport port number 1131 in one of the headers, then the corresponding IPFIX Exporting Process 1132 should be able to report the 'translated' value of these fields, as 1133 far as they have constant values for the particular traffic flow, in 1134 addition to the observed values of these fields. 1136 If the changed values are not constant for the particular traffic 1137 flow but still reporting is desired, then it is recommended that the 1138 general rule for Information Elements with changing values is 1139 applied: The reported value is the one that applies to the first 1140 packet observed for the reported flow. 1142 Note that the 'translated' value of the fields can be the values 1143 before or after the translation depending on the Flow direction and 1144 the location of the observation point with respect to the middlebox. 1145 We always call the value that is not the one observed at the 1146 observation point the translated value. 1148 Note also that a middlebox may change the same port number value of 1149 packets from the same flow to different values depending on the 1150 packet or other conditions. Also it is possible that packets of 1151 different uni-directional arriving flows with different source/ 1152 destination port number pairs may be mapped to a single flow with a 1153 single source/destination port number pair by the middlebox. In both 1154 cases there is a constant value for the port number pair to be 1155 observed on one side of the middlebox, but on the other side the 1156 values may vary. In such a case reliable reporting of the port 1157 number pairs on the 'other' side of the middlebox is not possible. 1159 According to the IPFIX information model [I-D.ietf-ipfix-info], the 1160 first value observed for each port number is reported by the IPFIX 1161 protocol in that case. 1163 This recommendation applies to NAT (1), NAT-PT (2), SOCKS gateway (3) 1164 and involuntary packet redirection (21) middleboxes. It may also be 1165 applied to anonymizers (22), though it should be noted that this 1166 carries the risk of losing the effect of anonymisation. 1168 8. Security Guidelines 1170 8.1. Introduction to TLS and DTLS for IPFIX implementers 1172 TLS [RFC4346] and DTLS [RFC4347] are the REQUIRED protocols for 1173 securing network traffic exported with IPFIX (see Section 11 of 1174 [I-D.ietf-ipfix-protocol]). TLS requires a reliable transport 1175 channel and is selected as the security mechanism for TCP. DTLS is a 1176 version of TLS capable of securing datagram traffic and is selected 1177 for UDP, SCTP and PR-SCTP. 1179 When mapping TLS terminology used in [RFC4346] to IPFIX terminology, 1180 keep in mind that the IPFIX Exporting Process, as it is the 1181 connection initiator, corresponds to the TLS Client, and the IPFIX 1182 Collecting Process corresponds to the TLS Server. These terms apply 1183 only to the bidirectional TLS handshakes done at Transport Session 1184 Establishment and completion time; aside from TLS connection set up 1185 between the Exporting Process and the Collecting Process, and 1186 teardown at the end of the session, the unidirectional flow of 1187 messages from Exporting Process to Collecting Process operates over 1188 TLS just as over any other transport layer for IPFIX. 1190 8.2. X.509-based Identity Verification for IPFIX over TLS or DTLS 1192 When using TLS or DTLS to secure an IPFIX Transport Session, the 1193 Collecting Process and Exporting Process must use strong mutual 1194 authentication. In other words, each IPFIX endpoint must have its 1195 own X.509 certificate [RFC3280] and private key, and the Collecting 1196 Process, which acts as the TLS or DTLS Server, must send a 1197 Certificate Request to the Exporting Process to the Exporting Process 1198 during the TLS handshake, and fail to establish a session if the 1199 Exporting Process does not present a valid certificate. 1201 Each of the Exporting Process and the Collecting Process must verify 1202 the identity of its peer against a set of authorized peers. This may 1203 be done by configuring a set of authorized distinguished names and 1204 comparing the peer certificate's subject distinguished name against 1205 each name in the set. However, if a private certificate authority 1206 (CA) is used to sign the certificates identifying the Collecting 1207 Processes and Exporting Processes, and the set of certificates signed 1208 by that private CA may be restricted to those identifying peers 1209 authorized to communicate with each other, it is sufficient to merely 1210 verify that the peer's certificate is issued by this private CA. 1212 When verifying the identity of its peer, an IPFIX Exporting Process 1213 or Collecting Process must verify that the peer certificate's subject 1214 common name or subjectAltName extension dNSName matches the fully- 1215 qualified domain name (FQDN) of the peer. This involves retrieving 1216 the expected domain name from the peer certificate and the address of 1217 the peer, then verifying that the two match via a DNS lookup. Such 1218 verification should require both that forward lookups (FQDN to peer 1219 address) and reverse lookups (peer address to FQDN) match. In 1220 deployments absent DNS infrastructure, it is acceptable to represent 1221 the FQDN as an IPv4 dotted-quad or a textual IPv6 address as in 1222 [RFC1924]. 1224 8.3. Implementing IPFIX over TLS over TCP 1226 Of the security solutions specified for IPFIX, TLS over TCP is as of 1227 this writing the most mature and widely implemented. Until stable 1228 implementations of DTLS over SCTP are widely available (see 1229 Section 8.5, below), it is recommended that applications requiring 1230 secure transport for IPFIX messages use TLS over TCP. 1232 When using TLS over TCP, IPFIX Exporting Processes and Collecting 1233 Processes should behave in all other aspects as if using TCP as the 1234 transport protocol; especially as regards the handling of templates 1235 and template withdrawals. 1237 8.4. Implementing IPFIX over DTLS over UDP 1239 An implementation of the DTLS protocol version 1, described in 1240 [RFC4347] and required to secure IPFIX over UDP, is available in 1241 OpenSSL [OPENSSL] as of version 0.9.8. However, DTLS support is as 1242 of this writing under active development and certain implementations 1243 might be unstable. We recommend extensive testing of DTLS based 1244 IPFIX implementations to build confidence in the DTLS stack over 1245 which your implementation runs. 1247 When using DTLS over UDP, IPFIX Exporting Processes and Collecting 1248 Processes should behave in all other aspects as if using UDP as the 1249 transport protocol; especially as regards the handling of templates 1250 and template timeouts. 1252 Note that the selection of IPFIX message sizes for DTLS over UDP must 1253 account for overhead per packet introduced by the DTLS layer. 1255 8.5. Implementing IPFIX over DTLS over SCTP 1257 As of this writing, there is no publicly available implementation of 1258 DTLS over SCTP as described in [RFC4347] and 1259 [I-D.tuexen-dtls-for-sctp]. 1261 When using DTLS over SCTP, IPFIX Exporting Processes and Collecting 1262 Processes should behave in all other aspects as if using SCTP as the 1263 transport protocol; especially as regards the handling of templates, 1264 the use of reliable transport for template and scope information. 1266 An implementation of the DTLS protocol version 1, described in 1267 [RFC4347] and required to secure IPFIX over SCTP, is available in 1268 OpenSSL [OPENSSL] as of version 0.9.8. However, DTLS support is as 1269 of this writing under active development and certain implementations 1270 might be unstable. We recommend extensive testing of DTLS based 1271 IPFIX implementations to build confidence in the DTLS stack over 1272 which your implementation runs. 1274 9. Extending the Information Model 1276 IPFIX supports two sets of Information Elements: IETF specified 1277 Information Elements and enterprise-specific Information Elements. 1278 New Information Elements can be added to both sets as described in 1279 this section. If an Information Element is considered of general 1280 interest, it should be added to the set of IETF specified Information 1281 Elements that is maintained by IANA. 1283 Alternatively, private enterprises can define proprietary Information 1284 Elements for internal purposes. There are several potential reasons 1285 for doing so. For example, the Information Element might only relate 1286 to proprietary features of a device or protocol of the enterprise. 1287 Also pre-standard product delivery or commercially sensitive product 1288 features might cause the need for enterprise-specific Information 1289 Elements. 1291 The Information Model [I-D.ietf-ipfix-info] document contains an XML- 1292 based specification of Template, abstract data types and IPFIX 1293 Information Elements, which may be used to create consistent machine- 1294 readable extensions to the IPFIX information model. This description 1295 can be used for automatically checking syntactic correctness of the 1296 specification of IPFIX Information Elements and for generating code 1297 that deals with processing IPFIX Information Elements. 1299 9.1. Adding new IETF specified Information Elements 1301 New IPFIX Information Elements that are considered to be of general 1302 interest should be added to the set of IETF specified Information 1303 Elements that is maintained by IANA. 1305 The introduction of new Information Elements in the IANA registry is 1306 subject to expert review. As described in section 7.1 of 1307 [I-D.ietf-ipfix-info] an expert review is performed by one of a group 1308 of experts designated by an IETF Operations and Management Area 1309 Director. The experts will initially be drawn from the Working Group 1310 Chairs and document editors of the IPFIX and PSAMP Working Groups. 1311 The group of experts must double check the Information Elements 1312 definitions with already defined Information Elements for 1313 completeness, accuracy, redundancy, and correct naming following the 1314 naming conventions in [I-D.ietf-ipfix-info] section 2.3. 1316 The specification of new IPFIX Information Elements must use the 1317 template specified in [I-D.ietf-ipfix-info] section 2.1 and must be 1318 published using a well established and persistent publication medium. 1320 9.2. Adding enterprise-specific Information Elements 1322 Enterprises or other organizations holding a registered SMI network 1323 management private enterprise code number can specify enterprise- 1324 specific Information Elements. Their identifiers can be chosen 1325 arbitrarily within the range of 1-32767 and have to be coupled with a 1326 Private Enterprise Identifier [PEN]. Enterprise identifiers MUST be 1327 registered as SMI network management private enterprise code numbers 1328 with IANA. The registry can be found at 1329 http://www.iana.org/assignments/enterprise-numbers. 1331 10. Common Implementation Mistakes 1333 The issues listed in this section were identified during 1334 implementation and interoperability testing. They do not stem from 1335 insufficient clarity in the protocol, but each of these was an actual 1336 mistake made in a tested IPFIX implementation. They are listed here 1337 for the convenience of future implementers. 1339 10.1. IPFIX and Netflow version 9 1341 A large group of mistakes stems from the fact that many implementers 1342 started implementing IPFIX from an existing version of NetFlow 1343 version 9 [RFC3954]. Despite their similarity, the two protocols 1344 differ in many aspects. We list here some of the most important 1345 differences.. 1347 o Transport protocol: NetFlow version 9 initially ran over UDP while 1348 IPFIX must have a congestion aware transport protocol. IPFIX 1349 specifies PR-SCTP as its mandatory protocol, while TCP and UDP are 1350 optional. 1352 o IPFIX differentiates between IETF and non-IETF defined Information 1353 Elements. Non-IETF Information Elements can be specified by 1354 coupling the non IETF Information Element identifier with an 1355 Enterprise ID (corresponding to the vendor that defined the 1356 Information Element). 1358 o Option Templates: in IPFIX, an Option Template must have a scope 1359 and the scope is not allowed to be of length zero. The NetFlow 1360 version 9 specifications [RFC3954] don't specify that the scope 1361 must not be of length zero. 1363 Message header: 1365 o Set ID: Even if the packet headers are different between IPFIX and 1366 NetFlow version 9, some of the fields are used in both of them. 1367 The difference between the two protocols is in the values that 1368 these fields can assume. A typical example is the Set ID values: 1369 the Set ID values of 0 and 1 are used in NetFlow version 9, while 1370 they are not used in IPFIX. 1372 o Length field: in NetFlow version 9, this field (called count) 1373 contains the number of Records. In IPFIX, it indicates the total 1374 length of the IPFIX message, measured in octets (including message 1375 header and Set(s)). 1377 o Timestamp: NetFlow version 9 has an additional timestamp: 1378 sysUpTime. It indicates the time in milliseconds since the last 1379 reboot of the Exporting Process. 1381 o The version number is different. NetFlow version 9 uses the 1382 version number 9 while IPFIX uses the version number 10. 1384 10.2. Padding of the Data Set 1386 [I-D.ietf-ipfix-protocol] specifies that the Exporting Process MAY 1387 insert some octets for set padding to align Data Sets within a 1388 Message. The padding length must be shorter than any allowable 1389 Record in that set. 1391 It is important to respect this limitation: if the padding length is 1392 equal to or longer than the length of the shortest Record, it will be 1393 interpreted as another Record. 1395 An alternative is to use the paddingOctets Information Element in the 1396 Template definition. 1398 10.3. Field ID Numbers 1400 Information Element numbers in IPFIX have the range 0-32767 (0 - 1401 0x7FFF). Information Element numbers outside this range (i.e., with 1402 the high bit set) are taken to be enterprise-specific Information 1403 Elements, which have an additional four-byte Private Enterprise 1404 Number following the Information Element number and length. 1405 Inadvertently setting the high bit of the Information Element number 1406 by selecting a number out of this range will therefore cause template 1407 scanning errors. 1409 10.4. Template ID Numbers 1411 Template IDs are generated as required by the Exporting Process. 1412 When the same set of Information Elements is exported at different 1413 times, the corresponding Template is usually identified by different 1414 Template IDs. Similarly, if multiple co-existing Templates are 1415 composed of the same set of Information Elements, they are also 1416 identified by different Template IDs. The collecting process does 1417 not know in advance which Template ID a particular Template will use. 1419 11. Security Considerations 1421 This document describes the implementation guidelines of IPFIX. The 1422 security requirements for the IPFIX target applications are addressed 1423 in the IPFIX requirements draft [RFC3917]. These requirements are 1424 considered for the specification of the IPFIX protocol, for which a 1425 security considerations section exits [I-D.ietf-ipfix-protocol]. 1427 Section 7 recommends that IPFIX Exporting Processes report internals 1428 about middleboxes. These internals may be security-relevant and the 1429 reported information needs to be protected appropriately for reasons 1430 given below. 1432 Reporting of packets dropped by firewalls and other packet-dropping 1433 middleboxes carries the risk that this information can be used by 1434 attackers for analyzing the configuration of the middlebox and for 1435 developing attacks against it. Address translation may be used for 1436 hiding the network structure behind an address translator. If an 1437 IPFIX Exporting Process reports the translations performed by an 1438 address translator, then parts of the network structure may be 1439 revealed. If an IPFIX Exporting Process reports the translations 1440 performed by an anonymizer, the main function of the anonymizer may 1441 be compromised. 1443 Note that there exist vulnerabilities in DTLS over SCTP as specified 1444 in the IPFIX Protocol, such that a third party could cause messages 1445 to be undetectably lost, or an SCTP Association to shut down. These 1446 vulnerabilities are addressed by [I-D.tuexen-dtls-for-sctp]; however, 1447 it is unclear whether initial OpenSSL-based implementations of DTLS 1448 over SCTP will contain the required fixes. DTLS over SCTP should be 1449 used with caution in production environments until these issues are 1450 completely addressed. 1452 12. IANA Considerations 1454 This document has no actions for IANA. 1456 13. Acknowledgments 1458 We would like to thank the MoMe project for organising two IPFIX 1459 Interoperability Events in July 2005 and in March 2006, and 1460 Fraunhofer Fokus for organising the third one in November 2006. The 1461 Interoperability Events provided us precious input for this document. 1462 Thanks to Brian Trammell for his contributions to the SCTP section 1463 and the security guidelines and for the multiple thorough reviews. 1464 We would also like to thank Benoit Claise, Carsten Schmoll, and 1465 Gerhard Muenz for the technical review and feedback, and Michael 1466 Tuexen Randall Stewart, and Peter Lei for reviewing the SCTP section. 1468 14. References 1470 14.1. Normative References 1472 [I-D.ietf-ipfix-protocol] 1473 Claise, B., "Specification of the IPFIX Protocol for the 1474 Exchange of IP Traffic Flow Information", 1475 draft-ietf-ipfix-protocol-26 (work in progress), 1476 September 2007. 1478 [I-D.ietf-ipfix-info] 1479 Quittek, J., "Information Model for IP Flow Information 1480 Export", draft-ietf-ipfix-info-15 (work in progress), 1481 February 2007. 1483 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1484 Requirement Levels", BCP 14, RFC 2119, March 1997. 1486 14.2. Informative References 1488 [I-D.ietf-ipfix-as] 1489 Zseby, T., "IPFIX Applicability", draft-ietf-ipfix-as-12 1490 (work in progress), July 2007. 1492 [I-D.ietf-ipfix-architecture] 1493 Sadasivan, G., "Architecture for IP Flow Information 1494 Export", draft-ietf-ipfix-architecture-12 (work in 1495 progress), September 2006. 1497 [I-D.ietf-ipfix-reducing-redundancy] 1498 Boschi, E., "Reducing Redundancy in IP Flow Information 1499 Export (IPFIX) and Packet Sampling (PSAMP) Reports", 1500 draft-ietf-ipfix-reducing-redundancy-04 (work in 1501 progress), May 2007. 1503 [I-D.tuexen-dtls-for-sctp] 1504 Tuexen, M., "Datagram Transport Layer Security for Stream 1505 Control Transmission Protocol", 1506 draft-tuexen-dtls-for-sctp-01 (work in progress), 1507 October 2006. 1509 [I-D.ietf-tsvwg-udp-guidelines] 1510 Eggert, L. and G. Fairhurst, "UDP Usage Guidelines for 1511 Application Designers", draft-ietf-tsvwg-udp-guidelines-03 1512 (work in progress), September 2007. 1514 [RFC1305] Mills, D., "Network Time Protocol (Version 3) 1515 Specification, Implementation", RFC 1305, March 1992. 1517 [RFC1924] Elz, R., "A Compact Representation of IPv6 Addresses", 1518 RFC 1924, April 1996. 1520 [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, 1521 S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., 1522 Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, 1523 S., Wroclawski, J., and L. Zhang, "Recommendations on 1524 Queue Management and Congestion Avoidance in the 1525 Internet", RFC 2309, April 1998. 1527 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 1528 "Definition of the Differentiated Services Field (DS 1529 Field) in the IPv4 and IPv6 Headers", RFC 2474, 1530 December 1998. 1532 [RFC3234] Carpenter, B. and S. Brim, "Middleboxes: Taxonomy and 1533 Issues", RFC 3234, February 2002. 1535 [RFC3280] Housley, R., Polk, W., Ford, W., and D. Solo, "Internet 1536 X.509 Public Key Infrastructure Certificate and 1537 Certificate Revocation List (CRL) Profile", RFC 3280, 1538 April 2002. 1540 [RFC3758] Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P. 1541 Conrad, "Stream Control Transmission Protocol (SCTP) 1542 Partial Reliability Extension", RFC 3758, May 2004. 1544 [RFC3917] Quittek, J., Zseby, T., Claise, B., and S. Zander, 1545 "Requirements for IP Flow Information Export (IPFIX)", 1546 RFC 3917, October 2004. 1548 [RFC3954] Claise, B., "Cisco Systems NetFlow Services Export Version 1549 9", RFC 3954, October 2004. 1551 [RFC4346] Dierks, T. and E. Rescorla, "The Transport Layer Security 1552 (TLS) Protocol Version 1.1", RFC 4346, April 2006. 1554 [RFC4347] Rescorla, E. and N. Modadugu, "Datagram Transport Layer 1555 Security", RFC 4347, April 2006. 1557 [RFC4960] Stewart, R., "Stream Control Transmission Protocol", 1558 RFC 4960, September 2007. 1560 [OPENSSL] OpenSSL, "http://www.openssl.org/". 1562 [PEN] IANA Private Enterprise Numbers registry, 1563 "http://www.iana.org/assignments/enterprise-numbers". 1565 Authors' Addresses 1567 Elisa Boschi 1568 Hitachi Europe SAS 1569 Immeuble Le Theleme 1570 1503 Route les Dolines 1571 06560 Valbonne 1572 France 1574 Phone: +33 4 89874100 1575 Email: elisa.boschi@hitachi-eu.com 1576 Lutz Mark 1577 Fraunhofer FOKUS 1578 Kaiserin Augusta Allee 31 1579 10589 Berlin 1580 Germany 1582 Phone: +49 30 34637306 1583 Email: lutz.mark@fokus.fraunhofer.de 1585 Juergen Quittek 1586 NEC Europe Ltd. Network Laboratories 1587 Kurfuersten-Anlage 36 1588 69115 Heidelberg 1589 Germany 1591 Phone: +49 6221 90511-15 1592 Email: quittek@netlab.nec.de 1594 Martin Stiemerling 1595 NEC Europe Ltd. Network Laboratories 1596 Kurfuersten-Anlage 36 1597 69115 Heidelberg 1598 Germany 1600 Phone: +49 6221 90511-13 1601 Email: stiemerling@netlab.nec.de 1603 Paul Aitken 1604 Cisco Systems 1605 96 Commercial Quay 1606 Edinburgh EH6 6LX 1607 Scotland 1609 Phone: +44 131 561 3616 1610 Email: paitken@cisco.com 1612 Full Copyright Statement 1614 Copyright (C) The IETF Trust (2007). 1616 This document is subject to the rights, licenses and restrictions 1617 contained in BCP 78, and except as set forth therein, the authors 1618 retain all their rights. 1620 This document and the information contained herein are provided on an 1621 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 1622 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND 1623 THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS 1624 OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF 1625 THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 1626 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 1628 Intellectual Property 1630 The IETF takes no position regarding the validity or scope of any 1631 Intellectual Property Rights or other rights that might be claimed to 1632 pertain to the implementation or use of the technology described in 1633 this document or the extent to which any license under such rights 1634 might or might not be available; nor does it represent that it has 1635 made any independent effort to identify any such rights. Information 1636 on the procedures with respect to rights in RFC documents can be 1637 found in BCP 78 and BCP 79. 1639 Copies of IPR disclosures made to the IETF Secretariat and any 1640 assurances of licenses to be made available, or the result of an 1641 attempt made to obtain a general license or permission for the use of 1642 such proprietary rights by implementers or users of this 1643 specification can be obtained from the IETF on-line IPR repository at 1644 http://www.ietf.org/ipr. 1646 The IETF invites any interested party to bring to its attention any 1647 copyrights, patents or patent applications, or other proprietary 1648 rights that may cover technology that may be required to implement 1649 this standard. Please address the information to the IETF at 1650 ietf-ipr@ietf.org. 1652 Acknowledgment 1654 Funding for the RFC Editor function is provided by the IETF 1655 Administrative Support Activity (IASA).