idnits 2.17.1 draft-deri-impl-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** The document seems to lack a 1id_guidelines paragraph about 6 months document validity. == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a Security Considerations section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. ** There are 3 instances of too long lines in the document, the longest one being 64 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (July 2004) is 7218 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) No issues found here. Summary: 6 errors (**), 0 flaws (~~), 2 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 IPFIX working group 2 Internet Draft L. Deri 3 Document: draft-deri-impl-00.txt NETikos S.p.A. 4 Expires: January 2005 July 2004 6 IPFIX Implementation Notes 8 Status of this Memo 10 This document is an Internet-Draft and is in full conformance with 11 all provisions of Section 10 of RFC2026. 13 Internet-Drafts are working documents of the Internet Engineering 14 Task Force (IETF), its areas, and its working groups. Note that 15 other groups may also distribute working documents as Internet- 16 Drafts. 18 The list of current Internet-Drafts can be accessed at 19 http://www.ietf.org/ietf/1id-abstracts.txt 20 The list of Internet-Draft Shadow Directories can be accessed at 21 http://www.ietf.org/shadow.html. 23 Abstract 25 Providing an early implementation report is important in order to 26 help validating a standard before its finalization. This document 27 describes the lessons learnt while implementing both an IPFIX probe 28 and collector. Its goals are manifold and include feedback to the 29 IPFIX working group, in addition to suggestions for tackling open 30 protocol issues that can be addressed in future IPFIX drafts. 32 Conventions used in this document 34 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 35 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 36 document are to be interpreted as described in RFC-2119. 38 Table of Contents 40 1. Introduction...................................................2 41 2. IPFIX Implementation...........................................2 42 2.1 General Implementation Notes...............................3 43 2.2 Vendor-specific Extensions.................................4 44 2.3 SCTP Protocol Implementation...............................4 45 3. Implementation Evaluation......................................6 46 4. Open IPFIX Issues..............................................7 47 5. Final Remarks..................................................9 48 6. Code Availability..............................................9 49 References........................................................9 50 Acknowledgments..................................................10 51 Author's Addresses...............................................10 53 1. Introduction 55 The IETF IPFIX working group [ipfix-wg] is working towards the 56 definition of a standard for the export of IP flows. At present a 57 draft of the IPFIX protocol specification has been released. 59 The interest of the author for IPFIX is because in the past years he 60 has developed two open source applications: 61 - A passive traffic monitoring application named ntop [ntop] that 62 can also act as a NetFlow collector. 64 - A libpcap-based [pcap] NetFlow v5/v9 probe named nProbe [nprobe]. 66 In order to both evaluate IPFIX and provide feedback to the working 67 group, the author has extended both applications with IPFIX support 68 so that it is possible to have control over both the probe and the 69 collector. The author believes that this is a typical scenario at 70 least for early IPFIX applications, as they will likely be derived 71 from existing NetFlow applications, considering that IPFIX is very 72 similar to NetFlow v9. For this reason the paper does not cover 73 NetFlow or general flow-based implementation issues. Instead it 74 focuses on the effort required to enhance an existing NetFlow 75 application so that it can also support IPFIX. 77 Since IPFIX uses SCTP [sctp] as default transport, in order to ease 78 the development cycle until SCTP will be supported in all the main 79 operating systems, Linux has been used as reference platform. However 80 this report should be considered platform-neutral, as both probe and 81 collector do not use Linux-only facilities. 83 The author is not aware of any IPFIX implementations so 84 interoperability issues have not been addressed here. 86 2. IPFIX Implementation 88 Implementing IPFIX is rather straightforward if the starting point is 89 a NetFlow v9 application. This is because NetFlow v9 has been used as 90 base for the IPFIX protocol [ipfix-eval]. In a nutshell [ipfix-nf- 91 eval] IPFIX is basically NetFlow v9 over SCTP with little differences 92 such as templates definition when non-IETF defined flow fields are 93 used. 95 As many NetFlow implementations do not support v9 yet, it is worth to 96 spend a few words about the complexity required to enhance an 97 existing NetFlow application (e.g. based on v5) to v9 [nfv9-spec]. 98 The main difference between v5 and v9 is the template concept, in 99 addition to other minor issues such as little changes in the flow 100 header (e.g. the sequence number). Implementing templates is a major 101 issue and it definitively costs more, in terms of code complexity, to 102 move from NetFlow v5 to v9 than from v9 to IPFIX. This is for several 103 reasons: 104 - The probe cannot guess in advance the template formats so all the 105 possible fields combinations are possible. 106 Due to the dynamic flow size, the probe must carefully allocate 107 buffers and make sure that exported flows can fit in a single UDP 108 packet. 110 - Optimizing template transmission is a complex activity in 111 particular when the probe needs to send out flows to various 112 collectors. For this reasons it is recommended that the probe does 113 not send more that a few different flow/template formats otherwise 114 the code complexity can increase significantly. 116 - From the performance point of view, the probe must be able to 117 collect all the possible flow information even if very few fields 118 can be emitted. For instance VLAN tags were not supported in 119 NetFlow v5 so the VLAN tag was ignored. Instead VLANs are supported 120 in V9, so the probe must take them into account even if the VLAN 121 tag will not be exported in the flow. Optimizing these kind of 122 activities is very time consuming and often the results are not 123 that great in term of performance and memory gain. 125 - Collectors must be prepared to handle all the possible flows 126 without breaking their logic. For instance a collector should not 127 assume that the source/destination IP address are contained in the 128 received flow as with previous NetFlow versions. 130 The following sections describe the effort required to implement 131 IPFIX starting from NetFlow v9 according to the role, either probe or 132 collector. 134 2.1 General Implementation Notes 136 This section contains general implementation notes that are applied 137 to both probe and collector. For both sides implementing IPFIX 138 support basically means to: 139 - Implement vendor-specific extensions. 140 - Support the SCTP protocol. 142 IPFIX support has been implemented according to the following 143 documents: [ipfix-info] and [ipfix-spec]. Technical documentation 144 about IPFIX is rather straightforward to undertstand in particular 145 for those who are familiar with NetFlow as IPFIX is often a 146 1:1 mapping with the corresponding NetFlow v9 specification. 148 2.2 Vendor-specific Extensions 150 Contrary to NetFlow v9, IPFIX vendor-specific extensions are 151 specified by means of the vendor specified field type defined using 152 the IANA enterprise number (PEN) [pen]. Although the use of PEN 153 numbers is general (e.g. it has been used into the SNMP MIB II for 154 the sysObjectID [mib2]), it has the drawback of requiring a different 155 template format from IETF-defined fields where a numeric identifier 156 is used. This difference requires that both probe and collector 157 support both formats, with and without the PEN. It should be 158 investigated whether it is feasible to merge both template formats 159 (IETF and vendor-specific extensions) into one, so that the 160 implementation complexity is reduced. For instance the IETF format 161 can be replaced with the vendor-specific one, using an enterprise 162 number for IETF. Although this is a minor issue, it definitively 163 simplifies the development work and troubleshooting. 165 2.3 SCTP Protocol Implementation 167 The SCTP protocol is a reliable, congestion-aware transport protocol 168 based on IP. It sports several features such as acknowledged error- 169 free retransmission of messages, detection of data corruption and 170 duplication. It is basically a general-purpose transport for message- 171 oriented applications such as flow-based applications based on IPFIX. 172 The flow peers handle SCTP very similar to TCP as they need to 173 - Bind to a port. 174 - Connect with a peer. 175 - Start sending data. 176 - Close the connection. 178 The main difference with TCP is that in case of network events such 179 as disconnection of the remote peer, the SCTP layer continues to work 180 even if it reports the problem to the sender. This means that an 181 application does not have to handle reconnection with the remote peer 182 as this is done transparently by the transport when the remote peer 183 is available although some emitted flows might be lost. Still the 184 application must register with SCTP in order to receive events about 185 the underlying network connection. This is because in case of 186 reconnection, the application has to retransmit the template records 187 so it is necessary to be informed about the state of the network 188 transport. 190 From the programming point of view, SCTP is very similar to TCP/UDP 191 so little code needs to be changed while migrating existing 192 applications to SCTP. The main issues with SCTP support are related 193 to the shift from a pure message oriented protocol such as UDP to a 194 connection-oriented protocol. In fact UDP-based applications do not 195 have to keep track of per-connection information nor handle transport 196 messages. This has definitively a major impact on application 197 complexity and performance that depends on the role of the 198 application as described below. 200 Probe Side 201 Probes need to keep track of peer connections that means: 202 - Templates need to be retransmitted per-connection every time a 203 connection is reestablished; using a connectionless protocol such 204 as UDP, this was a periodic operation as there is no simple way to 205 detect whether a collector disconnected. Hence using SCTP, template 206 sending can be optimized, as it is not really necessary to 207 periodically send templates, because it is now possible to detected 208 whether the collector has disconnected. 209 On the other hand as a probe can send flows to several collectors 210 (e.g. in round-robin or as a reflector) it must keep track of per- 211 collector templates transmission. This means that if collector X 212 reconnects, the probe must send the template only to this collector 213 and not to all collectors. 215 - The IPFIX specification allows a probe to open a connection for 216 sending templates and another for sending flows, making the whole 217 setup even trickier. 219 - Transmitting templates is a time consuming activity for both the 220 application and the transport, as they need to be sent quite often. 221 For instance Cisco IOS sends NetFlow v9 templates every 10 flow 222 packets (ip flow-export template refresh-rate IOS parameter 223 [nfv9]). Sending templates using SCTP is a little more costly than 224 with UDP, in particular if IPSec is used, so every transmission 225 that is not necessary should be carefully avoided. Nevertheless, 226 sending templates only when there is a reconnection it 227 significantly reduces the network traffic with respect to a 228 connectionless transport. 230 Collector Side 231 There are no major changes to be supported from the transport point 232 of view with respect to NetFlow v9, beside little coding necessary 233 to support SCTP. 235 As stated before, SCTP is currently supported only on some selected 236 OSs such as Linux and Solaris. At the moment it is not clear whether 237 both popular operating systems such as Windows will ever support it 238 natively (Microsoft has no plan to support SCTP in the current and 239 next Windows version, codename Longhorn), and network manufacturers 240 will implement it on their routers. In addition, being SCTP a totally 241 new protocol, it is unclear what kind of problems it will suffer when 242 deployed in production environments or whether it will be vulnerable 243 to network attacks. This makes the IPFIX future not too bright, as 244 SCTP is the mandatory transport protocol. It is clear that the IPFIX 245 WG is betting a lot on this protocol, however making it mandatory is 246 probably a very strong position that will prevent IPFIX diffusion at 247 least in its early days. 249 3. Implementation Evaluation 251 From the coding point of view, implementing IPFIX without vendor- 252 specific extensions is not much different from NetFlow v9 support. In 253 fact the only major difference is just the transport that has to be 254 handled slightly differently from UDP but that has the advantage of 255 reducing template periodic transmission. 256 In order to evaluate differences between transports, some packets 257 (the sample contained 50'000 packets sent from the same address 258 towards random destinations) have been captured and reproduced using 259 nProbe. In order to evaluate the two transports regardless of the 260 template transmission, templates are sent periodically on both 261 transports, although templates transmission affects the test very 262 little. Note that as explained before, periodic template transmission 263 is not really necessary when SCTP is used, unless there is a 264 transport disconnection/reconnection. The test outcome is shown in 265 the following table. 267 +-------------+-----------+-----------------+ 268 | | UDP | SCTP | 269 +-------------+-----------+-----------------+ 270 | Packets | 1'667 | 2'514 [+150%] | 271 +-------------+-----------+-----------------+ 272 | Bytes | 2'509'540 | 2'595'952 [+3%] | 273 +-------------+-----------+-----------------+ 275 Table 1 - UDP vs. SCTP Network Usage 277 Basically with SCTP the number of packets is significantly increased 278 with respect to UDP. This is because with SCTP the flow collector 279 acknowledges almost immediately all the received packets. However as 280 these acknowledge packets are rather small (62 bytes) they have very 281 little impact on the total amount of data transfer but just on the 282 number of packets sent. Furthermore it is worth to remark that if 283 there is no traffic (i.e. flows are not sent) for some time (the 284 default is 30 seconds in the current Linux 2.6 implementation), the 285 SCTP stack on the probe side sends a heartbeat message that is 286 acknowledged by the collector side. Heartbeat messages are very small 287 (86 bytes) however they influence the amount of traffic exchanged. 289 From the performance point of view, IPFIX is basically as fast as 290 NetFlow v9. This is because there are not many differences between 291 the two protocols. However it has to be remarked that the use of 292 templates has some little performance drawbacks with respect to 293 'static' NetFlow as v5. This is because: 294 - Applications (probes and collectors) know in advance the data 295 format so they can be optimized and tuned in advance. 297 - Several routines that handle templates (definition, transmission, 298 check) are avoided, so there are less per-flow CPU cycles needed. 300 4. Open IPFIX Issues 302 As stated already, IPFIX is very similar to NetFlow v9. On one hand 303 this is an advantage as it allows existing NetFlow v9 applications to 304 be made IPFIX aware with little effort, however on the other hand 305 this is a serious limitation. In fact the NetFlow design has evolved 306 very little since its introduction in 90s, and most of the changes 307 are minor and often due to the need to support new features while 308 maintaining a wide compatibility with past versions. This explains 309 for instance why NetFlow v7 is very similar to v1, and because the 310 first Cisco IOS implementation of v9 is basically a "port" of v5 over 311 templates. 312 On the other hand, IPFIX is a brand new effort to define a standard 313 flow-based protocol that should capitalize on experience and lessons 314 learnt after ten years of NetFlow. This obviously did not fully 315 happen for many reasons, as this protocol is far too similar to v9 316 and not really introduces new features. The following list summarizes 317 some of the missing features that according to the author should be 318 introduced before the protocol is finalized: 319 - As with NetFlow, the IPFIX flow format is fixed. Consider the 320 following scenario: a user requires in-depth ICMP flow analysis 321 (e.g. ICMP code and type). There are basically two solutions: 322 either a probe/collector agree on two different templates (one for 323 ICMP and one for non-ICMP flows) or they agree on a common template 324 for all protocols with the outcome that some flow fields are empty 325 when non-ICMP flows are emitted. The ideal solution to this kind of 326 scenario is dynamic flows, where emitted flows can have all the 327 template fields or a subset of them according to the flow kind. 328 This means that non-ICMP flows will not have ICMP code and type, 329 that instead will be present on ICMP flows that will lack source 330 and destination ports, meaningless for ICMP. Unfortunately due to 331 the way templates and flows are defined, it is not possible to 332 simply handle dynamic flows, unless an "inline" template is 333 embedded in flows as with nFlow [nflow]. nFlow has no concept of 334 template and it embeds in flows the field identifier. This practice 335 slightly increases the flow size, however thanks to both on-the-fly 336 flow compression and lack of sending templates, the volume of data 337 transmitted is significantly reduced. This feature will save 338 processing time and bandwidth while enabling probes and collectors 339 to have a rich template format. 341 - IPFIX templates specify just the format of flows and cannot 342 modify the flow header. This means that it is not possible to 343 define flow header attributes. For instance it is not possible to 344 define attributes like MD5 flow message checksum for non- 345 repudiation of flows, or attributes that can be transported in 346 piggy-back instead of putting them into the options template. 347 Adding this capability would enable developers to have the same 348 level of freedom in both header and flow content. 350 - The SCTP protocol is definitively an enhancement over UDP, 351 although it has some severe drawbacks in terms of number of packets 352 transmitted. The main issue is that every packet is acknowledged 353 almost immediately, that can for instance significantly amplify the 354 effect of a DoS attack. Note that NetFlow amplifies the effects of 355 a DoS when small packets are used, because NetFlow flows are larger 356 than packets used for the attack. It should be explored whether 357 SCTP ACKs can be reduced for instance using mechanisms like those 358 used by TCP, in order to reduce the (almost) 1:1 traffic flow (one 359 ACK for each flow received) from the collector to the probe. 360 Finally it should be explored whether it still makes sense to have 361 a flow identifier in the flow header when SCTP is taking care of 362 issues such as flow retransmission or duplicate discard. 364 - IPFIX does not specify a complete, two-way protocol but just a 365 one-way (from probe to collector) format for exported flows. The 366 drawback is that the collector has no way to query the probe 367 configuration or to modify it. This statement is true unless an 368 offline protocol (e.g. via the CLI interface) is used. The probe to 369 inform the collector uses the options template/flow, although this 370 is not an effective solution as: 371 - It does not allow the collector to query the probe but it is the 372 probe that decides when to inform the collector and what to 373 report. 375 - It is used to periodically send information that could be 376 discarded by the collector. It would make more sense to allow the 377 collector to query the probe when needed. 378 It is not rich enough to let the probe inform the collector about 379 issues such as lack of memory for handling incoming flows. 381 A possible solution could either be the approach used in other flow 382 protocols (e.g. sFlow [sflow]) that defines an SNMP MIB implemented 383 by the probe, or enrich the IPFIX specification with a real two-way 384 protocol. 386 5. Final Remarks 387 This document reported about implementing IPFIX in both a probe and a 388 collector. It covers issues encountered during the implementation and 389 describes open issues that can be tackled before IPFIX is released 390 that include: 391 - Limited innovation, if any, with respect to existing technologies 392 such as NetFlow. 394 - Use of different data formats for IETF and non-IETF information. 396 - Use of SCTP requires additional coding and is not supported in 397 all platforms. 399 - Lack of configuration and monitoring that can be solved through 400 an SNMP MIB. 402 This paper has also proved that the IPFIX protocol is feasible to 403 implement in existing NetFlow applications and shown two open source 404 applications that can be used by those interested in evaluating IPFIX 405 and developing IPFIX-aware applications. 407 6. Code Availability 408 The applications described in this document are distributed under the 409 GPL2 license and available from the ntop home page 410 (http://www.ntop.org/). 412 References 414 [ipfix-wg] IETF, "IP Flow Information Export Working Group", 415 http://www.ietf.org/html.charters/ipfix-charter.html. 417 [ipfix-eval] S. Leinen, "Evaluation of Candidate Protocols for IP Flow 418 Information Export (IPFIX)", draft-leinen-ipfix-eval-contrib-02.txt, 419 January 2003. 421 [ipfix-info] P. Calato and others, "Information Model for IP Flow 422 Information Export", draft-ietf-ipfix-info-03.txt, February 2004. 424 [ipfix-nf-eval] B. Claise, "Evaluation Of NetFlow Version 9 Against 425 IPFIX Requirements", draft-claise-ipfix-eval-netflow-04.txt, February 426 2003. 428 [ipfix-spec] B. Claise and others, "IPFIX Protocol Specifications", 429 draft-ietf-ipfix-protocol-03.txt, January 2003. 431 [mib2] K. McCloghrie, and M. Rose, "Management Information Base for 432 Network Management of TCP/IP-based internets: MIB-II", RFC 1213, 433 March 1991. 435 [nfv9] Cisco Systems Inc., "NetFlow v9 Data Format", 436 http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/ 437 120newft/120limit/120s/120s24/nfv9expf.htm, 2003. 439 [nfv9-spec] B. Claise, "Cisco Systems NetFlow Services Export Version 440 9", draft-claise-netflow-9-08.txt, April 2004. 442 [ntop] L. Deri and others, "Monitoring Networks using ntop", 443 Proceedings of IM 2001, Seattle, May 2001. 445 [nprobe] L. Deri, "nProbe: an Open Source NetFlow Probe for Gigabit 446 Networks", Proceedings of Terena TNC 2003, Zagreb, May 2003. 448 [nflow] L. Deri, "nFlow", http://www.ntop.org/nFlow/, March 2003. 450 [pen] IANA, "Private Enterprise Numbers Registry", 451 http://www.iana.org/assignments/enterprise-numbers 453 [sctp] R. Stewart and others, "Stream Control Transmission Protocol 454 (SCTP)", RFC 2960, October 2000. 456 [sflow] P. Phaal and others, "sFlow: A Method for Monitoring Traffic 457 in Switched and Routed Networks", RFC 3176, September 2001. 459 Acknowledgments 461 This work was supported in part by the IST project SCAMPI (IST-2001- 462 32404) funded by the European Union. The author would like to thank 463 Fulvio Risso and Simon Leinen 464 for the valuable discussions and 465 suggestions. 467 Author's Addresses 469 Luca Deri 470 NETikos S.p.A. 471 Via del Brennero km 4 472 56123 Pisa 473 Italy 474 Phone: +39 050 80091 475 Email: luca.deri@netikos.com