idnits 2.17.1 draft-irtf-nmrg-snmp-measure-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 15. -- Found old boilerplate from RFC 3978, Section 5.5, updated by RFC 4748 on line 1030. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 1041. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1048. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1054. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (September 17, 2008) is 5699 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: '0-1' is mentioned on line 664, but not defined == Missing Reference: '1-3' is mentioned on line 664, but not defined == Missing Reference: '0-9' is mentioned on line 678, but not defined == Missing Reference: '1-9' is mentioned on line 664, but not defined == Missing Reference: '0-4' is mentioned on line 678, but not defined == Missing Reference: '0-5' is mentioned on line 678, but not defined == Missing Reference: '6-9' is mentioned on line 678, but not defined == Missing Reference: '3-9' is mentioned on line 678, but not defined Summary: 1 error (**), 0 flaws (~~), 9 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NMRG J. Schoenwaelder 3 Internet-Draft Jacobs University Bremen 4 Intended status: Informational September 17, 2008 5 Expires: March 21, 2009 7 SNMP Traffic Measurements and Trace Exchange Formats 8 draft-irtf-nmrg-snmp-measure-06.txt 10 Status of this Memo 12 By submitting this Internet-Draft, each author represents that any 13 applicable patent or other IPR claims of which he or she is aware 14 have been or will be disclosed, and any of which he or she becomes 15 aware will be disclosed, in accordance with Section 6 of BCP 79. 17 Internet-Drafts are working documents of the Internet Engineering 18 Task Force (IETF), its areas, and its working groups. Note that 19 other groups may also distribute working documents as Internet- 20 Drafts. 22 Internet-Drafts are draft documents valid for a maximum of six months 23 and may be updated, replaced, or obsoleted by other documents at any 24 time. It is inappropriate to use Internet-Drafts as reference 25 material or to cite them other than as "work in progress." 27 The list of current Internet-Drafts can be accessed at 28 http://www.ietf.org/ietf/1id-abstracts.txt. 30 The list of Internet-Draft Shadow Directories can be accessed at 31 http://www.ietf.org/shadow.html. 33 This Internet-Draft will expire on March 21, 2009. 35 Copyright Notice 37 Copyright (C) The IETF Trust (2008). 39 Abstract 41 The Simple Network Management Protocol (SNMP) is widely deployed to 42 monitor, control and (sometimes also) configure network elements. 43 Even though the SNMP technology is well documented, it remains 44 relatively unclear how SNMP is used in practice and what typical SNMP 45 usage patterns are. 47 This document describes an approach to carrying out large scale SNMP 48 traffic measurements in order to develop a better understanding how 49 SNMP is used in real world production networks. It describes the 50 motivation, the measurement approach, and the tools and data formats 51 needed to carry out such a study. 53 This document was produced within the IRTF's Network Management 54 Research Group (NMRG) and represents the consensus of all of the 55 active contributors to this group. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 60 2. Measurement Approach . . . . . . . . . . . . . . . . . . . . . 4 61 2.1. Capturing Traffic Traces . . . . . . . . . . . . . . . . . 5 62 2.2. Converting Traffic Traces . . . . . . . . . . . . . . . . 6 63 2.3. Filtering Traffic Traces . . . . . . . . . . . . . . . . . 7 64 2.4. Storing Traffic Traces . . . . . . . . . . . . . . . . . . 7 65 2.5. Analyzing Traffic Traces . . . . . . . . . . . . . . . . . 8 66 3. Analysis of Traffic Traces . . . . . . . . . . . . . . . . . . 8 67 3.1. Basic Statistics . . . . . . . . . . . . . . . . . . . . . 9 68 3.2. Periodic vs. Aperiodic Traffic . . . . . . . . . . . . . . 9 69 3.3. Message Size and Latency Distributions . . . . . . . . . . 9 70 3.4. Concurrency Levels . . . . . . . . . . . . . . . . . . . . 10 71 3.5. Table Retrieval Approaches . . . . . . . . . . . . . . . . 10 72 3.6. Trap-Directed Polling - Myths or Reality? . . . . . . . . 10 73 3.7. Popular MIB Definitions . . . . . . . . . . . . . . . . . 10 74 3.8. Usage of Obsolete Objects . . . . . . . . . . . . . . . . 11 75 3.9. Encoding Length Distributions . . . . . . . . . . . . . . 11 76 3.10. Counters and Discontinuities . . . . . . . . . . . . . . . 11 77 3.11. Spin Locks . . . . . . . . . . . . . . . . . . . . . . . . 11 78 3.12. Row Creation . . . . . . . . . . . . . . . . . . . . . . . 12 79 4. Trace Exchange Formats . . . . . . . . . . . . . . . . . . . . 12 80 4.1. XML Representation . . . . . . . . . . . . . . . . . . . . 12 81 4.2. CSV Representation . . . . . . . . . . . . . . . . . . . . 17 82 5. Security Considerations . . . . . . . . . . . . . . . . . . . 18 83 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 84 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 19 85 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 20 86 8.1. Normative References . . . . . . . . . . . . . . . . . . . 20 87 8.2. Informative References . . . . . . . . . . . . . . . . . . 20 88 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 22 89 Intellectual Property and Copyright Statements . . . . . . . . . . 23 91 1. Introduction 93 The Simple Network Management Protocol (SNMP) was introduced in the 94 late 1980s [RFC1052] and has since then evolved to what is known 95 today as the SNMP version 3 Framework (SNMPv3) [RFC3410]. While SNMP 96 is widely deployed, it is not clear what protocol versions are being 97 used, which protocol features are being used, how SNMP usage differs 98 in different types of networks or organizations, which information is 99 frequently queried, and what typical SNMP interactions patterns occur 100 in real world production networks. 102 There have been several publications in the recent past dealing with 103 the performance of SNMP in general [SM99][Mal02][Pat01], the impact 104 of SNMPv3 security [DSR01][CT04], or the relative performance of SNMP 105 compared to Web Services [PDMQ04][PFGL04]. While these papers are 106 generally useful to better understand the impact of various design 107 decisions and technologies, some of these papers lack a strong 108 foundation because authors typically assume certain SNMP interaction 109 patterns without having experimental evidence that the assumptions 110 are correct. In fact, there are many speculations on how SNMP is 111 being used in real world production networks and performance 112 comparisons are based on limited test cases, but no systematic 113 measurements have been performed and published so far. 115 Many authors use the ifTable of the IF-MIB [RFC2863] or the 116 tcpConnTable of the TCP-MIB [RFC4022] as a starting point for their 117 analysis and comparison. Despite the fact that there is no evidence 118 that operations on these tables dominate SNMP traffic, it is even 119 more unclear how these tables are read and which optimizations are 120 done (or not done) by real world applications. It is also unclear 121 what the actual traffic trade-off between periodic polling and more 122 aperiodic bulk data retrieval is. Furthermore, we do not generally 123 understand how much traffic is devoted to standardized MIB objects 124 and how much traffic deals with proprietary MIB objects and whether 125 the operation mix between these object classes differs between 126 different operational environments (e.g., backbone networks, access 127 networks, enterprise networks). 129 This document recommends an approach to collecting, codifying, and 130 handling SNMP traffic traces in order to find answers to some of 131 these questions. It describes the tools that have been developed to 132 allow network operators to collect traffic traces and to share them 133 with research groups interested in analyzing and modeling network 134 management interactions. 136 While the SNMP trace collection and analysis effort was initiated by 137 the research community, network operators can benefit from the SNMP 138 measurements too. Several new tools are being developed as part of 139 this effort that can be used to capture and analyze the traffic 140 generated by management stations. This resulting information can 141 then be used to improve the efficiency and scalability of management 142 systems. 144 The measurement approach described in this document is by design 145 limited to the study of SNMP traffic. Studies of other management 146 protocols or the impact of management protocols such as SNMP on other 147 traffic sharing the same network resources is left to future efforts. 149 This is an informational document, produced within the IRTF's Network 150 Management Research Group (NMRG) and represents the consensus of all 151 of the active contributors to this group. 153 2. Measurement Approach 155 This section outlines the process of doing SNMP traffic measurements 156 and analysis. The process consists of the following five basic 157 steps: 159 1. Capture raw SNMP traffic traces in pcap packet capture files [1]. 160 2. Convert the raw traffic traces into a structured machine and 161 human readable format. A suitable XML schema has been developed 162 for this purpose which captures all SNMP message details. 163 Another more compact comma separated values (CSV) format has been 164 developed which only keeps key information about SNMP messages. 165 3. Filter the converted traffic traces to hide or anonymize 166 sensitive information. While the filtering is conceptually a 167 separate step, filtering may actually be implemented as part of 168 the previous data conversion step for efficiency reasons. 169 4. Submit the filtered traffic traces to a repository from where 170 they can be retrieved and analyzed. Such a repository may be 171 public, it may be under the control of a research group, or it 172 may be under the control of a network operator who commits to run 173 analysis scripts on the repository on behalf of researchers. 174 5. Analyze the traces by creating and executing analysis scripts 175 which extract and aggregate information. 177 Several of the steps listed above require the involvement of network 178 operators supporting the SNMP measurement projects. In many cases, 179 the filtered XML and CSV representation of the SNMP traces will be 180 the interface between the researchers writing analysis scripts and 181 the operators involved in the measurement activity. It is therefore 182 important to have a well defined specification of these interfaces. 184 This section provides some advice and concrete hints how the steps 185 listed above can be carried out efficiently. Some special tools have 186 been developed to assist network operators and researchers so that 187 the time spent on supporting SNMP traffic measurement projects is 188 limited. The following sections describe the five steps and some 189 tools in more detail. 191 2.1. Capturing Traffic Traces 193 Capturing SNMP traffic traces can be done using packet sniffers such 194 as tcpdump [2], wireshark [3], or similar applications. Some care 195 must be taken to specify pcap filter expressions that match the SNMP 196 transport endpoints used to carry SNMP traffic (typically 'udp and 197 (port 161 or port 162)'). Furthermore, it is necessary to ensure 198 that full packets are captured, that is packets are not truncated 199 (tcpdump option -s 0). Finally, it is necessary to carefully select 200 the placement of the capturing probe within the network. Especially 201 on bridged LANs, it is important to ensure that all management 202 traffic is captured and that the probe has access to all virtual LANs 203 carrying management traffic. This usually requires placing the 204 probe(s) close to the management system(s) and to configure dedicated 205 monitoring ports on bridged networks. Some bridges have restrictions 206 concerning their monitoring capabilities and this should be 207 investigated and documented where necessary. 209 It is recommended to capture at least a full week of data to capture 210 diurnal patterns and one cycle of weekly behavior. Operators are 211 strongly encouraged to capture traces over even longer periods of 212 time. Tools such as tcpslice [2] or pcapmerge [4] can be used to 213 split or merge pcap trace files as needed. 215 Several operating systems can offload some of the TCP/IP processing 216 such as the calculation of transport layer checksum to network 217 interface cards. Traces that include traffic to/from a capturing 218 interface which supports TCP/IP offloading can include incorrect 219 transport layer checksums. The simplest solution is of course to 220 turn checksum offloading off while capturing traces (if that is 221 feasible without losing too many packets). The other solution is to 222 correct or ignore checksums during the subsequent conversion of the 223 raw pcap files. 225 It is important to note that the raw pcap files should ideally be 226 kept in permanent storage (e.g., compressed and encrypted on a CD ROM 227 or DVD). To verify measurements, it might be necessary to go back to 228 the original pcap files if, for example, bugs in the tools described 229 below have been detected and fixed. 231 For each captured trace, some meta data should be recorded and made 232 available. The meta data should include information such as where 233 the trace was collected (name of the network and name of the 234 organization owning the network, description of the measurement point 235 in the network topology where the trace was collected), when it was 236 collected, contact information, the size of the trace, any known 237 special events, equipment failures, or major infrastructure changes 238 during the data collection period and so on. It is also extremely 239 useful to provide a unique identification. There are special online 240 services such as DatCat [5] where meta data can be stored and which 241 provide unique identifiers. 243 2.2. Converting Traffic Traces 245 Raw traces in pcap format must be converted into a format that is 246 human readable while remaining also machine readable for efficient 247 post-processing. Human readability makes it easy for an operator to 248 verify that no sensitive data is left in a trace while machine 249 readability is needed to efficiently extract relevant information. 251 The natural choice here is to use an XML format since XML is human as 252 well as machine readable and there are many tools and high-level 253 scripting language application programming interfaces (APIs) that can 254 be used to process XML documents and to extract meaningful 255 information. However, XML is also pretty verbose which increases 256 processing overhead. In particular, the usage of XML streaming APIs 257 is strongly suggested since APIs that require an in memory 258 representation of XML documents do not handle large traces well. 260 Section 4.1 of this document defines a RELAX NG schema [OASISRNG] for 261 representing SNMP traffic traces in XML. The schema captures all 262 relevant details of an SNMP message in the XML format. Note that the 263 XML format retains some information about the original ASN.1/BER 264 encoding to support message size analysis. 266 A lightweight alternative to the full blown XML representation based 267 on comma separated values (CSV) is defined in Section 4.2. The CSV 268 format only captures selected parts of SNMP messages and is thus more 269 compact and faster to process. 271 As explained in the previous sections, analysis programs which 272 process raw pcap files should have an option to ignore incorrect 273 checksums caused by TCP/IP offloading. In addition, analysis 274 programs which process raw pcap files should be able to perform IP 275 reassembly for SNMP messages that were fragmented at the IP layer. 277 The snmpdump [6] package has been developed to convert raw pcap files 278 into XML and CSV format. The snmpdump program reads pcap, XML, or 279 CSV files as input and produces XML files or CSV files as output. 280 Specific elements can be filtered as required to protect sensitive 281 data. 283 2.3. Filtering Traffic Traces 285 Filtering sensitive data (e.g., access control lists or community 286 strings) can be achieved by manipulating the XML representation of an 287 SNMP trace. Standard XSLT processors (e.g., xsltproc [7]) can be 288 used for this purpose. People familiar with the scripting language 289 Perl might be interested in choosing a suitable Perl module to 290 manipulate XML documents [8]. 292 The snmpdump program, for example, can filter out sensitive 293 information, e.g., by deleting or clearing all XML elements whose 294 name matches a regular expression. Data type specific anonymization 295 transformations that maintain lexicographic ordering for values that 296 appear in instance identifiers [HS06] can be applied. Note that 297 anonymization transformations are often bound to an initialization 298 key and depend on the data being anonymized in an anonymization run. 299 As a consequence, users must be careful when they merge data from 300 independently anonymized traces. More information about network 301 traffic trace anonymization techniques can be found in [XFA02], 302 [FXAM04], [PAPL06], and [RW07]. 304 2.4. Storing Traffic Traces 306 The raw pcap traces as well as the XML / CSV formatted traces should 307 be stored in a stable archive or repository. Such an archive or 308 repository might either be maintained by research groups (e.g., the 309 NMRG) or by network operators or both. It is of key importance that 310 captured traces are not lost or modified as they may form the basis 311 of future research projects and may also be needed to verify 312 published research results. Access to the archive might be 313 restricted to those who have signed some sort of a non-disclosure 314 agreement. 316 While this document recommends that raw traces should be kept, it 317 must be noted that there are situations where this may not be 318 feasible. The recommendation to keep raw traces may be ignored for 319 example to comply with data protection laws or to protect a network 320 operator from being forced to provide the data to other 321 organizations. 323 Lossless compression algorithms embodied in programs such as gzip or 324 bzip2 can be used to compress even large trace files down to a size 325 where they can be burned on DVDs for cheap long-term storage. 327 It must be stressed again that it is important to keep the original 328 pcap traces in addition to the XML / CSV formatted traces since the 329 pcap traces are the most authentic source of information. 330 Improvements in the tool chain may require going back to the original 331 pcap traces and rebuilding all intermediate formats from them. 333 2.5. Analyzing Traffic Traces 335 Scripts that analyze traffic traces must be verified for correctness. 336 Ideally, all scripts used to analyze traffic traces will be 337 publically accessible so that third parties can verify them. 338 Furthermore, sharing scripts will enable other parties to repeat an 339 analysis on other traffic traces and to extend such analysis scripts. 340 It might be useful to establish and common, versioning repository for 341 analysis scripts. 343 Due to the availability of XML parsers and the simplicity of the CSV 344 format, trace files can be processed with tools written in almost any 345 programming language. However, in order to facilitate a common 346 vocabulary and to allow operators to easily read scripts they execute 347 on trace files, it is suggested that analysis scripts be written is 348 scripting languages such as Perl using suitable Perl modules to 349 manipulate XML documents . 350 Using a scripting language such as Perl instead of system programming 351 languages such as C or C++ has the advantage of reducing development 352 time and making scripts more accessible to operators who may want to 353 verify scripts before running them on trace files which may contain 354 sensitive data. 356 The snmpdump tool provides an API to process SNMP messages in C/C++. 357 While the coding of trace analysis programs in C/C++ should in 358 general be avoided for code readability, verifiability and 359 portability reasons, using C/C++ might be the only option in dealing 360 with very large traces efficiently. 362 Any results produced by analyzing a trace must be interpreted in the 363 context of the trace. The nature of the network, the attachment 364 point used to collect the trace, the nature of the applications 365 generating SNMP traffic, or the events that happened while the trace 366 was collected clearly influence the result. It is therefore 367 important to be careful when drawing general conclusions based on a 368 potentially (too) limited data set. 370 3. Analysis of Traffic Traces 372 This section discusses several questions that can be answered by 373 analyzing SNMP traffic traces. The questions raised in the following 374 subsections are meant to be illustrative and no attempt has been made 375 to provide a complete list. 377 3.1. Basic Statistics 379 Basic statistics cover things such as 380 o protocol version used, 381 o protocol operations used, 382 o message size distribution, 383 o error message type frequency, or 384 o usage of authentication and encryption mechanisms. 385 The OID names of the objects manipulated can be categorized into OID 386 subtrees, for example to identify 'standardized', 'proprietary', and 387 'experimental' objects. 389 3.2. Periodic vs. Aperiodic Traffic 391 SNMP is used to periodically poll devices as well as to retrieve 392 information at the request of an operator or application. The 393 periodic polling leads to periodic traffic patterns while on-demand 394 information retrieval causes more aperiodic traffic patterns. It is 395 worthwhile to understand what the relationship is between the amount 396 of periodic and aperiodic traffic. It will be interesting to 397 understand whether there are multiple levels of periodicity at 398 different time scales. 400 Periodic polling behavior may be dependent on the application and 401 polling engine it uses. For example, some management platforms allow 402 applications to specify how long polled values may be kept in a cache 403 before they are polled again. Such optimizations needs to be 404 considered when analyzing traces for periodic and aperiodic traffic. 406 3.3. Message Size and Latency Distributions 408 SNMP messages are size constrained by the transport mappings used and 409 the buffers provided by the SNMP engines. For the further evolution 410 of the SNMP framework, it would be useful to know what the actual 411 message size distributions are. It would be useful to understand the 412 latency distributions, especially the distribution of the processing 413 times by SNMP command responders. Some SNMP implementations 414 approximate networking delays by measuring request-response times and 415 it would be useful to understand to what extent this is a viable 416 approach. 418 Some SNMP implementations update their counters from the underlying 419 instrumentation following adaptive algorthms, not necessarily 420 periodically, and not necessarily on-demand. The granularity of 421 internal counter updates may impact latency measurements and should 422 be taken into account. 424 3.4. Concurrency Levels 426 SNMP allows management stations to retrieve information from multiple 427 agents concurrently. It will be interesting to identify what the 428 typical concurrency level is that can be observed on production 429 networks or whether management applications prefer more sequential 430 ways of retrieving data. 432 Furthermore, it will be interesting to analyze how many redundant 433 requests coming from applications are processed almost simultaneously 434 by a device. The concurrency level and the amount of redundant 435 requests has implications on caching strategies employed by SNMP 436 agents. 438 3.5. Table Retrieval Approaches 440 Tables can be read in several different ways. The simplest and most 441 inefficient approach is to retrieve tables object-by-object in 442 column-by-column order. More advanced approaches try to read tables 443 row-by-row or even multiple-rows-by-multiple-rows. The retrieval of 444 index elements can be suppressed in most cases or only a subset of 445 columns of a table are retrieved. It will be useful to know which of 446 these approaches are used on production networks since this has a 447 direct implication on agent implementation techniques and caching 448 strategies. 450 3.6. Trap-Directed Polling - Myths or Reality? 452 SNMP is built around a concept called trap-directed polling. 453 Management applications are responsible to periodically poll SNMP 454 agents to determine their status. SNMP agents can in addition send 455 traps to notify SNMP managers about events so that SNMP managers can 456 adapt their polling strategy and basically react faster than normal 457 polling would allow to do. 459 Analysis of SNMP traffic traces can identify whether trap-directed 460 polling is actually deployed. In particular, the question that 461 should be addressed is whether SNMP notifications lead to changes in 462 the short-term polling behavior of management stations. In 463 particular, it should be investigated to what extent SNMP managers 464 use automated procedures to track down the meaning of the event 465 conveyed by an SNMP notification. 467 3.7. Popular MIB Definitions 469 An analysis of object identifier prefixes can identify the most 470 popular MIB modules and the most important object types or 471 notification types defined by these modules. Such information would 472 be very valuable for the further maintenance and development of these 473 and related MIB modules. 475 3.8. Usage of Obsolete Objects 477 Several objects from the early days have been obsoleted because they 478 cannot properly represent today's networks. A typical example is the 479 ipRouteTable which was obsoleted because it was not able to represent 480 classless routing, introduced and deployed on the Internet in 1993. 481 Some of these obsolete objects are still mentioned in popular 482 publications as well as research papers. It will be interesting to 483 find out whether they are also still used by management applications 484 or whether management applications have been updated to use the 485 replacement objects. 487 Depending on the data recorded in a trace, it might be possible to 488 determine the age of devices by looking at values of objects such as 489 sysObjectID and sysDecr [RFC3418]. The age of a device can then be 490 taken into consideration when analyzing the use of obsolete and 491 deprecated objects. 493 3.9. Encoding Length Distributions 495 It will be useful to understand the encoding length distributions for 496 various data types. Assumptions about encoding length distributions 497 are sometimes used to estimate SNMP message sizes in order to meet 498 transport and buffer size constraints. 500 3.10. Counters and Discontinuities 502 Counters can experience discontinuities [RFC2578]. A widely used 503 discontinuity indicator is the sysUpTime scalar of the SNMPv2-MIB 504 [RFC3418], which can be reset through a warm start to indicate 505 counter discontinuities. Some MIB modules introduce more specific 506 discontinuity indicators, e.g., the ifCounterDiscontinuityTime of the 507 IF-MIB [RFC2863]. It will be interesting to study to what extent 508 these objects are actually used by management applications to handle 509 discontinuity events. 511 3.11. Spin Locks 513 Cooperating command generators can use advisory locks to coordinate 514 their usage of SNMP write operations. The snmpSetSerialNo scalar of 515 the SNMPv2-MIB [RFC3418] is the default course-grain coordination 516 object. It will be interesting to find out whether there are command 517 generators which coordinate themselves using these spin locks. 519 3.12. Row Creation 521 Row creation is an operation not natively supported by the protocol 522 operations. Instead, conceptual tables supporting row creation 523 typically provide a control column which uses the RowStatus textual 524 convention defined in the SNMPv2-TC [RFC2579] module. The RowStatus 525 itself supports different row creation modes, namely createAndWait 526 (dribble-mode) and createAndGo (one-shot mode). Different approaches 527 can be used to derive the instance identifier if it does not have 528 special semantics associated. It will be useful to study which of 529 the various row creation approaches are actually used by management 530 applications on production networks. 532 4. Trace Exchange Formats 534 4.1. XML Representation 536 The XML format has been designed to keep all information associated 537 with SNMP messages. The format is specified in RELAX NG compact 538 notation [OASISRNC]. Freely available tools such as trang [9] can be 539 used to convert RELAX NG compact syntax to other XML schema 540 notations. 542 The XML format can represent SNMPv1, SNMPv2c, and SNMPv3 messages. 543 In case a new version of SNMP is introduced in the future or existing 544 SNMP versions are extended in ways that require changes to the XML 545 format, a new XML format with a different namespace needs to be 546 defined (e.g., by incrementing the version number included in the 547 namespace URI). 549 # Relax NG grammar for the XML SNMP trace format. 550 # 551 # Published as part of RFC XXXX. 553 default namespace = "urn:ietf:params:xml:ns:snmp-trace-1.0" 555 start = 556 element snmptrace { 557 packet.elem* 558 } 560 packet.elem = 561 element packet { 562 element time-sec { xsd:unsignedInt }, 563 element time-usec { xsd:unsignedInt }, 564 element src-ip { ipaddress.type }, 565 element src-port { xsd:unsignedInt }, 566 element dst-ip { ipaddress.type }, 567 element dst-port { xsd:unsignedInt }, 568 snmp.elem 569 } 571 snmp.elem = 572 element snmp { 573 length.attrs?, 574 message.elem 575 } 577 message.elem = 578 element version { length.attrs, xsd:int }, 579 element community { length.attrs, xsd:hexBinary }, 580 pdu.elem 582 message.elem |= 583 element version { length.attrs, xsd:int }, 584 element message { 585 length.attrs, 586 element msg-id { length.attrs, xsd:unsignedInt }, 587 element max-size { length.attrs, xsd:unsignedInt }, 588 element flags { length.attrs, xsd:hexBinary }, 589 element security-model { length.attrs, xsd:unsignedInt } 590 }, 591 usm.elem?, 592 element scoped-pdu { 593 length.attrs, 594 element context-engine-id { length.attrs, xsd:hexBinary }, 595 element context-name { length.attrs, xsd:string }, 596 pdu.elem 597 } 599 usm.elem = 600 element usm { 601 length.attrs, 602 element auth-engine-id { length.attrs, xsd:hexBinary }, 603 element auth-engine-boots { length.attrs, xsd:unsignedInt }, 604 element auth-engine-time { length.attrs, xsd:unsignedInt }, 605 element user { length.attrs, xsd:hexBinary }, 606 element auth-params { length.attrs, xsd:hexBinary }, 607 element priv-params { length.attrs, xsd:hexBinary } 608 } 610 pdu.elem = 611 element trap { 612 length.attrs, 613 element enterprise { length.attrs, oid.type }, 614 element agent-addr { length.attrs, ipv4address.type }, 615 element generic-trap { length.attrs, xsd:int }, 616 element specific-trap { length.attrs, xsd:int }, 617 element time-stamp { length.attrs, xsd:int }, 618 element variable-bindings { length.attrs, varbind.elem* } 619 } 621 pdu.elem |= 622 element (get-request | get-next-request | get-bulk-request | 623 set-request | inform-request | snmpV2-trap | 624 response | report) { 625 length.attrs, 626 element request-id { length.attrs, xsd:int }, 627 element error-status { length.attrs, xsd:int }, 628 element error-index { length.attrs, xsd:int }, 629 element variable-bindings { length.attrs, varbind.elem* } 630 } 632 varbind.elem = 633 element varbind { length.attrs, name.elem, value.elem } 635 name.elem = 636 element name { length.attrs, oid.type } 638 value.elem = 639 element null { length.attrs, empty } | 640 element integer32 { length.attrs, xsd:int } | 641 element unsigned32 { length.attrs, xsd:unsignedInt } | 642 element counter32 { length.attrs, xsd:unsignedInt } | 643 element counter64 { length.attrs, xsd:unsignedLong } | 644 element timeticks { length.attrs, xsd:unsignedInt } | 645 element ipaddress { length.attrs, ipv4address.type } | 646 element octet-string { length.attrs, xsd:hexBinary } | 647 element object-identifier { length.attrs, oid.type } | 648 element opaque { length.attrs, xsd:hexBinary } | 649 element no-such-object { length.attrs, empty } | 650 element no-such-instance { length.attrs, empty } | 651 element end-of-mib-view { length.attrs, empty } 653 # The blen attribute indicates the number of octets used by the BER 654 # encoded tag / length / value triple. The vlen attribute indicates 655 # the number of octets used by the BER encoded value alone. 657 length.attrs = 658 ( attribute blen { xsd:unsignedShort }, 659 attribute vlen { xsd:unsignedShort } )? 661 oid.type = 662 xsd:string { 663 pattern = 664 "(([0-1](\.[1-3]?[0-9]))|(2.(0|([1-9]\d*))))" ~ 665 "(\.(0|([1-9]\d*))){0,126}" 666 } 668 # The types below are for IP addresses. Note that SNMP's buildin 669 # IpAddress type only supports IPv4 addresses; IPv6 addresses are only 670 # introduced to cover SNMP over IPv6 endpoints. 672 ipv4address.type = 673 xsd:string { 674 pattern = 675 "((0|(1[0-9]{0,2})" ~ 676 "|(2(([0-4][0-9]?)|(5[0-5]?)|([6-9]?)))|([3-9][0-9]?))\.){3}" ~ 677 "(0|(1[0-9]{0,2})" ~ 678 "|(2(([0-4][0-9]?)|(5[0-5]?)|([6-9]?)))|([3-9][0-9]?))" 679 } 681 ipv6address.type = 682 xsd:string { 683 pattern = 684 "(([0-9a-fA-F]+:){7}[0-9a-fA-F]+)|" ~ 685 "(([0-9a-fA-F]+:)*[0-9a-fA-F]+)?::(([0-9a-fA-F]+:)*[0-9a-fA-F]+)?" 686 } 688 ipaddress.type = ipv4address.type | ipv6address.type 690 The following example shows an SNMP trace file in XML format 691 containing an SNMPv1 get-next-request message for the OID 692 1.3.6.1.2.1.1.3 (sysUpTime) and the response message returned by the 693 agent. 695 696 697 1147212206 698 739609 699 192.0.2.1 700 60371 701 192.0.2.2 702 12345 703 704 1 705 7075626c6963 706 707 1804289383 708 0 709 0 710 711 712 1.3.6.1.2.1.1.3 713 714 715 716 717 718 719 720 1147212206 721 762891 722 192.0.2.2 723 12345 724 192.0.2.1 725 60371 726 727 1 728 7075626c6963 729 730 1804289383 731 0 732 0 733 734 735 1.3.6.1.2.1.1.3.0 736 26842224 737 738 739 740 741 742 744 4.2. CSV Representation 746 The comma separated values (CSV) format has been designed to capture 747 only the most relevant information about an SNMP message. In 748 situations where all information about an SNMP message must be 749 captured, the XML format defined above must be used. The CSV format 750 uses the following fields: 752 1. Time-stamp in the format seconds.microseconds since 1970, for 753 example "1137764769.425484". 754 2. Source IP address in dotted quad notation (IPv4), for example 755 "192.0.2.1", or compact hexadecimal notation (IPv6), for example 756 "2001:DB8::1". 757 3. Source port number represented as a decimal number, for example 758 "4242". 759 4. Destination IP address in dotted quad notation (IPv4), for 760 example "192.0.2.1", or compact hexadecimal notation (IPv6), for 761 example "2001:DB8::1". 762 5. Destination port number represented as a decimal number, for 763 example "161". 764 6. Size of the SNMP message (a decimal number) counted in octets, 765 for example "123". The size excludes all transport, network, 766 and link-layer headers. 767 7. SNMP message version represented as a decimal number. The 768 version 0 stands for SNMPv1, 1 for SNMPv2c, and 3 for SNMPv3, 769 for example "3". 770 8. SNMP protocol operation indicated by one of the keywords get- 771 request, get-next-request, get-bulk-request, set-request, trap, 772 snmpV2-trap, inform-request, response, report. 773 9. SNMP request-id in decimal notation, for example "1511411010". 774 10. SNMP error-status in decimal notation, for example "0". 775 11. SNMP error-index in decimal notation, for example "0". 776 12. Number of variable-bindings contained in the varbind-list in 777 decimal notation, for example "5". 778 13. For each varbind in the varbind list, three output elements are 779 generated 780 1. Object name given as object identifier in dotted decimal 781 notation, for example "1.3.6.1.2.1.1.3.0". 782 2. Object base type name or exception name, that is one of the 783 following: null, integer32, unsigned32, counter32, 784 counter64, timeticks, ipaddress, octet-string, object- 785 identifier, opaque, no-such-object, no-such-instance, and 786 end-of-mib-view. 787 3. Object value is printed as a number if the underlying base 788 type is numeric. An IPv4 addresses is rendered in the 789 dotted quad notation and an IPv6 address is rendered in the 790 usual hexadecimal notation. An octet string value is 791 printed in hexadecimal format while an object identifier 792 value is printed in dotted decimal notation. In case of an 793 exception, the object value is empty. 795 Note that the format does not preserve the information needed to 796 understand SNMPv1 traps. It is therefore recommended that 797 implementations are able to convert the SNMPv1 trap format into the 798 trap format used by SNMPv2c and SNMPv3, according to the rules 799 defined in [RFC3584]. The activation of trap format conversion 800 should be the user's choice. 802 The following example shows an SNMP trace file in CSV format 803 containing an SNMPv1 get-next-request message for the OID 804 1.3.6.1.2.1.1.3 (sysUpTime) and the response message returned by the 805 agent. (Note that the example uses backslash line continuation marks 806 in order to fit the example into the RFC format. Backslash line 807 continuations are not part of the CSV format.) 809 1147212206.739609,192.0.2.1,60371,192.0.2.2,12345,42,1,\ 810 get-next-request,1804289383,0,0,1,1.3.6.1.2.1.1.3,null, 811 1147212206.762891,192.0.2.2,12345,192.0.2.1,60371,47,1,\ 812 response,1804289383,0,0,1,1.3.6.1.2.1.1.3.0,timeticks,26842224 814 5. Security Considerations 816 SNMP traffic traces usually contain sensitive information. It is 817 therefore necessary to (a) remove unwanted information and (b) to 818 anonymize the remaining necessary information before traces are made 819 available for analysis. It is recommended to encrypt traces when 820 they are archived. 822 Implementations that generate CSV or XML traces from raw pcap files 823 should have an option to suppress or anonymize values. Note that 824 instance identifiers of tables also include values and it might 825 therefore be necessary to suppress or anonymize (parts of) the 826 instance identifiers. Similarly, the packet and message headers 827 typically contain sensitive information about the source and 828 destination of SNMP messages as well as authentication information 829 (community strings or user names). 831 Anonymization techniques can be applied to keep information in traces 832 which could otherwise reveal sensitive information. When using 833 anonymization, values should only be kept when the underlying data 834 type is known and an appropriate anonymization transformation is 835 available (filter-in principle). For values appearing in instance 836 identifiers, it is usually desirable to maintain the lexicographic 837 order. Special anonymization transformations which preserve this 838 property have been developed, although their anonymization strength 839 is usually reduced compared to transformations that do not preserve 840 lexicographic ordering [HS06]. 842 The meta data associated with traces and in particular information 843 about the organization owning a network and the description of the 844 measurement point in the network topology where a trace was collected 845 may be misused to decide/pinpoint where and how to attack a network. 846 Meta data therefore needs to be properly protected. 848 6. IANA Considerations 850 This document requests that IANA register a URI for the SNMP XML 851 trace format namespace in the IETF XML registry [RFC3688]. Following 852 the format in RFC 3688, the following registration is requested. 854 URI: Please assign the URI "urn:ietf:params:xml:ns:snmp-trace-1.0" 855 for use by the SNMP trace format. 857 Registrant Contact: The NMRG of the IRTF. 859 XML: N/A, the requested URI is an XML namespace. 861 7. Acknowledgements 863 This document was influenced by discussions within the Network 864 Management Research Group (NMRG). Special thanks to Remco van de 865 Meent for writing the initial Perl script that lead to the 866 development of the snmpdump software package and Matus Harvan for his 867 work on lexicographic order preserving anonymization transformations. 868 Aiko Pras contributed ideas to Section 3 while David Harrington 869 helped to improve the readability of this document. 871 Last call reviews have been received from Bert Wijnen, Aiko Pras, 872 Frank Strauss, Remco van de Meent, Giorgio Nunzi, Wes Hardacker, Liam 873 Fallon, Sharon Chisholm, David Perkins, Deep Medhi, Randy Bush, David 874 Harrington, Dan Romascanu, Luca Deri, and Marc Burgess. Karen R. 875 Sollins reviewed the document for the Internet Research Steering 876 Group (IRSG). Jari Arkko, Pasi Eronen, Chris Newman, and Tim Polk 877 provided helpful comments during the Internet Engineering Steering 878 Group (IESG) review. 880 Part of this work was funded by the European Commission under grant 881 FP6-2004-IST-4-EMANICS-026854-NOE. 883 8. References 884 8.1. Normative References 886 [RFC2578] McCloghrie, K., Perkins, D., and J. Schoenwaelder, 887 "Structure of Management Information Version 2 (SMIv2)", 888 STD 58, RFC 2578, April 1999. 890 [OASISRNG] 891 Clark, J. and M. Makoto, "RELAX NG Specification", 892 OASIS Committee Specification, December 2001. 894 [OASISRNC] 895 Clark, J., "RELAX NG Compact Syntax", OASIS Committee 896 Specification, November 2002. 898 [RFC3584] Frye, R., Levi, D., Routhier, S., and B. Wijnen, 899 "Coexistence between Version 1, Version 2, and Version 3 900 of the Internet-standard Network Management Framework", 901 RFC 3584, August 2003. 903 [RFC3688] Mealling, M., "The IETF XML Registry", RFC 3688, 904 January 2004. 906 8.2. Informative References 908 [RFC1052] Cerf, V., "IAB Recommendations for the Development of 909 Internet Network Management Standards", RFC 1052, 910 April 1998. 912 [RFC2579] McCloghrie, K., Perkins, D., and J. Schoenwaelder, 913 "Textual Conventions for SMIv2", STD 58, RFC 2579, 914 April 1999. 916 [RFC3418] Presuhn, R., Case, J., McCloghrie, K., Rose, M., and S. 917 Waldbusser, "Management Information Base (MIB) for the 918 Simple Network Management Protocol (SNMP)", STD 62, 919 RFC 3418, December 2002. 921 [RFC2863] McCloghrie, K. and F. Kastenholz, "The Interfaces Group 922 MIB", RFC 2863, June 2000. 924 [RFC3410] Case, J., Mundy, R., Partain, D., and B. Stewart, 925 "Introduction and Applicability Statements for Internet 926 Standard Management Framework", RFC 3410, December 2002. 928 [RFC4022] Raghunarayan, R., "Management Information Base for the 929 Transmission Control Protocol (TCP)", RFC 4022, 930 March 2005. 932 [PDMQ04] Pras, A., Drevers, T., van de Meent, R., and D. Quartel, 933 "Comparing the Performance of SNMP and Web Services based 934 Management", IEEE electronic Transactions on Network and 935 Service Management 1(2), November 2004. 937 [Pat01] Pattinson, C., "A Study of the Behaviour of the Simple 938 Network Management Protocol", Proc. 12th IFIP/IEEE 939 Workshop on Distributed Systems: Operations and 940 Management , October 2001. 942 [DSR01] Du, X., Shayman, M., and M. Rozenblit, "Implementation and 943 Performance Analysis of SNMP on a TLS/TCP Base", Proc. 7th 944 IFIP/IEEE International Symposium on Integrated Network 945 Management , May 2001. 947 [CT04] Corrente, A. and L. Tura, "Security Performance Analysis 948 of SNMPv3 with Respect to SNMPv2c", Proc. 2004 IEEE/IFIP 949 Network Operations and Management Symposium , April 2004. 951 [PFGL04] Pavlou, G., Flegkas, P., Gouveris, S., and A. Liotta, "On 952 Management Technologies and the Potential of Web 953 Services", IEEE Communications Magazine 42(7), July 2004. 955 [SM99] Sprenkels, R. and J. Martin-Flatin, "Bulk Transfers of MIB 956 Data", Simple Times 7(1), March 1999. 958 [Mal02] Malowidzki, M., "GetBulk Worth Fixing", Simple 959 Times 10(1), December 2002. 961 [HS06] Harvan, M. and J. Schoenwaelder, "Prefix- and 962 Lexicographical-order-preserving IP Address 963 Anonymization", IEEE/IFIP Network Operations and 964 Management Symposium NOMS 2006, April 2006. 966 [XFA02] Xu, J., Fan, J., and M. Ammar, "Prefix-Preserving IP 967 Address Anonymization: Measurement-based Security 968 Evaluation and a New Cryptography-based Scheme", 10th IEEE 969 International Conference on Network Protocols ICNP'02, 970 November 2002. 972 [FXAM04] Fan, J., Xu, J., Ammar, M., and S. Moon, "Prefix- 973 Preserving IP Address Anonymization", Computer 974 Networks 46(2), October 2004. 976 [PAPL06] Pang, R., Allman, M., Paxson, V., and J. Lee, "The Devil 977 and Packet Trace Anonymization", Computer Communication 978 Review 36(1), January 2006. 980 [RW07] Ramaswamy, R. and T. Wolf, "High-Speed Prefix-Preserving 981 {IP} Address Anonymization for Passive Measurement 982 Systems", IEEE Transactions on Networking 15(1), 983 February 2007. 985 URIs 987 [1] 989 [2] 991 [3] 993 [4] 995 [5] 997 [6] 999 [7] 1001 [8] 1003 [9] 1005 Author's Address 1007 Juergen Schoenwaelder 1008 Jacobs University Bremen 1009 Campus Ring 1 1010 28725 Bremen 1011 Germany 1013 Phone: +49 421 200-3587 1014 Email: j.schoenwaelder@jacobs-university.de 1016 Full Copyright Statement 1018 Copyright (C) The IETF Trust (2008). 1020 This document is subject to the rights, licenses and restrictions 1021 contained in BCP 78, and except as set forth therein, the authors 1022 retain all their rights. 1024 This document and the information contained herein are provided on an 1025 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 1026 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND 1027 THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS 1028 OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF 1029 THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 1030 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 1032 Intellectual Property 1034 The IETF takes no position regarding the validity or scope of any 1035 Intellectual Property Rights or other rights that might be claimed to 1036 pertain to the implementation or use of the technology described in 1037 this document or the extent to which any license under such rights 1038 might or might not be available; nor does it represent that it has 1039 made any independent effort to identify any such rights. Information 1040 on the procedures with respect to rights in RFC documents can be 1041 found in BCP 78 and BCP 79. 1043 Copies of IPR disclosures made to the IETF Secretariat and any 1044 assurances of licenses to be made available, or the result of an 1045 attempt made to obtain a general license or permission for the use of 1046 such proprietary rights by implementers or users of this 1047 specification can be obtained from the IETF on-line IPR repository at 1048 http://www.ietf.org/ipr. 1050 The IETF invites any interested party to bring to its attention any 1051 copyrights, patents or patent applications, or other proprietary 1052 rights that may cover technology that may be required to implement 1053 this standard. Please address the information to the IETF at 1054 ietf-ipr@ietf.org. 1056 Acknowledgment 1058 Funding for the RFC Editor function is provided by the IETF 1059 Administrative Support Activity (IASA).