idnits 2.17.1 draft-ietf-eos-snmpbulk-00.txt: -(178): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There are 4 instances of lines with non-ascii characters in the document. == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (April 2001) is 8384 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: 'RFC2119' on line 32 == Unused Reference: '1' is defined on line 455, but no explicit reference was found in the text == Unused Reference: '2' is defined on line 458, but no explicit reference was found in the text == Unused Reference: '3' is defined on line 461, but no explicit reference was found in the text == Unused Reference: '4' is defined on line 464, but no explicit reference was found in the text ** Downref: Normative reference to an Experimental RFC: RFC 1187 (ref. '1') -- Possible downref: Non-RFC (?) normative reference: ref. '2' -- Possible downref: Non-RFC (?) normative reference: ref. '3' -- Possible downref: Non-RFC (?) normative reference: ref. '4' Summary: 3 errors (**), 0 flaws (~~), 8 warnings (==), 6 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Evolution Of SNMP WG Satyen Chandragiri 3 INTERNET-DRAFT Ranch Networks, Inc 4 Expires October 2001 April 2001 6 Efficient Transfer of Bulk SNMP Data 7 draft-ietf-eos-snmpbulk-00.txt 9 Status of this Memo 11 This document is an Internet-Draft and is in full conformance 12 with all provisions of Section 10 of RFC2026. 14 Internet-Drafts are working documents of the Internet Engineering 15 Task Force (IETF), its areas, and its working groups. Note that 16 other groups may also distribute working documents as Internet- 17 Drafts. 19 Internet-Drafts are draft documents valid for a maximum of six 20 months and may be updated, replaced, or obsoleted by other documents 21 at any time. It is inappropriate to use Internet-Drafts as 22 reference material or to cite them other than as "work in progress." 24 The list of current Internet-Drafts can be accessed at 25 http://www.ietf.org/ietf/1id-abstracts.txt 26 The list of Internet-Draft Shadow Directories can be accessed at 27 http://www.ietf.org/shadow.html. 29 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 30 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in 31 this document are to be interpreted as described in [RFC2119]. 33 Abstract 35 Managing networks and network devices using the Internet Standard 36 Management Framework often requires the retrieval of significant 37 amounts of MIB data via SNMP, which can result in large latency, 38 increased network overhead, and other problems. This memo discusses 39 the need for an efficient mechanism for transferring large amounts 40 of SNMP data and explores possible solutions for overcoming the 41 current limitations of the protocol. 43 Table of Contents 45 Status of this Memo................................................1 46 Abstract...........................................................1 47 Overview...........................................................2 48 Previous Work......................................................3 49 Proposed Solution..................................................5 50 Summary............................................................9 51 Security Considerations............................................9 52 IANA Considerations................................................9 53 Author's Address...................................................9 54 Acknowledgements...................................................9 55 References........................................................10 56 Full Copyright Statement..........................................10 58 Overview 60 As network elements grow in size and complexity, so do the size of 61 MIB tables used to monitor and configure them. Since the first 62 introduction of SNMP in the late 1980s, the amount of MIB data that 63 is required to be transferred between a command generator and a 64 command responder has grown tremendously. For example, the size of 65 IP routing tables in a typical backbone router, or the subscriber 66 management tables in a cable modem termination system can be quite 67 substantial. Retrieving such data via SNMP may involve the transfer 68 of several hundred kilobytes of data across a network. Although SNMP 69 has evolved substantially, with version 3 providing many desirable 70 features such as security and access control, no enhancements have 71 been made that address the issue of bulk transfer of SNMP data. 73 Using SNMPv1, a command generator has to generate a series of 74 GetNextRequests to traverse the MIB tables. For large tables like 75 the ones mentioned above, this requires a very large sequence of 76 request-response exchanges between the two entities thereby 77 resulting in large latency. To address this problem, the 78 GetBulkRequest operation was introduced in SNMPv2. This operation 79 attempts to reduce the number of protocol exchanges required to 80 retrieve a large amount of MIB data by returning a series of 81 variable bindings in a single response. The command generator is 82 required to specify a "max-repetitions" count, and the command 83 responder then fills in as many variable bindings as it can without 84 exceeding either this count, or the maximum message size. 86 The main problem with retrieving tables using GetBulkRequest is that 87 the command generator typically does not know the number of rows in 88 the table, and hence cannot set max-repetitions to the optimal 89 value. As a result, it must either set max-repetitions to some very 90 large value, resulting in a potentially large waste of bandwidth 91 when many more variable bindings are returned than are needed (this 92 is referred to as the "overshoot" problem), or else it must issue 93 multiple GetBulkRequests sequentially to traverse a large table. 95 Another significant issue affecting the efficiency of bulk transfer 96 is network overhead [ref]. This refers to the amount of non-data 97 bytes (header information, encoding bytes, etc.) needed to be sent 98 along with each PDU. All current versions of SNMP use the Basic 99 Encoding Rules (BER) to encode PDUs. Though BER was selected because 100 of its simplicity and easy availability, it is quite inefficient in 101 terms of network overhead. Added to this is the repetitive 102 information contained in OIDs � there are multiple occurrences of 103 identical portions of identifiers in the OIDs of the MIB values sent 104 in a response. Transferring large amounts of MIB data further 105 compounds these problems. 107 Retrieving MIB table data (as versus MIB scalars) can pose some 108 special problems. The main reason for these problems is that SNMP 109 does not recognize table rows and columns and thus all protocol 110 operations have to deal with "conceptual" rows and columns. If a 111 table allows certain columnar objects of a conceptual row to be 112 absent, then it creates "holes" in the table. A command generator 113 that is performing a "GetNext" or "GetBulk" on all columnar rows 114 will be returned all elements of the following row, except if there 115 are "holes", in which case the first columnar object of a row that 116 does have this object are returned. The command receiver (generator) 117 has to realize that not all returned objects are from the same row, 118 and has to correctly reconstruct the MIB table while determining the 119 locations of the "holes". This can be very time consuming and 120 challenging for a network management application to implement. 121 Another problem with tables that have rapidly refreshed values (e.g. 122 packet counts, number of active connections, etc.) is that the 123 latency in retrieving table rows can create inconsistencies since by 124 the time a management application reads a value it may be obsolete 125 on the device. 127 Previous Work 129 Several solutions have been proposed and discussed in the past that 130 attempt to address the problems mentioned above. These range from 131 those requiring no changes to the existing protocol to evolutionary 132 changes to the framework to non-SNMP solutions. Some of these 133 proposals are described in this section. Each method has its own 134 merits and demerits. 136 Pipeline Retrieval 138 In RFC 1187 "Bulk Table Retrieval with the SNMP", the authors 139 propose a pipeline algorithm using multiple threads to traverse 140 different sections of a MIB table simultaneously. This improves 141 latency because several pieces of MIB data are gathered in parallel, 142 however it adds complexity on both the command generator and command 143 responder especially when packets are dropped and retransmission of 144 the request or response is required. It also does not reduce the 145 number of request-response exchanges required between the two 146 entities. 148 SNMP over TCP 150 Another proposal is to extend the transport mapping for SNMP by 151 sending the PDUs over a TCP connection rather than UDP. An immediate 152 benefit is that UDP's 64KB restriction on the SNMP maximum message 153 size is eliminated since TCP's windowing mechanism can be used to 154 send several segments of data in parallel. This reduces the number 155 of request-response exchanges thereby significantly lowering latency 156 as well as network overhead. On the other hand, the SNMP entities 157 now have to manage their TCP connections and be able to accommodate 158 larger buffers for packet processing. 160 Changing PDU Encoding 162 As previously mentioned, the BER encoding used for SNMP PDUs is 163 inefficient and is a major contributor to network overhead. Several 164 alternatives exist, but it should be noted that any other encoding 165 scheme in place of BER entails a major change in implementation and 166 reduces interoperability. One alternative scheme is "Packed Encoding 167 Rules" (PER) which has approximately 30% shorter encodings and 168 requires much less encoding buffer space than BER. Other 169 possibilities are "Lightweight Encoding Rules" (LER) which allows 170 quick encoding and decoding, thereby reducing latency; or 171 "Distinguished Encoding Rules" (DER) which have better encoding time 172 while also keeping the network overhead low. 174 Notification-based GetSubtree 176 The GetSubtree operation allows a management application to retrieve 177 "subtrees" of MIB data. It first specifies the root of the subtree 178 to be retrieved (it can be rooted anywhere � at the head of a table, 179 a specific column, etc.) and then triggers the retrieval operation. 180 The command responder must then retrieve the MIB data contained 181 lexicographically under the specified root and send the retrieved 182 values back to the management application. The responder stops when 183 it reaches the end of the subtree (thereby eliminating the overshoot 184 problem of GetBulk). The responses are sent back as a series of 185 Notifications to the management application. Multiple varbinds are 186 bundled in each trap packing in as many as allowed by the maximum 187 message size constraint. A sequence number is provided so that the 188 receiver can detect packet loss and request retransmission (via a 189 GetBulk request). The solution can be extended to allow multiple 190 subtrees to be retrieved in parallel if the command responder can 191 handle it. The limitations of this approach are that a) it requires 192 the management application to be registered as a notification 193 receiver, b) it tightly couples the command generator and 194 notification receiver, and c) in a lossy network this protocol 195 degenerates to GetBulk. 197 Non-SNMP Solutions 199 Several non-SNMP based solutions to the bulk MIB-data transfer 200 problem have been implemented or proposed. Prominent among them is 201 Cisco Systems' FTP-based solution, where the MIB data is retrieved 202 and stored in a file on the device, and then transferred to the 203 management application via FTP. Two MIB modules are involved: the 204 "CISCO-BULK-FILE-MIB" and "CISCO-FTP-CLIENT-MIB". The bulk file MIB 205 consists of three tables that are used to specify the MIB data 206 objects to retrieve and the name of the file, its storage type, and 207 encoding format (BER / binary / ASCII). The FTP client MIB has a 208 single table that allows the management application to specify the 209 FTP server details, and the name of the local file where the data 210 should be uploaded. This solution requires FTP capability on both 211 SNMP entities and since a non-SNMP transfer mechanism is used, 212 security considerations need to be taken into account by the 213 implementers. 215 Other non-SNMP alternatives for bulk transfer include using MIME or 216 XML Document Type Definition (DFD) to encode the MIB data. The data 217 can then be transferred via the well-known HTTP protocol since it is 218 well suited for bulk transfer of MIME-encapsulated data. However, 219 since HTTP is primarily intended for transferring World Wide Web 220 data, it has many features and options that are not required for 221 management data. However, a compliant implementation will 222 nevertheless have to implement those features thereby increasing the 223 size and complexity of the SNMP implementations without additional 224 benefit. In addition, future evolution of HTTP may add features or 225 requirements that make it unsuitable for transferring MIB data. 227 Proposed Solution 229 While each of the proposals above attempts to improve/eliminate one 230 or more problem areas (latency, overhead, table retrieval) none 231 addresses them all. Specifically, the question of how to handle 232 "holes" in MIB tables is not addressed. Moreover, for a solution to 233 be widely adopted by the development community it would have to be 234 easily implementable and not be overly complex or require special 235 resources. Similarly, solutions that are proprietary or rely on non- 236 SNMP mechanisms are also not good candidates for standardization as 237 an evolutionary change to the SNMP protocol. 239 A solution that fits these requirements and adequately addresses the 240 problems associated with bulk transfer is the "GetCols" operation 241 proposed by David Perkins. This solution requires the addition of a 242 new PDU type called "GetColsRequest" to SNMP that will operate as 243 explained below. This PDU type will have the same syntax as the 244 GetBulkRequest and Response PDUs thus minimizing the effect on 245 implementation. 247 Columnar objects in MIB tables are usually attributes of modeled 248 entities. Management applications often need to retrieve specific 249 columns of MIB data rather than all columns (i.e. entire rows are 250 not required). Moreover, only a certain segment of MIB data (called 251 a "slice") may be required rather than the entire table. The 252 GetColsRequest is used to specify columns of interest to the 253 management application. The command responder sends back only the 254 data of interest & terminates the PDU with an "end-of-data" marker. 256 The GetColsRequest PDU is identical to the GetBulk PDU, and is as 257 follows: 259 BulkPDU ::= 260 SEQUENCE { 261 request-id 262 Integer32, 264 non-repeaters 265 INTEGER (0..max-bindings), 267 max-repetitions 268 INTEGER (0..max-bindings), 270 Variable-bindings 271 VarBindList 272 } 274 The Response PDU is unchanged from that currently used, and is as 275 follows: 277 PDU ::= 278 SEQUENCE { 279 request-id 280 Integer32, 282 error-status -- sometimes ignored 283 INTEGER { 284 noError(0), 285 tooBig(1), 286 noSuchName(2), -- for proxy compatibility 287 badValue(3), -- for proxy compatibility 288 readOnly(4), -- for proxy compatibility 289 genErr(5), 290 noAccess(6), 291 wrongType(7), 292 wrongLength(8), 293 wrongEncoding(9), 294 wrongValue(10), 295 noCreation(11), 296 inconsistentValue(12), 297 resourceUnavailable(13), 298 commitFailed(14), 299 undoFailed(15), 300 authorizationError(16), 301 notWritable(17), 302 inconsistentName(18) 304 }, 306 error-index -- sometimes ignored 307 INTEGER (0..max-bindings), 309 variable-bindings -- values are sometimes ignored 310 VarBindList 311 } 313 GetCols Example: The Interfaces Table (ifTable) has 18 columns, but 314 a management application may only be interested in retrieving 315 ifIndex, ifType, ifAdminStatus and ifOperStatus for all the 316 interfaces on a router, and the value of ifTableLastChange. The 317 application could use a GetBulkRequest to retrieve the data, but not 318 knowing the number of instances present, it would not be able to set 319 the "max-repetitions" parameter to an optimal value. As explained 320 earlier, this may require the application to make multiple requests 321 until the entire table is traversed, or it may cause an overshoot 322 problem where a large number of unwanted data from past the end of 323 the table is retrieved. 325 The GetColsRequest operation can therefore be utilized here in place 326 of GetBulkRequest. The request would be constructed as follows: 328 GetColsRequest (, 329 1, 330 100, 331 ifTableLastChange:NULL, 332 ifIndex:NULL, 333 ifType:NULL, 334 ifAdminStatus:NULL, 335 ifOperStatus:NULL) 337 The command responder will reply with a list of varbinds containing 338 the value of ifTableLastChange and repetitions of the other varbinds 339 and values similar to a GetBulk response, BUT there are important 340 differences: 342 a. There will be no "holes" in the response. If an instance 343 of a columnar object does not exist, it will use a 344 "noSuchInstance" exception in its place rather than skip 345 over to the next instance for that column. Thus, each set 346 of repetitions in the response has the same instance value. 348 b. If the max-repetitions value exceeds the number of instances 349 available in the table, the command responder stops at the 350 end of the table and adds a "End-Of-Rows" marker in the 351 Response PDU rather than overshoot and provide irrelevant 352 MIB data that will be discarded by the requester. 354 c. The OIDs in the Response PDU are compressed to eliminate 355 redundancy. Since all repetitions of a columnar value have 356 a common OID prefix � differing only in the instance part, 357 the Response PDU only needs to use the suffix identifiers 358 to distinguish between instance values. 360 The OID compression is done as follows: Either two or three sub- 361 identifiers are used in place of the complete OID � 362 .. or 363 . 365 The represents the MIB table to which the instance 366 value belongs. In the example above, since all the varbinds belong 367 to the same MIB table (ifTable), would be '0'. Had 368 there been objects from other tables, for those 369 instances would be '1', '2', etc. 370 The represents the column to which the instances 371 belong. The first requested column will have a value of '0' for 372 , the second column will have a value of '1', and so 373 on. 374 The is only present for the first columnar instance 375 of a row and represents the index value. Since subsequent columnar 376 objects of that row have the same index value, it need not be 377 specified in the Response, and hence the is not 378 required for these objects. 380 An indication for a table is provided by setting the 381 for that table equal to one more than the number of 382 columns requested for the table. 384 The Response PDU to the GetCols request in the example presented 385 above would therefore be: 387 Response (, 388 noError, 389 0, 390 ifTableLastChange:, 391 0.0.1:, 0.1:, 0.2:, 0.3:, 392 0.0.2:, 0.1:, 0.2:, 0.3:, 393 ... 394 0.0.n:, 0.1:, 0.2:, 0.3:, 395 0.4:) 397 The Response contains the non-repeaters (ifTableLastChange in this 398 example) followed by a VarBindList for the repeaters. The compressed 399 OID values '0.0.x' represent the first requested column (ifIndex in 400 this example) where 'x' is the row index. Compressed OID '0.1' 401 represents the second requested column (ifType), '0.2' represents 402 the third requested column (ifAdminStatus), and '0.3' represents the 403 fourth requested column (ifOperStatus). Note that for the second, 404 third and fourth columns the row index is not included. The special 405 OID '0.4' marks the end of rows for this table (ifTable). Of course, 406 this is present only if the number of rows returned (n) is less than 407 or equal to the max-repetitions requested in the GetCols request. If 408 there are more rows in the table, the absence of this marker 409 notifies the command generator that there are more rows to be 410 retrieved. 412 Summary 414 The GetCols operation adds a new PDU type to the SNMP protocol, but 415 by reusing the same message formats it minimizes the implementation 416 effort required to add this feature to existing applications. It 417 provides many desirable features such as reduced latency and network 418 overhead, elimination of problems caused by holes in MIB tables, and 419 OID compression to greatly reduce the amount of data transmitted in 420 the Response PDU. The overshoot problem of GetBulk is also 421 eliminated because GetCols stops at the end of the table and hence 422 the management application can choose a very large value for the 423 max-repetitions parameter (constrained mainly by the maximum message 424 size limit). Calculations (presented by David Perkins) demonstrate 425 that significant performance improvements can be gained by using 426 GetCols versus GetBulk operations to retrieve large amounts of MIB 427 data. 429 Security Considerations 431 433 IANA Considerations 435 437 Author's Address 439 Satyen Chandragiri 440 Ranch Networks, Inc 441 65 Route 34 North 442 Suite 200 Phone: 1-732-817-1900 x264 443 Morganville, NJ 07751 Email: satyen@ranchnetworks.com 445 Acknowledgements 447 The author wishes to acknowledge Ron Sprenkels, Jean-Philippe 448 Martin-Flatin, Juergen Schoenwaelder and other members of the 449 Network Management Research Group (NMRG) of the IRTF for their prior 450 work in this area; and David Perkins for the GetCols proposal 451 presented in this document. 453 References 455 [1] M. Rose, K. McCloghrie, J. Davin, "Bulk Table Retrieval with the 456 SNMP", RFC 1187, October 1990 458 [2] R. Sprenkels, J. Martin-Flatin, "Bulk Transfers of MIB Data", 459 The Simple Times Volume 7 Number 1, March 1999 461 [3] D. Thaler, "Subtree Retrieval MIB", Presentation at the 48th 462 IETF meeting in Pittsburgh, PA, July 2000 464 [4] D. Perkins, "GetCols Operation", Presentation at the 50th IETF 465 meeting in Minneapolis, MN, March 2001 467 Full Copyright Statement 469 Copyright (C) The Internet Society (2001). All Rights Reserved. 471 This document and translations of it may be copied and furnished to 472 others, and derivative works that comment on or otherwise explain it 473 or assist in its implementation may be prepared, copied, published 474 and distributed, in whole or in part, without restriction of any 475 kind, provided that the above copyright notice and this paragraph 476 are included on all such copies and derivative works. However, this 477 document itself may not be modified in any way, such as by removing 478 the copyright notice or references to the Internet Society or other 479 Internet organizations, except as needed for the purpose of 480 developing Internet standards in which case the procedures for 481 copyrights defined in the Internet Standards process must be 482 followed, or as required to translate it into languages other than 483 English. 485 The limited permissions granted above are perpetual and will not be 486 revoked by the Internet Society or its successors or assigns. 488 This document and the information contained herein is provided on an 489 "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING 490 TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING 491 BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION 492 HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF 493 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.