idnits 2.17.1 draft-ietf-nfsv4-nfs-rdma-problem-statement-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 15. -- Found old boilerplate from RFC 3978, Section 5.5, updated by RFC 4748 on line 680. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 690. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 697. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 703. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (May 7, 2007) is 6199 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 3530 (Obsoleted by RFC 7530) ** Obsolete normative reference: RFC 1831 (Obsoleted by RFC 5531) Summary: 3 errors (**), 0 flaws (~~), 1 warning (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFSv4 Working Group Tom Talpey 3 Internet-Draft Network Appliance, Inc. 4 Intended status: Informational Chet Juszczak 5 Expires: November 8, 2007 May 7, 2007 7 NFS RDMA Problem Statement 8 draft-ietf-nfsv4-nfs-rdma-problem-statement-06 10 Status of this Memo 12 By submitting this Internet-Draft, each author represents that any 13 applicable patent or other IPR claims of which he or she is aware 14 have been or will be disclosed, and any of which he or she becomes 15 aware will be disclosed, in accordance with Section 6 of BCP 79. 17 Internet-Drafts are working documents of the Internet Engineering 18 Task Force (IETF), its areas, and its working groups. Note that 19 other groups may also distribute working documents as Internet- 20 Drafts. 22 Internet-Drafts are draft documents valid for a maximum of six 23 months and may be updated, replaced, or obsoleted by other 24 documents at any time. It is inappropriate to use Internet-Drafts 25 as reference material or to cite them other than as "work in 26 progress." 28 The list of current Internet-Drafts can be accessed at 29 http://www.ietf.org/ietf/1id-abstracts.txt 31 The list of Internet-Draft Shadow Directories can be accessed at 32 http://www.ietf.org/shadow.html. 34 This Internet-Draft will expire on November 8, 2007. 36 Copyright Notice 38 Copyright (C) The IETF Trust (2007). 40 Abstract 42 This draft addresses applying Remote Direct Memory Access to the 43 NFS protocols. NFS implementations historically incur significant 44 overhead due to data copies on end-host systems, as well as other 45 sources. The potential benefits of RDMA to these implementations 46 are explored, and the reasons why RDMA is especially well-suited to 47 NFS and network file protocols in general are evaluated. 49 Table Of Contents 51 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . 2 52 2. Problem Statement . . . . . . . . . . . . . . . . . . . . 4 53 3. File Protocol Architecture . . . . . . . . . . . . . . . . 5 54 4. Sources of Overhead . . . . . . . . . . . . . . . . . . . 7 55 4.1. Savings from TOE . . . . . . . . . . . . . . . . . . . . 8 56 4.2. Savings from RDMA . . . . . . . . . . . . . . . . . . . 9 57 5. Application of RDMA to NFS . . . . . . . . . . . . . . . . 10 58 6. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 10 59 Security Considerations . . . . . . . . . . . . . . . . . 11 60 IANA Considerations . . . . . . . . . . . . . . . . . . . 11 61 Acknowledgements . . . . . . . . . . . . . . . . . . . . . 11 62 Normative References . . . . . . . . . . . . . . . . . . . 11 63 Informative References . . . . . . . . . . . . . . . . . . 12 64 Authors' Addresses . . . . . . . . . . . . . . . . . . . . 14 65 Intellectual Property and Copyright Statements . . . . . . 14 66 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . 15 68 1. Introduction 70 The Network File System (NFS) protocol (as described in [RFC1094], 71 [RFC1813], and [RFC3530]) is one of several remote file access 72 protocols used in the class of processing architecture sometimes 73 called Network Attached Storage (NAS). 75 Historically, remote file access has proved to be a convenient, 76 cost-effective way to share information over a network, a concept 77 proven over time by the popularity of the NFS protocol. However, 78 there are issues in such a deployment. 80 As compared to a local (direct-attached) file access architecture, 81 NFS removes the overhead of managing the local on-disk filesystem 82 state and its metadata, but interposes at least a transport network 83 and two network endpoints between an application process and the 84 files it is accessing. This tradeoff has to date usually resulted 85 in a net performance loss as a result of reduced bandwidth, 86 increased application server CPU utilization, and other overheads. 88 Several classes of applications, including those directly 89 supporting enterprise activities in high performance domains such 90 as database applications and shared clusters, have therefore 91 encountered issues with moving to NFS architectures. While this 92 has been due principally to the performance costs of NFS versus 93 direct attached files, other reasons are relevant, such as the lack 94 of strong consistency guarantees being provided by NFS 95 implementations. 97 Replication of local file access performance on NAS using 98 traditional network protocol stacks has proven difficult, not 99 because of protocol processing overheads, but because of data copy 100 costs in the network endpoints. This is especially true since host 101 buses are now often the main bottleneck in NAS architectures 102 [MOG03] [CHA+01]. 104 The External Data Representation [RFC4506] employed beneath NFS and 105 RPC [RFC1831] can add more data copies, exacerbating the problem. 107 Data copy-avoidance designs have not been widely adopted for a 108 variety of reasons. [BRU99] points out that "many copy avoidance 109 techniques for network I/O are not applicable or may even backfire 110 if applied to file I/O." Other designs that eliminate unnecessary 111 copies, such as [PAI+00], are incompatible with existing APIs and 112 therefore force application changes. 114 In recent years, an effort to standardize a set of protocols for 115 Remote Direct Memory Access, RDMA, over the standard Internet 116 Protocol Suite has been chartered [RDDP]. Several drafts have been 117 proposed and are being considered for Standards Track. 119 RDMA is a general solution to the problem of CPU overhead incurred 120 due to data copies, primarily at the receiver. Substantial 121 research has addressed this and has borne out the efficacy of the 122 approach. An overview of this is the RDDP "Remote Direct Memory 123 Access (RDMA) over IP Problem Statement" document, [RFC4297]. 125 In addition to the per-byte savings of off-loading data copies, 126 RDMA-enabled NICs (RNICS) offload the underlying protocol layers as 127 well, e.g. TCP, further reducing CPU overhead due to NAS 128 processing. 130 1.1. Background 132 The RDDP Problem Statement [RFC4297] asserts: 134 "High costs associated with copying are an issue primarily for 135 large scale systems ... with high bandwidth feeds, usually 136 multiprocessors and clusters, that are adversely affected by 137 copying overhead. Examples of such machines include all 138 varieties of servers: database servers, storage servers, 139 application servers for transaction processing, for e- 140 commerce, and web serving, content distribution, video 141 distribution, backups, data mining and decision support, and 142 scientific computing. 144 Note that such servers almost exclusively service many 145 concurrent sessions (transport connections), which, in 146 aggregate, are responsible for > 1 Gbits/s of communication. 147 Nonetheless, the cost of copying overhead for a particular 148 load is the same whether from few or many sessions." 150 Note that each of the servers listed above could be accessing their 151 file data as an NFS client, or NFS serving the data to such 152 clients, or acting as both. 154 The CPU overhead of the NFS and TCP/IP protocol stacks (including 155 data copies or reduced copy workarounds) becomes a significant 156 matter in these clients and servers. File access using locally 157 attached disks imposes relatively low overhead due to the highly 158 optimized I/O path and direct memory access afforded to the storage 159 controller. This is not the case with NFS, which must pass data 160 to, and especially from, the network and network processing stack 161 to the NFS stack. Frequently, data copies are imposed on this 162 transfer, in some cases several such copies in each direction. 164 Copies are potentially encountered in an NFS implementation 165 exchanging data to and from user address spaces, within kernel 166 buffer caches, in XDR marshalling and unmarshalling, and within 167 network stacks and network drivers. Other overheads such as 168 serialization among multiple threads of execution sharing a single 169 NFS mount point and transport connection are additionally 170 encountered. 172 Numerous upper layer protocols achieve extremely high bandwidth and 173 low overhead through the use of RDMA. [MAF+02] show that the RDMA- 174 based Direct Access File System (with a user-level implementation 175 of the file system client) can outperform even a zero-copy 176 implementation of NFS [CHA+01] [CHA+99] [GAL+99] [KM02]. Also, 177 file data access implies the use of large ULP messages. These 178 large messages tend to amortize any increase in per-message costs 179 due to the offload of protocol processing incurred when using RNICs 180 while gaining the benefits of reduced per-byte costs. Finally, the 181 direct memory addressing afforded by RDMA avoids many sources of 182 contention on network resources. 184 2. Problem Statement 186 The principal performance problem encountered by NFS 187 implementations is the CPU overhead required to implement the 188 protocol. Primary among the sources of this overhead is the 189 movement of data from NFS protocol messages to its eventual 190 destination in user buffers or aligned kernel buffers. Due to the 191 nature of the RPC and XDR protocols, the NFS data payload arrives 192 at arbitrary alignment, necessitating a copy at the receiver, and 193 the NFS requests are completed in an arbitrary sequence. 195 The data copies consume system bus bandwidth and CPU time, reducing 196 the available system capacity for applications [RFC4297]. 197 Achieving zero-copy with NFS has, to date, required sophisticated, 198 version-specific "header cracking" hardware and/or extensive 199 platform-specific virtual memory mapping tricks. Such approaches 200 become even more difficult for NFS version 4 due to the existence 201 of the COMPOUND operation, which further reduces alignment and 202 greatly complicates ULP offload. 204 Furthermore, NFS will soon be challenged by emerging high-speed 205 network fabrics such as 10 Gbits/s Ethernet. Performing even raw 206 network I/O such as TCP is an issue at such speeds with today's 207 hardware. The problem is fundamental in nature and has led the 208 IETF to explore RDMA [RFC4297]. 210 Zero-copy techniques benefit file protocols extensively, as they 211 enable direct user I/O, reduce the overhead of protocol stacks, 212 provide perfect alignment into caches, etc. Many studies have 213 already shown the performance benefits of such techniques [SKE+01] 214 [DCK+03] [FJNFS] [FJDAFS] [KM02] [MAF+02]. 216 RDMA is compelling here for another reason; hardware offloaded 217 networking support in itself does not avoid data copies, without 218 resorting to implementing part of the NFS protocol in the NIC. 219 Support of RDMA by NFS enables the highest performance at the 220 architecture level rather than by implementation; this enables 221 ubiquitous and interoperable solutions. 223 By providing file access performance equivalent to that of local 224 file systems, NFS over RDMA will enable applications running on a 225 set of client machines to interact through an NFS file system, just 226 as applications running on a single machine might interact through 227 a local file system. 229 3. File Protocol Architecture 231 NFS runs as an ONC RPC [RFC1831] application. Being a file access 232 protocol, NFS is very "rich" in data content (versus control 233 information). 235 NFS messages can range from very small (under 100 bytes) to very 236 large (from many kilobytes to a megabyte or more). They are all 237 contained within an RPC message and follow a variable length RPC 238 header. This layout provides an alignment challenge for the data 239 items contained in an NFS call (request) or reply (response) 240 message. 242 In addition to the control information in each NFS call or reply 243 message, sometimes there are large "chunks" of application file 244 data, for example read and write requests. With NFS version 4 (due 245 to the existence of the COMPOUND operation) there can be several of 246 these data chunks interspersed with control information. 248 ONC RPC is a remote procedure call protocol that has been run over 249 a variety of transports. Most implementations today use UDP or 250 TCP. RPC messages are defined in terms of an eXternal Data 251 Representation (XDR) [RFC4506] which provides a canonical data 252 representation across a variety of host architectures. An XDR data 253 stream is conveyed differently on each type of transport. On UDP, 254 RPC messages are encapsulated inside datagrams, while on a TCP byte 255 stream, RPC messages are delineated by a record marking protocol. 256 An RDMA transport also conveys RPC messages in a unique fashion 257 that must be fully described if client and server implementations 258 are to interoperate. 260 The RPC transport is responsible for conveying an RPC message from 261 a sender to a receiver. An RPC message is either an RPC call from 262 a client to a server, or an RPC reply from the server back to the 263 client. An RPC message contains an RPC call header followed by 264 arguments if the message is an RPC call, or an RPC reply header 265 followed by results if the message is an RPC reply. The call 266 header contains a transaction ID (XID) followed by the program and 267 procedure number as well as a security credential. An RPC reply 268 header begins with an XID that matches that of the RPC call 269 message, followed by a security verifier and results. All data in 270 an RPC message is XDR encoded. 272 The encoding of XDR data into transport buffers is referred to as 273 "marshalling", and the decoding of XDR data contained within 274 transport buffers and into destination RPC procedure result 275 buffers, is referred to as "unmarshalling". The process of 276 marshalling takes place therefore at the sender of any particular 277 message, be it an RPC request or an RPC response. Unmarshalling, 278 of course, takes place at the receiver. 280 Normally, any bulk data is moved (copied) as a result of the 281 unmarshalling process, because the destination adddress is not 282 known until the RPC code receives control and subsequently invokes 283 the XDR unmarshalling routine. In other words, XDR-encoded data is 284 not self-describing, and it carries no placement information. This 285 results in a data copy in most NFS implementations. 287 One mechanism by which the RPC layer may overcome this is for each 288 request to include placement information, to be used for direct 289 placement during XDR encode. This "write chunk" can avoid sending 290 bulk data inline in an RPC message and generally results in one or 291 more RDMA Write operations. 293 Similarly, a "read chunk", where placement information referring to 294 bulk data which may be directly fetched via one or more RDMA Read 295 operations during XDR decode, may be conveyed. The "read chunk" 296 will therefore be useful in both RPC calls and replies, while the 297 "write chunk" is used solely in replies. 299 These "chunks" are the key concept in an existing proposal 300 [RPCRDMA]. They convey what are effectively pointers to remote 301 memory across the network. They allow cooperating peers to 302 exchange data outside of XDR encodings but still use XDR for 303 describing the data to be transferred. And, finally, through use 304 of XDR they maintain a large degree of on-the-wire compatibility. 306 The central concept of the RDMA transport is to provide the 307 additional encoding conventions to convey this placement 308 information in transport-specific encoding, and to modify the XDR 309 handling of bulk data. 311 Block Diagram 313 +------------------------+-----------------------------------+ 314 | NFS | NFS + RDMA | 315 +------------------------+----------------------+------------+ 316 | Operations / Procedures | | 317 +-----------------------------------------------+ | 318 | RPC/XDR | | 319 +--------------------------------+--------------+ | 320 | Stream Transport | RDMA Transport | 321 +--------------------------------+---------------------------+ 323 4. Sources of Overhead 325 Network and file protocol costs can be categorized as follows: 327 o per-byte costs - data touching costs such as checksum or data 328 copy. Today's network interface hardware commonly offloads 329 the checksum, which leaves the other major source of per-byte 330 overhead, data copy. 332 o per-packet costs - interrupts and lower-layer processing. 333 Today's network interface hardware also commonly coalesce 334 interrupts to reduce per-packet costs. 336 o per-message (request or response) costs - LLP and ULP 337 processing. 339 Improvement from optimization becomes more important if the 340 overhead it targets is a larger share of the total cost. As other 341 sources of overhead, such as the checksumming and interrupt 342 handling above are eliminated, the remaining overheads (primarily 343 data copy) loom larger. 345 With copies crossing the bus twice per copy, network processing 346 overhead is high whenever network bandwidth is large in comparison 347 to CPU and memory bandwidths. Generally with today's end-systems, 348 the effects are observable at network speeds at or above 1 Gbits/s. 350 A common question is whether increase in CPU processing power 351 alleviates the problem of high processing costs of network I/O. 352 The answer is no, it is the memory bandwidth that is the issue. 353 Faster CPUs do not help if the CPU spends most of its time waiting 354 for memory [RFC4297]. 356 TCP offload engine (TOE) technology aims to offload the CPU by 357 moving TCP/IP protocol processing to the NIC. However, TOE 358 technology by itself does nothing to avoid necessary data copies 359 within upper layer protocols. [MOG03] provides a description of 360 the role TOE can play in reducing per-packet and per-message costs. 361 Beyond the offloads commonly provided by today's network interface 362 hardware, TOE alone (w/o RDMA) helps in protocol header processing, 363 but this has been shown to be a minority component of the total 364 protocol processing overhead. [CHA+01] 366 Numerous software approaches to the optimization of network 367 throughput have been made. Experience has shown that network I/O 368 interacts with other aspects of system processing such as file I/O 369 and disk I/O. [BRU99] [CHU96] Zero-copy optimizations based on 370 page remapping [CHU96] can be dependent upon machine architecture, 371 and are not scaleable to multi-processor architectures. Correct 372 buffer alignment and sizing together are needed to optimize the 373 performance of zero-copy movement mechanisms [SKE+01]. The NFS 374 message layout described above does not facilitate the splitting of 375 headers from data nor does it facilitate providing correct data 376 buffer alignment. 378 4.1. Savings from TOE 380 The expected improvement of TOE specifically for NFS protocol 381 processing can be quantified and shown to be fundamentally limited. 382 [SHI+03] presents a set of "LAWS" parameters which serve to 383 illustrate the issues. In the TOE case, the copy cost can be 384 viewed as part of the application processing "a". Application 385 processing increases the LAWS "gamma", which is shown by the paper 386 to result in a diminished benefit for TOE. 388 For example, if the overhead is 20% TCP/IP, 30% copy and 50% real 389 application work, then gamma is 80/20 or 4, which means the maximum 390 benefit of TOE is 1/gamma, or only 25%. 392 For RDMA (with embedded TOE) and the same example, the "overhead" 393 (o) offloaded or eliminated is 50% (20%+30%). Therefore in the 394 RDMA case, gamma is 50/50 or 1, and the inverse gives the potential 395 benefit of 1 (100%), a factor of two. 397 CPU overhead reduction factor 399 No Offload TCP Offload RDMA Offload 400 -----------+-------------+------------- 401 1.00x 1.25x 2.00x 403 The analysis in the paper shows that RDMA could improve throughput 404 by the same factor of two, even when the host is (just) powerful 405 enough to drive the full network bandwidth without RDMA. It can 406 also be shown that the speedup may be higher if network bandwidth 407 grows faster than Moore's Law, although the higher benefits will 408 apply to a narrow range of applications. 410 4.2. Savings from RDMA 412 Performance measurements directly comparing an NFS over RDMA 413 prototype with conventional network-based NFS processing are 414 described in [CAL+03]. Comparisons of Read throughput and CPU 415 overhead were performed on two Gigabit Ethernet adapters, one 416 conventional and one with RDMA capability. The prototype RDMA 417 protocol performed all transfers via RDMA Read. 419 In these results, conventional network-based throughput was 420 severely limited by the client's CPU being saturated at 100% for 421 all transfers. Read throughput reached no more than 60MBytes/s. 423 I/O Type Size Read Throughput CPU Utilization 424 Conventional 2KB 20MB/s 100% 425 Conventional 16KB 40MB/s 100% 426 Conventional 256KB 60MB/s 100% 428 However, over RDMA, throughput rose to the theoretical maximum 429 throughput of the platform, while saturating the single-CPU system 430 only at maximum throughput. 432 I/O Type Size Read Throughput CPU Utilization 433 RDMA 2KB 10MB/s 45% 434 RDMA 16KB 40MB/s 70% 435 RDMA 256KB 100MB/s 100% 437 The lower relative throughput of the RDMA prototype at the small 438 blocksize may be attributable to the RDMA Read imposed by the 439 prototype protocol, which reduced the operation rate since it 440 introduces additional latency. As well, it may reflect the 441 relative increase of per-packet setup costs within the DMA portion 442 of the transfer. 444 5. Application of RDMA to NFS 446 Efficient file protocols require efficient data positioning and 447 movement. The client system knows the client memory address where 448 the application has data to be written or wants read data 449 deposited. The server system knows the server memory address where 450 the local filesystem will accept write data or has data to be read. 451 Neither peer however is aware of the others' data destination in 452 the current NFS, RPC or XDR protocols. Existing NFS 453 implementations have struggled with the performance costs of data 454 copies when using traditional Ethernet transports. 456 With the onset of faster networks, the network I/O bottleneck will 457 worsen. Fortunately, new transports that support RDMA have 458 emerged. RDMA excels at bulk transfer efficiency; it is an 459 efficient way to deliver direct data placement and remove a major 460 part of the problem: data copies. RDMA also addresses other 461 overheads, e.g. underlying protocol offload, and offers separation 462 of control information from data. 464 The current NFS message layout provides the performance enhancing 465 opportunity for an NFS over RDMA protocol that separates the 466 control information from data chunks while meeting the alignment 467 needs of both. The data chunks can be copied "directly" between 468 the client and server memory addresses above (with a single 469 occurrence on each memory bus) while the control information can be 470 passed "inline". [RPCRDMA] describes such a protocol. 472 6. Conclusions 474 NFS version 4 [RFC3530] has been granted "Proposed Standard" 475 status. The NFSv4 protocol was developed along several design 476 points, important among them: effective operation over wide- area 477 networks, including the Internet itself; strong security 478 integrated into the protocol; extensive cross-platform 479 interoperability including integrated locking semantics compatible 480 with multiple operating systems; and (this is key), protocol 481 extension. 483 NFS version 4 is an excellent base on which to add the needed 484 performance enhancements and improved semantics described above. 485 The minor versioning support defined in NFS version 4 was designed 486 to support protocol improvements without disruption to the 487 installed base. Evolutionary improvement of the protocol via minor 488 versioning is a conservative and cautious approach to current and 489 future problems and shortcomings. 491 Many arguments can be made as to the efficacy of the file 492 abstraction in meeting the future needs of enterprise data service 493 and the Internet. Fine grained Quality of Service (QoS) policies 494 (e.g. data delivery, retention, availability, security, ...) are 495 high among them. 497 It is vital that the NFS protocol continue to provide these 498 benefits to a wide range of applications, without its usefulness 499 being compromised by concerns about performance and semantic 500 inadequacies. This can reasonably be addressed in the existing NFS 501 protocol framework. A cautious evolutionary improvement of 502 performance and semantics allows building on the value already 503 present in the NFS protocol, while addressing new requirements that 504 have arisen from the application of networking technology. 506 7. Security Considerations 508 Security Considerations are not covered by this document. Please 509 refer to the appropriate protocol documents for any security 510 issues. 512 8. IANA Considerations 514 IANA Considerations are not covered by this document. Please refer 515 to the appropriate protocol documents for any IANA issues. 517 9. Acknowledgements 519 The authors wish to thank Jeff Chase who provided many useful 520 suggestions. 522 10. Normative References 524 [RFC3530] 525 S. Shepler, et. al., "NFS Version 4 Protocol", Standards Track 526 RFC 528 [RFC1831] 529 R. Srinivasan, "RPC: Remote Procedure Call Protocol 530 Specification Version 2", Standards Track RFC 532 [RFC4506] 533 M. Eisler, Ed. "XDR: External Data Representation Standard", 534 Standards Track RFC 536 [RFC1813] 537 B. Callaghan, B. Pawlowski, P. Staubach, "NFS Version 3 538 Protocol Specification", Informational RFC 540 11. Informative References 542 [BRU99] 543 J. Brustoloni, "Interoperation of copy avoidance in network 544 and file I/O", in Proc. INFOCOM '99, pages 534-542, New York, 545 NY, Mar. 1999., IEEE. Also available from 546 http://www.cs.pitt.edu/~jcb/publs.html 548 [CAL+03] 549 B. Callaghan, T. Lingutla-Raj, A. Chiu, P. Staubach, O. Asad, 550 "NFS over RDMA", in Proceedings of ACM SIGCOMM Summer 2003 551 NICELI Workshop. 553 [CHA+01] 554 J. S. Chase, A. J. Gallatin, K. G. Yocum, "Endsystem 555 optimizations for high-speed TCP", IEEE Communications, 556 39(4):68-74, April 2001. 558 [CHA+99] 559 J. S. Chase, D. C. Anderson, A. J. Gallatin, A. R. Lebeck, K. 560 G. Yocum, "Network I/O with Trapeze", in 1999 Hot 561 Interconnects Symposium, August 1999. 563 [CHU96] 564 H.K. Chu, "Zero-copy TCP in Solaris", Proc. of the USENIX 1996 565 Annual Technical Conference, San Diego, CA, January 1996 567 [DCK+03] 568 M. DeBergalis, P. Corbett, S. Kleiman, A. Lent, D. Noveck, T. 569 Talpey, M. Wittle, "The Direct Access File System", in 570 Proceedings of 2nd USENIX Conference on File and Storage 571 Technologies (FAST '03), San Francisco, CA, March 31 - April 572 2, 2003 574 [FJDAFS] 575 Fujitsu Prime Software Technologies, "Meet the DAFS 576 Performance with DAFS/VI Kernel Implementation using cLAN", 577 available from 578 http://www.pst.fujitsu.com/english/dafsdemo/index.html, 2001. 580 [FJNFS] 581 Fujitsu Prime Software Technologies, "An Adaptation of VIA to 582 NFS on Linux", available from 583 http://www.pst.fujitsu.com/english/nfs/index.html, 2000. 585 [GAL+99] 586 A. Gallatin, J. Chase, K. Yocum, "Trapeze/IP: TCP/IP at Near- 587 Gigabit Speeds", 1999 USENIX Technical Conference (Freenix 588 Track), June 1999. 590 [KM02] 591 K. Magoutis, "Design and Implementation of a Direct Access 592 File System (DAFS) Kernel Server for FreeBSD", in Proceedings 593 of USENIX BSDCon 2002 Conference, San Francisco, CA, February 594 11-14, 2002. 596 [MAF+02] 597 K. Magoutis, S. Addetia, A. Fedorova, M. Seltzer, J. Chase, D. 598 Gallatin, R. Kisley, R. Wickremesinghe, E. Gabber, "Structure 599 and Performance of the Direct Access File System (DAFS)", in 600 Proceedings of 2002 USENIX Annual Technical Conference, 601 Monterey, CA, June 9-14, 2002. 603 [MOG03] 604 J. Mogul, "TCP offload is a dumb idea whose time has come", 605 9th Workshop on Hot Topics in Operating Systems (HotOS IX), 606 Lihue, HI, May 2003. USENIX. 608 [NFSv4.1] 609 S. Shepler, ed., "NFSv4 Minor Version 1" Internet Draft work- 610 in-progress, draft-ietf-nfsv4-minorversion1 612 [PAI+00] 613 V. S. Pai, P. Druschel, W. Zwaenepoel, "IO-Lite: a unified I/O 614 buffering and caching system", ACM Trans. Computer Systems, 615 18(1):37-66, Feb. 2000. 617 [RDDP] 618 RDDP Working Group charter, 619 http://www.ietf.org/html.charters/rddp-charter.html 621 [RFC4297] 622 A. Romanow, J. Mogul, T. Talpey, S. Bailey, "Remote Direct 623 Memory Access (RDMA) over IP Problem Statement", Informational 624 RFC 626 [RFC1094] 627 Sun Microsystems, "NFS: Network File System Protocol 628 Specification" 630 [RPCRDMA] 631 T. Talpey, B. Callaghan, "RDMA Transport for ONC RPC", 632 Internet Draft Work in Progress, draft-ietf-nfsv4-rpcrdma 634 [SHI+03] 635 P. Shivam, J. Chase, "On the Elusive Benefits of Protocol 636 Offload", to be published in Proceedings of ACM SIGCOMM Summer 637 2003 NICELI Workshop, also available from 638 http://issg.cs.duke.edu/publications/niceli03.pdf 640 [SKE+01] 641 K.-A. Skevik, T. Plagemann, V. Goebel, P. Halvorsen, 642 "Evaluation of a Zero-Copy Protocol Implementation", in 643 Proceedings of the 27th Euromicro Conference - Multimedia and 644 Telecommunications Track (MTT'2001), Warsaw, Poland, September 645 2001. 647 Authors' Addresses 649 Tom Talpey 650 Network Appliance, Inc. 651 375 Totten Pond Road 652 Waltham, MA 02451 USA 654 Phone: +1 781 768 5329 655 Email: thomas.talpey@netapp.com 657 Chet Juszczak 658 Chet's Boathouse Co. 659 P.O. Box 1467 660 Merrimack, NH 03054 662 Email: chetnh@earthlink.net 664 Intellectual Property and Copyright Statements 666 Full Copyright Statement 668 Copyright (C) The IETF Trust (2007). 669 This document is subject to the rights, licenses and restrictions 670 contained in BCP 78, and except as set forth therein, the authors 671 retain all their rights. 673 This document and the information contained herein are provided on 674 an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE 675 REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE 676 IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL 677 WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY 678 WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE 679 ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS 680 FOR A PARTICULAR PURPOSE. 682 Intellectual Property 683 The IETF takes no position regarding the validity or scope of any 684 Intellectual Property Rights or other rights that might be claimed 685 to pertain to the implementation or use of the technology described 686 in this document or the extent to which any license under such 687 rights might or might not be available; nor does it represent that 688 it has made any independent effort to identify any such rights. 689 Information on the procedures with respect to rights in RFC 690 documents can be found in BCP 78 and BCP 79. 692 Copies of IPR disclosures made to the IETF Secretariat and any 693 assurances of licenses to be made available, or the result of an 694 attempt made to obtain a general license or permission for the use 695 of such proprietary rights by implementers or users of this 696 specification can be obtained from the IETF on-line IPR repository 697 at http://www.ietf.org/ipr. 699 The IETF invites any interested party to bring to its attention any 700 copyrights, patents or patent applications, or other proprietary 701 rights that may cover technology that may be required to implement 702 this standard. Please address the information to the IETF at ietf- 703 ipr@ietf.org. 705 Acknowledgment 706 Funding for the RFC Editor function is provided by the IETF 707 Administrative Support Activity (IASA).