idnits 2.17.1 draft-cel-nfsv4-rpcrdma-version-two-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (April 8, 2016) is 2940 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-11) exists of draft-ietf-nfsv4-rfc5666bis-04 == Outdated reference: A later version (-08) exists of draft-ietf-nfsv4-rpcrdma-bidirection-01 -- Obsolete informational reference (is this intentional?): RFC 5661 (Obsoleted by RFC 8881) -- Obsolete informational reference (is this intentional?): RFC 5666 (Obsoleted by RFC 8166) Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network File System Version 4 C. Lever, Ed. 3 Internet-Draft Oracle 4 Intended status: Standards Track D. Noveck 5 Expires: October 10, 2016 HPE 6 April 8, 2016 8 RPC-over-RDMA Version Two 9 draft-cel-nfsv4-rpcrdma-version-two-00 11 Abstract 13 This document specifies an improved protocol for conveying Remote 14 Procedure Call (RPC) messages on physical transports capable of 15 Remote Direct Memory Access (RDMA), based on RPC-over-RDMA Version 16 One. 18 Status of This Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at http://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on October 10, 2016. 35 Copyright Notice 37 Copyright (c) 2016 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents 42 (http://trustee.ietf.org/license-info) in effect on the date of 43 publication of this document. Please review these documents 44 carefully, as they describe your rights and restrictions with respect 45 to this document. Code Components extracted from this document must 46 include Simplified BSD License text as described in Section 4.e of 47 the Trust Legal Provisions and are provided without warranty as 48 described in the Simplified BSD License. 50 Table of Contents 52 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 53 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 54 2. Inline Threshold . . . . . . . . . . . . . . . . . . . . . . 3 55 2.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 56 2.2. Motivation . . . . . . . . . . . . . . . . . . . . . . . 4 57 2.3. Default Values . . . . . . . . . . . . . . . . . . . . . 4 58 3. Protocol Extensibility . . . . . . . . . . . . . . . . . . . 5 59 3.1. Optional Features . . . . . . . . . . . . . . . . . . . . 5 60 3.2. Documentation Requirements . . . . . . . . . . . . . . . 5 61 4. XDR Protocol Definition . . . . . . . . . . . . . . . . . . . 6 62 4.1. Code Component License . . . . . . . . . . . . . . . . . 7 63 4.2. RPC-Over-RDMA Version Two XDR . . . . . . . . . . . . . . 9 64 5. Protocol Version Negotiation . . . . . . . . . . . . . . . . 11 65 5.1. Responder Does Support RPC-over-RDMA Version Two . . . . 11 66 5.2. Responder Does Not Support RPC-over-RDMA Version Two . . 12 67 5.3. Requester Does Not Support RPC-over-RDMA Version Two . . 12 68 6. Security Considerations . . . . . . . . . . . . . . . . . . . 12 69 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 70 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 12 71 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 13 72 9.1. Normative References . . . . . . . . . . . . . . . . . . 13 73 9.2. Informative References . . . . . . . . . . . . . . . . . 13 74 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 14 76 1. Introduction 78 Remote Direct Memory Access (RDMA) [RFC5040] [RFC5041] [IB] is a 79 technique for moving data efficiently between end nodes. By 80 directing data into destination buffers as it is sent on a network, 81 and placing it via direct memory access by hardware, the benefits of 82 faster transfers and reduced host overhead are obtained. 84 A protocol already exists that enables RPC [RFC5531] messages to be 85 conveyed on RDMA transports. That protocol is RPC-over-RDMA Version 86 One, specified in [I-D.ietf-nfsv4-rfc5666bis]. RPC-over-RDMA Version 87 One is deployed and in use, though there are some shortcomings to 88 this protocol, such as: 90 o The use of small Receive buffers force the use of RDMA Read and 91 Write transfers for small payloads, and limit the size of 92 backchannel messages 94 o Lack of support for potential optimizations, such as remote 95 invalidation, that require changes to on-the-wire behavior 97 To address these issues in a way that is compatible with existing 98 RPC-over-RDMA Version One deployments, a new version of RPC-over-RDMA 99 is presented in this document. RPC-over-RDMA Version Two contains 100 only incremental changes over RPC-over-RDMA Version One to facilitate 101 adoption of Version Two by existing Version One implementations. 103 The major new feature in RPC-over-RDMA Version Two is extensibility 104 of the RPC-over-RDMA header. Extensibility enables narrow changes to 105 RPC-over-RDMA Version Two so that new optional capabilities can be 106 introduced without a protocol version change and while maintaining 107 interoperability with existing implementations. New capabilities can 108 be proposed and developed independently of each other, and 109 implementaters can choose among them. It should be straightforward 110 to create and document experimental features and then bring them 111 through the standards process. 113 In addition to extensibility, the default inline threshold value is 114 larger in RPC-over-RDMA Version Two. This change is driven by the 115 increase in average size of RPC messages containing common NFS 116 operations. With NFSv4.1 [RFC5661] and later, compound operations 117 convey more data per RPC message. The default 1KB inline threshold 118 in RPC-over-RDMA Version One prevents attaining the best possible 119 performance. 121 1.1. Requirements Language 123 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 124 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 125 document are to be interpreted as described in [RFC2119]. 127 2. Inline Threshold 129 2.1. Terminology 131 The term "inline threshold" is defined in Section 4 of 132 [I-D.ietf-nfsv4-rfc5666bis]. A receiver's "inline threshold" value 133 is the largest message size (in octets) that the receiver can accept 134 via an RDMA Receive operation. Each connection has two inline 135 threshold values, one for each peer receiver. Inline threshold 136 values are not advertised to peers via the base RPC-over-RDMA Version 137 Two protocol. 139 A connection's inline threshold determines when full RDMA Read or 140 Write operations are required because the RPC message to be send is 141 larger than the peer's Receive buffer. If an RPC message does not 142 contain DDP-eligible data items, a requester prepares a Long Call or 143 Reply to convey the whole RPC message using an RDMA Read or Write. 145 2.2. Motivation 147 RDMA Read and Write operations require that each data payload resides 148 in a region of memory that is registered with the RNIC. When an RPC 149 is complete, that region is unregistered, fencing it from the 150 responder. 152 Both registration and unregistration have a latency cost which is 153 insignificant compared to data handling costs. When a data payload 154 is small, however, the cost of registering and unregistering the 155 memory where it resides becomes a relatively significant part of 156 total RPC latency. Therefore the most efficient operation of RPC- 157 over-RDMA occurs when RDMA Read and Write operations are used for 158 large payloads, and avoided for small payloads. 160 When RPC-over-RDMA Version One was conceived, the typical size of RPC 161 messages that did not involve a significant data payload was under 162 500 bytes. A 1024-byte inline threshold adequately minimized the 163 frequency of inefficient Long Calls and Replies. 165 Starting with NFSv4.1 [RFC5661], NFS COMPOUND RPC messages are larger 166 and more complex than before. With a 1024-byte inline threshold, 167 RDMA Read or Write operations are needed for frequent operations that 168 do not bear a data payload, such as GETATTR and LOOKUP, reducing the 169 efficiency of the transport. To reduce the need to use Long Calls 170 and Replies, RPC-over-RDMA Version Two quadruples the default inline 171 threshold size. This also increases the maximum size of backward 172 direction RPC messages. 174 2.3. Default Values 176 RPC-over-RDMA Version Two receiver implementations MUST support an 177 inline threshold of 4096 bytes, but MAY support larger inline 178 threshold values. A mechanism for discovering a peer's preferred 179 inline threshold value may be used to optimize RDMA Send operations 180 further. In the absense of such a mechanism, senders MUST assume a 181 receiver's inline threshold is 4096 bytes. 183 The new default inline threshold size is no larger than the size of a 184 hardware page on typical platforms. This conserves the resources 185 needed to Send and Receive base level RPC-over-RDMA Version Two 186 messages, enabling RPC-over-RDMA Version Two to be used on a broad 187 base of hardware. 189 3. Protocol Extensibility 191 The core RPC-over-RDMA Version Two header format is specified in 192 Section 4 as a complete and stand-alone piece of XDR. Any change to 193 this XDR description requires a protocol version number change. 195 3.1. Optional Features 197 RPC-over-RDMA Version Two introduces the ability to extend the core 198 protocol via optional features. Extensibility enables minor protocol 199 issues to be addressed and incremental enhancements to be made 200 without the need to change the protocol version. The key capability 201 is that both sides can detect whether a feature is supported by their 202 peer or not. With this ability, OPTIONAL features can be introduced 203 over time to an otherwise stable protocol. 205 The rdma_opttype field carries a 32-bit unsigned integer. The value 206 in this field denotes an optional operation that MAY be supported by 207 the receiver. The values of this field and their meaning are defined 208 in other Standards Track documents. 210 The rdma_optinfo field carries opaque data. The content of this 211 field is data meaningful to the optional operation denoted by the 212 value in rdma_opttype. The content of this field is not defined in 213 the base RPC-over-RDMA Version Two protocol, but is defined in other 214 Standards Track documents 216 When an implementation does not recognize or support the value 217 contained in the rdma_opttype field, it MUST send an RPC-over-RDMA 218 message with the rdma_xid field set to the same value as the 219 erroneous message, the rdma_proc field set to RDMA_ERROR, and the 220 rdma_err field set to RDMA_ERR_INVAL_OPTION. 222 3.2. Documentation Requirements 224 RPC-over-RDMA Version Two may be extended by defining a new 225 rdma_opttype value and XDR description of the corresponding 226 rdma_optinfo content. 228 A set of such new protocol elements may be introduced by a Standards 229 Track document and are together considered an OPTIONAL feature. 230 nfsv4 Working Group and IESG review, together with appropriate 231 testing of prototype implementations, should ensure continued 232 interoperation with existing implementations. 234 Documents describing extensions to RPC-over-RDMA Version Two should 235 contain: 237 o An explanation of the purpose and use of each new protocol element 238 added 240 o An XDR description and a script to extract it 242 o A mechanism for reporting errors to the message sender that are 243 outside the available choices in the base RPC-over-RDMA Version 244 Two protocol 246 o A requirement that an RPC Payload Stream MUST or MUST NOT follow 247 the Transport Header Stream for each new opttype code added 249 o A description of interactions with existing features (e.g., 250 requirements that another OPTIONAL feature needs to be present and 251 supported for newly added features to work) 253 Implementers concatenate the XDR description of the new feature with 254 the XDR description of the base protocol, extracted from this 255 document, to produce a combined XDR description for the RPC-over-RDMA 256 Version Two protocol with the specified extension. 258 4. XDR Protocol Definition 260 This section contains a description of the core features of the RPC- 261 over-RDMA Version Two protocol, expressed in the XDR language 262 [RFC4506]. 264 This description is provided in a way that makes it simple to extract 265 into ready-to-compile form. The reader can apply the following shell 266 script to this document to produce a machine-readable XDR description 267 of the RPC-over-RDMA Version One protocol without any OPTIONAL 268 extensions. 270 272 #!/bin/sh 273 grep '^ *///' | sed 's?^ /// ??' | sed 's?^ *///$??' 275 277 That is, if the above script is stored in a file called "extract.sh" 278 and this document is in a file called "spec.txt" then the reader can 279 do the following to extract an XDR description file: 281 283 sh extract.sh < spec.txt > rpcrdma_corev2.x 285 287 Optional extensions to RPC-over-RDMA Version Two, published as 288 Standards Track documents, will have similar means of providing XDR 289 that describes those extensions. Once XDR for all desired extensions 290 is also extracted, it can be appended to the XDR description file 291 extracted from this document to produce a consolidated XDR 292 description file reflecting all extensions selected for an RPC-over- 293 RDMA implementation. 295 4.1. Code Component License 297 Code components extracted from this document must include the 298 following license text. When the extracted XDR code is combined with 299 other complementary XDR code which itself has an identical license, 300 only a single copy of the license text need be preserved. 302 304 /// /* 305 /// * Copyright (c) 2010, 2016 IETF Trust and the persons 306 /// * identified as authors of the code. All rights reserved. 307 /// * 308 /// * The authors of the code are: 309 /// * B. Callaghan, T. Talpey, C. Lever, and D. Noveck. 310 /// * 311 /// * Redistribution and use in source and binary forms, with 312 /// * or without modification, are permitted provided that the 313 /// * following conditions are met: 314 /// * 315 /// * - Redistributions of source code must retain the above 316 /// * copyright notice, this list of conditions and the 317 /// * following disclaimer. 318 /// * 319 /// * - Redistributions in binary form must reproduce the above 320 /// * copyright notice, this list of conditions and the 321 /// * following disclaimer in the documentation and/or other 322 /// * materials provided with the distribution. 323 /// * 324 /// * - Neither the name of Internet Society, IETF or IETF 325 /// * Trust, nor the names of specific contributors, may be 326 /// * used to endorse or promote products derived from this 327 /// * software without specific prior written permission. 328 /// * 329 /// * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS 330 /// * AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED 331 /// * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 332 /// * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 333 /// * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO 334 /// * EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 335 /// * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 336 /// * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 337 /// * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 338 /// * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 339 /// * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 340 /// * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 341 /// * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING 342 /// * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 343 /// * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 344 /// */ 346 348 4.2. RPC-Over-RDMA Version Two XDR 350 The XDR defined in this section is used to encode the Transport 351 Header Stream in each RPC-over-RDMA Version Two message. The terms 352 "Transport Header Stream" and "RPC Payload Stream" are defined in 353 Section 4 of [I-D.ietf-nfsv4-rfc5666bis]. 355 357 /// struct rpcrdma2_segment { 358 /// uint32 rdma_handle; 359 /// uint32 rdma_length; 360 /// uint64 rdma_offset; 361 /// }; 362 /// 363 /// struct rpcrdma2_read_segment { 364 /// uint32 rdma_position; 365 /// struct rpcrdma2_segment rdma_target; 366 /// }; 367 /// 368 /// struct rpcrdma2_read_list { 369 /// struct rpcrdma2_read_segment rdma_entry; 370 /// struct rpcrdma2_read_list *rdma_next; 371 /// }; 372 /// 373 /// struct rpcrdma2_write_chunk { 374 /// struct rpcrdma2_segment rdma_target<>; 375 /// }; 376 /// 377 /// struct rpcrdma2_write_list { 378 /// struct rpcrdma2_write_chunk rdma_entry; 379 /// struct rpcrdma2_write_list *rdma_next; 380 /// }; 381 /// 382 /// struct rpcrdma2_chunk_lists { 383 /// struct rpcrdma2_read_list *rdma_reads; 384 /// struct rpcrdma2_write_list *rdma_writes; 385 /// struct rpcrdma2_write_chunk *rdma_reply; 386 /// }; 387 /// 388 /// enum rpcrdma2_errcode { 389 /// RDMA_ERR_VERS = 1, 390 /// RDMA_ERR_BAD_HEADER = 2, 391 /// RDMA_ERR_INVAL_OPTION = 3 392 /// }; 393 /// 394 /// struct rpcrdma2_err_vers { 395 /// uint32 rdma_vers_low; 396 /// uint32 rdma_vers_high; 397 /// }; 398 /// 399 /// union rpcrdma2_error switch (rpcrdma2_errcode rdma_err) { 400 /// case RDMA_ERR_VERS: 401 /// rpcrdma2_err_vers rdma_vrange; 402 /// case RDMA_ERR_BAD_HEADER: 403 /// void; 404 /// case RDMA_ERR_INVAL_OPTION: 405 /// void; 406 /// }; 407 /// 408 /// struct rpcrdma2_optional { 409 /// uint32 rdma_opttype; 410 /// opaque rdma_optinfo<>; 411 /// }; 412 /// 413 /// enum rpcrdma2_proc { 414 /// RDMA_MSG = 0, 415 /// RDMA_NOMSG = 1, 416 /// RDMA_MSGP = 2, /* Reserved */ 417 /// RDMA_DONE = 3, /* Reserved */ 418 /// RDMA_ERROR = 4, 419 /// RDMA_OPTIONAL = 5 420 /// }; 421 /// 422 /// union rpcrdma2_body switch (rpcrdma2_proc rdma_proc) { 423 /// case RDMA_MSG: 424 /// rpcrdma2_chunks rdma_chunks; 425 /// case RDMA_NOMSG: 426 /// rpcrdma2_chunks rdma_chunks; 427 /// case RDMA_MSGP: 428 /// void; 429 /// case RDMA_DONE: 430 /// void; 431 /// case RDMA_ERROR: 432 /// rpcrdma2_error rdma_error; 433 /// case RDMA_OPTIONAL: 434 /// rpcrdma2_optional rdma_optional; 435 /// }; 436 /// 437 /// struct rpcrdma2_xprt_hdr { 438 /// uint32 rdma_xid; 439 /// uint32 rdma_vers; 440 /// uint32 rdma_credit; 441 /// rpcrdma2_body rdma_body; 442 /// }; 443 445 When the rdma_proc field has the value RDMA_MSG, an RPC Payload 446 Stream MUST follow the Transport Header Stream in the Send buffer. 447 When the rdma_proc field has the value RDMA_OPTIONAL, an RPC Payload 448 Stream MAY follow the Transport header Stream in the Send buffer. 449 When the rdma_proc field has any other value, an RPC Payload Stream 450 MUST NOT follow the Transport Header Stream. 452 Implementations of RPC-over-RDMA Version Two are REQUIRED to support 453 backwards direction operation as described in 454 [I-D.ietf-nfsv4-rpcrdma-bidirection]. 456 Error handling works the same way in RPC-over-RDMA Version Two as it 457 does in RPC-over-RDMA Version One, with one change described in 458 Section 3.1. Version One error handling is described in Section 5 of 459 [I-D.ietf-nfsv4-rfc5666bis]. 461 5. Protocol Version Negotiation 463 When an RPC-over-RDMA Version Two requester establishes a connection 464 to a responder, the first order of business is to determine the 465 responder's highest supported protocol version. 467 As with RPC-over-RDMA Version One, a requester MUST assume the 468 ability to exchange only a single RPC-over-RDMA message at a time 469 until it receives a non-error RPC-over-RDMA message from the 470 responder that reports the responder's actual credit limit. 472 First, the requester sends a single valid RPC-over-RDMA message with 473 the value two (2) in the rdma_vers field. Because the responder 474 might support only RPC-over-RDMA Version One, this initial message 475 can be no larger than the Version One default inline threshold of 476 1024 bytes. 478 5.1. Responder Does Support RPC-over-RDMA Version Two 480 If the responder does support RPC-over-RDMA Version Two, it sends an 481 RPC-over-RDMA message back to the requester with the same XID 482 containing a valid non-error response. Subsequently, both peers use 483 the default inline threshold value for RPC-over-RDMA Version Two 484 connections (4096 bytes). 486 5.2. Responder Does Not Support RPC-over-RDMA Version Two 488 If the responder does not support RPC-over-RDMA Version Two, 489 [I-D.ietf-nfsv4-rfc5666bis] REQUIRES that it send an RPC-over-RDMA 490 message to the requester with the same XID, with RDMA_ERROR in the 491 rdma_proc field, and with the error code RDMA_ERR_VERS. This message 492 also reports a range of protocol versions that the responder 493 supports. To continue operation, the requester selects a protocol 494 version in the range of responder-supported versions for subsequent 495 messages on this connection. 497 If the connection is lost immediately after the RDMA_ERROR reply is 498 received, a requester can avoid a possible version negotiation loop 499 when re-establishing another connection by assuming that particular 500 responder does not support RPC-over-RDMA Version Two. A requester 501 can assume the same situation (no responder support for RPC-over-RDMA 502 Version Two) if the initial negotiation message is lost or dropped. 504 Once the negotiation exchange is complete, both peers use the default 505 inline threshold value for the protocol version that will be used for 506 the remainder of the connection lifetime. To permit inline threshold 507 values to change during negotiation of protocol version, RPC-over- 508 RDMA Version Two implementations MUST allow inline threshold values 509 to change without triggering a connection loss. 511 5.3. Requester Does Not Support RPC-over-RDMA Version Two 513 [I-D.ietf-nfsv4-rfc5666bis] REQUIRES that a responder MUST send 514 Replies with the same RPC-over-RDMA protocol version that the 515 requester uses to send its Calls. 517 6. Security Considerations 519 The security considerations for RPC-over-RDMA Version Two are the 520 same as those for RPC-over-RDMA Version One. 522 7. IANA Considerations 524 There are no IANA considerations at this time. 526 8. Acknowledgments 528 The authors gratefully acknowledge the work of Brent Callaghan and 529 Tom Talpey on the original RPC-over-RDMA Version One specification 530 [RFC5666]. The authors also wish to thank Bill Baker, Greg Marsden, 531 and Matt Benjamin for their support of this work. 533 The extract.sh shell script and formatting conventions were first 534 described by the authors of the NFSv4.1 XDR specification [RFC5662]. 536 Special thanks go to nfsv4 Working Group Chair Spencer Shepler and 537 nfsv4 Working Group Secretary Thomas Haynes for their support. 539 9. References 541 9.1. Normative References 543 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 544 Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/ 545 RFC2119, March 1997, 546 . 548 [RFC4506] Eisler, M., Ed., "XDR: External Data Representation 549 Standard", STD 67, RFC 4506, DOI 10.17487/RFC4506, May 550 2006, . 552 [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol 553 Specification Version 2", RFC 5531, DOI 10.17487/RFC5531, 554 May 2009, . 556 9.2. Informative References 558 [I-D.ietf-nfsv4-rfc5666bis] 559 Lever, C., Simpson, W., and T. Talpey, "Remote Direct 560 Memory Access Transport for Remote Procedure Call", draft- 561 ietf-nfsv4-rfc5666bis-04 (work in progress), March 2016. 563 [I-D.ietf-nfsv4-rpcrdma-bidirection] 564 Lever, C., "Size-Limited Bi-directional Remote Procedure 565 Call On Remote Direct Memory Access Transports", draft- 566 ietf-nfsv4-rpcrdma-bidirection-01 (work in progress), 567 September 2015. 569 [IB] InfiniBand Trade Association, "InfiniBand Architecture 570 Specifications", . 572 [RFC5040] Recio, R., Metzler, B., Culley, P., Hilland, J., and D. 573 Garcia, "A Remote Direct Memory Access Protocol 574 Specification", RFC 5040, DOI 10.17487/RFC5040, October 575 2007, . 577 [RFC5041] Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct 578 Data Placement over Reliable Transports", RFC 5041, DOI 579 10.17487/RFC5041, October 2007, 580 . 582 [RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., 583 "Network File System (NFS) Version 4 Minor Version 1 584 Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010, 585 . 587 [RFC5662] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., 588 "Network File System (NFS) Version 4 Minor Version 1 589 External Data Representation Standard (XDR) Description", 590 RFC 5662, DOI 10.17487/RFC5662, January 2010, 591 . 593 [RFC5666] Talpey, T. and B. Callaghan, "Remote Direct Memory Access 594 Transport for Remote Procedure Call", RFC 5666, DOI 595 10.17487/RFC5666, January 2010, 596 . 598 Authors' Addresses 600 Charles Lever (editor) 601 Oracle Corporation 602 1015 Granger Avenue 603 Ann Arbor, MI 48104 604 USA 606 Phone: +1 734 274 2396 607 Email: chuck.lever@oracle.com 609 David Noveck 610 Hewlett Packard Enterprise 611 165 Dascomb Road 612 Andover, MA 01810 613 USA 615 Phone: +1 978 474 2011 616 Email: davenoveck@gmail.com